UIUC CS-588 Autonomous Vehicle System Engineering Outline

This page contains what we've done

Texts:

Applied Machine Learning D.A. Forsyth, Springer, 2019

This is purely for background material on deep learning and other useful data tricks.

Important The PDF is a free download from the UIUC library (you have to be on the intranet to download it, I think). Or you could buy a paper copy.
The New Computer Vision D.A. Forsyth, Some publisher, The distant future

This is the source of various compact enrichment notes. They've been proofread, but not particularly carefully, and anyhow I'm a lousy proofreader. But they're at a point where they should be helpful.

Records

Week of	Topic	Reading Materials	Various enrichments
18 Jan	18 Jan: Admin+intro 21 Jan: Safety lookout training at High Bay	My intro slides Vehicle safety rules	Old safety lookout video, for reference
25 Jan	25 Jan: ROS and start PID control 27 Jan: I'm absent Simple neural networks Deep nets: Units; AML: 16.1 Deep nets: Gradients; AML: 16.2 Deep nets: Backpropagation; AML 16.3 Deep nets: Gradient tricks; AML 16.4 Deep nets: Convolutional layers; AML 17.1 Deep nets: More conv. layers; AML 17.1, 17.2 Basic object detection: AML 19	ROS and PACMOD: "My" ROS and PACMOD slides Original detailed slides, with examples, etc.from Roi Yehoshua at Bar-Ilan U. The Simulator Lecture on the simulator in 2021 by Xiaoming Zhao Basic control: Intro to PID control	25 Jan we're on mediaspace; the movie of 25 Jan is here. For routine class movies, I won't post link - look them up on mediaspace. Movies for 27 Jan Units Gradients Backprop Gradient tricks Convolutional layers More conv layers Image classifiers Object detectors
1 Feb	1 Feb: Finish PID control, start motion planning 3 Feb: more simple motion planning	Basic control: Intro to PID control (these are the slides I used 25 Jan, but I've skipped some) Simple motion planning
8 Feb	8 Feb: Point set registration and IRLS 10 Feb: Cameras	Basic Registration: Simple point set registration and iterated closest points Robustness and iteratively reweighted least squares Maps and very simple bundle adjustment Cameras: Basic camera properties Mathematical models of cameras	Enrichment movie Compact movie on M-estimators and IRLS Enrichment notes Compact notes on registration Compact notes on basic camera models Compact notes on camera geometry Weighted least squares solutions for rotation translation; very nice notes from Olga Sorkine-Hornung and Michael Rabinovitch see also Notes on robustness and iteratively reweighted least squares see also Robust registration of point sets using iteratively reweighted least squares P. Bergstrom and A.Edlund, Computational Optimization and Applications July 2014, Volume 58, Issue 3, pp 543–561 (click from inside UIUC intranet!) Geometrically Stable Sampling for the ICP AlgorithmN. Gelfand; L. Ikemoto; S. Rusinkiewicz; M. Levoy, (3DIMPVT) 2003 Efficient Variants of the ICP AlgorithmS.Rusinkiewicz and M. Levoy, (3DIMPVT), 2001 Translation Synchronization via Truncated Least SquaresX Huang, Z Liang, C. Bajaj, Q. Huang, NeurIPS17 Uncertainty quantification for multi-scan registration X Huang, Z Liang, Q Huang ACM Transactions on Graphics (TOG) 39 (4), 130: 1-130: 24
15 Feb	15 Feb: Camera Calibration, start pairs of cameras 17 Feb: Pairs of cameras and Visual Odometry	Calibration: A sketch of camera calibration Pairs of Cameras: The Geometry of two cameras, with fundamental and essential matrices and RANSAC Simple odometry from two views or getting something out of the fundamental matrix and calibration Interest points or what to match	Enrichment notes Compact notes on camera calibration Multiple View Geometry in Computer Vision Second Edition Richard Hartley and Andrew Zisserman, Cambridge University Press, March 2004. SuperPoint: Self-Supervised Interest Point Detection and Description DeTone, Malisciewicz, Rabinovich, 2018 An implementation of SuperPoint Tutorial on visual slam at CVPR2014 I found the following particularly helpful Visual odometry Stephan Weiss Stereo Visual odometryChris Beall Dense odometryRichard Newcombe
22 Feb	22 Feb: Filtering and Kalman Filter 24 Feb: I'm on Travel - review notes	Filtering A simple, important example; filtering in 1D The general story of filtering The Kalman filter in ND	Enrichment Kris Kitani's Kalman filtering slides Pieter Abbeel's Kalman filtering slides
1 Mar	1 Mar: Finish Kalman Filter; start EKF 3 Mar: Finish EKF and EKF-SLAM	The Extended Kalman Filter The Extended Kalman Filter (EKF) EKF-slam	Great notes on EKF-SLAM by Joan Sola, with Matlab code, etc
8 Mar	8 Mar: Travel, sorry 10 Mar: Direct and semi-direct methods	Direct and Semi-direct methods Semi-direct Methods for VO and SLAM	Resources Lucas-Kanade 20 Years On: A Unifying Framework: Part 1 Simon Baker and Iain Matthews, 2002 Christian Forster, Zichao Zhang, Michael Gassner, Manuel Werlberger, Davide Scaramuzza SVO: Semi-Direct Visual Odometry for Monocular and Multi-Camera Systems IEEE Transactions on Robotics, Vol. 33, Issue 2, pages 249-265, Apr. 2017. Dense visual slam an earlier version of lsd-slam for RGB-D cameras, code too Dense Visual SLAM for RGB-D Cameras (C. Kerl, J. Sturm and D. Cremers), In Proc. of the Int. Conf. on Intelligent Robot Systems (IROS), 2013.
23 Mar	Harder Planning Problems	Slides My slides on higher dimensional motion planning My slides on planning in the presence of dynamics, with a bit of stuff about motion graphs	Other Slides Slides on imitation learning by Katerina Fragkiadaki Slides from Tom Mitchell on Imitation learning(First part is rather similar to Fragkiadaki's slides, above; but the second part is different!) Very nice CMU motion planning slides, I believe by H. Choset A Planning Library The open mpl library and app Papers on Lane Boundaries US Patent 9081385 (lane boundaries) Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection Lucas Tabelini, Rodrigo Berriel, Thiago M. Paixão, Claudine Badue, Alberto F. De Souza, Thiago Oliveira-Santos Code Off-road Path Following using Region Classification and Geometric Projection Constraints Y. Alon; A. Ferencz; A. Shashua, CVPR 06 Detection and tracking of boundary of unmarked roads Young-Woo Seo; Ragunathan Raj Rajkumar, 17'th international conf on information fusion, 2014 Geometric Constrained Joint Lane Segmentation and Lane Boundary Detection Jie Zhang, Yi Xu, Bingbing Ni, Zhenyu Duan; Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 486-502 Lane detection using lane boundary marker network with road geometry constraints Hussam Ullah Khan, Afsheen Rafaqat Ali, Ali Hassan, Ahmed Ali, Wajahat Kazmi, Aamer Zaheer; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 1834-1843 Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks Qin Zou, Hanwen Jiang, Qiyu Dai, Yuanhao Yue, Long Chen, Qian Wang Horizon Detection Resources Horizon detection page by Nathan Jacobs, including horizon data and some nice papers on horizon detection. Dagger Efficient reductions for imitation learning Ross and Bagnell, AIStats, 10 (problems with imitation learning exposed) A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ross, Gordon and Bagnell, AIStats-11, (Dagger) ALVINN: an autonomous land vehicle in a neural network Dean Pomerlau, NIPS, 1989 (early - first?- autonomous driver) End to End Learning for Self-Driving CarsMariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, Karol Zieba (heavily improved and modernised alvinn; note the clever use of extra cameras to supply somewhat off-policy views) Planning in high dimensional spaces Notes on sampling based motion planning, chapter 5 of Lavalle's book "Planning Algorithms" Dubins, L.E. (July 1957). "On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents". American Journal of Mathematics. 79 (3): 497–516. J. A. Reeds and L. A. Shepp. Optimal paths for a car that goes both forwards and backwards. Pacific Journal of Mathematics, 145(2):367-393, 1990. Planning in the presence of Dynamics Notes on planning in the presence of dynamics , chapter 14 of Lavalle's book "Planning Algorithms"
29 Mar	Harder Planning Problems	Slides My slides on higher dimensional motion planning My slides on planning in the presence of dynamics, with a bit of stuff about motion graphs	Other Slides Very nice CMU motion planning slides, I believe by H. Choset A Planning Library The open mpl library and app Planning in high dimensional spaces Notes on sampling based motion planning, chapter 5 of Lavalle's book "Planning Algorithms" Planning in the presence of Dynamics Notes on planning in the presence of dynamics , chapter 14 of Lavalle's book "Planning Algorithms" This is not light reading, and goes farther than I do. Notes on systems and dynamics, including BVPs , chapter 15 of Lavalle's book "Planning Algorithms" This is not for the faint-hearted, and goes very much farther than I do. Motion Graphs Interactive motion generation from examples O. Arikan and D.A. Forsyth, 2002 Motion graphs L. Kovar, M. Gleicher and F.Pighin, 2002 Interactive control of avatars animated with human motion data Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, Nancy S Pollard, 2002 BEV papers and code Learning to Look around Objects for Top-View Representations of Outdoor Scenes Samuel Schulter, Menghua Zhai, Nathan Jacobs, Manmohan Chandraker A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker BEVFormer github page BEVFormer: a Cutting-edge Baseline for Camera-based DetectionLi, Zhiqi and Wang, Wenhai and Li, Hongyang and Xie, Enze and Sima, Chonghao and Lu, Tong and Qiao, Yu and Dai, Jifeng, 2022 TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, Andreas Geiger, 2022 TransFuser github page Depth from Single Image History Learning Depth from Single Monocular Images Ashutosh Saxena, Sung H. Chung, Andrew Y. Ng. In Neural Information Processing Systems (NIPS) 18, 2005 Automatic Photo Popup D. Hoiem, A. Efros, M. Hebert, 2005 Make3D: Learning 3D Structure from a Single Still Image A. Saxena, M. Sun, A. Ng, 2008 Recovering the spatial layout of cluttered roomsV. Hedau, D. Forsyth, D.Hoiem, 2009 Depth from Single Image - probe into SOTA Papers with code Midas github page Omnidata home page Normal from single image history A Theory of Origami World T. Kanade, 1980 Unfolding an indoor origami world D. Fouhey, A.Gupta, M. Hebert, 2014 Designing deep networks for surface normal estimationX. Wang, D. Fouhey, A.Gupta, 2015 Normal from single image - probe into SOTA Papers with code page Bae et al github page Omnidata page
29 Mar	Bird's Eye Views	Slides My slides on BEV's	BEV papers and code Learning to Look around Objects for Top-View Representations of Outdoor Scenes Samuel Schulter, Menghua Zhai, Nathan Jacobs, Manmohan Chandraker A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker BEVFormer github page BEVFormer: a Cutting-edge Baseline for Camera-based DetectionLi, Zhiqi and Wang, Wenhai and Li, Hongyang and Xie, Enze and Sima, Chonghao and Lu, Tong and Qiao, Yu and Dai, Jifeng, 2022 TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, Andreas Geiger, 2022 TransFuser github page Semantic segmentation The Role of Context for Object Detection and Semantic Segmentation in the Wild Roozbeh Mottaghi et al. (CVPR), 2014, pp. 891-898 Urban 3D Semantic Modelling Using Stereo Vision Sunando Sengupta Eric Greveson, Ali Shahrokni2 Philip H. S. Torr, ICRA, 2013 FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY Timo Hackel, Jan D. Wegner, Konrad Schindler "Fully Convolutional Networks for Semantic Segmentation" Jonathan Long∗ Evan Shelhamer∗ Trevor Darrell, 2014 The Stixel world: A medium-level representation of traffic scenes Marius Cordts, et al., ArXiv, 2017 Towards a Global Optimal Multi-Layer Stixel Representation of Dense 3D Data David Pfeiffer, Uwe Franke, BMVC 2011 Papers with code page Depth from Single Image History Learning Depth from Single Monocular Images Ashutosh Saxena, Sung H. Chung, Andrew Y. Ng. In Neural Information Processing Systems (NIPS) 18, 2005 Automatic Photo Popup D. Hoiem, A. Efros, M. Hebert, 2005 Make3D: Learning 3D Structure from a Single Still Image A. Saxena, M. Sun, A. Ng, 2008 Recovering the spatial layout of cluttered roomsV. Hedau, D. Forsyth, D.Hoiem, 2009 Depth from Single Image - probe into SOTA Papers with code Midas github page Omnidata home page Normal from single image history A Theory of Origami World T. Kanade, 1980 Unfolding an indoor origami world D. Fouhey, A.Gupta, M. Hebert, 2014 Designing deep networks for surface normal estimationX. Wang, D. Fouhey, A.Gupta, 2015 Normal from single image - probe into SOTA Papers with code page Bae et al github page Omnidata page
5 April	More Bird's Eye Views; Stereo and Optic Flow	Slides My slides on BEV technologies My slides on actual BEV's	Inpainting Texture synthesis by non-parametric sampling A. Efros and T. Leung, 1999 Scene completion using millions of photographs J. Hays and A. Efros, 2007 Object removal by exemplar-based inpainting A. Criminisi, P. Perez, K. Toyama, 2003 Region filling and object removal by exemplar based image inpainting A. Criminisi, P.Perez, K. Toyama, 2004 Image Inpainting via Generative Multi-column Convolutional Neural Networks Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, Jiaya Jia , 2018 Papers with code page Extracting and composing robust features with denoising autoencodersP.Vincent, H.Larochelle, Y.Bengio, P-A. Manzagol, Adversarial Losses Generative Adversarial Networks Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, 2014 NIPS 2016 Tutorial: Generative Adversarial NetworksI. Goodfellow, 2016 More BEV Papers Understanding Road Layout from Videos as a WholeB. Liu, B. Zhuang, S. Schulter, P. Ji, M. Chandraker, 2020 Weakly But Deeply Supervised Occlusion-Reasoned Parametric Road Layouts Liu, Zhuang Chandraker, 2022 Learning to Detect Mobile Objects from LiDAR ScansWithout LabelsYurong You, Katie Z Luo, Cheng Perng Phoo, Wei-Lun Chao, Wen Sun, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger AutoLay: Benchmarking amodal layout estimation for autonomous drivingKaustubh Mani, N. Sai Shankar, Krishna Murthy Jatavallabhula, K. Madhava Krishna MonoLayout: Amodal scene layout from a single imageKaustubh Mani, Swapnil Daga, Shubhika Garg, N. Sai Shankar, Krishna Murthy Jatavallabhula, K. Madhava Krishna Understanding Road Layout from Videos as a WholeBuyu Liu1 Bingbing Zhuang Samuel Schulter Pan Ji1 Manmohan Chandraker Learning to Look around Objects for Top-View Representations of Outdoor ScenesSamuel Schulter, Menghua Zhai, Nathan Jacobs, Manmohan Chandraker Weakly But Deeply Supervised Occlusion-Reasoned Parametric Road LayoutsBuyu Liu, Bingbing Zhuang, Manmohan Chandraker MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View MapsPengxiang Wu, Siheng Chen, Dimitris Metaxas BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous DrivingYunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, Jiwen Lu
12 April	Stereo, Optic Flow and Structure from Motion; Simulators	Slides My slides on stereo, optic flow, and structure from motion Movie This is in lieu of lecture 14 April - we will not meet that day! German Ros' lecture on simulators for autonomous driving	Resources - Books and Papers Multiple View Geometry in Computer Vision Second Edition Richard Hartley and Andrew Zisserman, Cambridge University Press, March 2004. Y. Boykov, O. Veksler and R. Zabih (2001), "Fast approximate energy minimisation via graph cuts", IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1222–1239. Li Hong and G. Chen, "Segment-based stereo matching using graph cuts," Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Washington, DC, USA, 2004, pp. I-I, doi: 10.1109/CVPR.2004.1315016. R. Szeliski et al., "A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 1068-1080, June 2008, doi: 10.1109/TPAMI.2007.70844. T. Brox, C. Bregler and J. Malik, "Large displacement optical flow", CVPR 2009 High Accuracy Optical Flow Estimation Based on a Theory for Warping, Thomas Brox, Andr´es Bruhn, Nils Papenberg, and Joachim Weickert, ECCV, 2004 Resources - Web: Middlebury Stereo Page (methods ranked in competition; datasets; software) Papers with Code Stereo Depth Estimation (various methods competing; various datasets; some implementations) Middlebury Optical Flow page (methods ranked in competition; datasets; software) Papers with Code Optic Flow(various methods competing; various datasets; some implementations) OpenSFM (open source structure from motion) MicMac (photogrammetry - SFM - from France's IGN) ColMap (another standard open source structure from motion library)
19 April	Weather	Effects at surfaces(these are revised and expanded) Scattering effects in the air(these are revised and expanded) Dealing with fog(these are revised and expanded) Dealing with rain(these are revised and expanded) Fog and rain affect lidar(these are revised and expanded)	General reading Great general reading with super pix Color and Light in Nature, David K. Lynch, William Livingston, Cambridge University Press, Jun 29, 1995 More great general readingLight and Color in the Outdoors by Marcel Minnaert (Author), L. Seymour (Translator), Springer, 1995 Haze and dehazing Vision in bad weather S.K. Nayar and S. Narasimhan, ICCV 1995 Single image dehazing Raanan Fattal, SIGGRAPH 08 DehazeNet: An End-to-End System for Single Image Haze Removal Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, Dacheng Tao, 2016 FFA-Net: Feature Fusion Attention Network for Single Image Dehazing Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, Huizhu Jia FD-GAN: Generative Adversarial Networks with Fusion-discriminator for Single Image Dehazing Yu Dong, Yihao Liu, He Zhang, Shifeng Chen, Yu Qiao Implicit Euler ODE Networks for Single-Image Dehazing Jiawei Shen, Zhuoyan Li, Lei Yu, Gui-Song Xia, Wen Yang CVPR 20 workshops ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding Christos Sakaridis, Dengxin Dai, Luc Van Gool Lidar and haze LIBRE: The Multiple 3D LiDAR Dataset Alexander Carballo, Jacob Lambert, Abraham Monrroy-Cano, David Robert Wong, Patiphon Narksri, Yuki Kitsukawa, Eijiro Takeuchi, Shinpei Kato, Kazuya Takeda Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather Martin Hahner, Christos Sakaridis, Dengxin Dai, Luc Van Gool Pointillism: accurate 3D bounding box estimation with multi-radars Kshitiz Bansal, Keshav Rungta,Siyuan Zhu,Dinesh Bharadia Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, Felix Heide Rain effects Object Detection Under Rainy Conditions for Autonomous Vehicles: A Review of State-of-the-Art and Emerging Techniques Mazin Hnewa, Hayder Radha, 2021 Vision and Rain K. Garg and S.K. Nayar International Journal of Computer Vision volume 75, pages3–27(2007) Programmable Automotive Headlights Robert Tamburo, Eriko Nurvitadhi, Abhishek Chugh, Mei Chen, Anthony Rowe, Takeo Kanade, Srinivasa G. Narasimhan Rain simulation Rain rendering for evaluating and improving robustness to bad weather Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, Jean-François Lalonde Rain removal Restoring An Image Taken Through a Window Covered with Dirt or Rain David Eigen Dilip Krishnan Rob Fergus, ICCV 13 Depth-attentional Features for Single-image Rain Removal Xiaowei Hu, Chi-Wing Fu, Lei Zhu2 and Pheng-Ann Heng, CVPR 2019 Rain Streak Removal Using Layer Priors Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, Michael S. Brown, CVPR 16 Semi-supervised Transfer Learning for Image Rain Removal Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, Ying Wu, CVPR 19 Deep Joint Rain Detection and Removal from a Single Image Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan, CVPR 17
26 April	More Learned Controll	Deeper into imitation learning for vehicles Setting up inverse reinforcement learning Inverse Reinforcement Learning	Harder imitation learning Learning by Watching Jimuyang Zhang, Eshed Ohn-Bar, 2021 Exploring Data Aggregation in Policy Learning for Vision-Based Urban Autonomous Driving Aditya Prakash, Aseem Behl, Eshed Ohn-Bar, Kashyap Chitta, Andreas Geiger, 2020 End-to-end Driving via Conditional Imitation LearningFelipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, Alexey Dosovitskiy
26 April	Reinforcement learning ideas to set up inverse reinforcement learning; inverse reinforcement learning	Slides Value iteration, policy iteration, Q learning, REINFORCE and actor critic, setting up a language Inverse optimal control and inverse reinforcement learning	Resources Slides from Fei-Fei, Johnson and Yeung course (used a couple of these) Slides from Abbeel on inverse reinforcement learning(used several of these) An inverse optimal control approach to human motion modeling Katja Mombaur, Jean-Paul Laumond and Anh Truong ISRR 2009 (rather a nice account of inverse optimal control) Learning to Search: Functional Gradient Techniques for Imitation Learning Nathan D. Ratliff, David Silver, J. Andrew Bagnell (super paper, which most people find quite tough going; goes further than we will in class) Maximum margin planning Nathan D. Ratliff, J. Andrew Bagnell Martin A. Zinkevich (another super paper, also goes further than we will in class)