UIUC CS-588 Autonomous Vehicle System Engineering Outline

This page contains what we've done

Texts:

Applied Machine Learning D.A. Forsyth, Springer, 2019

Important The PDF is a free download from the UIUC library (you have to be on the intranet to download it, I think). Or you could buy a paper copy.

Mediaspace Channel

The class recordings appear here, and should be accessible to you. When I remember to wear the microphone, you can hear the lecture, too!

Week Starting	Topic	Reading Materials	Movie List
25 Aug	Admin	Nothing so far!	These are 2020 movies; the 2021 movies are on mediaspace. but you might find you prefer these, or it may amuse you to check whether camera geometry has changed over the last year, etc.
25 Aug	Safety	Vehicle safety rules My slides on intro and safety	Movie of the first day goes here
25 Aug	Intellectual Context	Technical Intro slides	Intro lecture recording will go here.
25 Aug	(Y) Basic Convolutional Neural Networks	Deep nets: Units; AML: 16.1 Deep nets: Gradients; AML: 16.2 Deep nets: Backpropagation; AML 16.3 Deep nets: Gradient tricks; AML 16.4 Deep nets: Convolutional layers; AML 17.1 Deep nets: More on convolutional layers; AML 17.1, 17.2	Units Gradients Backprop Gradient tricks Convolutional layers More conv layers
27 Aug	Safety lookout training	Vehicle safety rules	The safety lookout video
27 Aug	Image classification and simple detection	Basic object detection: AML 19	Image classifiers Object detectors
WE 3 Sep	Simple PID Control	My PID control slides (now fixed so as to reflect class) Rather nice pocket guide to tuning PID controllers	Old PID Control lecture Old PID Control lecture -II Old PID Control lecture - leftover bits
WE 10 Sep	Point set registration, ROS, PACMOD and the simulator	I'm away 10 Sep In person Point set registration (3 Sep) Slides on aligning point sets see also Weighted least squares solutions for rotation translation; very nice notes from Olga Sorkine-Hornung and Michael Rabinovitch Slides on IRLS (best if you've watched the robust regression movie) see also Notes on robustness and iteratively reweighted least squares see also Robust registration of point sets using iteratively reweighted least squares P. Bergstrom and A.Edlund, Computational Optimization and Applications July 2014, Volume 58, Issue 3, pp 543–561 (click from inside UIUC intranet!) Simple bundle adjustment Enrichment Geometrically Stable Sampling for the ICP AlgorithmN. Gelfand; L. Ikemoto; S. Rusinkiewicz; M. Levoy, (3DIMPVT) 2003 Efficient Variants of the ICP AlgorithmS.Rusinkiewicz and M. Levoy, (3DIMPVT), 2001 Translation Synchronization via Truncated Least SquaresX Huang, Z Liang, C. Bajaj, Q. Huang, NeurIPS17 Uncertainty quantification for multi-scan registration X Huang, Z Liang, Q Huang ACM Transactions on Graphics (TOG) 39 (4), 130: 1-130: 24 In person The vehicle simulator (10 Sep) by video:ROS and PACMOD "My" ROS and PACMOD slides Original detailed slides, with examples, etc.from Roi Yehoshua at Bar-Ilan U.	Romping through ROS; Closed form estimates of rotation and translation from point correspondences Robust regression using IRLS This sets up my IRLS lecture IRLS and bundle adjustment
WE 17 Sep	Finish point set registration; Kalman, extended Kalman and particle filters	In person Filtering An analogy between visual tracking and localization Slides on Kalman filters and EKFs Enrichment Kris Kitani's Kalman filtering slides Pieter Abbeel's Kalman filtering slides Particle filter slides by Pieter Abbeel Particle filter slides by Cyrill Stachniss Particle filter notes by Drew Bagnell, scribes Greg Seyfarth, Zachary Batts, David Fouhey A chapter on particle filtersthat didn't make it into "Computer Vision: A Modern Approach"	Simplest tracking and the Kalman filter Basic Particle Filters
WE 24 Sep	Particle Filters; Cameras and pairs of cameras	I'm away 24 Sep Below is a video lecture in lieu. This does cameras and homogeneous coordinates and camera matrices in some detail. I'll do pairs of cameras in person. 24 Sep video lecture on cameras, big version (500M, but sharper) 24 Sep video lecture on cameras, smaller version (200M) Slides and references My slides on particle filters Slides on perspective and affine camera models Brush up your homogeneous coordinates with wikipedia Slides on pairs of cameras	Finish off particle filters; start cameras Finish basic cameras; start pairs of cameras
WE 1 Oct	Pairs of cameras; flow and stereo; SFM, SLAM and visual odometry	Slides on pairs of cameras (slightly revised) Multiple View Geometry in Computer Vision Second Edition Richard Hartley and Andrew Zisserman, Cambridge University Press, March 2004. Y. Boykov, O. Veksler and R. Zabih (2001), "Fast approximate energy minimisation via graph cuts", IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1222–1239. Li Hong and G. Chen, "Segment-based stereo matching using graph cuts," Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Washington, DC, USA, 2004, pp. I-I, doi: 10.1109/CVPR.2004.1315016. R. Szeliski et al., "A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 1068-1080, June 2008, doi: 10.1109/TPAMI.2007.70844. T. Brox, C. Bregler and J. Malik, "Large displacement optical flow", CVPR 2009 High Accuracy Optical Flow Estimation Based on a Theory for Warping, Thomas Brox, Andr´es Bruhn, Nils Papenberg, and Joachim Weickert, ECCV, 2004 My slides on very simplest SFM and SLAM My slides on basic visual odometry Tutorial on visual slam at CVPR2014 I found the following particularly helpful Visual odometry Stephan Weiss Stereo Visual odometryChris Beall Dense odometryRichard Newcombe	Finish off pairs of cameras Factorization based SFM; simple monocular odometry
WE 8 Oct	EKFSLAM, FastSLAM, Direct methods	Slides on Features and Interest points Significantly revised slides on EKF SLAM Great notes on EKF-SLAM by Joan Sola, with Matlab code, etc	EKF-Slam and starting FASTSLAM finish FASTSLAM; interest points and features; direct methods and SVO
WE 15 Oct	finish EKFSLAM, Direct methods, FAST SLAM (movie); start path planning	EKF SLAM Significantly revised slides on EKF SLAM Great notes on EKF-SLAM by Joan Sola, with Matlab code, etc Direct methods Slides on Direct SLAM Christian Forster, Zichao Zhang, Michael Gassner, Manuel Werlberger, Davide Scaramuzza SVO: Semi-Direct Visual Odometry for Monocular and Multi-Camera Systems IEEE Transactions on Robotics, Vol. 33, Issue 2, pages 249-265, Apr. 2017. Dense visual slam an earlier version of lsd-slam for RGB-D cameras, code too Dense Visual SLAM for RGB-D Cameras (C. Kerl, J. Sturm and D. Cremers), In Proc. of the Int. Conf. on Intelligent Robot Systems (IROS), 2013. FAST SLAM Slides on FastSLAM Slides from Ben Kuipers on FastSLAM Slides from Burgard et al on FastSLAM FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem Michael Montemerlo and Sebastian Thrun and Daphne Koller and Ben Wegbreit AAAI 02 FastSLAM 2.0: an improved particle filtering algorithm for simultaneous localization and mapping that provably converges Michael Steven Montemerlo and Sebastian B Thrun and Daphne Koller and Ben Wegbreit IJCAI -03 lsdSLAM website - there is a pointer to a github site with code here. SVO (fast semi-direct visual odometry) there is a pointer to code here, too.	Edited movie in lieu of 15 Oct lecture HR version (but big 500M!) LR version (smaller, but not as much as I'd like) Last years movies EKF-Slam and starting FASTSLAM finish FASTSLAM; interest points and features; direct methods and SVO
WE 22 Oct	Motion planning	My slides on motion planning - I My slides on higher dimensional motion planning My slides on planning in the presence of dynamics, with a bit of stuff about motion graphs Notes on sampling based motion planning, chapter 5 of Lavalle's book "Planning Algorithms" Notes on planning in the presence of dynamics , chapter 14 of Lavalle's book "Planning Algorithms" Dubins, L.E. (July 1957). "On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents". American Journal of Mathematics. 79 (3): 497–516. J. A. Reeds and L. A. Shepp. Optimal paths for a car that goes both forwards and backwards. Pacific Journal of Mathematics, 145(2):367-393, 1990. The open mpl library and app Very nice CMU motion planning slides, I believe by H. Choset	Basic motion planning; cells, potential fields, etc.
WE 29 Oct	Lane boundaries to scenes	Slides Lane boundary detection Building scene maps - I Semantic segmentation - overview GrabCut, MRFs, CRFs, graph cut, and variational inference Background Movies Boltzmann machines Discrete Markov random fields Variational Free Energy Simple variational inference: example Papers Lane Boundaries US Patent 9081385 (lane boundaries) Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection Lucas Tabelini, Rodrigo Berriel, Thiago M. Paixão, Claudine Badue, Alberto F. De Souza, Thiago Oliveira-Santos Code Off-road Path Following using Region Classification and Geometric Projection Constraints Y. Alon; A. Ferencz; A. Shashua, CVPR 06 Detection and tracking of boundary of unmarked roads Young-Woo Seo; Ragunathan Raj Rajkumar, 17'th international conf on information fusion, 2014 Geometric Constrained Joint Lane Segmentation and Lane Boundary Detection Jie Zhang, Yi Xu, Bingbing Ni, Zhenyu Duan; Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 486-502 Lane detection using lane boundary marker network with road geometry constraints Hussam Ullah Khan, Afsheen Rafaqat Ali, Ali Hassan, Ahmed Ali, Wajahat Kazmi, Aamer Zaheer; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 1834-1843 Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks Qin Zou, Hanwen Jiang, Qiyu Dai, Yuanhao Yue, Long Chen, Qian Wang Horizon detection page by Nathan Jacobs, including horizon data and some nice papers on horizon detection. Simple Maps Learning from Maps: Visual Common Sense for Autonomous Driving Ari Seff, Jianxiong Xiao 3D Traffic Scene Understanding from Movable Platforms. Andreas Geiger. Martin Lauer. Christian Wojek. Christoph Stiller. Raquel Urtasun Semantic segmentation The Role of Context for Object Detection and Semantic Segmentation in the Wild Roozbeh Mottaghi et al. (CVPR), 2014, pp. 891-898 Urban 3D Semantic Modelling Using Stereo Vision Sunando Sengupta Eric Greveson, Ali Shahrokni2 Philip H. S. Torr, ICRA, 2013 FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY Timo Hackel, Jan D. Wegner, Konrad Schindler "Fully Convolutional Networks for Semantic Segmentation" Jonathan Long∗ Evan Shelhamer∗ Trevor Darrell, 2014 The Stixel world: A medium-level representation of traffic scenes Marius Cordts, et al., ArXiv, 2017 Towards a Global Optimal Multi-Layer Stixel Representation of Dense 3D Data David Pfeiffer, Uwe Franke, BMVC 2011 Graph cuts GREIG, D., PORTEOUS, B., AND SEHEULT, A. 1989. Exact MAP estimation for binary images. J. Roy. Stat. Soc. B. 51, 271–279.
WE 5 Nov	finish Semantic segmentation	Slides My slides on non-local means, bilateral filters, and fully connected CRFS Papers Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials Philipp Krähenbühl, Vladlen Koltun The Fast Bilateral Solver Jonathan T. Barron, Ben Poole Fast Bilateral-Space Stereo for Synthetic DefocusJonathan T Barron Andrew Adams YiChang Shih Carlos Hernández CVPR (2015) Fast high-dimensional filtering using the permutohedral lattice.A. Adams, J. Baek, and M. A. Davis. Computer Graphics Forum, 29(2), 2010. 2, 5	Setting up FCCRFs and non-local means
12 Nov	Weather	My slides on physical effects on appearance, rain and on intrinsic images(these are revised and expanded) General reading Great general reading with super pix Color and Light in Nature, David K. Lynch, William Livingston, Cambridge University Press, Jun 29, 1995 More great general readingLight and Color in the Outdoors by Marcel Minnaert (Author), L. Seymour (Translator), Springer, 1995 Haze and dehazing Vision in bad weather S.K. Nayar and S. Narasimhan, ICCV 1995 Single image dehazing Raanan Fattal, SIGGRAPH 08 DehazeNet: An End-to-End System for Single Image Haze Removal Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, Dacheng Tao, 2016 FFA-Net: Feature Fusion Attention Network for Single Image Dehazing Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, Huizhu Jia FD-GAN: Generative Adversarial Networks with Fusion-discriminator for Single Image Dehazing Yu Dong, Yihao Liu, He Zhang, Shifeng Chen, Yu Qiao Implicit Euler ODE Networks for Single-Image Dehazing Jiawei Shen, Zhuoyan Li, Lei Yu, Gui-Song Xia, Wen Yang CVPR 20 workshops ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding Christos Sakaridis, Dengxin Dai, Luc Van Gool Lidar and haze LIBRE: The Multiple 3D LiDAR Dataset Alexander Carballo, Jacob Lambert, Abraham Monrroy-Cano, David Robert Wong, Patiphon Narksri, Yuki Kitsukawa, Eijiro Takeuchi, Shinpei Kato, Kazuya Takeda Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather Martin Hahner, Christos Sakaridis, Dengxin Dai, Luc Van Gool Pointillism: accurate 3D bounding box estimation with multi-radars Kshitiz Bansal, Keshav Rungta,Siyuan Zhu,Dinesh Bharadia Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, Felix Heide Rain effects Object Detection Under Rainy Conditions for Autonomous Vehicles: A Review of State-of-the-Art and Emerging Techniques Mazin Hnewa, Hayder Radha, 2021 Vision and Rain K. Garg and S.K. Nayar International Journal of Computer Vision volume 75, pages3–27(2007) Programmable Automotive Headlights Robert Tamburo, Eriko Nurvitadhi, Abhishek Chugh, Mei Chen, Anthony Rowe, Takeo Kanade, Srinivasa G. Narasimhan Rain simulation Rain rendering for evaluating and improving robustness to bad weather Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, Jean-François Lalonde Rain removal Restoring An Image Taken Through a Window Covered with Dirt or Rain David Eigen Dilip Krishnan Rob Fergus, ICCV 13 Depth-attentional Features for Single-image Rain Removal Xiaowei Hu, Chi-Wing Fu, Lei Zhu2 and Pheng-Ann Heng, CVPR 2019 Rain Streak Removal Using Layer Priors Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, Michael S. Brown, CVPR 16 Semi-supervised Transfer Learning for Image Rain Removal Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, Ying Wu, CVPR 19 Deep Joint Rain Detection and Removal from a Single Image Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan, CVPR 17	Appearance physics and rain; rain removal methods Intrinsic images, curtailed by Senate Meeting
19 Nov	Intrinsic images and ground maps	My slides on intrinsic images and adversarial losses (these are revised and expanded) My slides on ground maps Intrinsic Image and Adversarial Loss readings Intrinsic image decomposition using paradigms D.A. Forsyth and Jason Rock, 2021 Slides by Roger Grosse on Generative Adversarial Networks GAN slides by Kushboo Thaker Ground Map Readings 3D Traffic Scene Understanding from Movable Platforms. Andreas Geiger. Martin Lauer. Christian Wojek. Christoph Stiller. Raquel Urtasun Learning to Look around Objects for Top-View Representations of Outdoor Scenes Samuel Schulter, Menghua Zhai, Nathan Jacobs, Manmohan Chandraker A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker Scene flow Object Scene Flow for Autonomous VehiclesMoritz Menze and Andreas Geiger Just Go With the Flow: Self-Supervised Scene Flow Estimation Himangi Mittal, Brian Okorn, David Held Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes Fabian Brickwedde, Steffen Abraham, Rudolf Mester Self-Supervised Monocular Scene Flow Estimation Junhwa Hur, Stefan Roth Upgrading Optical Flow to 3D Scene Flow Through Optical Expansion Gengshan Yang, Deva Ramanan FlowNet3D: Learning Scene Flow in 3D Point Clouds Xingyu Liu, Charles R. Qi, Leonidas J. Guibas Clustering Unsupervised Deep Embedding for Clustering Analysis Junyuan Xie, Ross Girshick, Ali Farhadi Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes Fabian Brickwedde, Steffen Abraham, Rudolf Mester Just Go with the Flow: Self-Supervised Scene Flow Estimation Himangi Mittal, Brian Okorn, David Held Object Scene Flow for Autonomous Vehicles M. Menze and A. Geiger Self-Supervised Monocular Scene Flow Estimation Junhwa Hur, Stefan Roth A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker Attribute discovery via predictable discriminative binary codes. M Rastegari, A Farhadi, D Forsyth.
3 Dec	Learning to control	Slides "My" slides on reinforcement and imitation learning heavily revised to emphasize the link to structured prediction Slides on reinforcement learning by Fei-Fei Li, Justin Johnson and Serena Yeung Slides on imitation learning by Katerina Fragkiadaki Slides from Tom Mitchell on Imitation learning(First part is rather similar to Fragkiadaki's slides, above; but the second part is different!) Movies German Ros' lecture on simulators for autonomous driving Structured prediction in a little more detail Movie used for AML Structured prediction in a little more detail -II (optimization) Another Movie used for AML Papers Efficient reductions for imitation learning Ross and Bagnell, AIStats, 10 (problems with imitation learning exposed) A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ross, Gordon and Bagnell, AIStats-11, (Dagger) ALVINN: an autonomous land vehicle in a neural network Dean Pomerlau, NIPS, 1989 (early - first?- autonomous driver) Reinforcement and Imitation Learning via Interactive No-Regret Learning Stephane Ross, J. Andrew Bagnell, NIPS 2014 (This is AGGREVATE) Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, J. Andrew Bagnell (this is the AGGREVATED paper; who could have guessed from title?) Learning to Search: Functional Gradient Techniques for Imitation LearningNathan Ratliffe, David Silver, Andrew Bagnell (Learning to search=LEARCH) End to End Learning for Self-Driving CarsMariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, Karol Zieba (heavily improved and modernised alvinn; note the clever use of extra cameras to supply somewhat off-policy views)	Imitiation learning - I Imitiation learning - II, and Intrinsic images finish off