UIUC CS-588 Autonomous Vehicle System Engineering Outline

This page contains what we've done

Texts:

Applied Machine Learning D.A. Forsyth, Springer, 2019

This is purely for background material on deep learning and other useful data tricks.

Important The PDF is a free download from the UIUC library (you have to be on the intranet to download it, I think). Or you could buy a paper copy.

Past Mediaspace Channel

The class recordings for last year appear here, and should be accessible to you. When I remember to wear the microphone, you can hear the lecture, too!

Current Mediaspace Channel

The class recordings for this year should appear here, and should be accessible to you. When I remember to wear the microphone, you can hear the lecture, too!

Outline based on last year

Week Starting	Topic	Reading Materials	Movie List
24 Aug	Admin	Nothing so far!	These are 2020 movies; the 2022 movies are on mediaspace. but you might find you prefer these, or it may amuse you to check whether camera geometry has changed over the last years, etc.
24 Aug	Safety	Vehicle safety rules My slides on intro and safety	Movie of the first day goes here
24 Aug	Intellectual Context	Technical Intro slides	Intro lecture recording will go here.
26 Aug	Safety lookout training AT HIGH BAY	Vehicle safety rules	The safety lookout video
31 Aug	(Y) Basic Convolutional Neural Networks	I'm away 31 Aug Deep nets: Units; AML: 16.1 Deep nets: Gradients; AML: 16.2 Deep nets: Backpropagation; AML 16.3 Deep nets: Gradient tricks; AML 16.4 Deep nets: Convolutional layers; AML 17.1 Deep nets: More on convolutional layers; AML 17.1, 17.2	Units Gradients Backprop Gradient tricks Convolutional layers More conv layers
2 Sep	Image classification and simple detection	I'm away 2 Sep Basic object detection: AML 19	Image classifiers Object detectors
2 Sep	ROS, PACMOD and the simulator	I'm away 2 Sep by video:ROS and PACMOD "My" ROS and PACMOD slides Original detailed slides, with examples, etc.from Roi Yehoshua at Bar-Ilan U. by video: The Simulator Lecture on the simulator in 2021 by Xiaoming Zhao	Romping through ROS; Closed form estimates of rotation and translation from point correspondences
7 Sep	Point set registration	In person Point set registration (3 Sep) Slides on aligning point sets see also Weighted least squares solutions for rotation translation; very nice notes from Olga Sorkine-Hornung and Michael Rabinovitch Slides on IRLS (best if you've watched the robust regression movie) see also Notes on robustness and iteratively reweighted least squares see also Robust registration of point sets using iteratively reweighted least squares P. Bergstrom and A.Edlund, Computational Optimization and Applications July 2014, Volume 58, Issue 3, pp 543–561 (click from inside UIUC intranet!) Simple bundle adjustment Enrichment Geometrically Stable Sampling for the ICP AlgorithmN. Gelfand; L. Ikemoto; S. Rusinkiewicz; M. Levoy, (3DIMPVT) 2003 Efficient Variants of the ICP AlgorithmS.Rusinkiewicz and M. Levoy, (3DIMPVT), 2001 Translation Synchronization via Truncated Least SquaresX Huang, Z Liang, C. Bajaj, Q. Huang, NeurIPS17 Uncertainty quantification for multi-scan registration X Huang, Z Liang, Q Huang ACM Transactions on Graphics (TOG) 39 (4), 130: 1-130: 24	Point set registration from 2021 (a) Robust regression using IRLS This sets up my IRLS lecture
9 Sep	Simple PID Control	I'm away 9 Sep My PID control slides (now fixed so as to reflect class) Rather nice pocket guide to tuning PID controllers	PID control lecture -1 on video from 2021 PID control lecture - 2 on video from 2021 Old PID Control lecture Old PID Control lecture -II Old PID Control lecture - leftover bits
14 Sep	Finish point set registration; start Tracking and Kalman Filtering	Point set registration material is under 7 Sep Filtering An analogy between visual tracking and localization Slides on Kalman filters and EKFs Enrichment Kris Kitani's Kalman filtering slides Pieter Abbeel's Kalman filtering slides Particle filter slides by Pieter Abbeel Particle filter slides by Cyrill Stachniss Particle filter notes by Drew Bagnell, scribes Greg Seyfarth, Zachary Batts, David Fouhey A chapter on particle filtersthat didn't make it into "Computer Vision: A Modern Approach"	Simplest tracking and the Kalman filter Basic Particle Filters
16 Sep	Finish Kalman Filtering and set up Kalman Filtering	Slides, etc. for Kalman, EKF are under 14 Sep	Simplest tracking and the Kalman filter Basic Particle Filters
21 Sep	skip Particle Filters; Cameras and pairs of cameras	Slides and references Basic camera models Camera matrices, intrinsics and extrinsics Camera calibration Chapters on cameras and calibration from new book Brush up your homogeneous coordinates with wikipedia	Finish off particle filters; start cameras Finish basic cameras; start pairs of cameras
23 Sep	Pairs of cameras and visual odometry	Movie The lecture took a movie form, as I was unwell and didn't want to pass on whatever I had. Movie is here Slides The geometry of two cameras Visual odometry with just one camera Resources Multiple View Geometry in Computer Vision Second Edition Richard Hartley and Andrew Zisserman, Cambridge University Press, March 2004. Visual odometry Stephan Weiss Stereo Visual odometryChris Beall Dense odometryRichard Newcombe
28 Sep	Feature points; Simple SFM ideas; EKF-SLAM	Slides on Features and Interest points Simplest structure from motion and SLAM Significantly revised slides on EKF SLAM Resources SuperPoint: Self-Supervised Interest Point Detection and Description DeTone, Malisciewicz, Rabinovich, 2018 An implementation of SuperPoint Tutorial on visual slam at CVPR2014 I found the following particularly helpful Visual odometry Stephan Weiss Stereo Visual odometryChris Beall Dense odometryRichard Newcombe Great notes on EKF-SLAM by Joan Sola, with Matlab code, etc
30 Sep	EKF-SLAM; quick stereo and optical flow	Significantly revised slides on EKF SLAM Slides on Stereo and Optical Flow Resources SuperPoint: Self-Supervised Interest Point Detection and Description DeTone, Malisciewicz, Rabinovich, 2018 An implementation of SuperPoint Tutorial on visual slam at CVPR2014 I found the following particularly helpful Visual odometry Stephan Weiss Stereo Visual odometryChris Beall Dense odometryRichard Newcombe Great notes on EKF-SLAM by Joan Sola, with Matlab code, etc
5 Oct	More quick stereo and optical flow; semi-direct SLAM	Significantly revised slides on Semi Direct SLAM Resources Christian Forster, Zichao Zhang, Michael Gassner, Manuel Werlberger, Davide Scaramuzza SVO: Semi-Direct Visual Odometry for Monocular and Multi-Camera Systems IEEE Transactions on Robotics, Vol. 33, Issue 2, pages 249-265, Apr. 2017. Dense visual slam an earlier version of lsd-slam for RGB-D cameras, code too Dense Visual SLAM for RGB-D Cameras (C. Kerl, J. Sturm and D. Cremers), In Proc. of the Int. Conf. on Intelligent Robot Systems (IROS), 2013.
7 Oct	Start motion planning	My slides on motion planning - I My slides on higher dimensional motion planning The open mpl library and app Very nice CMU motion planning slides, I believe by H. Choset
12 Oct	More motion planning	My slides on higher dimensional motion planning My slides on planning in the presence of dynamics, with a bit of stuff about motion graphs Notes on sampling based motion planning, chapter 5 of Lavalle's book "Planning Algorithms" Notes on planning in the presence of dynamics , chapter 14 of Lavalle's book "Planning Algorithms" Dubins, L.E. (July 1957). "On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents". American Journal of Mathematics. 79 (3): 497–516. J. A. Reeds and L. A. Shepp. Optimal paths for a car that goes both forwards and backwards. Pacific Journal of Mathematics, 145(2):367-393, 1990. The open mpl library and app Very nice CMU motion planning slides, I believe by H. Choset
14 Oct	End planning with dynamics; start lane boundaries to scenes	Slides Lane boundary detection Papers US Patent 9081385 (lane boundaries) Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection Lucas Tabelini, Rodrigo Berriel, Thiago M. Paixão, Claudine Badue, Alberto F. De Souza, Thiago Oliveira-Santos Code Off-road Path Following using Region Classification and Geometric Projection Constraints Y. Alon; A. Ferencz; A. Shashua, CVPR 06 Detection and tracking of boundary of unmarked roads Young-Woo Seo; Ragunathan Raj Rajkumar, 17'th international conf on information fusion, 2014 Geometric Constrained Joint Lane Segmentation and Lane Boundary Detection Jie Zhang, Yi Xu, Bingbing Ni, Zhenyu Duan; Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 486-502 Lane detection using lane boundary marker network with road geometry constraints Hussam Ullah Khan, Afsheen Rafaqat Ali, Ali Hassan, Ahmed Ali, Wajahat Kazmi, Aamer Zaheer; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 1834-1843 Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks Qin Zou, Hanwen Jiang, Qiyu Dai, Yuanhao Yue, Long Chen, Qian Wang Horizon detection page by Nathan Jacobs, including horizon data and some nice papers on horizon detection.
18 Oct	We did not meet
21 Oct	Finish lane boundaries; simple scene maps	Slides Simple scene maps Ideas of Semantic Segmentation Papers Simple Maps Learning from Maps: Visual Common Sense for Autonomous Driving Ari Seff, Jianxiong Xiao Basic Semantic segmentation The Role of Context for Object Detection and Semantic Segmentation in the Wild Roozbeh Mottaghi et al. (CVPR), 2014, pp. 891-898 Urban 3D Semantic Modelling Using Stereo Vision Sunando Sengupta Eric Greveson, Ali Shahrokni2 Philip H. S. Torr, ICRA, 2013 FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY Timo Hackel, Jan D. Wegner, Konrad Schindler "Fully Convolutional Networks for Semantic Segmentation" Jonathan Long∗ Evan Shelhamer∗ Trevor Darrell, 2014 The Stixel world: A medium-level representation of traffic scenes Marius Cordts, et al., ArXiv, 2017 Towards a Global Optimal Multi-Layer Stixel Representation of Dense 3D Data David Pfeiffer, Uwe Franke, BMVC 2011
26 Oct	More sophisticated scene maps	Notes Comprehensive notes on single camera estimates of ground planes Slides More sophisticated scene maps Papers Ground Map Readings 3D Traffic Scene Understanding from Movable Platforms. Andreas Geiger. Martin Lauer. Christian Wojek. Christoph Stiller. Raquel Urtasun Learning to Look around Objects for Top-View Representations of Outdoor Scenes Samuel Schulter, Menghua Zhai, Nathan Jacobs, Manmohan Chandraker A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker
28 Oct	Intermission: Basic learned control and imitation learning	Slides basic ideas of learned control; simple imitation learning; DAGGER Papers Efficient reductions for imitation learning Ross and Bagnell, AIStats, 10 (problems with imitation learning exposed) A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ross, Gordon and Bagnell, AIStats-11, (Dagger) ALVINN: an autonomous land vehicle in a neural network Dean Pomerlau, NIPS, 1989 (early - first?- autonomous driver)
2 Nov	More sophisticated scene maps	Slides More sophisticated scene maps (revised) Scene Flow Papers Ground Map Readings 3D Traffic Scene Understanding from Movable Platforms. Andreas Geiger. Martin Lauer. Christian Wojek. Christoph Stiller. Raquel Urtasun Learning to Look around Objects for Top-View Representations of Outdoor Scenes Samuel Schulter, Menghua Zhai, Nathan Jacobs, Manmohan Chandraker A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker Scene Flow Readings Object Scene Flow for Autonomous VehiclesMoritz Menze and Andreas Geiger Just Go With the Flow: Self-Supervised Scene Flow Estimation Himangi Mittal, Brian Okorn, David Held Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes Fabian Brickwedde, Steffen Abraham, Rudolf Mester Self-Supervised Monocular Scene Flow Estimation Junhwa Hur, Stefan Roth Upgrading Optical Flow to 3D Scene Flow Through Optical Expansion Gengshan Yang, Deva Ramanan FlowNet3D: Learning Scene Flow in 3D Point Clouds Xingyu Liu, Charles R. Qi, Leonidas J. Guibas Clustering Unsupervised Deep Embedding for Clustering Analysis Junyuan Xie, Ross Girshick, Ali Farhadi Attribute discovery via predictable discriminative binary codes. M Rastegari, A Farhadi, D Forsyth.
4 Nov	Simulation We will not meet	Movie German Ros' lecture on simulators for autonomous driving
9 Nov	Finish Scene FLow; Weather	My slides on physical effects on appearance, rain and on intrinsic images(these are revised and expanded) General reading Great general reading with super pix Color and Light in Nature, David K. Lynch, William Livingston, Cambridge University Press, Jun 29, 1995 More great general readingLight and Color in the Outdoors by Marcel Minnaert (Author), L. Seymour (Translator), Springer, 1995 Haze and dehazing Vision in bad weather S.K. Nayar and S. Narasimhan, ICCV 1995 Single image dehazing Raanan Fattal, SIGGRAPH 08 DehazeNet: An End-to-End System for Single Image Haze Removal Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, Dacheng Tao, 2016 FFA-Net: Feature Fusion Attention Network for Single Image Dehazing Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, Huizhu Jia FD-GAN: Generative Adversarial Networks with Fusion-discriminator for Single Image Dehazing Yu Dong, Yihao Liu, He Zhang, Shifeng Chen, Yu Qiao Implicit Euler ODE Networks for Single-Image Dehazing Jiawei Shen, Zhuoyan Li, Lei Yu, Gui-Song Xia, Wen Yang CVPR 20 workshops ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding Christos Sakaridis, Dengxin Dai, Luc Van Gool Lidar and haze LIBRE: The Multiple 3D LiDAR Dataset Alexander Carballo, Jacob Lambert, Abraham Monrroy-Cano, David Robert Wong, Patiphon Narksri, Yuki Kitsukawa, Eijiro Takeuchi, Shinpei Kato, Kazuya Takeda Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather Martin Hahner, Christos Sakaridis, Dengxin Dai, Luc Van Gool Pointillism: accurate 3D bounding box estimation with multi-radars Kshitiz Bansal, Keshav Rungta,Siyuan Zhu,Dinesh Bharadia Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, Felix Heide Rain effects Object Detection Under Rainy Conditions for Autonomous Vehicles: A Review of State-of-the-Art and Emerging Techniques Mazin Hnewa, Hayder Radha, 2021 Vision and Rain K. Garg and S.K. Nayar International Journal of Computer Vision volume 75, pages3–27(2007) Programmable Automotive Headlights Robert Tamburo, Eriko Nurvitadhi, Abhishek Chugh, Mei Chen, Anthony Rowe, Takeo Kanade, Srinivasa G. Narasimhan Rain simulation Rain rendering for evaluating and improving robustness to bad weather Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, Jean-François Lalonde Rain removal Restoring An Image Taken Through a Window Covered with Dirt or Rain David Eigen Dilip Krishnan Rob Fergus, ICCV 13 Depth-attentional Features for Single-image Rain Removal Xiaowei Hu, Chi-Wing Fu, Lei Zhu2 and Pheng-Ann Heng, CVPR 2019 Rain Streak Removal Using Layer Priors Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, Michael S. Brown, CVPR 16 Semi-supervised Transfer Learning for Image Rain Removal Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, Ying Wu, CVPR 19 Deep Joint Rain Detection and Removal from a Single Image Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan, CVPR 17
11 Nov	More Weather, papers above	There are enough papers in previous cell!
16 Nov	Finish Weather, Intrinsic images	Intrinsic image slides An intrinsic image paper(which is a start to a huge subject) Intrinsic image decomposition using paradigms D.A. Forsyth and Jason Rock, 2021
18 Nov	Finish Intrinsic images; MDPs	Basics of Markov Decision Processes
30 Nov	More reinforcement learning, and start inverse reinforcement learning	Slides Value iteration, policy iteration, Q learning, REINFORCE and actor critic! Inverse optimal control and inverse reinforcement learning Resources Slides from Fei-Fei, Johnson and Yeung course (used a couple of these) Slides from Abbeel on inverse reinforcement learning(used several of these) An inverse optimal control approach to human motion modeling Katja Mombaur, Jean-Paul Laumond and Anh Truong ISRR 2009 (rather a nice account of inverse optimal control) Learning to Search: Functional Gradient Techniques for Imitation Learning Nathan D. Ratliff, David Silver, J. Andrew Bagnell (super paper, which most people find quite tough going; goes further than we will in class) Maximum margin planning Nathan D. Ratliff, J. Andrew Bagnell Martin A. Zinkevich (another super paper, also goes further than we will in class)