UIUC CS-588 Autonomous Vehicle System Engineering Outline

This page sketches out roughly what we're going to do, based on last year. I intend to have a second page describing what we've actually done, but it doesn't exist because we haven't done anything yet. Main page has pointer.

Texts:

Applied Machine Learning D.A. Forsyth, Springer, 2019

Important The PDF is a free download from the UIUC library (you have to be on the intranet to download it, I think). Or you could buy a paper copy.

Week Starting	Topic	Reading Materials	Movie List
25 Aug	Admin	Nothing so far!	Nothing so far!
25 Aug	Intellectual Context	Nothing so far!	Here is an intro lecture, with some remarks about safety.
25 Aug	(Y) Basic Convolutional Neural Networks	Deep nets: Units; AML: 16.1 Deep nets: Gradients; AML: 16.2 Deep nets: Backpropagation; AML 16.3 Deep nets: Gradient tricks; AML 16.4 Deep nets: Convolutional layers; AML 17.1 Deep nets: More on convolutional layers; AML 17.1, 17.2	Units Gradients Backprop Gradient tricks Convolutional layers More conv layers
27 Aug	Safety	Slides and pix for safety and vehicle stuff	Some safety and vehicle stuff
27 Aug	Safety lookout training	No reading	The safety lookout video
27 Aug	Image classification and simple detection	Basic object detection: AML 19	Image classifiers Object detectors
WE 3 Sep	Simple PID Control	My PID control slides	PID Control lecture PID Control lecture -II PID Control lecture - leftover bits
WE 10 Sep	ROS and PACMOD and Registration	"My" ROS and PACMOD slides Original detailed slides, with examples, etc.from Roi Yehoshua at Bar-Ilan U. Slides on aligning point sets see also Weighted least squares solutions for rotation translation; very nice notes from Olga Sorkine-Hornung and Michael Rabinovitch Slides on IRLS (best if you've watched the robust regression movie) see also Notes on robustness and iteratively reweighted least squares see also Robust registration of point sets using iteratively reweighted least squares P. Bergstrom and A.Edlund, Computational Optimization and Applications July 2014, Volume 58, Issue 3, pp 543–561 (click from inside UIUC intranet!) Simple bundle adjustment	Romping through ROS; Closed form estimates of rotation and translation from point correspondences Robust regression using IRLS This sets up my IRLS lecture IRLS and bundle adjustment
WE 17 Sep	Kalman and Particle Filters	Slides on basic ideas in visual tracking Slides on Kalman filters, in the context of tracking Slides on Particle filters Particle filter slides by Pieter Abbeel Particle filter slides by Cyrill Stachniss Particle filter notes by Drew Bagnell, scribes Greg Seyfarth, Zachary Batts, David Fouhey A chapter on particle filtersthat didn't make it into "Computer Vision: A Modern Approach"	Simplest tracking and the Kalman filter Basic Particle Filters
WE 24 Sep	Finish Particle Filters; Cameras and pairs of cameras	Slides on Particle filters (slightly revised) Slides on perspective and affine camera models Brush up your homogeneous coordinates with wikipedia Slides on pairs of cameras	Finish off particle filters; start cameras Finish basic cameras; start pairs of cameras
WE 1 Oct	Finish pairs of cameras; flow and stereo; SFM, SLAM and visual odometry	Slides on pairs of cameras (slightly revised) Multiple View Geometry in Computer Vision Second Edition Richard Hartley and Andrew Zisserman, Cambridge University Press, March 2004. Y. Boykov, O. Veksler and R. Zabih (2001), "Fast approximate energy minimisation via graph cuts", IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1222–1239. Li Hong and G. Chen, "Segment-based stereo matching using graph cuts," Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Washington, DC, USA, 2004, pp. I-I, doi: 10.1109/CVPR.2004.1315016. R. Szeliski et al., "A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 1068-1080, June 2008, doi: 10.1109/TPAMI.2007.70844. T. Brox, C. Bregler and J. Malik, "Large displacement optical flow", CVPR 2009 High Accuracy Optical Flow Estimation Based on a Theory for Warping, Thomas Brox, Andr´es Bruhn, Nils Papenberg, and Joachim Weickert, ECCV, 2004 My slides on very simplest SFM and SLAM My slides on basic visual odometry Tutorial on visual slam at CVPR2014 I found the following particularly helpful Visual odometry Stephan Weiss Stereo Visual odometryChris Beall Dense odometryRichard Newcombe	Finish off pairs of cameras Factorization based SFM; simple monocular odometry
WE 8 Oct	EKFSLAM, FastSLAM, Direct methods	Slides on EKF SLAM Slides on FastSLAM Slides on Features and Interest points Slides on Direct SLAM Great notes on EKF-SLAM by Joan Sola, with Matlab code, etc Slides from Ben Kuipers on FastSLAM Slides from Burgard et al on FastSLAM FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem Michael Montemerlo and Sebastian Thrun and Daphne Koller and Ben Wegbreit AAAI 02 FastSLAM 2.0: an improved particle filtering algorithm for simultaneous localization and mapping that provably converges Michael Steven Montemerlo and Sebastian B Thrun and Daphne Koller and Ben Wegbreit IJCAI -03 lsdSLAM website - there is a pointer to a github site with code here. SVO (fast semi-direct visual odometry) there is a pointer to code here, too. Christian Forster, Zichao Zhang, Michael Gassner, Manuel Werlberger, Davide Scaramuzza SVO: Semi-Direct Visual Odometry for Monocular and Multi-Camera Systems IEEE Transactions on Robotics, Vol. 33, Issue 2, pages 249-265, Apr. 2017. Dense visual slam an earlier version of lsd-slam for RGB-D cameras, code too Dense Visual SLAM for RGB-D Cameras (C. Kerl, J. Sturm and D. Cremers), In Proc. of the Int. Conf. on Intelligent Robot Systems (IROS), 2013.	EKF-Slam and starting FASTSLAM finish FASTSLAM; interest points and features; direct methods and SVO
WE 15 Oct	Fall viruses and Semantic Segmentation	My semantic segmentation slides A simple two-class CRF example with basic variational inference The Role of Context for Object Detection and Semantic Segmentation in the Wild Roozbeh Mottaghi et al. (CVPR), 2014, pp. 891-898 Urban 3D Semantic Modelling Using Stereo Vision Sunando Sengupta Eric Greveson, Ali Shahrokni2 Philip H. S. Torr, ICRA, 2013 FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY Timo Hackel, Jan D. Wegner, Konrad Schindler "Fully Convolutional Networks for Semantic Segmentation" Jonathan Long∗ Evan Shelhamer∗ Trevor Darrell, 2014 The Stixel world: A medium-level representation of traffic scenes Marius Cordts, et al., ArXiv, 2017 Towards a Global Optimal Multi-Layer Stixel Representation of Dense 3D Data David Pfeiffer, Uwe Franke, BMVC 2011 GREIG, D., PORTEOUS, B., AND SEHEULT, A. 1989. Exact MAP estimation for binary images. J. Roy. Stat. Soc. B. 51, 271–279.	Semantic segmentation; simple MRFs and GrabCut
WE 22 Oct	Finish Semantic segmentation; start motion planning	My slides on fully connected CRFS, with bilateral filters My slides on motion planning - I My slides on higher dimensional motion planning Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials Philipp Krähenbühl, Vladlen Koltun The Fast Bilateral Solver Jonathan T. Barron, Ben Poole Fast Bilateral-Space Stereo for Synthetic DefocusJonathan T Barron Andrew Adams YiChang Shih Carlos Hernández CVPR (2015) Fast high-dimensional filtering using the permutohedral lattice.A. Adams, J. Baek, and M. A. Davis. Computer Graphics Forum, 29(2), 2010. 2, 5	Setting up FCCRFs and non-local means Basic motion planning; cells, potential fields, etc.
29 Oct	Motion planning	Notes on sampling based motion planning, chapter 5 of Lavalle's book "Planning Algorithms" Notes on planning in the presence of dynamics , chapter 14 of Lavalle's book "Planning Algorithms" Dubins, L.E. (July 1957). "On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents". American Journal of Mathematics. 79 (3): 497–516. J. A. Reeds and L. A. Shepp. Optimal paths for a car that goes both forwards and backwards. Pacific Journal of Mathematics, 145(2):367-393, 1990. The open mpl library and app Very nice CMU motion planning slides, I believe by H. Choset	Motion planning in the presence of dynamics and motion graphs
5 Nov	Scene representations I: generalities and lane estimates	Some general points about scenes and representation Lane boundary detection US Patent 9081385 (lane boundaries) Scene slides by Aude Oliva Off-road Path Following using Region Classification and Geometric Projection Constraints Y. Alon; A. Ferencz; A. Shashua, CVPR 06 Detection and tracking of boundary of unmarked roads Young-Woo Seo; Ragunathan Raj Rajkumar, 17'th international conf on information fusion, 2014 Geometric Constrained Joint Lane Segmentation and Lane Boundary Detection Jie Zhang, Yi Xu, Bingbing Ni, Zhenyu Duan; Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 486-502 Lane detection using lane boundary marker network with road geometry constraints Hussam Ullah Khan, Afsheen Rafaqat Ali, Ali Hassan, Ahmed Ali, Wajahat Kazmi, Aamer Zaheer; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 1834-1843 Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks Qin Zou, Hanwen Jiang, Qiyu Dai, Yuanhao Yue, Long Chen, Qian Wang Horizon detection page by Nathan Jacobs, including horizon data and some nice papers on horizon detection.	Generalities on scene understanding; lane boundary detection
12 Nov	Scene representations II: maps of the ground; and More SFM	Maps of the ground, scene flow Learning from Maps: Visual Common Sense for Autonomous Driving Ari Seff, Jianxiong Xiao 3D Traffic Scene Understanding from Movable Platforms. Andreas Geiger. Martin Lauer. Christian Wojek. Christoph Stiller. Raquel Urtasun A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker Unsupervised Deep Embedding for Clustering Analysis Junyuan Xie, Ross Girshick, Ali Farhadi Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes Fabian Brickwedde, Steffen Abraham, Rudolf Mester Just Go with the Flow: Self-Supervised Scene Flow Estimation Himangi Mittal, Brian Okorn, David Held Object Scene Flow for Autonomous Vehicles M. Menze and A. Geiger Self-Supervised Monocular Scene Flow Estimation Junhwa Hur, Stefan Roth A Parametric Top-View Representation of Complex Road Scenes Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker Attribute discovery via predictable discriminative binary codes. M Rastegari, A Farhadi, D Forsyth. Derek Hoiem's structure from motion slides My shortened version of above, with additions Structure-from-Motion Revisited Johannes L. Sch¨onberger1;2, Jan-Michael Frahm Colmap github page	Ground plane label maps as scene representations Finish label maps; Incremental SFM
19 Nov	Weather, rain, and intrinsic images	My slides on physical effects on appearance, rain and on intrinsic images Great general reading with super pix Color and Light in Nature, David K. Lynch, William Livingston, Cambridge University Press, Jun 29, 1995 More great general readingLight and Color in the Outdoors by Marcel Minnaert (Author), L. Seymour (Translator), Springer, 1995 Vision in bad weather S.K. Nayar and S. Narasimhan, ICCV 1995 Vision and Rain K. Garg and S.K. Nayar International Journal of Computer Vision volume 75, pages3–27(2007) Programmable Automotive Headlights Robert Tamburo, Eriko Nurvitadhi, Abhishek Chugh, Mei Chen, Anthony Rowe, Takeo Kanade, Srinivasa G. Narasimhan Restoring An Image Taken Through a Window Covered with Dirt or Rain David Eigen Dilip Krishnan Rob Fergus, ICCV 13 Depth-attentional Features for Single-image Rain Removal Xiaowei Hu, Chi-Wing Fu, Lei Zhu2 and Pheng-Ann Heng, CVPR 2019 Rain Streak Removal Using Layer Priors Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, Michael S. Brown, CVPR 16 Semi-supervised Transfer Learning for Image Rain Removal Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, Ying Wu, CVPR 19 Implicit Euler ODE Networks for Single-Image Dehazing Jiawei Shen, Zhuoyan Li, Lei Yu, Gui-Song Xia, Wen Yang CVPR 20 workshops Deep Joint Rain Detection and Removal from a Single Image Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan, CVPR 17 Intrinsic Images in the Wild Sean Bell, Kavita Bala, Noah Snavely Cornell University ACM Transactions on Graphics (SIGGRAPH 2014) Learning lightness from human judgement on relative reflectance T. Narihira, M. Maire, and S. X. Yu, in Proceedings of the IEEE CVPR, 2015. Revisiting deep intrinsic image decompositions Q. Fan, J. Yang, G. Hua, B. Chen, and D. Wipf, in CVPR, 2018. Deep hybrid real and synthetic training for intrinsic decomposition S. Bi, N. K. Kalantari, and R. Ramamoorthi, in Eurographics Symposium on Rendering, 2018. Intrinsic Image Decompositions using Paradigms D.A. Forsyth and Jason J. Rock, 2020	Appearance physics and rain; rain removal methods Intrinsic images, curtailed by Senate Meeting
3 Dec	Learning to control	"My" slides on reinforcement and imitation learning Slides on reinforcement learning by Fei-Fei Li, Justin Johnson and Serena Yeung Slides on imitation learning by Katerina Fragkiadaki Efficient reductions for imitation learning Ross and Bagnell, AIStats, 10 (problems with imitation learning exposed) A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ross, Gordon and Bagnell, AIStats-11, (Dagger) ALVINN: an autonomous land vehicle in a neural network Dean Pomerlau, NIPS, 1989 (early - first?- autonomous driver) Structured prediction in a little more detail Movie used for AML Structured prediction in a little more detail -II (optimization) Another Movie used for AML Slides from Tom Mitchell on Imitation learning(First part is rather similar to Fragkiadaki's slides, above; but the second part is different!) Reinforcement and Imitation Learning via Interactive No-Regret Learning Stephane Ross, J. Andrew Bagnell, NIPS 2014 (This is AGGREVATE) Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, J. Andrew Bagnell (this is the AGGREVATED paper; who could have guessed from title?) Learning to Search: Functional Gradient Techniques for Imitation LearningNathan Ratliffe, David Silver, Andrew Bagnell (Learning to search=LEARCH) End to End Learning for Self-Driving CarsMariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, Karol Zieba (heavily improved and modernised alvinn; note the clever use of extra cameras to supply somewhat off-policy views)	Imitiation learning - I Imitiation learning - II, and Intrinsic images finish off