CS-598 AICG-21 - Records

Records of our meetings.

Class Date	Lecture movie	Slides	Sources and Readings
25 Jan	Ray tracing as integration	Ray tracing in all its glory; plenoptic functions	Adelson, E.H., Bergen, J.R. (1991). "The Plenoptic Function and the Elements of Early Vision", In Computation Models of Visual Processing, M. Landy and J.A. Movshon, eds., MIT Press, Cambridge, 1991, pp. 3–20. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M. (1996). "The Lumigraph", Proc. ACM SIGGRAPH, ACM Press, pp. 43–54. Levoy, M., Hanrahan, P. (1996). "Light Field Rendering", Proc. ACM SIGGRAPH, ACM Press, pp. 31–42. M. Pharr, W. Jakob, G. Humphreys, "Physically based rendering" Morgan Kauffmann, ( Note: third edition is online free; 4e is a physical book not quite out yet) H.W. Jensen, Realistic Image Synthesis Using Photon Mapping AK Peters, 2001 ( Note: Might be hard to find - try a library) Slides on perspective and affine camera models (Note:if you haven't seen these) Brush up your homogeneous coordinates with wikipedia Some material on camera geometry towards END of this More material on camera geometry and pairs of cameras
27 Jan	Some radiometry, and light paths	revised later slides from 25 Jan	F. Durand and G. Drettakis and C. Puech "The 3D Visibility Complex", ACM Transactions on Graphics, Association for Computing Machinery, 2002, 21 (2), pp.176-206. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, ECCV20
1 Feb	More paths and radiometry; visibility complexes; starting light fields	Slides from 1 Feb	GPU Gems ch39, Volume rendering techniques, M. Ilkits, J. Kniss, A. Lefohn, C. Hansen, Collection of recent NeRF papers by Yen Chen Lin Optical models for direct volume rendering Nelson Max, Optical models for direct volume rendering N Max - IEEE Transactions on Visualization and Computer Graphics, 1995 Volume Visualization and Volume Rendering Techniques M. Meißner , H. Pfister , R. Westermann , and C.M. Wittenbrink, Notes from Eurographics Tutorial, 2000
3 Feb	Volume rendering and NeRF	Slides from 3 Feb	Wikipedia on CT (thoroughly helpful) Computerized Tomography: The New Medical X-Ray Technology L. A. Shepp and J. B. Kruskal The American Mathematical Monthly Vol. 85, No. 6 (Jun. - Jul., 1978), pp. 420-439 (20 pages)( Note: older article, but accessible) An introduction to X-ray tomography and Radon transforms ET Quinto - Proceedings of symposia in Applied Mathematics, 2006 ( Warning:this is an intro. to the hardcore math of Tomography; not for faint-hearted, though very helpful) Generalized transforms of Radon type and their applications P Kuchment - Proceedings of Symposia in Applied Mathematics, 2006( Warning:this is an intro. to the hardcore math of Tomography; not for faint-hearted, though very helpful) Image Reconstruction is a New Frontier of Machine Learning Ge Wang; Jong Chu Ye; Klaus Mueller; Jeffrey A. Fessler (Intro to Special issue on deep learning in tomography - likely read all of this) IEEE Transactions on Medical Imaging ( Volume: 37, Issue: 6, June 201)
8 Feb	More NeRF and spatial frequencies	Revised slides from 3 Feb for 8 Feb	Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, Ren Ng Neural sparse voxel fields Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, Christian Theobalt SSD-GAN: Measuring the Realness in the Spatial and Spectral Domains Yuanqi Chen, Ge Li, Cece Jin, Shan Liu, Thomas Li
10 Feb	Some Tomography and some differentiable rendering	Revised slides from 8 Feb for 10 Feb Differentiable Rendering slides	Differentiable Rendering: A Survey Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, Adrien Gaidon
15 Feb	Image denoising with CNNs; setting up denoising IBR	Neural rendering slides Differentiable Rendering slides	State of the Art on Neural Rendering Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer Phototourism project page (Snavely et al 06) Facade project page (Debevec et al 96) Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder Chakravarty R Alla Chaitanya Anton Kaplanyan , Christoph Schied, Marco Salvi, Aaron E Lefohn, Derek Nowrouzezahrai , Timo Aila A Machine Learning Approach for Filtering Monte Carlo NoiseNima Khademi Kalantari, Steve Bako Pradeep Sen IGNOR: Image-guided Neural Object Rendering Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner Deep blending for free-viewpoint image-based rendering Peter Hedman, Julien Philip, True Price, Jan Michael Frahm, George Drettakis, Gabriel J Brostow
17 Feb	No class	No class	No class
22 Feb	Beauty rendering IBR	Expanded neural rendering slides	Virtualized Reality: Constructing Virtual Worlds from Real ScenesT. Kanade, P. Rander and P.J. Narayanan, 1997 Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and BodiesHanbyul Joo, Tomas Simon, Yaser Sheikh, CVPR 2018 The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting KAIWEN GUO, PETER LINCOLN∗, PHILIPDAVIDSON∗, JAY BUSCH†, XUEMING YU†,MATTWHALEN†, GEOFF HARVEY†, SERGIO ORTS-ESCOLANO, ROHIT PANDEY, JASON DOURGARIAN, DANHANG TANG, ANASTASIA TKACH, ADARSH KOWDLE, EMILY COOPER, MINGSONG DOU, SEAN FANELLO‡, GRAHAMFYFFE‡, CHRISTOPHRHEMANN‡, JONATHAN TAYLOR‡, PAUL DEBEVEC§, andSHAHRAM IZADI§, SIGGRAPH 2019
24 Feb	Beauty rendering IBR	Expanded neural rendering slides	LookinGood: Enhancing Performance Capture with Real-time Neural Re-RenderingRICARDO MARTIN-BRUALLA∗, ROHIT PANDEY∗, SHUORANYANG, PAVEL PIDLYPENSKYI, JONATHAN TAYLOR, JULIEN VALENTIN, SAMEH KHAMIS, PHILIP DAVIDSON, ANASTASIA TKACH, PETER LINCOLN, ADARSH KOWDLE, CHRISTOPH RHEMANN, DAN B GOLDMAN, CEM KESKIN, STEVE SEITZ, SHAHRAM IZADI, and SEAN FANELLO, SIGGRAPH 2018 Image-to-Image Translation with Conditional Adversarial NetworksPhillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A. Efros, CVPR 2016 Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros
1 Mar	Deep CG2Real; SPADE; FID and bias	Yet more expanded neural rendering slides	Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement Sai Bi, Kalyan Sunkavalli, Federico Perazzi, Eli Shechtman, Vladimir Kim, Ravi Ramamoorthi Semantic Image Synthesis with Spatially-Adaptive Normalization Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu SinGAN: Learning a Generative Model from a Single Natural Image Tamar Rott Shaham, Tali Dekel, Tomer Michaeli High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro Effectively Unbiased FID and Inception Score and where to find them Min Jin Chong, David Forsyth pixelNeRF: Neural Radiance Fields from One or Few Images Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa
3 Mar	Control in DeepCG2Real; PixelNerf; Reshading ideas	Reshading Slides	pixelNeRF: Neural Radiance Fields from One or Few Images Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa On the relationship between radiance and irradiance: determining the illumination from a convex object R. Ramamoorthi and P. Hanrahan, 2001 Frequency-Space Decomposition and Acquisition of Light Transport under Spatially Varying IlluminationReddy, Ramamoorthi, Curless, 2012 A Signal-Processing Framework for Inverse RenderingR. Ramamoorthi and P. Hanrahan, 2001
8 Mar	Inferring illumination representations from images; some reshading	Revised and expanded Reshading Slides	Automatic Scene Inference for 3D Object CompositingKevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, Manmohan Chandraker Neural Inverse Rendering of an Indoor Scene from a Single ImageSoumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, Jan Kautz Lighthouse: Predicting Lighting Volumes for Spatially-Coherent IlluminationPratul P. Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T. Barron, Richard Tucker, Noah Snavely DeepLight: Learning Illumination for Unconstrained Mobile Mixed RealityChloe LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, Paul Debevec Learning to Predict Indoor Illumination from a Single ImageMarc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, Jean-François Lalonde Fast Spatially-Varying Indoor Lighting EstimationMathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr Neural Illumination: Lighting Prediction for Indoor EnvironmentsShuran Song, Thomas Funkhouser An Approximate Shading Model for Object RelightingZicheng Liao, Kevin Karsch, David A. Forsyth An Approximate Shading Model with Detail Decomposition for Object RelightingZicheng Liao, Kevin Karsch, Hongyi Zhang, David Forsyth
10 Mar	Intrinsic images and conditional models	Revised and expanded Reshading Slides	An Uncertain Future: Forecasting from Static Images using Variational Autoencoders Jacob Walker, Carl Doersch, Abhinav Gupta, Martial Hebert Tutorial on Variational AutoencodersCarl Doersch Learning Diverse Image ColorizationAditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, David Forsyth Intrinsic Image Decomposition using ParadigmsD. A. Forsyth, Jason J. Rock
15 Mar	Some reshading; some point based rendering; neat trick for representing hair which might extend	Revised and expanded Reshading Slides	Differentiable Surface Splatting for Point-based Geometry Processing Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, Olga Sorkine-Hornung Neural Opacity Point Cloud Cen Wang; Minye Wu; Ziyu Wang; Liao Wang; Hao Sheng; Jingyi Yu
17 Mar	We did not meet; ICCV deadline		However, there's a lot of reading above
22 Mar	3D representations, inc points, implicit surfaces, metaballs, CSG and Marching cubes	Shape representations and PointNet	Nice slides on CSG and procedural modelling by Stelian Coros PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas Data structure for soft objects. Wyvill, G., McPheeters, C., & Wyvill, B. (1986). Visual Computer 2(4), 227-234. Should be free from inside UIUC "Marching cubes: A high resolution 3D surface construction algorithm"Lorensen, William E.; Cline, Harvey E. (1 August 1987). ACM SIGGRAPH Computer Graphics. 21 (4): 163–169.Note citation rate! Practical considerations on Marching Cubes 33 topological correctnessCustodio, Lis; Etiene, Tiago; Pesco, Sinesio; Silva, Claudio (November 2013). . Computers & Graphics. 37 (7): 840–850. Should be free from inside UIUC
24 Mar	No class	No class	No class
29 Mar	Pointnet, CvxNet and Farkas' lemma	Shape representations, PointNet, CvxNet	Wikipedia on Farkas' lemma Lectures on Polytopes, G.M. Ziegler, 1995 No link, sorry - try library for download This goes much much further than we need, but chapter 1, particularly the bit on Farkas' lemma, is illuminating. CvxNet: Learnable Convex Decomposition Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, Andrea Tagliasacchi Read this carefully there is a silly bit, and a lot has been left on the table, but there is room for a lot more.
31 Mar	Parsing and generation - 1	Generating and parsing shapes	Primitives Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs Maxim Tatarchenko, Alexey Dosovitskiy, Thomas Brox PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas Occupancy Networks: Learning 3D Reconstruction in Function Space Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas Geiger Learning Representations and Generative Models for 3D Point Clouds Panos Achlioptas 1 Olga Diamanti Ioannis Mitliagkas 2 Leonidas Guibas Parsing and generation Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models Daniel Ritchie, Kai Wang, Yu-an Lin PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, Hao Su Learning Shape Abstractions by Assembling Volumetric Primitives Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, Jitendra Malik Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboids Despoina Paschalidou, Ali Osman Ulusoy, Andreas Geiger CSGNet: Neural Shape Parser for Constructive Solid Geometry Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, Subhransu Maji GRAINS: Generative Recursive Autoencoders for INdoor Scenes Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, Hao Zhang Learning to Infer and Execute 3D Shape Programs Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J. Mitra, Daniel Ritchie Learning Generative Models of 3D Structures Siddhartha Chaudhuri Daniel Ritchie Jiajun Wu4 Kai Xu Hao Zhang
5 Apr	More on generativev shape models	Generating and parsing shapes, updated	Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboids Despoina Paschalidou, Ali Osman Ulusoy, Andreas Geiger Revisiting Superquadric Fitting: A numerically stable formulation Narunas Vaskevicius and Andreas Birk Learning to Infer and Execute 3D Shape Programs Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu
7 Apr	We saw Jiajun Wu in Yuxiong Wang's class; here's the link to the recording	Jiajun Wu's slides
12 Apr	Shape programs; simple betti number constructions	Point cloud constructions, tending to persistent homology	Metrics for 3D Rotations: Comparison and Analysis Du Q. Huynh BARCODES: THE PERSISTENT TOPOLOGY OF DATA ROBERT GHRIST This is well ahead of where I'll go - not for the faint-hearted A roadmap for the computation of persistent homology Nina Otter, Mason A Porter, Ulrike Tillmann, Peter Grindrod & Heather A Harrington Another well ahead of where I'll go - but again...
14 Apr	More simplicial stuff	complexes for convexes; homology; Betti numbers some figures to help	Very nice notes on ways to make complexes Tamal Dey Fairly comprehensive notes on homologyJakob Jonsson Class notes on simplicial homologyScribe: Marc Khoury
19 Apr	Simplicial stuff as a way to build unsupervised point set repns	various constructions of complexes; why we care some figures to help	Witness complexes Jean-Daniel Boissonnat
21 Apr	Basic animation, with a view to motion capture	PDF of slides Keynote of slides	a really useful motion capture dataset, from CMU which started most of the thread below (thanks Jessica Hodgins!) Motion graphs L. Kovar, M. Gleicher, F. Pighin, SIGGRAPH 02 Interactive motion generation from examples O. Arikan and D.A. Forsyth, SIGGRAPH 02 Interactive control of avatars animated with human motion data J Lee, J Chai, PSA Reitsma, JK Hodgins, NS Pollard, SIGGRAPH 02 Motion synthesis from annotations O Arikan, DA Forsyth, JF O'Brienâ€¦ - ACM Transactions on â€¦, 2003 - dl.acm.org Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces A Safonova, JK Hodgins, NS Pollard ACM Transactions on Graphics (ToG) 23 (3), 514-521 Knowing when to put your foot down L.Ikemoto, O. Arikan, D.A. Forsyth "Pushing People Around". Okan Arikan, David A. Forsyth, and James F. O'Brien.In ACM SIGGRAPH/Eurographics Symposium on Computer Animation 2005, pages 56–66, July 2005. Mapping Optical Motion Capture Data to Skeletal Motion Using a Physical Model Zordan, V. B., Horst, N. C., ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2003.
26 Apr	Basic animation, with a view to motion capture	Revised PDF of slides Revised Keynote of slides	Motion capture assisted animation: texturing and synthesisK. Pullen and C. Bregler, 2004 Style-based Inverse Kinematics Keith Grochow, Steven L. Martin Aaron Hertzmann Zoran Popović Generalizing motion edits with Gaussian processesL. Ikemoto, O. Arikan, D.A. Forsyth, 2009 Character Controllers Using Motion VAEs ACM Transactions on Graphics (SIGGRAPH 2020) HUNG YU LING, University of British Columbia FABIO ZINNO, Electronic Arts Vancouver GEORGE CHENG, Electronic Arts Vancouver MICHIEL VAN DE PANNE, University of British Columbia
3 May	Chunks of motion, linear interpolation, speculation	PDF of slides	Evaluating motion graphs for character animation P. Reitsma and N. Pollard, 2007 Construction and optimal search of interpolated motion graphs Alla Safonova Jessica K. Hodgins January 2007 Analyzing the Physical Correctness of Interpolated Human Motion Alla Safonova and Jessica K. Hodgins Dynamic Future Net: Diversified Human Motion Generation Wenheng Chen, He Wang, Yi Yuan, Tianjia Shao, Kun Zhou Contact and Human Dynamics from Monocular Video Davis Rempe1;2, Leonidas J. Guibas1, Aaron Hertzmann2, Bryan Russell2, Ruben Villegas2, and Jimei Yang2 Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling He Wang , Edmond S. L. Ho , Hubert P. H. Shum , and Zhanxing Zhu Slides on imitation learning by Katerina Fragkiadaki Efficient reductions for imitation learning Ross and Bagnell, AIStats, 10 (problems with imitation learning exposed) A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ross, Gordon and Bagnell, AIStats-11, (Dagger) ALVINN: an autonomous land vehicle in a neural network Dean Pomerlau, NIPS, 1989 (early - first?- autonomous driver) Structured prediction in a little more detail Movie used for AML Structured prediction in a little more detail -II (optimization) Another Movie used for AML Slides from Tom Mitchell on Imitation learning(First part is rather similar to Fragkiadaki's slides, above; but the second part is different!) Reinforcement and Imitation Learning via Interactive No-Regret Learning Stephane Ross, J. Andrew Bagnell, NIPS 2014 (This is AGGREVATE) Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, J. Andrew Bagnell (this is the AGGREVATED paper; who could have guessed from title?) Learning to Search: Functional Gradient Techniques for Imitation LearningNathan Ratliffe, David Silver, Andrew Bagnell (Learning to search=LEARCH) Imitiation learning - I Imitiation learning - II, and Intrinsic images finish off