CS-598 AI and Graphics
D.A. Forsyth --- 3310 Siebel Center
Lecture Times Time: MW 11h00-12h15, Location: ONLINE
A closer look at Positional Encoding, NeRF and Tomography
Groups, rules, etc. You can do this in groups of up to five. There will be four such
homeworks. Each student should have done: all exercises; at least one challenge; at least one project. Project
ideas are ideas - you can do different projects, by discussion with me. You may use code from any source, as long as you acknowledge it. Due date: last day of semester for everything.
Exercise (for everyone)
Investigate the effect of using random Fourier features for image encoding, as in the Tancik et al paper.
For at least five images (your choice; be sensible)
- represent the image as an MLP without the embedding. Use Tancik et al's choice
of number of layers, dimension of layer - I've had good luck with it.
- represent the image as an MLP using the embedding, for various sigma and dimension.
Use Tancik et al's choice
of number of layers, dimension of layer - I've had good luck with it.
- Now prepare a plot showing the effects of sigma and dimension on the held out error. Fit your representation to 2/3 of image
pixels, chosen at random, and look at error on remaining 1/3. In the plot, you should show the effect of not using the embedding. You should also show the training error of the representation (RMSE is good), so that it is possible to tell how well the method smoothes images.
Challenge
Produce a reconstruction of the 2D Shepp Logan phantom, at 512 x 512 resolution, from 20 angular projections,
as sketched in the Tancik paper. Represent the phantom as an MLP applied to a random Fourier feature embedding.
My experience: I've found it hard to get good reconstructions.
Possible project
Make a learned sampler for a NeRF representation