CS-598 AI and Graphics

D.A. Forsyth --- 3310 Siebel Center

Lecture Times Time: MW 11h00-12h15, Location: ONLINE

Beauty rendering of various kinds

Groups, rules, etc. You can do this in groups of up to five. There will be four such homeworks. Each student should have done: all exercises; at least one challenge; at least one project. Project ideas are ideas - you can do different projects, by discussion with me. You may use code from any source, as long as you acknowledge it. Due date: last day of semester for everything.

Exercise (for everyone)

Produce a semantically controlled set of synthesized images (I'd use SPADE; code on Github) of a cow in a field with a tree on the left and a pond to the right of the image. The view will be looking north, so the sun will rise in the east (as it tends to) which is the right side of the image and set in the west (as it tends to) which is the left side of the image. I want four images, distinguished by lighting, as below

Early morning, with some fog, and some evidence of sunrise. If you can get dew on the grass, boast about it.
Noon on a bright day
Noon on an overcast day
Sunset, with an orange/red sky. If you can ensure that "The plowman homeward plods his weary way," boast about it.

Variant

Here is a variant that means you don't need to train the whole SPADE again (you mean you don't have multiple GPU weeks at your disposal?) and which works for me - Thanks to Aiyu Cui for suggestion.

first use the SPADE (the one without encoder) to get an image without lighting and without the cow;
insert a cow by some harmonization algorithm/model; and
run some style transfer model to get the right time of a day (e.g. sunrise in a orange sky.

BTW, it is a genuinely interesting qn if this works well; if it does, why bother with hard stuff?

Challenge

Write an auxiliary network that accepts a time of day and weather specification, and applies it to outdoor scenes synthesized using SPADE or the like successfully (easy to do a bodged job on this; hard to do it nicely).

Possible project

Build a local patchGAN for a pixelNerf representation that can be trained with reasonable effort.