CS 544 Optimization methods in vision and learning

Office hours:

• DAF: 14h00-15h00 Thur or see if I'm busy (note change!)
• Xiaoming: 14h00-15h00 Tue

MP's

• There will be five in total, modelled after last time's MPS
• Final Versions:
  1. MP-1 due 14 Feb
  2. MP-2 due 28 Feb
  3. MP-3 due 31 Mar
  4. MP-4 due 6 May
  5. MP-5 due 6 May
• Drafts:

Current Notes

1. Week 1:
   • Basic continuous optimization (descent directions, coordinate descent, Newton's method); My notes
2. Week 2:
   • More basic continuous optimization (descent directions, steplength selection, assorted quasi-Newton methods; notes same as week 1).
   • Conjugate Gradient My notes
3. Week 3:
   • More basic continuous optimization Finish conjugate gradient, notes above.
   • Various Quasi-Newton Methods My notes
4. Week 4: Constrained optimization basics
   • Lagrangians, quadratic penalty method, augmented lagrangian method (My notes:)
   • first look at duals, simple splitting, setting up Lasso (My notes:)
   • Lasso by ADMM and total variation denoising ( My Notes
5. Week 5: Inequalities, first look
   • Linear programs (My Notes)
   • Inequality constraints, KKT and duals (My Notes)
   • An interesting dual - the SVM (My Notes)
   • Box constraints (My Notes)
6. Week 6: Interior Point Methods
• Structure learning - an example with lots of inequalities ( My Notes)
• Working sets(My Notes)
• Central path following ( My Notes)
• Rough intro to semidefinite programming ( My Notes)
7. Week 7: Finish interior Point Methods; start with discrete problems
• Primal-dual methods, I ( My Notes)
• Primal-dual methods, II ( My Notes)
• Basic discrete problems ( My Notes)
• Chapter from AML book on Markov chains, HMMS and Dynamic Programming (should you wish, library will allow download of PDF. Applied Machine Learning, D.A. Forsyth, Springer, can't remember date)
8. Week 8: Various discrete problems
• Examples of discrete problems that can be relaxed successfully, and why (My Notes)
• The DETR paper I use as an example is: End-to-End Object Detection with Transformers Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko, 2020
• Flows, cuts, linear programs and quadratic programs ( My notes)
9. Week 9: More discrete problems and graphical models
• Push-relabel as a flow algorithm My notes
• Setting up interactive segmentation as a flow-cut problem My slides
• What happens when there are more than two classes? My notes
• Papers:
  • GrabCut: Interactive foreground extraction using iterated graph cuts C. Rother, V. Kolmogorov, and A. Blake, ACM Trans. Graph., vol. 23, pp. 309–314, 2004.
  • Boykov, Y. and O. Veksler, Graph cuts in vision and graphics: theories and applications, Handbook of Math. Models of Computer Vision, Paragios, Chen, Faugeras (eds)
  • Boykov, Y.   Veksler, O.   Zabih, R.  Fast approximate energy minimization via graph cuts, PAMI, 23, 11, 1222-1239, 2001
  • Which Energy Minimization for my MRF/CRF? A Cheat-Sheet, Bogdan Alexe, Thomas Deselaers, Marcin Eicher, Vittorio Ferrari, Peter Gehler, Alain Lehmann, Stefano Pellegrini, Alessandro Prest
10. Week 10: Graphical models and other approximate solution techniques
    • Graphical models ( My notes)
    • Message passing and loopy belief propagation ( My notes)
    • Some equivalences ( My notes)
    • Nice slides from Martin Wainwright dealing with loopy belief propagation and a variety of other topics; I use some of these, but they go far beyond what we'll do
    • Mean field inference ( My notes)
    • Fitting a tree and loopy belief propagation ( My notes)
    Papers

    These go far beyond where I'll go in lecture, but lectures will set you up to read these.

    • Convergent Tree-Reweighted Message Passing for Energy Minimization, V. Kolmogorov, PAMI 2006
    • An introduction to LP relaxations for MAP inference slides by Adrial Weller, 2017
    • Graphical Models, Exponential Families, and Variational Inference MJ Wainwright and MIJordan, Foundations and trends in Machine Learning, 2008 The mother lode; everything is here, done rigorously, but quite hard work.
    • Understanding Belief Propagation and its Generalizations Jonathan S. Yedidia, William T. Freeman, and Yair Weiss, IJCAI 2001 another mother lode, for bp and loopy bp
11. Week 11: Finish graphical models; start various first order methods
    • Gradient descent, convergence, and discontents (My notes, which have changed since 2 April)
    • Regret and online descent (My notes, which have changed since 2 April)
    Papers and notes

    These go far beyond where I'll go in lecture, but lectures will set you up to read these.

    • The Gradient Descent Framework a really nice chapter draft, from a CMU course; it isn't named, but I believe the author to be Anupam Gupta.
    • Online convex programming and generalized infinitesimal gradient ascent M.Zinkievich, 2003. This is the paper laying out the idea of regret.
    • An Improved Analysis of Stochastic Gradient Descent with Momentum, Y. Liu, W. Gao, Y. Lin, Neurips, 2020. Worth a look; I haven't lectured this, but....
    • Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, J. Duchi, E. Hazan, Y. Singer, JMLR, 2010. This is AdaGrad.
    • ADAM: A Method for Stochastic Optimization, D. Kingma and J.L. Ba, ICLR, 2015
12. Week 12: More first order methods
    • Subgradients (My notes)
    • Proximal methods(My notes which have changed since 9 April and which draw heavily on Tibshirani notes)
    Papers and notes

    These go slightly beyond where I'll go in lecture.

    • Notes on proximal gradient descent very clear and helpful notes by Ryan Tibshirani of CMU
13. Week 13: Yet more first order methods
    • Acceleration (My notes, which draw heavily on Tibshirani notes above)
    • Striking behavior of gradient descent: linear predictors ( My notes, which draw heavily from first couple of pages of Soudry et al 2018, below)
    • Striking behavior of gradient descent: matrix problems ( My notes, which draw heavily from first couple of pages of Gunarskar et al 2017, below)
    Papers and notes

    These go notably beyond where I'll go in lecture.

    • The Implicit Bias of Gradient Descent on Separable Data Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, Nathan Srebro, 2018
    • Implicit Regularization in Matrix Factorization Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, Nathan Srebro, 2017
14. Week 14: Gradient boost and a taste of Bayesian Optimization
    • Striking behavior of gradient descent: matrix problems ( My notes, which draw heavily from first couple of pages of Gunarskar et al 2017, below)
    • Gradient Boost (my terse notes; more detailed notes with some stuff on xgboost)
    • Bayesian optimization ( My notes)
    Papers and notes

    These go notably beyond where I'll go in lecture.

    • Greedy function approximation: A gradient boosting machine. Jerome H. Friedman, 2001. Very heavily cited paper full of details about how to boost.
    • Book: Bayesian Optimization, Roman Garnett, CUP 2023 As you can see from notes, I found this extremely helpful. I've cut out a small part of the book for notes. Library might have this, but I doubt in PDF - sorry. BUT the book's website implies that PDF downloads from there are possible.
    • Bayesian optimization: a tutorial , P.I. Frazier, 2018

Resources

1. Numerical Optimization (Springer Series in Operations Research and Financial Engineering) by Jorge Nocedal and Stephen Wright, 2006
2. Convex Optimization by Stephen Boyd and Lieven Vandenberghe, Cambridge, 2004
3. Understanding and Using Linear Programming, J. Matousek and B. Gartner, Springer, 2007
4. Introduction to Optimization, P. Pedregal, Springer, 2004
5. Optimization for Machine Learning, Sra, Nowozin and Wright, MIT Press, 2011
6. Foundations of Optimization, Guler, Springer, 2010
7. Nonlinear Programming by Dimitri P. Bertsekas, Athena, 1999
8. Practical Methods of Optimization by R. Fletcher, Wiley, 2000
9. Practical Optimization by Philip E. Gill, Walter Murray, Margaret H. Wright, Academic, 1982
10. The EM Algorithm and Extensions 2nd Edition by Geoffrey McLachlan (Author), Thriyambakam Krishnan (Author)

Notes From Last Time

1. Week 1: Basic continuous optimization (descent directions, coordinate ascent, EM as coordinate ascent, Newton's method, stabilized Newton's method); My notes
2. Week 2, Week 3: More basic continuous optimization (trust regions, dogleg method, subspace method); My notes: (Conjugate gradient Old notes) (Conjugate gradient and Polak Ribiere Notes- use these for conjugate gradient as well);(Quasi Newton methods for big problems, including DFP, BFGS, limited memory methods) My notes:
3. Week 4, 5 and 6: Constrained optimization methods
• Lagrangians, quadratic penalty method, augmented lagrangian method (My notes:)
• Linear programs as an example (My notes)
• Inequality constraints in Lagrangians and KKT; relations between primal and dual (My notes)
• Active set methods and box constraints (my notes)
• Logistic regression as a classifier (my notes)
• The SVM as an example (My notes)
• Stochastic gradient descent (my notes)
• Semi-definite programs (My notes - these really do start at page 5!)
• Interior point methods (My notes); merit functions and the Maratos effect (my notes)
• L1 logistic regression, homotopy paths, subgradients and pursuit methods (my notes)
• Quadratic programming and sequential quadratic programming (my notes)
4. Initial remarks on Combinatorial optimization (my notes); Flow and cuts (my notes)
5. alpha-expansion, alpha-beta swaps; MRF's as cuts (my notes):
6. Stochastic Average Gradient (my notes)
7. Matchings (my notes); bipartite graph matching as a linear program (my notes)
8. Dual decomposition methods (my notes); ADMM (my notes there really are two page 13's, sorry)
9. The graphical models block
   • alpha-expansion, alpha-beta swaps; MRF's as cuts (my notes):
   • Simple belief propagation ( My notes )
   • Simple mean field inference (My notes
   • Basic marginal polytope (My notes
   • Variational inference using a tree; the local polytope; exponential models ( My notes)
   • ADMM for discrete problems (my notes)
10. Proximal Algorithms (my notes)
11. Gradient Boost (my terse notes; more detailed notes with some stuff on xgboost)
12. Striking behavior of gradient descent My notes
13. Gradient Boost (my terse notes; more detailed notes with some stuff on xgboost)
14. The Linear Quadratic Regulator (my notes)
15. Markov Decision Processes (my notes)
16. Learning from an expert, I (my notes)
17. Structure Learning (my notes)
Papers