CS 598 Optimization methods in vision and learning

Notes

1. Week 1: No lectures
2. Week 2: Basic continuous optimization (descent directions, coordinate ascent, EM as coordinate ascent, Newton's method, stabilized Newton's method); My notes
3. Week 3: More basic continuous optimization (trust regions, dogleg method, subspace method); My notes: (Conjugate gradient, approximate Hessian methods, BFGS, limited memory methods My notes:
4. Week 5 and 6: Constrained optimization methods (Lagrangians, lagrange duals, SVMs, quadratic penalty method, augmented lagrangian method) My notes:    (inequality constraints; boxes, interior point methods) My notes:   (Interior point methods) My notes
5. Week 8:  Flow and cuts (my notes)
6. Week 9:  More flow and cuts; matchings (my notes)

Resources

1. Numerical Optimization (Springer Series in Operations Research and Financial Engineering) by Jorge Nocedal and Stephen Wright, 2006
2. Convex Optimization by Stephen Boyd and Lieven Vandenberghe, Cambridge, 2004
3. Nonlinear Programming by Dimitri P. Bertsekas, Athena, 1999
4. Practical Methods of Optimization by R. Fletcher, Wiley, 2000
5. Practical Optimization by Philip E. Gill, Walter Murray, Margaret H. Wright, Academic, 1982

Papers

• Discrete optimization resources

1. M. Goemans and D.P Williamson, Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming, Journal of the ACM, Volume 42 ,  Issue 6, 1115 - 1145, 1995
2. L.Vandenberghe and S.Boyd, Semidefinite programming, SIAM review, 38, 1, 49-95, 1996
   

• Discrete optimization applications

1. Y Rubner, C Tomasi, LJ Guibas, The Earth Mover's Distance as a Metric for Image Retrieval, International Journal of Computer Vision, 2000
2. Grauman, K.   Darrell, T., Fast contour matching using approximate earth mover's distance. CVPR 2004
3. E Levina, P Bickel, The earth mover’s distance is the Mallows distance: Some insights from statistics, Proc. ICCV, 2001
4. S Belongie, J Malik, J Puzicha, Matching shapes, Proc. of ICCV, 2001
5. Boykov, Y. and O. Veksler, Graph cuts in vision and graphics: theories and applications, Handbook of Math. Models of Computer Vision, Paragios, Chen, Faugeras (eds)
6. Boykov, Y.   Veksler, O.   Zabih, R.  Fast approximate energy minimization via graph cuts, PAMI, 23, 11, 1222-1239, 2001
7. Boykov, Y.   Veksler, O. Zabih, R., Markov random fields with efficient approximations, CVPR, 1998
8. Maciel, J.; Costeira, J.P.; A global solution to sparse correspondence problems, Pattern Analysis and Machine Intelligence, IEEE Transactions on, Volume 25,  Issue 2,  Feb. 2003 Page(s):187 - 19
9. A.Berg, T. Berg and J. Malik, Shape matching and object recognition using low distortion correspondence, CVPR, 2005
10. P. Kohli and M. Pawan Kumar and P.H.S. Torr, P^3 and Beyond: Solving Energies with Higher Order Cliques, CVPR 2007
11. C.Olsson A.P. Eriksson and F. Kahl, Solving Large Scale Binary Quadratic Problems: Spectral Methods vs Semidefinite Programming, CVPR 2007

• Continuous optimization resources

1. Johathan Shewchuk, An Introduction to the conjugate gradient method without the agonizing pain, 1994

• Continuous optimization applications

Iterative scaling and the like

1. The improved iterative scaling algorithm: A gentle introduction
   A Berger - Unpublished manuscript, 1997
2. Iain Bancarz, M. Osborne, Improved iterative scaling can yield multiple globally optimal models with radically differing performance levels, Proceedings of the 19th international conference on Computational linguistics, 1 - 7, 2002
   
3. Robert Malouf, A comparison of algorithms for maximum entropy parameter estimation,
   proceeding of the 6th conference on Natural language learning - Volume 20, 1 - 7,  2002
4. F. Sha and F. Pereira, Shallow parsing with conditional random fields, Proc HLT-NAACL, Main papers, pp 134-141, 2003
5. Hanna Wallach, Efficient Training of Conditional Random Fields, University of Edinburgh, 2002.

Bundle adjustment

1.   Bill Triggs, Philip F. McLauchlan, Richard I. Hartley and Andrew W. Fitzgibbon,   Bundle Adjustment -- A Modern Synthesis, Vision Algorithms: Theory and Practice: International Workshop on Vision Algorithms, Corfu, Greece, September 1999, pp153-177,
   

Matrix factorization

1.   Buchanan, A.M.; Fitzgibbon, A.W., Damped Newton algorithms for matrix factorization with missing data, Computer Vision and Pattern Recognition, 2005, 316 - 322

Registration

1. A.W. Fitzgibbon, Robust registration of 2D and 3D point sets, Image and Vision Computing, Volume 21, Issues 13-14, 1 December 2003, Pages 1145-1153

SVM's

1. C.J.C. Burges, ``A Tutorial on Support Vector Machines for Pattern Recognition, '' Data Mining and Knowledge Discovery, 2, 121-167 (1998)
2. I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, Large Margin Methods for Structured and Interdependent Output Variables, Journal of Machine Learning Research (JMLR), 6(Sep):1453-1484, 2005.
3. Sequential minimal optimization: A fast algorithm for training support vector machines
   J Platt - Advances in Kernel Methods-Support Vector Learning, 1999
4. S. S. Keerthi, S. K. Shevade,C. Bhattacharyya, K. R. K. Murthy, Improvements to Platt's SMO Algorithm for SVM Classifier Design,  Neural Computation. 2001;13:637-649, 2001
5. K. Zhang, I.W. Tsang, J.T. Kwok, Maximum margin clustering made practical, Proceedings of the 24th international conference on Machine learning,  1119 - 1126, 2007   

Matrix Factorization and clustering

1. D.D. Lee and H. S. Seung, Algorithms for Non-negative Matrix Factorization, NIPS, 556-562, 2000
2. M. Heiler, C. Schnorr, Learning Sparse Representations by Non-Negative Matrix Factorization and Sequential Cone Programming   The Journal of Machine Learning Research, Volume 7 ,  (December 2006), 1385 - 1407 
3. F.Bach and M.I. Jordan, Learning Spectral Clustering, With Application to Speech Separation, JMLR, 7, 2006, 1963-2001
4. T. Finley and T.Joachims, Supervised clustering with support vector machines, Proceedings of the 22nd international conference on Machine learning,  217 - 224, 2005