CS 598 Topics in Statistical Learning

Notes

1. Week 1: There was no week 1 meeting
2. Week 2: Linear Regression

   My notes

3. Week 3: Linear Methods in Classification

   My notes

Reading

1. The Kernel Trick
   • C.J.C. Burges, ``A Tutorial on Support Vector Machines for Pattern Recognition, '' Data Mining and Knowledge Discovery, 2, 121Ð167 (1998)
   • C. Campbell, ``Kernel methods:a survey of current techniques,'' Neurocomputing 48 (2002) 63Ð84
   • B. Scholkopf, A. Smola, K-R Muller, ``Nonlinear Component Analysis as a Kernel Eigenvalue Problem,'' Neural Computation 10, 1299Ð1319 (1998)
   • S. Mika, G. Ratsch, J. Weston, B. Scholkopf, K.R. Mullers, ``Fisher discriminant analysis with kernels,'' Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop 41-48
2. The Relevance Vector Machine
   • M. E. Tipping ``Sparse bayesian learning and the relevance vector machine'', The Journal of Machine Learning Research Volume 1 , 211 - 244, 2001
3. Discriminative Parsing
   • B Taskar, D Klein, M Collins, D Koller, C Manning, "Max-margin parsing" Proc. EMNLP, 2004
   • M Collins, N Duffy, "Convolution kernels for natural language," Advances in Neural Information Processing Systems, 2002
   • M. Collins and N. Duffy, "Parsing with a Single Neuron: Convolution Kernels for Natural Language Problems", Technical Report, University of California at Santa Cruz., 2001
4. SVM's, Loss, and RKHS
   • Theodoros Evgeniou, Massimiliano Pontil and Tomaso Poggio, "Regularization Networks and Support Vector Machines," Advances in Computational Mathematics 13 (2000) 1Ð50 1
5. Manifold Learning - 1
   • Joshua Tenenbaum, Vin de Silva,John C. Langfor, A Global Geometric Framework for Nonlinear Dimensionality Reduction, Science, 290, 2319-2323, 2000
   • Sam T. Roweis and Lawrence K. Saul, Nonlinear Dimensionality Reduction by Locally Linear Embedding, Science, 290, 2323-2326, 2000
   • Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifolds Lawrence K. Saul, SamT. Roweis, Journal of Machine Learning Research 4 (2003) 119-155
   • Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering Yoshua Bengio, Jean-Francüois Paiement and Pascal Vincent D«epartement dÕInformatique et Recherche Op«erationnelle Universit«e de Montr«eal Montr«eal, Qu«ebec, Canada, H3C 3J7 {bengioy,paiemeje,vincentp}@iro.umontreal.ca Technical Report 1238, D«epartement dÕInformatique et Recherche Op«erationnelle July 25, 2003
6. Structure Learning and related
   • An End-to-End Discriminative Approach to Machine Translation, P. Liang, Alexandre Bouchard-Cote, D. Klein and B. Taskar. Association for Computational Linguistics (ACL06), Sydney, Australia, July 2006.
   • Word Alignment via Quadratic Assignment, S. Lacoste-Julien, B. Taskar, D. Klein, and M. Jordan. Human Language Technology conference - North American chapter of the Association for Computational Linguistics (HLT-NAACL06), New York, June 2006.
   • Structured Prediction, Dual Extragradient and Bregman Projections, B. Taskar, S. Lacoste-Julien, and M. Jordan. Journal of Machine Learning Research (JMLR), Special Topic on Machine Learning and Large Scale Optimization. Max-Margin Markov Networks, B. Taskar, C. Guestrin, V. Chatalbashev and D. Koller. Journal of Machine Learning Research (JMLR), to appear.
   • Structured Prediction via the Extragradient Method, B. Taskar, S. Lacoste-Julien, and M. Jordan, Neural Information Processing Systems Conference (NIPS05), Vancouver, British Columbia, December 2005. [Longer version]
   • Subgradient Methods for Maximum Margin Structured Learning Nathan D. Ratliff J. Andrew Bagnell Martin A. Zinkevich
   • Discriminative Learning of Markov Random Fields for Segmentation of 3D Scan Data. D. Anguelov, B. Taskar, V. Chatalbashev, D. Koller, D. Gupta, G. Heitz, A. Ng. International Conference on Computer Vision and Pattern Recognition (CVPR05), San Diego, CA, June 2005.
7. Semi-Supervised Learning and related
   • A. Blum and T. Mitchell, Combining labeled and unlabeled data with co-training Proceedings of the eleventh annual conference on Computational learning, 92 - 100, 1998
   • K. Barnard, P. Duygulu, D. Forsyth, N. de Freitas, D. Blei, and M. Jordan, Matching Words and Pictures, Journal of Machine Learning Research 3 (2003) 1107Ð1135
   • Kamal Nigam1 , Andrew Kachites Mccallum2, 3 , Sebastian Thrun4 and Tom Mitchell5 Classification from Labeled and Unlabeled Documents using EM Journal Machine Learning Issue Volume 39, Numbers 2-3 / May, 2000
   • Unsupervised improvement of visual detectors using cotraining Levin, A. Viola, P. Freund, Y. This paper appears in: Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on Publication Date: 13-16 Oct. 2003
8. Some more Manifold learning
   • The Geometric Basis of Semi-supervised Learning. V. Sindhwani, M. Belkin, and P. Niyogi Book Chapter in Semi-supervised Learning, (Chapelle, Schoelkopf, Zien: editors) MIT Press, 2006.
   • Beyond the Point Cloud: From Transductive to Semisupervised Learning. V. Sindhwani, P. Niyogi, and M. Belkin Proc. of International Conference on Machine Learning, 2005.
9. Some more Structure Learning
   • I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, Large Margin Methods for Structured and Interdependent Output Variables, Journal of Machine Learning Research (JMLR), 6(Sep):1453-1484, 2005.
   • F. Kang, R. Jin, and R. Sukthankar, "Correlated Label Propagation with Applications to Multi-Label Learning", CVPR 2006
10. Transductive Learning
    • Transductive Inference for Text Classification using Support Vector Machines (1999) Thorsten Joachims Proceedings of ICML-99, 16th International Conference on Machine Learning
    • Gert R. G. Lanckriet, Nello Cristianini, Peter Bartlett, Laurent El Ghaoui, Michael I. Jordan, Learning the Kernel Matrix with Semidefinite Programming The Journal of Machine Learning Research Volume 5 , 27 - 72, 2004
11. Variational Methods
    • Michael I. Jordan , Zoubin Ghahramani , Tommi S. Jaakkola and Lawrence K. Saul, An Introduction to Variational Methods for Graphical Models Machine Learning Volume 37, Number 2 / November, 1999 Pages 183-233
12. Gaussian Processes
    • My notes, after Rasmussen +Williams' book "Gaussian Processes"
    • My brief notes on model selection, etc