Course CS506 Special Topics: Problem Solving with Large Clusters

Problem Solving with Large Clusters
Izhak Shafran & Richard Sproat
CSLU, OHSU

News and Assignments Course Information Lectures and Readings Textbooks Schedule Links

News and Assignments

Posted Homework 4, Solution
Posted Homework 3, Solution
Posted Homework 2, Solution
Posted Homework 1, Solution
First meeting on 3/29.

Course Information

This course aims to provide theoretical foundations and practical experience in distributed algorithms. The techniques covered in this course have wide application. Examples will be drawn from speech and language processing, machine learning, optimization, and graph theory. The course will be a combination of:

Introductory lectures
Reading discussions: Students will take turns presenting papers and will be responsible for up to 2 papers.
Assignments
In-Class discussion of assignment solutions by students.
Course project: There will be no final exam. Instead, the course requires a final project of interest to student, chosen in consultation with the instructor. The project requires a written report and a final presentation. In many cases, the data, software toolkit, and key components for the project will be made available.

Prerequisites: A graduate level course on machine learning or probability and statistics. Students should be comfortable coding in at least one programming language.

Lectures and Readings

Session 1 (3/29)
- Course Overview, Infrastructure at CSLU
- Distributed file system Slides
  The Google File System, Sanjay Ghemawat et. al., ACM Symposium on Operating Systems Principles (SOSP), pg. 29-43, 2003.
  Lustre File System, White Paper, Sun Microsystems.
  Data Center As a Computer, Luiz Andre Barroso and Urs Holzle, Synthesis Lectures on Computer Architecture, 2009.
  Challenges in building large-scale information retrieval systems, Jeffrey Dean, Invited talk, Proc. of the Second ACM International Conference on Web Search and Data Mining, 2009.
- Map Reduce Slides
  Textbook Chapter 1-3
  MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat, Communications of the ACM, vol. 51, no. 1, pg. 107-113, 2008.
  MapReduce: A Flexible Data Processing Tool, Jeffrey Dean and Sanjay Ghemawat, Communications of the ACM, vol 53, no. 1, pg. 72-77, 2010.
  MapReduce: The Programming Model and Practice, J. Zhao et. al, SIGMETRICS'09 Tutorial, 2009.
  MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat, OSDI, 2004.
  Designs, Lessons and Advice from Building Large Distributed Systems, Jeff Dean, Keynote LADIS 2009.
Session 2 (4/5): Inverted Index, Language for Machine Translation, Multicore MapReduce (Zak, Richard, Masoud & Aaron)
- Review of Google File System
- Review of MapReduce Framework
- Large Language Models in Machine Translation, Thorsten Brants et. al., Proc. Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP), pg. 858-867, 2007.
- Evaluating Map-Reduce for Multicore and Multiprocessor Systems, Proc. High Performance Computer Architecture (HPCA), Best Paper Award, 2007, Phoneix Code.
For further reading
- Other publications related to Phoeinix can be found here.
- Optimizing MapReduce for Multicore Architectures, Yandong Mao et al, Technical Report, MIT-CSAIL-TR-2010-020, MIT.
- Tiled-MapReduce: optimizing resource usages of data-parallel applications on multicore with tiling, R. Chen et al., Proc International Conference on Parallel Architectures and Compilation Techniques, 2010.
- Where does the speedup go: Quantitative modeling of performance losses in shared-memory programs, Seon Wook Kim et al, Parallel Processing Letters, 2001.
Session 3 (4/12) (Nate, Richard & Zak)
- Fast, Easy, and Cheap: Construction of Statistical Machine Translation Models with MapReduce, Christopher Dyer et. al., Proc. ACL Workshop on Statistical Machine Translation, pg. 199-207, 2008. slides
- Pairwise Document Similarity in Large Collections with MapReduce, Tamer Elsayed et. al., Association for Computational Linguistics (ACL), pg. 265-268, 2008.
- Asynchronous Distributed Learning of Topic Models, Arthur Asuncion et. al., Proc. Neural Information Processing Systems (NIPS), pg. 81-88, 2008.
Further reading
- Distributed Inference for Latent Dirichlet Allocation, David Newman et. al., Proc Neural Information Processing Systems (NIPS), vol. 20, pg. 1081-1088, 2007.
- Tutorial on Latent Dirichlet Allocation, David Blei, Machine Learning Summer School, 2009 (with video, excellent!)
- Inverted Files for Text Search Engines, Justin Zobel et. al., ACM Computing Survey, vol. 38, no. 2, 2006.
- Term weighted approach in automatic text retrieval, G. Salton and C. Buckley, Information Processing and Management, vol 24, no 5, pp 513--523, 1988.
Session 4 (4/19): Machine Learning (Maider, Alireza & Zak)
- Distributed Training Strategies for the Structured Perceptron, Ryan McDonald et. al., Proc NAACL, 2010.
- Parallelizing SVM on Distributed Computers, Edward Y. Chang et. al., Proc. Neural Information Processing Systems (NIPS), vol. 20, 2007. code
- Stochastic Gradient Boosted Decision Trees, Jerry Le et. al., Proc. ACM Conference on Information and Knowledge Management (CIKM), no. 4, pg. 2061-2064, 2009.
Further reading
- A GPU-Tailored Approach for Training Kernelized SVMs, KDD (under review, protected w/ course password), 2011.
- On Explicit Approximations of the Gaussian Kernel, ICML (under review, protected w/ course password), 2011.
- Large Vocabulary Continuous Speech Recognition Using WFST-based Linear Classifier for Structured Data, Interspeech, 2010.
- Efficient L1 Regularized Logistic RegressionSu-In Lee et al, AAAI, 2006.
- Parallel Machine Learning for Hadoop/MapReduce with Code
Session 5 (4/26) Graphs (Richard, Russ & Andrew)
- Graph Twiddling in a MapReduce World, Jonathan Cohen, Computing in Science and Engineering, vol. 11, no. 4, pg. 29-41, 2009. Trusses Barycentric
- Pregel: A System for Large-Scale Graph Processing, Grzegorz Malewicz et. al., Proc. International Conference on Management of Data (SIGMOD), no. 12, pg. 135-146, 2010.
- Link Mining: A Survey, Lise Getoor et. al., 7(2):3-12, SIGKDD Exploratory Newsletter, 2005.
Further reading
- Scalable Community Detection using Label Propagation and MapReduce, to appear, 2011.
- MapReduce-Based Pattern Finding Algorithm Applied in Motif Detection for Prescription Compatibility Network, Advanced Parallel Processing Technologies, vol. 5737, pg. 341, 2009. (Ulon's paper on motifs)
Session 6 (5/3): Genomics (Ranjani Slides & Chris slides)
- An Overview of the Hadoop/MapReduce/HBase Framework and its Current Applications in Bioinformatics, Ronald Taylor, BMC Bioinformatics, 2010.
- CloudBurst: Highly Sensitive Read Mapping with MapReduce, Michael Schatz, Bioinformatics, 25(11), 1363-1369, 2009.slides.
- Low cost, scalable proteomics data analysis using amazon's cloud computing services and open source search algorithms, Brian D. Halligan et. al., Journal of Proteome Research, 8(6):3148-3153, 2009.
- The Genome Analysis Toolkit: A MapReduce framework for analyzing next-generation DNA sequencing data, Genome, 2010.
- SNP detection for massively parallel whole-genome resequencing, Genome Research, 2009.
Session 7 (5/10): Linear Algebra (Zak & Meysam)
- Matrix Multiplication with MapReduce
- HAMA: An efficient matrix computation with the MapReduce framework, Sangwon et. al., Proc. IEEE Conference on Clound Computing Technology and Science, 2007.
- Ensemble Nystrom Method, S. Kumar et. al., Proc. Neural Information Processing Systems (NIPS), 2010, Winner of Best Student Paper at the New York Academy of Sciences 2009 Symposium on ML.
- Parallel Spectral Clustering, Wen-Yen Chen et. al., IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010. code
Session 8 (5/17): Genomics (contd., Brian & Zak)
- Meta-Alignment with Crumble and Prune: Partitioning very large alignment problems for performance and parallelizationBMC Bioinformatics, 2011.
- A Novel Approach to Multiple Sequence Alignment using Hadoop Data Grids, Proc. ACM Massive Data Analytics on the Cloud (MDAC), 2:1-7, 2010.
- An Efficient Parallel Processing Approach For Multiple Biological Sequence Alignment , Md. Maruf Monwar and Siamak Rezaei. IAENG International Journal of Computer Science, 33:2.
- Searching for SNPs with Cloud Computing, Ben Langmead et. al., Genome Biology, no. 10, 2009.
- Ultrafast and memory-efficient alignment of short DNA sequences to the human genome, Ben Langmead et. al., Genome Biology, 2009 (slides).
Session 9 (5/24): Data Base Algorithms (Richard & Geza)
- BigTable: A Distributed storage system for structured data Fay Chang et. al., Proc. Symposium on Operating System Design and Implementation (OSDI), pages 205-218, 2006.
- MapReduce and Parallel DBMSs: Friends or Foes?, Michael Stonebraker et. al., Communications of the ACM, 53(1):64-71, 2010.
- Interpreting the data: Parallel analysis with Sawzall, Rob Pike et. al., Scientific Programming Journal, 13(4):277-298, 2005.
Session 10 (5/31): Asynchronous Consensus and Averaging, Alternate Frameworks (Zak)
- Network-based Consensus Averaging with General Noisy Channels, IEEE Transactions on Signal Processing, vol. 59, no. 1, 2011.
- Nonparametric Decentralized Detection using Kernel Methods, IEEE Transactions on Signal Processing, pp 4053-4066, 2005; Outstanding ICML student paper award.
- Weighted Gossip: Distributed Averaging Using Non-Doubly Stochastic Matrices, IEEE International Symposium on Information Theory, 2010. slides on related topic
- Dual Averaging for Distributed optimization: Convergence analysis and network scaling, Technical Report, UC Berkeley, 2010.
- GraphLab: A New Parallel Framework for Machine Learning, Yucheng Low et. al., Proc. Conference on Uncertainity in Artificial Intelligence (UAI), 2010, code.
- Conditional Probability Tree Estimation Analysis and Algorithm, Proc. Uncertainity in Artificial Intelligence (UAI), 51-58, 2009, Vowpal Wabbit (code)
Session 11 (6/7): Final presentations

Textbook and Other Useful Resources

Note: For recently developed techniques, we will rely on selected papers, which will be provided in required readings.

Schedule

Meetings	Tue 930 - 1230 hrs
Venue	WCC 403
Office hours	By appointment (request by email)

Links

Relevant Software Tools & Other Resources

This page is maintained by Zak Shafran. Last updated on Feb 4, 2010.