Instructor: Prof. INDIKA Rajapakse

Teaching Assistant: STEPHEN Lindsly

Class Time: Tuesday and Thursday, 4:00 - 5:30PM, Office Hours: Tuesday and Thursday, 5:30 - 7:00PM

Syllabus

Timeline

Simplicity, Rigor, and Magic

References of Interest

• Digital Library

• Matrix World

• Poincaré Diagram

• Eigshow in MATLAB

• SVD and Image Compression

• The Fundamental Theorem of Linear Algebra

• Euclidean Distance Matrices

PROBLEM SETS

Problem Set 1: Solution

Problem Set 2: Solution

Problem Set 3: Solution

Problem Set 3b: Solution

Problem Set 4: Code Solution, Math Solution

Problem Set 5: Solution

Final Problem Set: Solution

PROBLEM OF THE DAY

POD1: Missing Entries Solution

POD2: Class Expectations

POD3: Least squares

POD4: Clustering

POD5: Learning

NOTES, SLIDES, AND PAPERS

Date: 1-21-2021

Topics

• Basic introduction to data representation: Vectors, matrices, and tensors

• Introduction to Eigenvalues and Eigenvectors

Notes

• Singular Value Decomposition (SVD)

Extra slides: The following slides are the data I showed you on 1-19-2021, with a bit more explanation to help you understand where the data comes from.

• Biological Data

Papers

1. Donoho D. "50 years of data science." Journal of Computational and Graphical Statistics. 2017 Oct 2;26(4):745-66.

2. Lieberman-Aiden, Erez, ..., Groudine Mark, ..., Lander Eric. "Comprehensive mapping of long-range interactions reveals folding principles of the human genome." Science 326.5950 (2009): 289-293.

Date: 1-26-2021

Notes

• Singular Value Decomposition (SVD)

Resources

• ImageSVD (.zip folder with MATLAB code)

Extra slides: The following slides contain the guest lecturers, helpful advice, and the course syllabus

• Mathematics of Data Introduction

Date: 1-28-2021

Guest Lecture: Dr. Cleve Moler

• How SVD Saved the Universe

Date: 2-2-2021

Notes

• Vector and Matrix Norms

• KL Divergence and Stein Divergence

Resources

• Network Data and Visualization

Papers

Classics (just browse)

1. Turk, Matthew, and Alex Pentland. "Eigenfaces for recognition." Journal of cognitive neuroscience 3.1 (1991): 71-86.

2. Bott, Raoul. "Morse theory indomitable." Publications Mathématiques de l'IHÉS 68 (1988): 99-114.

Preparation for Spectral Clustering

1. Ng, Andrew Y., Michael I. Jordan, and Yair Weiss. "On spectral clustering: Analysis and an algorithm." Advances in neural information processing systems 2 (2002): 849-856.

2. Von Luxburg, Ulrike. "A tutorial on spectral clustering." Statistics and computing 17.4 (2007): 395-416

Date: 2-4-2021

Notes

• Vector and Matrix Norms

• KL Divergence and Stein Divergence

Papers

1. Gavish, Matan, and David L. Donoho. "The optimal hard threshold for singular values is 4/sqrt(3)." IEEE Transactions on Information Theory 60.8 (2014): 5040-5053. (Amazing Paper!)

1. • Supplementary Code

   • Python Code

2. Chen, Jie, Alfred O. Hero III, and Indika Rajapakse. "Spectral identification of topological domains." Bioinformatics 32.14 (2016): 2151-2158.

Date: 2-9-2021

Resources

• Simplicity, Rigor, and Magic

Notes

• Singular Value Decomposition (SVD)

• Laplacian Matrix

• Inverse and Pseudoinverse

Date: 2-11-2021

Guest Lecture: Prof. Gilbert Strang

• Three Great Factorizations

Resources

• From Cleve’s Corner: Gil Strang and the CR Matrix Factorization and Notes on CR and west0479

Date: 2-16-2021

Notes

• k-Means Clustering

• Spectral Clustering, Laplacian Eigenmaps, and Entropy

Papers

1. Ng, Andrew Y., Michael I. Jordan, and Yair Weiss. "On spectral clustering: Analysis and an algorithm." Advances in neural information processing systems 2 (2002): 849-856.

2. Belkin, Mikhail, and Partha Niyogi. "Laplacian eigenmaps and spectral techniques for embedding and clustering." Advances in neural information processing systems. 2002.

3. Von Luxburg, Ulrike. "A tutorial on spectral clustering." Statistics and computing 17.4 (2007): 395-416. (Excellent Review!)

Date: 2-18-2021

Guest Lecture: Col. Chris Macedonia

• Adventures in Data

Date: 2-23-2021

Notes

• Dynamic Mode Decomposition (DMD)

Book:

• Kutz, J. Nathan, et al. Dynamic mode decomposition: data-driven modeling of complex systems. Society for Industrial and Applied Mathematics, 2016.

Chapter 1: Dynamic Mode Decomposition: An Introduction

Nice Video!

• Introduces Topological Data Analysis

Date: 2-25-2021

Notes

• Dynamic Mode Decomposition (DMD)

Book:

• Kutz, J. Nathan, et al. Dynamic mode decomposition: data-driven modeling of complex systems. Society for Industrial and Applied Mathematics, 2016.

Chapter 1: Dynamic Mode Decomposition: An Introduction

Code

• DMD.m

• DMD_intro.m

Papers

• Champion K, Lusch B, Kutz JN, Brunton SL. "Data-driven discovery of coordinates and governing equations." Proceedings of the National Academy of Sciences. 2019 Nov 5;116(45):22445-51.

• Ronquist S, Patterson G, Muir LA, Lindsly S, Chen H, Brown M, Wicha MS, Bloch A, Brockett R, Rajapakse I. "Algorithm for cellular reprogramming." Proceedings of the National Academy of Sciences. 2017 Nov 7;114(45):11832-7.

Date: 3-2-2021

Notes

• Network Centrality

Slides

• A Matrix

• Controllability

Papers

• Hirsh, Seth M., et al. "Centering data improves the dynamic mode decomposition." SIAM Journal on Applied Dynamical Systems 19.3 (2020): 1920-1955.

• Proctor, Joshua L., Steven L. Brunton, and J. Nathan Kutz. "Dynamic mode decomposition with control." SIAM Journal on Applied Dynamical Systems 15.1 (2016): 142-161.

Date: 3-4-2021

Book:

• Kutz, J. Nathan. Data-driven modeling & scientific computation: methods for complex systems & big data. Oxford University Press, 2013.

Independent Component Analysis

Papers

Data-guided Control (DGC)

• Ronquist, Scott, et al. "Algorithm for cellular reprogramming." Proceedings of the National Academy of Sciences 114.45 (2017): 11832-11837.

Non-negative Matrix Factorization (NMF or NNMF)

• Lee, Daniel D., and H. Sebastian Seung. "Learning the parts of objects by non-negative matrix factorization." Nature 401.6755 (1999): 788-791.

DMD with Control

• Proctor, Joshua L., Steven L. Brunton, and J. Nathan Kutz. "Dynamic mode decomposition with control." SIAM Journal on Applied Dynamical Systems 15.1 (2016): 142-161.

Video: Dynamic Mode Decomposition (DMD) with Control

Date: 3-9-2021

Guest Lecture: Dr. Reza Ghanadan from Google

• Assuring AI for Real-World Decision Making with Robust AI System Design

Date: 3-11-2021

Notes

• KL Divergence and Stein Divergence

• Dimension Reduction of Data (PCA, t-SNE, UMAP)

Date: 3-16-2021

Guest Lecture: Dr. David Heckerman from Amazon

• Methods for Causal Discovery

Date: 3-18-2021

Notes

• Mathematics of Optimization

• Nonlinear Dimension Reduction

Papers

• Kobak, Dmitry, and George C. Linderman. "Initialization is critical for preserving global data structure in both t-SNE and UMAP." Nature Biotechnology (2021): 1-2.

• McInnes, Leland, John Healy, and James Melville. "Umap: Uniform manifold approximation and projection for dimension reduction." arXiv preprint arXiv:1802.03426 (2018).

• Van der Maaten, Laurens, and Geoffrey Hinton. "Visualizing data using t-SNE." Journal of machine learning research 9.11 (2008).

Date: 3-25-2021

Notes

• Mathematics of Optimization

• Convex Optimization Tools: CVX

• Clustering (Code)

Papers

• Candès, Emmanuel J., et al. "Robust principal component analysis?." Journal of the ACM (JACM) 58.3 (2011): 1-37.

Date: 3-30-2021

Notes

• Clustering (Code)

• Matrix Completion

Papers

• Udell, Madeleine, and Alex Townsend. "Why are big data matrices approximately low rank?." SIAM Journal on Mathematics of Data Science 1.1 (2019): 144-160.

Date: 4-1-2021

Slides

• Tensor Decompositions

Notes

• Tensors and Hypergraphs

Book:

• Eldén, Lars. Matrix methods in data mining and pattern recognition. Society for Industrial and Applied Mathematics, 2007.

Chapter 8: Tensor Decomposition

Papers

• 1. Bryan, Kurt, and Tanya Leise. "The $25,000,000,000 eigenvector: The linear algebra behind Google." SIAM review 48.3 (2006): 569-581.

  2. Kolda, Tamara G., and Brett W. Bader. "Tensor decompositions and applications." SIAM review 51.3 (2009): 455-500. (Excellent Review!)

  3. Chen C, Rajapakse I. "Tensor Entropy for Uniform Hypergraphs." IEEE Transactions on Network Science and Engineering 7.4 (2020): 2889-2900.

  4. Benson, Austin R., David F. Gleich, and Desmond J. Higham. "Higher-order Network Analysis Takes Off, Fueled by Classical Ideas and New Data." arXiv preprint arXiv:2103.05031 (2021). (Nice Read!)

Date: 4-6-2021

Slides

• Tensors and Hypergraphs Slides

Notes

• Tensors and Hypergraphs Notes

• Tensor Toolbox

Book:

• Eldén, Lars. Matrix methods in data mining and pattern recognition. Society for Industrial and Applied Mathematics, 2007.

Chapter 8: Tensor Decomposition

Papers

1. Kolda, Tamara G., and Brett W. Bader. "Tensor decompositions and applications." SIAM review 51.3 (2009): 455-500. (Excellent Review!)

2. Chen C, Rajapakse I. "Tensor Entropy for Uniform Hypergraphs." IEEE Transactions on Network Science and Engineering 7.4 (2020): 2889-2900.

3. Sweeney P, Chen C, Rajapakse I, Cone R. "Network Dynamics of Hypothalamic Feeding Neurons." Proceedings of the National Academy of Sciences (2021).

4. Williams, Alex H., et al. "Unsupervised discovery of demixed, low-dimensional neural dynamics across multiple timescales through tensor component analysis." Neuron 98.6 (2018): 1099-1115.

5. Wolf, Michael M., Alicia M. Klinvex, and Daniel M. Dunlavy. "Advantages to modeling relational data using hypergraphs versus graphs." 2016 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2016.

Date: 4-8-2021

Slides

• Topological Data Analysis

• Multi-Correlation Summary

Papers

1. Carlsson, Gunnar. "The shape of biomedical data." Current Opinion in Systems Biology 1 (2017): 109-113.

2. Carlsson, Gunnar. "Topology and data." Bulletin of the American Mathematical Society 46.2 (2009): 255-308.

3. Lum, Pek Y., et al. "Extracting insights from the shape of complex data using topology." Scientific reports 3.1 (2013): 1-8.

Date: 4-13-2021

Notes

• Turing System and Bifurcations

• Series Expansions

Papers

1. Brunton, Steven L., Joshua L. Proctor, and J. Nathan Kutz. "Discovering governing equations from data by sparse identification of nonlinear dynamical systems." Proceedings of the national academy of sciences 113.15 (2016): 3932-3937.

Date: 4-15-2021

Data-guided Control

Notes

• Data-Guided Controllability

• Perturbation Theory and Condition Number

• Google matrix and Perron

Papers

1. Ronquist S, Patterson G, Muir LA, Lindsly S, Chen H, Brown M, Wicha M, Bloch A, Brockett R and Rajapakse I. "Algorithm for Cellular Reprogramming." Proceedings of the National Academy of Sciences 114.45 (2017): 11832-11837.

2. Liu, Yang-Yu, Jean-Jacques Slotine, and Albert-László Barabási. "Controllability of complex networks." Nature 473.7346 (2011): 167-173.

Date: 4-20-2021

Slides

• Ethics of Data Interpretation

Notes

• Data Visualization

• Dissimilarity Measures