\documentclass[11pt]{article}

\usepackage{classDM}
\usepackage{hyperref}
\usepackage[normalem]{ulem}

\title{Asmt 6: Regression}
\author{Turn in through Canvas by 2:45pm: \\
Wednesday, March 25 \\
100 points}
\date{}



\begin{document}
\maketitle

%\end{titlepage}




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Overview}

In this assignment you will explore regression techniques on high-dimensional data.  

You will use a few data sets for this assignment:
\begin{itemize} \denselist
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/X.csv}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/X.csv}}
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/y.csv}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/y.csv}}
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/M.csv}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/M.csv}}
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/W.csv}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/W.csv}}
\end{itemize}

%These data sets are in matrix format and can be loaded into MATLAB or OCTAVE.  By calling 
%\\
%\texttt{load filename} (for instance \texttt{load X.dat})
%\\
%it will put in memory the data in the file, for instance in the above example the matrix %\texttt{X}.  You can then display this matrix by typing 

%\texttt{X}


For python, you can use the following approach to load the data:

\texttt{X = np.loadtxt('X.csv', delimiter=',')}

\texttt{y = np.loadtxt('y.csv', delimiter=',')}

%\includegraphics[width=.6\linewidth]{python-read.png}

%\vspace{.1in}

\emph{As usual, it is recommended that you use LaTeX for this assignment (or similar way to properly typeset math).  If you do not, you may lose points if your assignment is difficult to read or hard to follow.  Find a sample form in this directory:
\url{http://www.cs.utah.edu/~jeffp/teaching/latex/}}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Linear Regression \& Cross-Validation (100 points)}

We will find coefficients \texttt{alpha} to estimate \texttt{X*alpha $\approx$ y}, using the provided datasets \texttt{X} and \texttt{y}.  We will compare two approaches \emph{least squares} and \emph{ridge regression}.   (e.g., in python as)

\begin{itemize} \denselist
\item[\textsf{Least Squares:} ]  Set \texttt{alpha = LA.inv(X.T @ X) @ X.T @ y.T}
\item[\textsf{Ridge Regression:} ] Set \texttt{alphas = LA.inv(X.T @ X + s*np.identity(50)) @ X.T @ y.T}
\end{itemize}

\paragraph{A (30 points): }
Solve for the coefficients \texttt{alpha} (or \texttt{alphas}) using Least Squares and Ridge Regression with 
$s \in \{0.2,0.4,0.8,1.0,1.2,1.4,1.6\}$
(i.e. $s$ will take on one of those $7$ values each time you try, say obtaining \texttt{alpha04} for $s=0.4$).  
For each set of coefficients, report the error in the estimate $\hat{y}$ of $y$ as 
\texttt{norm(y - X*alpha,2)}.

\paragraph{B (30 points): }
Create three row-subsets of \texttt{X} and \texttt{Y}
\begin{itemize} \denselist
\item \texttt{X1 = X[:66,:]} and \texttt{Y1 = Y[:66]}
\item \texttt{X2 = X[33:,:]} and \texttt{Y2 = Y[33:]}
\item \texttt{X3 = np.vstack((X[:33,:], X[66:,:]))} and \texttt{Y3 = np.vstack((Y[:33], Y[66:]))]}
\end{itemize}

Repeat the above procedure on these subsets and \emph{cross-validate} the solution on the remainder of \texttt{X} and \texttt{Y}.  Specifically, learn the coefficients \texttt{alpha} using, say, \texttt{X1 and Y1} and then measure \texttt{np.norm(Y[66:] - X[66:,:] @ alpha,2)}.  

\paragraph{C (15 points): }
Which approach works best (averaging the results from the three subsets): Least Squares, or for which value of $s$ using Ridge Regression?  



\paragraph{D (15 points): }

Use the same $3$ test / train splits, taking their average errors, to estimate the average squared error on each predicted data point.  

What is problematic about the above estimate, especially for the best performing parameter value $s$?  

\paragraph{E (10 points): }
Even circumventing the issue raised in part \textbf{D}, what \emph{assumptions} about how the data set $(\texttt{X,y})$ is generated are needed in an assessment based on cross-validation?  


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Bonus: Matching Pursuit (5 points)}

Consider a linear equation \texttt{W = M*S} where \texttt{M} is a measurement matrix filled with random values $\{-1, 0, +1\}$ (although now that they are there, they are no longer random), and \texttt{W} is the output of the sparse signal \texttt{S} when measured by \texttt{M}.  

Use Matching Pursuit (as described in the book as \textbf{Algorithm 5.5.1}) to recover the non-zero entries from \texttt{S}.  Record the order in which you find each entry and the residual vector after each step.  



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section*{BONUS: $L_1$ ball geometry (2 points)}
%In class we examined pictures of $L_1$-balls in $\mathbb{R}^2$, but in higher dimensions their geometry is more wild.  Its boundary has "facets" of $k$-dimensions for each $k \in \{0, \ldots, d-1\}$, these are linear subspaces of of rank $k$.  

%As a function of $d$ and $k$, derive how many different facets there are of $k$-dimensions when on the boundary of $L_1$-balls in $\mathbb{R}^d$.  Explain how you derive your answer.  




\end{document}
