Solve inconsistent systems optimally and expose the geometry of any matrix
Real data rarely satisfies every linear equation exactly. Least squares finds the vector Ax̂ closest to b. The singular value decomposition then provides a universal coordinate system for understanding rank, conditioning, compression, pseudoinverses, and principal directions.
Least-squares geometry
Replace an impossible equation with a closest-point problem
When b is not in C(A), no exact solution to Ax=b exists. Least squares chooses x̂ to minimize ‖Ax−b‖². The fitted vector p=Ax̂ is the orthogonal projection of b onto C(A), so the residual r=b−Ax̂ is orthogonal to every column of A.
Normal equations can magnify conditioning problems
Forming AᵀA squares the 2-norm condition number. QR factorization or SVD is usually preferred for serious numerical work.
Linear regression as a matrix problem
Fit slope and intercept simultaneously
For points (xᵢ,yᵢ), model y≈b+mx. The design matrix has a column of ones and a column of x-values. Solving least squares chooses b and m so the vertical residual sum of squares is minimized.
Interactive least-squares line fitting
Enter data and verify the orthogonality conditions
Least-squares regression laboratory
Enter one x,y pair per line. Residual segments and the best-fit line are drawn automatically.
Singular value decomposition
Factor any matrix into orthogonal directions and nonnegative stretches
Input rotation/reflection
Right singular vectors define orthonormal input directions.
Axis scaling
Nonnegative singular values scale mutually perpendicular axes.
Output rotation/reflection
Left singular vectors orient the resulting axes in the output space.
The nonzero singular values are square roots of nonzero eigenvalues of AᵀA. Their count is rank(A). The largest and smallest singular values determine maximum and minimum stretch.
Interactive singular-value geometry
Map a unit circle into an ellipse and measure conditioning
2×2 singular-value analyzer
The unit circle is transformed into an ellipse whose semiaxis lengths are σ₁ and σ₂.
Pseudoinverse, compression, and PCA
Use SVD as a universal structural tool
| Application | SVD role |
|---|---|
| Moore–Penrose pseudoinverse | Invert nonzero singular values to compute minimum-norm least-squares solutions. |
| Low-rank approximation | Keep the largest singular values and corresponding singular vectors; truncated SVD gives the best rank-k approximation in standard norms. |
| Conditioning | κ₂(A)=σmax/σmin for full-rank square A. Large ratios indicate sensitivity. |
| PCA | After centering data, right singular vectors identify principal directions and squared singular values relate to explained variance. |
| Noise filtering | Small singular components may represent weak or noise-dominated directions, though truncation must be justified by context. |
Review questions
Test least-squares geometry and SVD structure