The least squared solution of a problem is one such that minimizes the least square error of the problem. For a given [[Set|set]] of [[Point|points]] $\set{(x_{1},y_{1}),\dots,(x_{n},y_{n})}$ some function $f(x)$ is trying to interpolate through, the least squared [[Numerical Analysis Error Bounds|error]] is calculated as: $\huge \epsilon\set{f} = \sum_{i=1}^{n}{ (f(x_{i}) - y_{i})^{2} } $ >[!note] >The most common type of function used with this method is a [[Linear Transformation|Linear]] [[Function]] $f(x)=a+bx$ > >$\huge >\epsilon(a,b) = >\sum_{i=1}^{n}{ >(f(a,b;x_{i}) - y_{i})^{2} >} >$ > More information about [[Linear Transformation|Linear]] systems here: [[Overdetermined Linear System]]. This form of solution is desirable for many reasons - one such is that the points we are trying to minimize error around don't need to constitute a [[Function|Function]] (which would break [[Interpolation]] methods), and we can have multiple points with equal $x$-values. To find a solution from this is a type of [[Optimization]] problem where we want to minimize the [[Numerical Analysis Error Bounds|error]] [[Function|function]]. One possible way one can do this is through [[Gradient Descent]] using the [[Gradient]] of $\epsilon$. However in the case of when we opearte [[Least Squared Solutions]] on [[Linear Combination|linear combinations]], we can explicitly solve this problem: $\huge \begin{align} \epsilon &= \sum_{i=1}^{n}(a+bx_{i}-y_{i})^{2} \\ \pderiv{\epsilon}{a} &= \sum_{i=1}^{n}{ 2(a+bx_{i}-y_{i}) \\ } \\ &= 2(na + b \sigma_{x}- \sigma_{y}) \\ \pderiv{\epsilon}{b} &= \sum_{i=1}^{n}2x_{i}(a+bx_{i}-y_{i}) \\ &= 2(a \sigma_{x} + b\sigma_{x x} - \sigma_{xy}) \end{align}$ $\huge \begin{cases} na &+ b\sigma_{x} &- \sigma_{y} &= 0 \\ a \sigma_{x} &+ b \sigma_{x x} &- \sigma_{x y} &= 0 \end{cases} $ $\huge \begin{align} \mat{n & \sigma_{x}\\\sigma_{x} & \sigma_{x x}}\mat{a \\b} &= \mat{ \sigma_{y} \\ \sigma _{x y} } \\ \mat{a\\b} &= \mat{n & \sigma_{x}\\\sigma_{x} & \sigma_{x x}}^{-1} \mat{\sigma_{y}\\\sigma_{x y}} \\ &= \frac{1}{n\sigma_{x x}- \sigma_{x}^{2}}\mat{ \sigma_{x x}\sigma_{y} - \sigma_{x} \sigma_{x y } \\ - \sigma_{x} \sigma_{y} + n\sigma_{x y } } \end{align}$ Note that the coefficient is actually the [[Variance]] of $x$. $\op{Var}(x_{i})= \frac{1}{n \sigma_{x x}- \sigma_{x}^{2}}$