Blog

Span and Subspace (Section 9.7 in the textbook)

Definition  Given vectors \(\uu_1,\ldots,\uu_k\) in a vector space \(S\), the set of all their linear combinations, \[ \uu=\alpha_1\uu_1+\cdots+\alpha_k\uu_k,\quad\text{for }\alpha_i\in\R,\ 1\leq i\leq k, \] is called the span of \(\uu_1,\ldots,\uu_k\), and is written \[ \operatorname{span}\{\uu_1,\ldots,\uu_k\}. \] The set \(\{\uu_1,\ldots,\uu_k\}\) is called a generating set of this span.

Examples

1. The span of the single vector \(\uu_1=\begin{bmatrix}4\\2\end{bmatrix}\in\R^2\) is all of its scalar multiples. In the plane this is the straight line through the origin in the direction of \(\uu_1\): \[ \operatorname{span}\{\uu_1\}=\left\{\uu:\uu=\alpha\begin{bmatrix}4\\2\end{bmatrix},\alpha\in\R\right\}. \]

2. Adding a second vector does not always enlarge the span. Here \(\uu_2=-2\uu_1\) points along the same line, so \[ \operatorname{span}\left\{\underbrace{\begin{bmatrix}4\\2\end{bmatrix}}_{\uu_1},\underbrace{\begin{bmatrix}-8\\-4\end{bmatrix}}_{\uu_2}\right\}=\operatorname{span}\left\{\begin{bmatrix}4\\2\end{bmatrix}\right\}. \] Indeed, every combination of the two vectors is a multiple of \(\uu_1\): \[ \uu=\alpha_1\uu_1+\alpha_2\uu_2=\alpha_1\uu_1-2\alpha_2\uu_1=(\alpha_1-2\alpha_2)\uu_1=\beta\cdot\uu_1. \]

Definition  A non-empty subset \(T\) of a vector space \(S\) is a subspace of \(S\) if it satisfies both of the following:

1. It is closed under addition: for all \(\uu\) and \(\vv\) in \(T\), the sum \(\uu+\vv\) is in \(T\).

2. It is closed under scalar multiplication: for all \(\uu\in T\) and \(\alpha\in\R\), the multiple \(\alpha\uu\) is in \(T\).

Only these two conditions need to be checked. The other axioms of a vector space are inherited from \(S\), so a subspace is itself a vector space.

Theorem 9.7.1

Every span is a subspace.

Proof  The span is non-empty, since it contains \(\nul\) and each \(\uu_i\). Now take two elements of it. If \(\uu\in\operatorname{span}\{\uu_1,\ldots,\uu_k\}\), there are scalars \(\alpha_1,\ldots,\alpha_k\) with \(\uu=\alpha_1\uu_1+\cdots+\alpha_k\uu_k\).

Likewise, if \(\vv\in\operatorname{span}\{\uu_1,\ldots,\uu_k\}\), there are scalars \(\beta_1,\ldots,\beta_k\) with \(\vv=\beta_1\uu_1+\cdots+\beta_k\uu_k\). Adding gives \[ \uu+\vv=(\alpha_1+\beta_1)\uu_1+\cdots+(\alpha_k+\beta_k)\uu_k\in\operatorname{span}\{\uu_1,\ldots,\uu_k\}, \] so the span is closed under addition. For any \(c\in\R\), \[ c\uu=c\alpha_1\uu_1+\cdots+c\alpha_k\uu_k\in\operatorname{span}\{\uu_1,\ldots,\uu_k\}, \] so it is closed under scalar multiplication too. Hence \(\operatorname{span}\{\uu_1,\ldots,\uu_k\}\) is a subspace of \(S\). \(\blacksquare\)

Linear Dependence

Definition  A set \(\{\uu_1,\ldots,\uu_k\}\) is linearly dependent (LD) if at least one of the vectors can be written as a linear combination of the others. If none of them can, the set is linearly independent (LI). Informally, a dependent set contains a vector that is redundant.

Examples

1. The set \(\left\{\underbrace{\begin{bmatrix}1\\0\end{bmatrix}}_{\uu_1},\underbrace{\begin{bmatrix}1\\1\end{bmatrix}}_{\uu_2},\underbrace{\begin{bmatrix}1\\-1\end{bmatrix}}_{\uu_3}\right\}\) is linearly dependent, because \(\uu_1=\tfrac12\uu_2+\tfrac12\uu_3\).

2. The set \(\left\{\underbrace{\begin{bmatrix}1\\1\\0\end{bmatrix}}_{\uu_1},\underbrace{\begin{bmatrix}0\\1\\1\end{bmatrix}}_{\uu_2}\right\}\) is linearly independent, since neither vector is a multiple of the other.

3. The set \(\left\{\underbrace{\begin{bmatrix}1\\1\\0\end{bmatrix}}_{\uu_1},\underbrace{\begin{bmatrix}0\\1\\1\end{bmatrix}}_{\uu_2},\underbrace{\begin{bmatrix}0\\0\\0\end{bmatrix}}_{\uu_3}\right\}\) is linearly dependent, because \(\uu_3=0\cdot\uu_1+0\cdot\uu_2\). A set containing the zero vector is always dependent.

Theorem 9.8.1

The definition of dependence is hard to test directly, because we would have to try every vector as a combination of the others. The next theorem gives an equivalent test that is easier to use: look for non-trivial ways of combining the vectors to get zero.

Proof  We prove both directions of the equivalence.

Both directions hold. Therefore the two conditions are equivalent. \(\blacksquare\)

Example Is \(\left\{\underbrace{\begin{bmatrix}2\\0\\1\\-3\end{bmatrix}}_{\uu_1},\underbrace{\begin{bmatrix}0\\1\\1\\1\end{bmatrix}}_{\uu_2},\underbrace{\begin{bmatrix}2\\2\\3\\0\end{bmatrix}}_{\uu_3}\right\}\) linearly independent?

By Theorem 9.8.1, the set is LI exactly when \(\alpha_1\uu_1+\alpha_2\uu_2+\alpha_3\uu_3=\nul\) has only the trivial solution \(\alpha_1=\alpha_2=\alpha_3=0\). Writing this out component by component gives a homogeneous linear system: \[ \left.\begin{aligned}2\alpha_1+2\alpha_3&=0\\\alpha_2+2\alpha_3&=0\\\alpha_1+\alpha_2+3\alpha_3&=0\\-3\alpha_1+\alpha_2&=0\end{aligned}\right\} \] We reduce its augmented matrix. The columns are \(\uu_1,\uu_2,\uu_3\) and the zero right-hand side. \[ \underset{\begin{matrix}\uu_1&\uu_2&\uu_3&\nul\end{matrix}}{\begin{bmatrix}2&0&2&0\\0&1&2&0\\1&1&3&0\\-3&1&0&0\end{bmatrix}} \xrightarrow[\text{Elimination}]{\text{Gauss}} \begin{bmatrix}\boxed{2}&0&2&0\\0&\boxed{1}&2&0\\0&0&\boxed{1}&0\\0&0&0&0\end{bmatrix}. \] The echelon form (reached after interchanging the last two rows) has three pivots and no free variables, so the only solution is the trivial one. Therefore \(\{\uu_1,\uu_2,\uu_3\}\) is linearly independent.

Theorem 9.8.4

For a linearly independent set, a combination of the vectors can be written in only one way. Two combinations that give the same vector must use the same coefficients.

Proof  Subtract the two sides of the equation from each other: \[ \alpha_1\uu_1+\cdots+\alpha_k\uu_k=\beta_1\uu_1+\cdots+\beta_k\uu_k \Rightarrow(\alpha_1-\beta_1)\uu_1+\cdots+(\alpha_k-\beta_k)\uu_k=\nul. \] Because \(\{\uu_1,\ldots,\uu_k\}\) is LI, the only combination that gives \(\nul\) is the trivial one, so \(\alpha_i-\beta_i=0\), that is, \(\alpha_i=\beta_i\) for \(1\leq i\leq k\). \(\blacksquare\)

Theorem 9.8.5

Orthogonal vectors point in completely different directions, so none of them can be built from the others.

Proof  Suppose, for contradiction, that a set \(\{\uu_1,\ldots,\uu_k\}\) of non-zero orthogonal vectors were LD. Then for some \(j\), \[ \uu_j=\sum_{i\ne j}\alpha_i\uu_i. \] Take the dot product of both sides with \(\uu_j\). Every term on the right vanishes, because \(\uu_i\cdot\uu_j=0\) for \(i\ne j\): \[ \uu_j\cdot\uu_j=\left(\sum_{i\ne j}\alpha_i\uu_i\right)\cdot\uu_j =\sum_{i\ne j}\alpha_i\underbrace{(\uu_i\cdot\uu_j)}_{0}=0. \] So \(\uu_j\cdot\uu_j=0\), which forces \(\uu_j=\nul\). This contradicts the assumption that all the vectors are non-zero. \(\blacksquare\)

Important Observation a set that contains the zero vector is always linearly dependent. For \(U=\{\uu_1,\ldots,\uu_k\}\) with, say, \(\uu_j=\nul\), we can take every coefficient to be \(0\) except the one in front of \(\nul\), which can be any non-zero number such as \(5\): \[ 0\uu_1+0\uu_2+\cdots+5\nul+\cdots+0\uu_k=\nul. \] This is a combination with a non-zero coefficient that equals \(\nul\).

Bases, Expansions, Dimension

Definition  A finite set of vectors \(\{\ev_1,\ldots,\ev_k\}\) in a vector space \(S\) is a basis for \(S\) if each vector in \(S\) can be written in exactly one way as \[ \uu=\alpha_1\ev_1+\cdots+\alpha_k\ev_k=\sum_{i=1}^k\alpha_i\ev_i. \] A basis is a set of building blocks: every vector is made from them, and made in only one way.

Theorem 9.9.1

There is an easier test for a basis than checking uniqueness directly.

Proof  Let \(\uu\) be any vector in \(S\). Since \(\Sp\{\ev_1,\ldots,\ev_k\}=S\), we can write \(\uu=\alpha_1\ev_1+\cdots+\alpha_k\ev_k\) for some scalars \(\alpha_i\).

To conclude that \(\{\ev_1,\ldots,\ev_k\}\) is a basis, we still have to show that this expansion of \(\uu\) is unique. If \[ \uu=\beta_1\ev_1+\cdots+\beta_k\ev_k=\alpha_1\ev_1+\cdots+\alpha_k\ev_k \] then \(\beta_i=\alpha_i\) for every \(i\), by Theorem 9.8.4, because \(\{\ev_1,\ldots,\ev_k\}\) is LI. \(\blacksquare\)

Example In \(\R^2\), the vectors \(\ev_1=\col{1\\2}\) and \(\ev_2=\col{1\\-1}\) form a basis. To see how a vector is expanded in it, take \(\uu=\col{u_1\\u_2}\) and look for the coefficients: \[ \col{u_1\\u_2}=\alpha_1\col{1\\2}+\alpha_2\col{1\\-1}\Rightarrow \begin{aligned}\alpha_1+\alpha_2&=u_1\\2\alpha_1-\alpha_2&=u_2.\end{aligned} \] The augmented coefficient matrix reduces as \[ \col{1&1&u_1\\2&-1&u_2}\xrightarrow[\text{Elimination}]{\text{Gauss}} \col{1&1&u_1\\0&-3&u_2-2u_1}. \] Back substitution gives \begin{align*} \alpha_2&=-\tfrac13u_2+\tfrac23u_1\\ \alpha_1&=u_1-\alpha_2=u_1+\tfrac13u_2-\tfrac23u_1=\tfrac13u_1+\tfrac13u_2. \end{align*} So every vector of \(\R^2\) is expanded in this basis as \[ \col{u_1\\u_2}=\frac{u_1+u_2}{3}\col{1\\2}+\frac{2u_1-u_2}{3}\col{1\\-1}. \] The coefficients are unique, as they must be, because \(\ev_1\) and \(\ev_2\) are LI.

Definition  The dimension of a vector space counts how many independent directions it has. If the greatest number of LI vectors that can be found in \(S\) is \(k\), with \(1\leq k<\infty\), then \(S\) is \(k\)-dimensional, and we write \[ \dim S=k. \] If \(S=\{\nul\}\), then \(\dim S=0\). If arbitrarily many LI vectors can be found in \(S\), then \(S\) is infinite-dimensional.

Theorem 9.9.2

The dimension can be read off from any basis.

Proof  Let \(\{\ev_1,\ldots,\ev_k\}\) be the basis. It is LI, because it is a basis, so \(\dim S\geq k\).

Now suppose \(\dim S>k\). Then there is an LI set \(\{\uu_1,\ldots,\uu_{k+1}\}\) of \(k+1\) vectors in \(S\). Since \(\{\ev_1,\ldots,\ev_k\}\) is a basis, each \(\uu_j\) can be written uniquely in terms of it: \[ \uu_j=\sum_{i=1}^k\alpha_{ji}\ev_i,\qquad1\leq j\leq k+1. \] Now consider a combination of the \(\uu_j\) that equals zero, and rewrite it in terms of the \(\ev_i\): \begin{align*} \nul&=\beta_1\uu_1+\cdots+\beta_{k+1}\uu_{k+1}=\sum_{j=1}^{k+1}\beta_j\uu_j\\ &=\sum_{j=1}^{k+1}\beta_j\sum_{i=1}^k\alpha_{ji}\ev_i =\sum_{i=1}^k\ev_i\left[\sum_{j=1}^{k+1}\alpha_{ji}\beta_j\right]. \end{align*} Since the \(\ev_i\) are LI, every bracket must be zero: \[ \Rightarrow\sum_{j=1}^{k+1}\alpha_{ji}\beta_j=0\quad\text{for }1\leq i\leq k. \] Written out in full, these are \(k\) equations in the \(k+1\) unknowns \(\beta_1,\ldots,\beta_{k+1}\): \[ \left.\begin{aligned} \alpha_{11}\beta_1+\alpha_{21}\beta_2+\cdots+\alpha_{k+1,1}\beta_{k+1}&=0\\ \alpha_{1,2}\beta_1+\alpha_{2,2}\beta_2+\cdots+\alpha_{k+1,2}\beta_{k+1}&=0\\&\vdots\\ \alpha_{1,k}\beta_1+\alpha_{2,k}\beta_2+\cdots+\alpha_{k+1,k}\beta_{k+1}&=0 \end{aligned}\right\}\begin{gathered}k\text{ equations in}\\k+1\text{ unknowns.}\end{gathered} \] A homogeneous system with more unknowns than equations always has non-trivial solutions (Theorem 8.3.4). So there are \(\beta_j\), not all zero, with \(\beta_1\uu_1+\cdots+\beta_{k+1}\uu_{k+1}=\nul\). This contradicts the assumption that \(\{\uu_1,\ldots,\uu_{k+1}\}\) is LI. Hence \(\dim S=k\). \(\blacksquare\)

Theorem 9.9.3

Proof  Take the standard basis vectors \(\ev_1=\col{1\\0\\\vdots\\0}\), \(\ldots\), \(\ev_n=\col{0\\\vdots\\0\\1}\). They are LI. Every \(\uu\in\R^n\) equals \(\sum_{i=1}^n\alpha_i\ev_i\) with \(\alpha_i=u_i\), so \(\Sp\{\ev_1,\ldots,\ev_n\}=\R^n\). Hence \(\{\ev_1,\ldots,\ev_n\}\) is a basis for \(\R^n\), and \(\dim\R^n=n\). \(\blacksquare\)

Theorem 9.9.4

The dimension of a span equals the number of vectors in the generating set that are really needed.

Proof  Let \(m\) be the greatest number of linearly independent vectors in \(\{\uu_1,\ldots,\uu_k\}\), and number the vectors so that the first \(m\) are LI. Each of the remaining vectors \(\uu_{m+1},\ldots,\uu_k\) must then depend linearly on \(\uu_1,\ldots,\uu_m\): \[ \uu_i=\alpha_{i1}\uu_1+\cdots+\alpha_{im}\uu_m,\quad m+1\leq i\leq k. \] Now take any \(\vv\in\Sp\{\uu_1,\ldots,\uu_k\}\), say \(\vv=\sum_{i=1}^k\beta_i\uu_i\). Replacing each of the last \(k-m\) vectors by its expression gives \begin{align*} \Rightarrow\quad\vv&=\sum_{i=1}^m\beta_i\uu_i+\sum_{j=m+1}^k\beta_j\uu_j\\ \vv&=\sum_{i=1}^m\beta_i\uu_i+\sum_{j=m+1}^k\beta_j\left[\sum_{i=1}^m\alpha_{ji}\uu_i\right]\\ \vv&=\sum_{i=1}^m\left[\beta_i+\sum_{j=m+1}^k\beta_j\alpha_{ji}\right]\uu_i\in\Sp\{\uu_1,\ldots,\uu_m\}. \end{align*} So \(\Sp\{\uu_1,\ldots,\uu_k\}=\Sp\{\uu_1,\ldots,\uu_m\}\). Since \(\{\uu_1,\ldots,\uu_m\}\) is LI and spans this space, it is a basis for \(\Sp\{\uu_1,\ldots,\uu_k\}\), and the dimension is \(m\). \(\blacksquare\)

Orthogonal Bases

A vector space of dimension two or more has infinitely many bases. In practice we prefer orthogonal bases, because the coefficients of a vector are then easy to find.

Let \(\{\ev_1,\ldots,\ev_k\}\) be an orthogonal basis for \(S\). Any \(\uu\in S\) can be written as \(\uu=\alpha_1\ev_1+\cdots+\alpha_k\ev_k\). To find \(\alpha_j\), take the dot product with \(\ev_j\). By orthogonality, every term except the \(j\)-th vanishes: \[ \uu\cdot\ev_j=(\alpha_1\ev_1+\cdots+\alpha_k\ev_k)\cdot\ev_j=\alpha_j\ev_j\cdot\ev_j. \] Solving for \(\alpha_j\) gives the coefficient formula, where the denominator \(\ev_i\cdot\ev_i=\|\ev_i\|^2\) is the normalization factor:

Dividing each basis vector by its length, \(\hat{\ev}_i=\ev_i/\|\ev_i\|\), turns the formula above into \(\uu=\sum_{i=1}^k(\uu\cdot\hat{\ev}_i)\hat{\ev}_i\). If the basis is orthonormal, that is, orthogonal with every vector of length one, then the denominators are all \(1\) and the coefficients are just dot products: \[ \alpha_i=\frac{\uu\cdot\ev_i}{\underbrace{\ev_i\cdot\ev_i}_{1}}=\uu\cdot\ev_i. \]

Best Approximation

In many engineering applications we are given a vector \(\uu\) and want its best approximation inside a subspace. Let \(S\) be a vector space and \(V\) a subspace of \(S\), and let \(\{\hat{\ev}_1,\ldots,\hat{\ev}_N\}\) be an orthonormal basis for \(V\): \[ \hat{\ev}_i\cdot\hat{\ev}_j=\delta_{ij},\quad1\leq i,j\leq N, \qquad\Sp\{\hat{\ev}_1,\ldots,\hat{\ev}_N\}=V. \] The picture shows the idea. The vector \(\uu_V\) in \(V\) should be as close to \(\uu\) as possible, so the error \(\err=\uu-\uu_V\) should be as short as possible.

We want to minimize the squared length of the error by choosing \(\uu_V\in V\): \[ \min_{\uu_V}\|\err\|_2^2=\min_{\uu_V}\|\uu-\uu_V\|_2^2. \] Every \(\uu_V\in V\) can be written in the orthonormal basis as \(\uu_V=\sum_{i=1}^N\alpha_i\hat{\ev}_i\), so choosing \(\uu_V\) means choosing the numbers \(\alpha_i\). The problem becomes one of minimizing a cost function \(J\) of these numbers: \[ \min_{\alpha_1,\ldots,\alpha_N}\underbrace{\left\|\uu-\sum_{i=1}^N\alpha_i\hat{\ev}_i\right\|_2^2}_{J(\alpha_1,\ldots,\alpha_N)}. \] Expanding the squared length as a dot product, and using the fact that the basis is orthonormal, gives \begin{align*} J(\alpha_1,\ldots,\alpha_N)&=\left(\uu-\sum_{i=1}^N\alpha_i\hat{\ev}_i\right)\cdot\left(\uu-\sum_{i=1}^N\alpha_i\hat{\ev}_i\right)\\ &=\uu\cdot\uu-2\sum_{i=1}^N\alpha_i\uu\cdot\hat{\ev}_i+\underbrace{\left(\sum_{i=1}^N\alpha_i\hat{\ev}_i\right)\cdot\left(\sum_{i=1}^N\alpha_i\hat{\ev}_i\right)}_{\sum_{i=1}^N\alpha_i^2}. \end{align*}

In short, \(J\) is a quadratic function of the \(\alpha_i\): \[ J(\alpha_1,\ldots,\alpha_N)=\underbrace{\uu\cdot\uu-2\sum_{i=1}^N\alpha_i\uu\cdot\hat{\ev}_i+\sum_{i=1}^N\alpha_i^2}_{\text{quadratic function of the }\alpha_i}. \] Its minimum is where all the partial derivatives are zero: \[ \frac{\partial J}{\partial\alpha_i}=-2\uu\cdot\hat{\ev}_i+2\alpha_i=0\Rightarrow\alpha_i=\uu\cdot\hat{\ev}_i. \] So the best approximation of \(\uu\) in the subspace \(V\) is \[ \uu_V=\sum_{i=1}^N(\uu\cdot\hat{\ev}_i)\hat{\ev}_i. \] If \(\uu\) already lies in \(V\), its best approximation is \(\uu\) itself.

Now let us look at the error of this best approximation: \[ \err=\uu-\uu_V=\uu-\sum_{i=1}^N(\uu\cdot\hat{\ev}_i)\hat{\ev}_i. \] Claim  The error \(\err\) is orthogonal to \(V\), that is, orthogonal to every vector in \(V\).

Proof  Any vector in \(V\) can be written \(\vv=\sum_{j=1}^N\beta_j\hat{\ev}_j\), because \(\Sp\{\hat{\ev}_1,\ldots,\hat{\ev}_N\}=V\). Then \begin{align*} \err\cdot\vv&=\left(\uu-\sum_{i=1}^N(\uu\cdot\hat{\ev}_i)\hat{\ev}_i\right)\cdot\left(\sum_{j=1}^N\beta_j\hat{\ev}_j\right)\\ &=\sum_{j=1}^N\beta_j\,\uu\cdot\hat{\ev}_j-\sum_{i=1}^N\sum_{j=1}^N(\uu\cdot\hat{\ev}_i)\beta_j\,\hat{\ev}_i\cdot\hat{\ev}_j\\ &=\sum_{j=1}^N\beta_j\,\uu\cdot\hat{\ev}_j-\sum_{i=1}^N(\uu\cdot\hat{\ev}_i)\beta_i=0, \end{align*} where the last step uses \(\hat{\ev}_i\cdot\hat{\ev}_j=\delta_{ij}\). \(\blacksquare\)

Because the error is perpendicular to \(V\), we call \(\uu_V\) the orthogonal projection of \(\uu\) onto \(V\):

This formula needs an orthonormal basis of \(V\). What if we are only given a basis that is not orthonormal?

Answer  We can use the Gram–Schmidt orthogonalization process.

The Gram-Schmidt Orthogonalization Process

Let \(\{\vv_1,\ldots,\vv_N\}\) be any basis for a subspace \(V\). We want to build from it an orthonormal basis \(\{\hat{\ev}_1,\ldots,\hat{\ev}_N\}\) for the same subspace. The process works one vector at a time, and each new vector is made orthogonal to all the previous ones.

Step 1  Take the first vector and normalize it: \(\hat{\ev}_1=\dfrac{\vv_1}{\|\vv_1\|}\). Then \(\hat{\ev}_1\cdot\hat{\ev}_1=1\).

Step 2  (i) Project \(\vv_2\) onto \(\Sp\{\hat{\ev}_1\}\) to get \(\vv_{2p}=(\vv_2\cdot\hat{\ev}_1)\hat{\ev}_1\). (ii) Compute the projection error \(\err_2=\vv_2-\vv_{2p}\); it is orthogonal to \(\hat{\ev}_1\). Normalize it: \[ \hat{\ev}_2=\dfrac{\err_2}{\|\err_2\|},\qquad\hat{\ev}_2\cdot\hat{\ev}_1=0,\quad\|\hat{\ev}_2\|=1. \] Step 3  Repeat for the remaining vectors, \(2<i\leq N\). Project \(\vv_i\) onto the span of the vectors found so far, \(\{\hat{\ev}_1,\ldots,\hat{\ev}_{i-1}\}\): \[ \vv_{ip}=\sum_{j=1}^{i-1}(\vv_i\cdot\hat{\ev}_j)\hat{\ev}_j. \] The projection error is \(\err_i=\vv_i-\vv_{ip}\). Then set \[ \hat{\ev}_i=\frac{\err_i}{\|\err_i\|}. \] The new vector \(\hat{\ev}_i\) is orthogonal to \(\hat{\ev}_1,\ldots,\hat{\ev}_{i-1}\), because \(\err_i\) is the error of projecting onto their span. It has length one by construction. The division by \(\|\err_i\|\) is allowed because \(\err_i\ne\nul\): the \(\vv_i\) are LI, so \(\vv_i\) does not lie in the span of the previous ones. So Gram–Schmidt turns any basis into an orthonormal basis.