Section 2 of 11 · Notes pages 7–21
Linear Systems, Matrices and Vector Spaces
Solving linear systems by Gauss and Gauss–Jordan elimination, when solutions exist and when they are unique, and the first look at vector spaces, norms, dot products and the Cauchy–Schwarz inequality.
Systems of Linear Algebraic Equations
Many physical systems are modeled by linear equations, so being able to set up and solve them is of fundamental importance in engineering. This is where linear algebra begins.
The general form of a system of \(m\) linear algebraic equations in \(n\) unknowns is:
Here the \(a_{ij}\) are the coefficients and the \(c_i\) are the right-hand sides; both can be real or complex. The \(x_j\) are the unknowns, which can also be real or complex.
A few words describe the possible outcomes:
If replacing \(x_1,\ldots,x_n\) by numbers \(s_1,\ldots,s_n\) makes all the equations true, then \(s_1,\ldots,s_n\) is a solution of the system.
A system with at least one solution is consistent.
If there is exactly one solution, the solution is unique.
If there is more than one solution, the solutions are non-unique.
A system with no solution is inconsistent.
The collection of all solutions is the solution set.
Examples Take the case \(m=n=2\) with real coefficients: \[ \begin{aligned} a_{11}x_1+a_{12}x_2&=c_1\\ a_{21}x_1+a_{22}x_2&=c_2. \end{aligned} \] Each of these equations describes a line in the plane, and a solution of the system is a point that lies on both lines. There are three possibilities, shown in the figure: the lines cross at a single point, they are parallel and never meet, or they are the same line.
Consistent with a unique solution, since the lines cross once: \(2x_1-x_2=5,\ x_1+3x_2=-1\).
Inconsistent, with no solution, since the lines are parallel: \(x_1+3x_2=1,\ x_1+3x_2=2\).
Consistent with non-unique solutions, since the second equation is three times the first and so describes the same line: \(x_1+3x_2=1,\ 3x_1+9x_2=3\).
In general, each equation defines a hyperplane in \(n\)-dimensional space. The system is consistent if these hyperplanes have a common point, and the solution is unique if they meet in exactly one point.
Obtaining solutions: Gauss or Gaussian Elimination
Gaussian elimination solves a system by simplifying it step by step, using only operations that do not change the solutions, until the answer can be read off. The example below starts with \(m=n=3\), eliminates one unknown at a time, and finishes with back substitution.
Original system
Begin with three equations in the three unknowns \(x_1,x_2,x_3\).
Eliminate \(x_1\)
Add \(-3\) times equation 1 to equation 2, and \(-1\) times equation 1 to equation 3:
The first unknown disappears from the second and third equations.
Eliminate \(x_2\)
Subtract equation 2 from equation 3:
This produces a triangular system: every row begins farther to the right than the row above it.
Make the second leading coefficient \(1\)
Divide equation 2 by \(-2\):
Read the last equation
The triangular system gives the final unknown immediately:
Substitute into equation 2
Insert \(x_3=1\) into \(x_2-2x_3=-3\):
Substitute into equation 1
Insert \(x_2=-1\) and \(x_3=1\) into the first equation:
The system has the unique solution
Important Observation the method has two stages. First, Gauss elimination turns the original system into a triangular one. Second, back substitution solves the triangular system from the bottom up. \[ \boxed{\begin{gathered}\text{Original System of}\\\text{Equations}\end{gathered}} \xrightarrow[\text{Elimination}]{\text{Gauss}} \boxed{\begin{gathered}\text{Triangular}\\\text{System of Equations}\end{gathered}} \xrightarrow[\text{Substitution}]{\text{Back}}\boxed{\text{Solution}} \] Question Does the triangular system have the same solutions as the original system? We need this, otherwise the method would solve the wrong problem.
Answer Yes. Gauss elimination uses only the three elementary row operations below, and none of them changes the solution set.
Definition The elementary row operations are:
Adding a multiple of one equation to another, \((\mathrm{eq.}\,j)\longrightarrow(\mathrm{eq.}\,j)+\alpha(\mathrm{eq.}\,k)\) with \(j\ne k\).
Multiplying an equation by a nonzero constant, \((\mathrm{eq.}\,j)\longrightarrow\alpha(\mathrm{eq.}\,j)\).
Interchanging two equations, \((\mathrm{eq.}\,j)\longleftrightarrow(\mathrm{eq.}\,k)\).
If one linear system is obtained from another by a finite number of elementary row operations, then the two systems are equivalent, that is, they have the same solution set.
Proof Follow the induction argument one step at a time.
Set up the induction. Call the original system \(LS_1\) and the system obtained by the row operations \(LS_2\), so
\[ LS_1\xrightarrow{E_1}\cdots\xrightarrow{E_q}LS_2, \]where \(E_1,\ldots,E_q\) are elementary row operations. We use induction on \(q\), the number of operations.
Base case: one operation. For \(q=1\),
\[ LS_1\xrightarrow{E_1}LS_2. \]Write \(S_{LS_1}\) and \(S_{LS_2}\) for the solution sets. For each elementary operation, we prove both inclusions \(S_{LS_1}\subset S_{LS_2}\) and \(S_{LS_2}\subset S_{LS_1}\). Together they give \(S_{LS_1}=S_{LS_2}\).
(i) Add a multiple of another equation.
\[ (\mathrm{eq.}\,j)\longrightarrow(\mathrm{eq.}\,j)+\alpha(\mathrm{eq.}\,k). \]Take \((s_1,\ldots,s_n)\in S_{LS_1}\). It satisfies equations \(j\) and \(k\), so it also satisfies their stated combination. Every other equation is unchanged. Therefore \((s_1,\ldots,s_n)\in S_{LS_2}\), proving \(S_{LS_1}\subset S_{LS_2}\).
Reverse direction for operation (i). Take \((s_1,\ldots,s_n)\in S_{LS_2}\). It satisfies equation \(k\) and the new equation \(j\). Subtracting \(\alpha(\mathrm{eq.}\,k)\) recovers the original equation \(j\):
\[ (\mathrm{eq.}\,j)+\alpha(\mathrm{eq.}\,k)-\alpha(\mathrm{eq.}\,k)=(\mathrm{eq.}\,j). \]Hence \(S_{LS_2}\subset S_{LS_1}\), and therefore \(S_{LS_1}=S_{LS_2}\).
(ii) Multiply an equation by a nonzero constant.
\[ (\mathrm{eq.}\,j)\longrightarrow\alpha(\mathrm{eq.}\,j),\qquad\alpha\ne0. \]If \((s_1,\ldots,s_n)\in S_{LS_1}\), multiplying equation \(j\) by \(\alpha\) keeps it true. Thus \((s_1,\ldots,s_n)\in S_{LS_2}\), and \(S_{LS_1}\subset S_{LS_2}\).
Reverse direction for operation (ii). If \((s_1,\ldots,s_n)\in S_{LS_2}\), it satisfies \(\alpha(\mathrm{eq.}\,j)\). Because \(\alpha\ne0\), divide by \(\alpha\) to recover
\[ \frac1\alpha\,\alpha(\mathrm{eq.}\,j)=(\mathrm{eq.}\,j). \]Therefore \(S_{LS_2}\subset S_{LS_1}\), so again \(S_{LS_1}=S_{LS_2}\).
(iii) Interchange two equations.
\[ (\mathrm{eq.}\,j)\longleftrightarrow(\mathrm{eq.}\,k). \]The two systems contain exactly the same equations in a different order. Reordering cannot change which points satisfy all equations, so \(S_{LS_1}=S_{LS_2}\). This completes the base case \(q=1\).
Induction step. Assume the result holds after \(q\) operations, so \(S_{LS_1}=S_{LS_q}\). The system \(LS_{q+1}\) comes from \(LS_q\) by one more elementary operation. The base case gives \(S_{LS_{q+1}}=S_{LS_q}\), and therefore
\[ S_{LS_{q+1}}=S_{LS_q}=S_{LS_1}. \]The result holds for every finite number of row operations. \(\blacksquare\)
Examples
Example An inconsistent system. We apply the row operations step by step. After each system, the operations used to reach the next one are written on its right. \[ \begin{aligned} 2x_1+3x_2-2x_3&=4\\x_1-2x_2+x_3&=3\\7x_1-x_3&=2 \end{aligned}\qquad(\mathrm{eq.}\,1)\longrightarrow\tfrac12(\mathrm{eq.}\,1) \] \[ \begin{aligned} x_1+\tfrac32x_2-x_3&=2\\x_1-2x_2+x_3&=3\\7x_1-x_3&=2 \end{aligned}\qquad\begin{aligned} (\mathrm{eq.}\,2)&\longrightarrow(\mathrm{eq.}\,2)-(\mathrm{eq.}\,1)\\ (\mathrm{eq.}\,3)&\longrightarrow(\mathrm{eq.}\,3)-7(\mathrm{eq.}\,1) \end{aligned} \] \[ \begin{aligned} x_1+\tfrac32x_2-x_3&=2\\-\tfrac72x_2+2x_3&=1\\-\tfrac{21}2x_2+6x_3&=-12 \end{aligned}\qquad\begin{aligned} (\mathrm{eq.}\,2)&\longrightarrow-\tfrac27(\mathrm{eq.}\,2)\\ (\mathrm{eq.}\,3)&\longrightarrow(\mathrm{eq.}\,3)+\tfrac{21}2(\mathrm{eq.}\,2) \end{aligned} \] \[ \begin{aligned} x_1+\tfrac32x_2-x_3&=2\\x_2-\tfrac47x_3&=-\tfrac27\\0&=-15 \end{aligned} \quad\leftarrow\quad\begin{gathered}\text{cannot be satisfied by}\\\text{any choice of }x_1,x_2\text{ and }x_3.\end{gathered} \] The last equation, \(0=-15\), can never hold, so the system is inconsistent.
The right-hand side of the third equation decided this. If it had been \(17\) instead of \(2\), the same elimination would have ended with \(0=0\) instead: \[ \begin{aligned} \overset{\text{(p)}}{x_1}+\tfrac32\overset{\text{(p)}}{x_2}-\overset{\text{(f)}}{x_3}&=2\\ x_2-\tfrac47x_3&=-\tfrac27\\0&=0. \end{aligned} \] Now the system is consistent. The equation \(0=0\) says nothing, so only two equations remain for three unknowns. We call \(x_1\) and \(x_2\) the pivot variables (p), because they sit at the start of an equation, and \(x_3\) a free variable (f), because it can be chosen freely. Setting \(x_3=\alpha\) and solving upward gives \begin{align*} x_3&=\alpha\quad\text{(a free parameter)}\\ x_2&=-\tfrac27+\tfrac47\alpha\\ x_1&=\tfrac{17}7+\tfrac17\alpha. \end{align*} Every value of \(\alpha\) gives a solution, so the solution set is a whole line of points:
Suppose a consistent system has \(r\) pivot variables and \(n\) unknowns, with \(r<n\). Then only \(r\) of the unknowns are determined by the equations, and the remaining \(n-r\) are free. The solution set can be written with \(n-r\) free parameters, and it is called an “\((n-r)\)-parameter family of solutions”. Every choice of the parameters gives one “particular” solution.
Theorem 8.3.2 Existence/Uniqueness of Solutions for Linear Systems
The following theorem answers, for any linear system, whether a solution exists and whether it is unique. Consider a linear system (\(LS\)) of \(m\) equations in \(n\) unknowns: \[ \left.\begin{aligned} a_{11}x_1+\cdots+a_{1n}x_n&=c_1\\&\vdots\\a_{m1}x_1+\cdots+a_{mn}x_n&=c_m \end{aligned}\right\}\ m\text{ equations with }n\text{ unknowns}. \] Apply Gauss elimination to obtain an upper triangular (row echelon) system, and let \(r\) be the number of pivot variables. Then:
If \(r=m\), the system is always consistent.
If \(r<m\), the system is consistent exactly when the last \((m-r)\) equations have zero right-hand sides.
If \(r<m\) and any of the last \((m-r)\) equations has a non-zero right-hand side, the system is inconsistent.
If the system is consistent and \(r=n\), the solution is unique.
If the system is consistent and \(r<n\), the solution set is an \((n-r)\)-parameter family of solutions.
In words: after elimination, an equation that reads \(0=\text{(non-zero number)}\) makes the system impossible, and unknowns without a pivot are free.
Theorem 8.3.4 Existence and Uniqueness of Solutions for \(c_1=\cdots=c_m=0\)
A system whose right-hand sides are all zero is called homogeneous. It is a special case of Theorem 8.3.2 and has a simpler answer.
A homogeneous system is always consistent, because \(x_1=\cdots=x_n=0\) is a solution.
If \(r<n\), the solution is non-unique: there are non-trivial solutions as well.
Matrix Notation
Writing out all the unknowns at every step is tedious. Only the numbers matter, so we collect them into a coefficient matrix and a vector of right-hand sides: \[ \left.\begin{aligned}a_{11}x_1+\cdots+a_{1n}x_n&=c_1\\&\vdots\\a_{m1}x_1+\cdots+a_{mn}x_n&=c_m\end{aligned}\right\} \longrightarrow \underbrace{\begin{bmatrix}a_{11}&a_{12}&\cdots&a_{1n}\\\vdots&\vdots&&\vdots\\a_{m1}&a_{m2}&\cdots&a_{mn}\end{bmatrix}}_{\text{coefficient matrix}},\quad \underbrace{\begin{bmatrix}c_1\\\vdots\\c_m\end{bmatrix}}_{\text{vector of right-hand sides}}. \] Gauss elimination is easiest to carry out on the augmented coefficient matrix, which puts the right-hand sides next to the coefficients: \[ \begin{bmatrix}a_{11}&a_{12}&\cdots&a_{1n}&c_1\\\vdots&\vdots&&\vdots&\vdots\\a_{m1}&a_{m2}&\cdots&a_{mn}&c_m\end{bmatrix} \] The elementary row operations act on the rows of this matrix exactly as they acted on the equations.
Definition Two matrices are called row-equivalent if one can be obtained from the other by finitely many elementary row operations.
Gauss-Jordan Reduction or Elimination
Gauss–Jordan reduction is another way to find the solution set. Instead of stopping at a triangular form, it keeps going until every pivot is \(1\) and is the only non-zero entry in its column. This is well suited to hand computation. (For large problems solved on a computer, Gauss elimination followed by back substitution needs fewer operations.)
Example Solve \[ \begin{aligned}x_1+2x_2+x_3&=4\\3x_1+8x_2+7x_3&=20\\2x_1+7x_2+9x_3&=23.\end{aligned} \] We form the augmented coefficient matrix and reduce it. The first three steps below are Gauss elimination and end in the echelon form; the remaining steps clear the entries above the pivots. \begin{gather*} \begin{bmatrix}1&2&1&4\\3&8&7&20\\2&7&9&23\end{bmatrix} \xrightarrow[{-2R_1+R_3\to R_3}]{{-3R_1+R_2\to R_2}} \begin{bmatrix}1&2&1&4\\0&2&4&8\\0&3&7&15\end{bmatrix} \xrightarrow{\frac12R_2\to R_2} \begin{bmatrix}1&2&1&4\\0&1&2&4\\0&3&7&15\end{bmatrix}\\[8pt] \xrightarrow{-3R_2+R_3\to R_3} \underbrace{\begin{bmatrix}1&2&1&4\\0&1&2&4\\0&0&1&3\end{bmatrix}}_{\substack{\text{Result of Gauss elimination}\\\text{(row echelon form)}}} \xrightarrow{-2R_2+R_1\to R_1} \begin{bmatrix}1&0&-3&-4\\0&1&2&4\\0&0&1&3\end{bmatrix}\\[8pt] \xrightarrow[{-2R_3+R_2\to R_2}]{{3R_3+R_1\to R_1}} \underbrace{\begin{bmatrix}1&0&0&5\\0&1&0&-2\\0&0&1&3\end{bmatrix}}_{\substack{\text{Result of Jordan elimination}\\\text{(reduced row echelon form)}}}. \end{gather*} The solution can now be read directly from the last column: \(x_1=5\), \(x_2=-2\), \(x_3=3\).
Example A system with free variables. Solve \[ \begin{aligned} x_1+x_2+x_3+x_4&=12\\x_1+2x_2+5x_4&=17\\3x_1+2x_2+4x_3-x_4&=31. \end{aligned} \] Reducing the augmented matrix gives \begin{gather*} \begin{bmatrix}1&1&1&1&12\\1&2&0&5&17\\3&2&4&-1&31\end{bmatrix} \xrightarrow[{-3R_1+R_3\to R_3}]{{-R_1+R_2\to R_2}} \begin{bmatrix}1&1&1&1&12\\0&1&-1&4&5\\0&-1&1&-4&-5\end{bmatrix}\\[8pt] \xrightarrow{R_2+R_3\to R_3} \begin{bmatrix}1&1&1&1&12\\0&1&-1&4&5\\0&0&0&0&0\end{bmatrix} \xrightarrow{-R_2+R_1\to R_1} \underbrace{\begin{bmatrix}1&0&2&-3&7\\0&1&-1&4&5\\0&0&0&0&0\end{bmatrix}}_{\text{reduced row echelon form}}. \end{gather*} The pivots are in the first two columns, so \(x_1\) and \(x_2\) are the pivot variables and \(x_3\) and \(x_4\) are the free variables. The row of zeros means the system is consistent. We give each free variable a parameter, \(x_3=\alpha\) and \(x_4=\beta\), and solve the two non-zero rows for the pivot variables: \begin{align*} x_1&=-2\alpha+3\beta+7\\ x_2&=\alpha-4\beta+5. \end{align*} Collecting the constants and the coefficients of each parameter shows the structure of the solution set: a fixed particular solution plus combinations of two direction vectors.
Partial Pivoting
Sometimes the entry we want as the next pivot is zero.
Example Suppose Gauss elimination has reached \[ \begin{bmatrix}1&2&4&7\\0&0&5&8\\0&3&6&9\end{bmatrix}. \] The entry in the second row and second column is \(0\), so it cannot serve as a pivot. We swap the second and third rows and carry on as usual: \[ \xrightarrow{R_2\leftrightarrow R_3}\begin{bmatrix}1&2&4&7\\0&3&6&9\\0&0&5&8\end{bmatrix}. \] Interchanging rows like this is called pivoting. When several rows could be swapped in, we choose the one whose entry has the largest absolute value as the pivot; this is called partial pivoting.
Vector Spaces
So far the unknowns have been numbers. We now describe a more general setting, in which the unknowns are vectors. A vector in three dimensions can be written either as an ordered triple or as a column of its components:
\[ \boldsymbol{v}=(a,b,c)=\begin{bmatrix}a\\b\\c\end{bmatrix}. \] We will use the column notation, and call \(a\), \(b\) and \(c\) the components of \(\boldsymbol{v}\).
Definition Vector addition adds the vectors component by component: \[ \boldsymbol{v}=\begin{bmatrix}v_1\\v_2\\v_3\end{bmatrix},\quad\boldsymbol{u}=\begin{bmatrix}u_1\\u_2\\u_3\end{bmatrix},\qquad \boldsymbol{v}+\boldsymbol{u}=\begin{bmatrix}v_1+u_1\\v_2+u_2\\v_3+u_3\end{bmatrix}. \] Geometrically, \(\boldsymbol{u}+\boldsymbol{v}\) is the diagonal of the parallelogram spanned by \(\boldsymbol{u}\) and \(\boldsymbol{v}\).
Definition Scalar multiplication multiplies every component by the same real number \(c\): \[ \boldsymbol{v}=\begin{bmatrix}v_1\\\vdots\\v_n\end{bmatrix},\quad c\cdot\boldsymbol{v}=\begin{bmatrix}cv_1\\\vdots\\cv_n\end{bmatrix},\qquad c\in\R. \] Definition Vector space. A set \(V\) of vectors, with vector addition and scalar multiplication defined on it, is a vector space if, for all \(\boldsymbol{u},\boldsymbol{v},\boldsymbol{w}\in V\) and all \(r,s\in\R\), the following axioms hold (and \(V\) is closed under both operations: \(\boldsymbol{u}+\boldsymbol{v}\in V\) and \(r\boldsymbol{u}\in V\)):
\(\boldsymbol{u}+\boldsymbol{v}=\boldsymbol{v}+\boldsymbol{u}\)
Commutativity of vector addition.
\(\boldsymbol{u}+(\boldsymbol{v}+\boldsymbol{w})=(\boldsymbol{u}+\boldsymbol{v})+\boldsymbol{w}\)
Associativity of vector addition.
\(\uu+\nul=\nul+\uu=\uu\)
Existence of a zero element.
\(\uu+(-\uu)=\nul\)
Existence of an additive inverse.
\(r(\uu+\vv)=r\uu+r\vv\)
Scalar multiplication distributes over vector addition.
\((r+s)\uu=r\uu+s\uu\)
A sum of scalars distributes over a vector.
\(r(s\uu)=(rs)\uu\)
Associativity of scalar multiplication.
\(1\cdot\uu=\uu\)
Multiplicative identity.
These are the familiar rules of arithmetic, written for vectors. The most important example is the space of all columns of \(n\) real numbers.
\[ \R^n=\left\{\vv=\begin{bmatrix}v_1\\\vdots\\v_n\end{bmatrix}:v_i\in\R,\ 1\leq i\leq n\right\} \] is a vector space with the vector addition and scalar multiplication defined earlier.
Proof Each of the axioms (a) to (h) holds for \(\R^n\) because the operations act on one component at a time, and each component is an ordinary real number, which obeys the same rules. \(\blacksquare\)
Three more facts follow from the axioms, for any \(\alpha\in\R\) and any \(\uu\in\R^n\):
\(\alpha\cdot\nul=\nul\)
Indeed, \(\alpha\nul=\alpha(\nul+\nul)=\alpha\nul+\alpha\nul\), and subtracting \(\alpha\nul\) from both sides leaves \(\nul=\alpha\nul\).
\(0\cdot\uu=\nul\)
The same argument works: \(0\uu=(0+0)\uu=0\uu+0\uu\).
\(-\uu=(-1)\cdot\uu\)
Indeed, \(\uu+(-1)\uu=(1+(-1))\uu=0\cdot\uu=\nul\), so \((-1)\uu\) is the additive inverse of \(\uu\).
Definition A norm measures the length of a vector. A function \(f\) from a vector space \(V\) to the real numbers is a norm if, for all \(\uu,\vv\in V\) and \(\alpha\in\R\):
\(f(\uu)>0\) if \(\uu\ne\nul\), and \(f(\uu)=0\) if \(\uu=\nul\).
Positive definiteness. Only the zero vector has length zero.
\(f(\alpha\uu)=|\alpha|f(\uu)\)
Absolute homogeneity. Scaling a vector scales its length.
\(f(\uu+\vv)\leq f(\uu)+f(\vv)\)
Triangle inequality.
Examples In the vector space \(\R^n\), the following functions are norms:
The Euclidean norm, the usual length of a vector: \(f(\uu)=\sqrt{u_1^2+u_2^2+\cdots+u_n^2}=\left[\sum_{i=1}^n u_i^2\right]^{1/2}=\|\uu\|_2\).
The \(p\)-norm, for any \(p\geq1\): \(f(\uu)=\left[\sum_{i=1}^n |u_i|^p\right]^{1/p}=\|\uu\|_p\).
Definition The dot product of two vectors in \(\R^n\) multiplies matching components and adds the results: \[ \uu\cdot\vv=\sum_{i=1}^n u_iv_i. \] Note that the dot product of a vector with itself is its squared Euclidean length, \(\|\uu\|_2=\sqrt{\uu\cdot\uu}\).
The properties of the dot product
\(\uu\cdot\vv=\vv\cdot\uu\) (commutativity).
\(\uu\cdot\uu>0\) for all \(\uu\ne\nul\), and \(\uu\cdot\uu=0\) for \(\uu=\nul\).
\((\alpha\uu+\beta\vv)\cdot\ww=\alpha(\uu\cdot\ww)+\beta(\vv\cdot\ww)\) (linearity).
Example Linearity lets us expand a dot product just like a product of brackets. To expand \((3\uu-\vv)\cdot(5\ww+\boldsymbol{t})\): \begin{align*} (3\uu-\vv)\cdot(5\ww+\boldsymbol{t}) &=(3\uu-\vv)\cdot5\ww+(3\uu-\vv)\cdot\boldsymbol{t}\\ &=15(\uu\cdot\ww)-5(\vv\cdot\ww)+3(\uu\cdot\boldsymbol{t})-(\vv\cdot\boldsymbol{t}). \end{align*}
Theorem: The Cauchy–Schwarz inequality
The dot product of two vectors can never be larger in size than the product of their lengths.
Proof If \(\vv=\nul\), both sides are zero, so assume \(\vv\ne\nul\). The idea is to look at the squared length of \(\uu+\alpha\vv\) as \(\alpha\) varies. A squared length is never negative: \[ J(\alpha)=(\uu+\alpha\vv)\cdot(\uu+\alpha\vv)\geq0\quad\text{for any }\alpha\in\R. \]
Expanding with the properties of the dot product shows that \(J(\alpha)\) is a quadratic function of \(\alpha\), whose graph is a parabola that never goes below zero: \[ J(\alpha)=(\uu+\alpha\vv)\cdot(\uu+\alpha\vv)=\underbrace{(\uu\cdot\uu)+2\alpha(\uu\cdot\vv)+\alpha^2(\vv\cdot\vv)}_{\text{a quadratic function of $\alpha$}}. \] Its lowest point is where the derivative is zero: \[ \frac{d}{d\alpha}J(\alpha)=2\uu\cdot\vv+2\alpha\underbrace{(\vv\cdot\vv)}_{\|\vv\|_2^2}=0 \quad\Rightarrow\quad\alpha=-\frac{\uu\cdot\vv}{\|\vv\|^2}. \] This is a minimum, not a maximum, because the second derivative is \(\frac{d^2}{d\alpha^2}J(\alpha)=2\|\vv\|_2^2>0\) for \(\vv\ne\nul\). Substituting this \(\alpha\) back into \(J\) gives the smallest value of the parabola, and that value must still be \(\geq0\): \begin{align*} &\left(\uu-\frac{\uu\cdot\vv}{\|\vv\|^2}\vv\right)\cdot\left(\uu-\frac{\uu\cdot\vv}{\|\vv\|^2}\vv\right)\\ &\qquad=\|\uu\|^2-\frac{2(\uu\cdot\vv)^2}{\|\vv\|^2}+\frac{(\uu\cdot\vv)^2}{\|\vv\|^2}\\ &\qquad=\|\uu\|^2-\frac{(\uu\cdot\vv)^2}{\|\vv\|^2}\geq0.\\ \Rightarrow\quad&\|\uu\|^2\|\vv\|^2\geq(\uu\cdot\vv)^2\\ \Rightarrow\quad&|\uu\cdot\vv|\leq\|\uu\|\cdot\|\vv\|.\quad\blacksquare \end{align*} Definition The angle between two non-zero vectors in \(\R^n\) is
By the Cauchy–Schwarz inequality the ratio inside \(\cos^{-1}\) lies in \([-1,1]\), so the angle is well defined.
Definition
Two vectors are orthogonal (perpendicular) if \(\uu\cdot\vv=0\).
A set \(\{\uu_1,\ldots,\uu_k\}\) is an orthogonal set if every two different vectors in it are orthogonal: \(\uu_i\cdot\uu_j=0\) for \(1\leq i\ne j\leq k\).
Question Given a vector \(\uu\ne\nul\), how can we find a vector \(\hat{\uu}\) of length one that points in the same direction as \(\uu\), so that the angle between them is zero?
Answer Only vectors of the form \(\alpha\uu\) with \(\alpha>0\) make a zero angle with \(\uu\). So we look for the \(\alpha\) for which \(\|\alpha\uu\|=1\): \[ 1=\|\alpha\uu\|^2=(\alpha\uu)\cdot(\alpha\uu)=\alpha^2(\uu\cdot\uu)=\alpha^2\|\uu\|^2 \quad\Rightarrow\quad\alpha=\frac1{\|\uu\|}. \] So \(\hat{\uu}=\dfrac{\uu}{\|\uu\|}\). This process is called vector normalization.
Definition A set \(\{\uu_1,\ldots,\uu_k\}\) is a set of orthonormal vectors if the vectors are mutually orthogonal and each has length one: \[ \uu_i\cdot\uu_j=0\quad\text{for }1\leq i\ne j\leq k \quad\text{and}\quad\|\uu_i\|=1\quad\text{for }1\leq i\leq k. \] Both conditions can be written in one line with the Kronecker delta, which is \(1\) when the two indices are equal and \(0\) otherwise: \[ \uu_i\cdot\uu_j=\delta_{ij}=\begin{cases}1&\text{if }i=j\\0&\text{if }i\ne j.\end{cases} \]
More Generalizations
The ideas above reach well beyond columns of numbers.
Not every vector space is finite dimensional. For example, the real-valued functions on the interval \([0,1]\) form a vector space, with addition and scalar multiplication done pointwise: \[ (\uu+\vv)(x)\equiv u(x)+v(x),\qquad(\alpha\uu)(x)\equiv\alpha u(x). \]
Not every vector space comes with a dot (“inner”) product and a norm; these are extra structure.
If a vector space has a dot product, we can use it to define a norm, \(\|\uu\|=\sqrt{\uu\cdot\uu}\), called the induced norm.
A common alternative to the ordinary dot product is the weighted dot product, in which each term gets a positive weight \(w_i\): \[ \uu\cdot\vv=\sum_{i=1}^n u_iv_iw_i,\quad\text{where }w_i>0,\ 1\leq i\leq n. \]
Extension to Complex Case
Vectors and matrices can have complex entries, with complex numbers as scalars instead of real ones.
This is very useful in electrical engineering, where linear systems with complex coefficients are common.
With complex vectors, the dot product has to be defined with care. If it were defined as in the real case, a non-zero complex vector could have a squared length that is zero or even negative. The fix is to conjugate the entries of the second vector. For \(\uu\in\C^n\) and \(\vv\in\C^n\),
Now the squared length is a sum of the non-negative numbers \(|u_i|^2\), as it should be.
Example Linear systems with complex numbers are solved with the same row operations, now using complex arithmetic. Find the solution set of \[ \begin{aligned}(1+j2)x_1+(2-j)x_2&=3+j8\\(2-j)x_1+(1+j)x_2&=4+j3.\end{aligned} \] We reduce the augmented coefficient matrix. The first step divides the first row by \(1+j2\); the two divisions needed are worked out below the matrices. \begin{gather*} \begin{bmatrix}1+j2&2-j&3+j8\\2-j&1+j&4+j3\end{bmatrix} \xrightarrow{R_1/(1+j2)\to R_1} \begin{bmatrix}1&-j&\frac{19+j2}{5}\\2-j&1+j&4+j3\end{bmatrix}\\[8pt] \xrightarrow{R_2-(2-j)R_1\to R_2} \begin{bmatrix}1&-j&\frac{19+j2}{5}\\0&2+j3&-4+j6\end{bmatrix} \xrightarrow{R_2/(2+j3)\to R_2} \begin{bmatrix}1&-j&\frac{19+j2}{5}\\0&1&\frac{10+j24}{13}\end{bmatrix}\\[8pt] \xrightarrow{R_1+jR_2\to R_1} \begin{bmatrix}1&0&\frac{127+j76}{65}\\0&1&\frac{10+j24}{13}\end{bmatrix}. \end{gather*} Here we used \(\dfrac{2-j}{1+j2}=\dfrac{(2-j)(1-j2)}{5}=-j\) and \(\dfrac{3+j8}{1+j2}=\dfrac{(3+j8)(1-j2)}{5}=\dfrac{19+j2}{5}\). The solution is read from the last column: \[ x_1=\frac{127+j76}{65},\qquad x_2=\frac{10+j24}{13}. \] Substituting back confirms it: \((1+j2)x_1+(2-j)x_2=3+j8\) and \((2-j)x_1+(1+j)x_2=4+j3\). As this example shows, arithmetic with complex matrices takes considerably more work than with real ones.