A diagonal matrix is very easy to work with in matrix multiplication (Example HPDM) and an orthonormal basis also has many advantages (Theorem COB). How about converting a matrix to a diagonal matrix through a similarity transformation using a unitary matrix (i.e. build a diagonal matrix representation with an orthonormal matrix)? That'd be fantastic! When can we do this? We can always accomplish this feat when the matrix is normal, and normal matrices are the only ones that behave this way. Here is the theorem.
Suppose that $A$ is a square matrix. Then there is a unitary matrix $U$ and a diagonal matrix $D$, with diagonal entries equal to the eigenvalues of $A$, such that $\adjoint{U}AU=D$ if and only if $A$ is a normal matrix.
We can rearrange the conclusion of this theorem to read $A=UD\adjoint{U}$. Recall that a unitary matrix can be viewed as a geometry-preserving transformation (isometry), or more loosely as a rotation of sorts. Then a matrix-vector product, $A\vect{x}$, can be viewed instead as a sequence of three transformations. $\adjoint{U}$ is unitary, and so is a rotation. Since $D$ is diagonal, it just multiplies each entry of a vector by a scalar. Diagonal entries that are positive or negative, with absolute values bigger or smaller than 1 evoke descriptions like reflection, expansion and contraction. Generally we can say that $D$ “stretches” a vector in each component. Final multiplication by $U$ undoes (inverts) the rotation performed by $\adjoint{U}$. So a normal matrix is a rotation-stretch-rotation transformation.
The orthonormal basis formed from the columns of $U$ can be viewed as a system of mutually perpendicular axes. The rotation by $\adjoint{U}$ allows the transformation by $A$ to be replaced by the simple transformation $D$ along these axes, and then $D$ brings the result back to the original coordinate system. For this reason Theorem OD is known as the Principal Axis Theorem.
The columns of the unitary matrix in Theorem OD create an especially nice basis for use with the normal matrix. We record this observation as a theorem.
Suppose that $A$ is a normal matrix of size $n$. Then there is an orthonormal basis of $\complex{n}$ composed of eigenvectors of $A$.
In a vague way Theorem OBNM is an improvement on Theorem HMOE which said that eigenvectors of a Hermitian matrix for different eigenvalues are always orthogonal. Hermitian matrices are normal and we see that we can find at least one basis where every pair of eigenvectors is orthogonal. Notice that this is not a generalization, since Theorem HMOE states a weak result which applies to many (but not all) pairs of eigenvectors, while Theorem OBNM is a seemingly stronger result, but only asserts that there is one collection of eigenvectors with the stronger property.
Given an $n\times n$ matrix $A$, an orthonormal basis for $\complex{n}$, comprised of eigenvectors of $A$ is an extremely useful basis to have at the service of the matrix $A$. Why do we say this? We can consider the vectors of a basis as a preferred set of directions, known as “axes,” which taken together might also be called a “coordinate system.” The standard basis of Definition SUV could be considered the default, or prototype, coordinate system. When a basis is orthornormal, we can consider the directions to be standardized to have unit length, and we can consider the axes as being mutually perpendicular. But there is more — let us be a bit more formal.
Suppose $U$ is a matrix whose columns are an orthonormal basis of eigenvectors of the $n\times n$ matrix $A$. So, in particular $U$ is a unitary matrix (Theorem CUMOS). For a vector $\vect{x}\in\complex{n}$, use the notation $\hat{\vect{x}}$ for the vector representation of $\vect{x}$ relative to the orthonormal basis. So the entries of $\hat{\vect{x}}$, used in a linear combination of the columns of $U$ will create $\vect{x}$. With Definition MVP, we can write this relationship as \begin{align*} U\hat{\vect{x}} = \vect{x} \end{align*} Since $\adjoint{U}$ is the inverse of $U$ (Definition UM), we can rearrange this equation as \begin{align*} \hat{\vect{x}} = \adjoint{U}\vect{x} \end{align*} This says we can easily create the vector representation relative to the orthonormal basis with a matrix-vector product of the adjoint of $U$. Note that the adjoint is much easier to compute than a matrix inverse, which would be one general way to obtain a vector representation. This is our first observation about coordinatization relative to orthonormal basis. However, we already knew this, as we just have Theorem COB in disguise (see Exercise OD.T20).
We also know that orthonormal bases play nicely with inner products. Theorem UMPIP says unitary matrices preserve inner products (and hence norms). More geometrically, lengths and angles are preserved by multiplication by a unitary matrix. Using our notation, this becomes \begin{align*} \innerproduct{\vect{x}}{\vect{y}} =\innerproduct{U\hat{\vect{x}}}{U\hat{\vect{y}}} =\innerproduct{\hat{\vect{x}}}{\hat{\vect{y}}} \end{align*} So we can compute inner products with the original vectors, or with their representations, and obtain the same result. It follows that norms, lengths, and angles can all be computed with the original vectors or with the representations in the new coordinate system based on an orthonormal basis.
So far we have not really said anything new, nor has the matrix $A$, or its eigenvectors, come into play. We know that a matrix is really a linear transformation, so we express this view of a matrix as a function by writing generically that $A\vect{x}=\vect{y}$. The matrix $U$ will diagonalize $A$, creating the diagonal matrix $D$ with diagonal entries equal to the eigenvalues of $A$. We can write this as $\adjoint{U}AU = D$ and convert to $\adjoint{U}A=D\adjoint{U}$. Then we have \begin{align*} \hat{\vect{y}} =\adjoint{U}\vect{y} =\adjoint{U}A\vect{x} =D\adjoint{U}\vect{x} =D\hat{\vect{x}} \end{align*} So with the coordinatized vectors, the transformation by the matrix $A$ can be accomplished with multiplication by a diagonal matrix $D$. A moment's thought should convince you that a matrix-vector product with a diagonal matrix is exeedingly simple computationally. Geometrically, this is simply stretching, contracting and/or reflecting in the direction of each basis vector (“axis”). And the multiples used for these changes are the diagonal entries of the diagonal matrix, the eigenvalues of $A$.
So the new coordinate system (provided by the orthonormal basis of eigenvectors) is a collection of mutually perpendicular unit vectors where inner products are preserved, and the action of the matrix $A$ is described by multiples (eigenvalues) of the entries of the coordinatized versions of the vectors. Nice.