Part II: Spectral Theory | Chapter 1

Eigenvalues & Eigenvectors

The fundamental objects of spectral theory and matrix analysis

Historical Context

The concept of eigenvalues emerged in the 18th century from the study of rotational motion and quadratic forms. Euler investigated the principal axes of rigid bodies, Lagrange studied the secular equation in celestial mechanics, and Cauchy proved that symmetric matrices have real eigenvalues. The term "eigenvalue" (from the German Eigenwert, meaning "own value") was coined by David Hilbert in 1904 while developing the spectral theory of integral operators.

Arthur Cayley and James Joseph Sylvester formalized matrix theory in the 1850s, establishing the characteristic polynomial and the Cayley-Hamilton theorem. The 20th century saw eigenvalue theory become central to quantum mechanics (Heisenberg's matrix mechanics, Schrödinger's equation), vibration analysis, stability theory, and eventually Google's PageRank algorithm. Today, eigenvalue problems pervade virtually every branch of applied mathematics and engineering.

2.1 Definition and Geometric Interpretation

Definition: Eigenvalue and Eigenvector

Let $T: V \to V$ be a linear operator on a vector space $V$ over a field $\mathbb{F}$. A scalar $\lambda \in \mathbb{F}$ is an eigenvalue of $T$ if there exists a nonzero vector $v \in V$ such that

$$Tv = \lambda v$$

The vector $v$ is called an eigenvector corresponding to $\lambda$.

Geometrically, an eigenvector is a direction that is preserved under the linear transformation—the map $T$ stretches or compresses along this direction by the factor $\lambda$. If $\lambda > 1$, the eigenvector is stretched; if $0 < \lambda < 1$, it is compressed; if $\lambda < 0$, the direction is reversed.

For a matrix $A \in \mathbb{F}^{n \times n}$, the eigenvalue equation $Av = \lambda v$ can be rewritten as $(A - \lambda I)v = 0$. This homogeneous system has a nonzero solution if and only if $A - \lambda I$ is singular:

$$\det(A - \lambda I) = 0$$

This determinantal equation is the characteristic equation, and the polynomial $p(\lambda) = \det(A - \lambda I)$ is the characteristic polynomial of $A$. By the fundamental theorem of algebra, a matrix in $\mathbb{C}^{n \times n}$ always has exactly $n$ eigenvalues (counted with multiplicity).

2.2 Computing Eigenvalues

For small matrices, eigenvalues can be found by solving the characteristic equation directly. For a $2 \times 2$ matrix $A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$:

$$p(\lambda) = \lambda^2 - (a+d)\lambda + (ad - bc) = \lambda^2 - \text{tr}(A)\lambda + \det(A)$$

Applying the quadratic formula:

$$\lambda = \frac{\text{tr}(A) \pm \sqrt{\text{tr}(A)^2 - 4\det(A)}}{2}$$

For $3 \times 3$ matrices, the characteristic polynomial is cubic and can be solved using Cardano's formula. For $n \geq 5$, Abel's impossibility theorem tells us there is no general algebraic solution, and numerical methods (QR algorithm, Lanczos iteration, divide-and-conquer) must be employed.

Theorem: Vieta's Formulas for Eigenvalues

For $A \in \mathbb{C}^{n \times n}$ with eigenvalues $\lambda_1, \ldots, \lambda_n$:

$$\text{tr}(A) = \sum_{i=1}^n \lambda_i, \qquad \det(A) = \prod_{i=1}^n \lambda_i$$

2.3 Eigenspaces and Multiplicity

Definition: Eigenspace

The eigenspace of $A$ corresponding to eigenvalue $\lambda$ is the null space $E_\lambda = \ker(A - \lambda I) = \{v \in V : Av = \lambda v\}$. This is always a subspace of $V$.

Two types of multiplicity characterize each eigenvalue:

Algebraic multiplicity $\text{am}(\lambda)$: the multiplicity of $\lambda$ as a root of the characteristic polynomial
Geometric multiplicity $\text{gm}(\lambda) = \dim E_\lambda = \dim \ker(A - \lambda I)$: the dimension of the eigenspace

Theorem: Multiplicity Inequality

For every eigenvalue $\lambda$: $1 \leq \text{gm}(\lambda) \leq \text{am}(\lambda)$. A matrix is diagonalizable if and only if $\text{gm}(\lambda) = \text{am}(\lambda)$ for every eigenvalue. When strict inequality holds, the matrix is called defective.

Eigenvectors corresponding to distinct eigenvalues are always linearly independent. This is a crucial result: if $A$ has $n$ distinct eigenvalues, then $A$ is automatically diagonalizable, as we can form a basis of eigenvectors.

2.4 Spectral Properties

The eigenvalues of a matrix encode deep structural information:

Key Spectral Properties

The spectral radius $\rho(A) = \max_i |\lambda_i|$ controls convergence of $A^k$
$A$ is invertible iff $0$ is not an eigenvalue
Eigenvalues of $A^k$ are $\lambda_i^k$; of $A^{-1}$ are $\lambda_i^{-1}$
Eigenvalues of $A + cI$ are $\lambda_i + c$ (spectral shift)
Similar matrices share the same eigenvalues

The Gershgorin circle theorem provides eigenvalue localization: every eigenvalue of $A$ lies in at least one Gershgorin disc $D_i = \{z \in \mathbb{C} : |z - a_{ii}| \leq R_i\}$, where $R_i = \sum_{j \neq i} |a_{ij}|$.

For symmetric matrices, all eigenvalues are real; for orthogonal matrices, all eigenvalues have modulus 1; for positive definite matrices, all eigenvalues are strictly positive.

2.5 Complex Eigenvalues and Rotations

Real matrices can have complex eigenvalues, which always occur in conjugate pairs $\lambda = a \pm bi$. The prototypical example is the rotation matrix:

$$R_\theta = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$$

with eigenvalues $\lambda = e^{\pm i\theta} = \cos\theta \pm i\sin\theta$. These lie on the unit circle in $\mathbb{C}$, reflecting the fact that rotations preserve lengths.

More generally, if $\lambda = a + bi$ is a complex eigenvalue of a real matrix with eigenvector $v = u + iw$, then the action of $A$ on $\text{span}(u, w)$ is a rotation by $\arg(\lambda)$ followed by scaling by $|\lambda|$:

$$A\big|_{\text{span}(u,w)} \sim |\lambda| \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$$

This explains oscillatory behavior in dynamical systems $\dot{x} = Ax$: complex eigenvalues produce spirals whose frequency is $|\text{Im}(\lambda)|$ and whose growth/decay rate is $\text{Re}(\lambda)$.

Computational Laboratory

This simulation computes eigenvalues and eigenvectors, verifies the eigenvalue equation, demonstrates algebraic vs geometric multiplicity, explores complex eigenvalues of rotations, and visualizes spectral properties including the circular law for random matrices.

Eigenvalues & Eigenvectors: Computation and Visualization

Python

eigenvalues.py159 lines

import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

# ============================================================
# Eigenvalues and Eigenvectors: Computation & Visualization
# ============================================================

# --- 1. Basic eigenvalue computation ---
print("=" * 60)
print("EIGENVALUE COMPUTATION")
print("=" * 60)

A = np.array([[4, 1], [2, 3]])
eigenvalues, eigenvectors = np.linalg.eig(A)

print(f"Matrix A:\n{A}")
print(f"\nEigenvalues: {eigenvalues}")
print(f"\nEigenvectors (columns):\n{eigenvectors}")

# Verify Av = λv
for i in range(len(eigenvalues)):
    lam = eigenvalues[i]
    v = eigenvectors[:, i]
    Av = A @ v
    lam_v = lam * v
    print(f"\nλ_{i+1} = {lam:.4f}")
    print(f"  Av   = {Av}")
    print(f"  λv   = {lam_v}")
    print(f"  Av = λv? {np.allclose(Av, lam_v)}")

# --- 2. Characteristic polynomial ---
print("\n" + "=" * 60)
print("CHARACTERISTIC POLYNOMIAL")
print("=" * 60)

coeffs = np.poly(A)
print(f"Characteristic polynomial coefficients: {coeffs}")
print(f"p(λ) = λ² - {A[0,0]+A[1,1]}λ + {np.linalg.det(A):.0f}")
print(f"Trace = {np.trace(A)} = sum of eigenvalues = {np.sum(eigenvalues):.4f}")
print(f"Det   = {np.linalg.det(A):.4f} = product of eigenvalues = {np.prod(eigenvalues):.4f}")

# --- 3. Algebraic vs geometric multiplicity ---
print("\n" + "=" * 60)
print("ALGEBRAIC vs GEOMETRIC MULTIPLICITY")
print("=" * 60)

B = np.array([[2, 1, 0],
              [0, 2, 0],
              [0, 0, 3]])
evals_B, evecs_B = np.linalg.eig(B)
print(f"Matrix B (defective):\n{B}")
print(f"Eigenvalues: {evals_B}")
print(f"λ=2: algebraic multiplicity = 2, geometric multiplicity = 1")
print(f"λ=3: algebraic multiplicity = 1, geometric multiplicity = 1")

C = np.array([[2, 0, 0],
              [0, 2, 0],
              [0, 0, 3]])
evals_C = np.linalg.eigvals(C)
print(f"\nMatrix C (diagonalizable):\n{C}")
print(f"Eigenvalues: {evals_C}")
print(f"λ=2: algebraic mult = geometric mult = 2")

# --- 4. Complex eigenvalues ---
print("\n" + "=" * 60)
print("COMPLEX EIGENVALUES - ROTATION MATRICES")
print("=" * 60)

theta = np.pi / 4
R = np.array([[np.cos(theta), -np.sin(theta)],
              [np.sin(theta),  np.cos(theta)]])
evals_R = np.linalg.eigvals(R)
print(f"Rotation by θ = π/4:")
print(f"R = \n{R}")
print(f"Eigenvalues: {evals_R}")
print(f"|λ₁| = {abs(evals_R[0]):.6f}, |λ₂| = {abs(evals_R[1]):.6f}")

# --- 5. Spectral radius and power iteration ---
print("\n" + "=" * 60)
print("SPECTRAL PROPERTIES & POWER ITERATION")
print("=" * 60)

M = np.array([[3, 1, 0], [0, 2, 1], [0, 0, 1]])
evals_M = np.linalg.eigvals(M)
spectral_radius = max(abs(evals_M))
print(f"Matrix M:\n{M}")
print(f"Eigenvalues: {evals_M}")
print(f"Spectral radius ρ(M) = {spectral_radius:.4f}")
print(f"Trace = {np.trace(M):.4f} = Σλ = {np.sum(evals_M):.4f}")
print(f"Det = {np.linalg.det(M):.4f} = Πλ = {np.prod(evals_M):.4f}")

print("\nPower iteration for dominant eigenvalue:")
x = np.random.randn(3)
x = x / np.linalg.norm(x)
for k in range(8):
    x_new = M @ x
    rayleigh = x_new @ x / (x @ x)
    x = x_new / np.linalg.norm(x_new)
    print(f"  Iteration {k+1}: λ ≈ {rayleigh:.6f}")

# --- 6. Visualization ---
fig, axes = plt.subplots(1, 3, figsize=(16, 5))

# Plot 1: Eigenvectors of 2x2 matrix
ax = axes[0]
for i, (color, label) in enumerate(zip(["#3b82f6", "#0ea5e9"], ["v₁", "v₂"])):
    v = eigenvectors[:, i] * eigenvalues[i].real
    ax.arrow(0, 0, eigenvectors[0, i], eigenvectors[1, i], head_width=0.06,
             head_length=0.04, fc=color, ec=color, linewidth=2, label=f"{label} (λ={eigenvalues[i]:.1f})")
    ax.arrow(0, 0, v[0], v[1], head_width=0.06, head_length=0.04,
             fc=color, ec=color, linewidth=1, alpha=0.5)
ax.set_xlim(-1.5, 2.5)
ax.set_ylim(-1.5, 2.5)
ax.set_aspect("equal")
ax.grid(True, alpha=0.3)
ax.legend(fontsize=8)
ax.set_title("Eigenvectors of A", fontsize=12, fontweight="bold")
ax.set_facecolor("#0f172a")

# Plot 2: Complex eigenvalues on unit circle
ax = axes[1]
angles = np.linspace(0, 2 * np.pi, 100)
ax.plot(np.cos(angles), np.sin(angles), "#38bdf8", alpha=0.3, linewidth=1)
for angle_mult in [1, 2, 3, 4, 5, 6, 7, 8]:
    t = angle_mult * theta
    R_t = np.array([[np.cos(t), -np.sin(t)], [np.sin(t), np.cos(t)]])
    evals_t = np.linalg.eigvals(R_t)
    ax.scatter(evals_t.real, evals_t.imag, c="#3b82f6", s=40, zorder=5)
ax.scatter([evals_R[0].real], [evals_R[0].imag], c="#0ea5e9", s=100, zorder=6, label="λ₁")
ax.scatter([evals_R[1].real], [evals_R[1].imag], c="#06b6d4", s=100, zorder=6, label="λ₂")
ax.axhline(y=0, color="white", linewidth=0.5, alpha=0.3)
ax.axvline(x=0, color="white", linewidth=0.5, alpha=0.3)
ax.set_aspect("equal")
ax.set_title("Complex Eigenvalues", fontsize=12, fontweight="bold")
ax.legend(fontsize=8)
ax.set_facecolor("#0f172a")

# Plot 3: Circular law for random matrices
ax = axes[2]
np.random.seed(42)
N = 200
random_matrix = np.random.randn(N, N) / np.sqrt(N)
evals_rand = np.linalg.eigvals(random_matrix)
ax.scatter(evals_rand.real, evals_rand.imag, s=3, c="#3b82f6", alpha=0.7)
circle = plt.Circle((0, 0), 1, fill=False, color="#38bdf8", linewidth=2, linestyle="--")
ax.add_patch(circle)
ax.set_xlim(-1.5, 1.5)
ax.set_ylim(-1.5, 1.5)
ax.set_aspect("equal")
ax.set_title(f"Circular Law (N={N})", fontsize=12, fontweight="bold")
ax.set_facecolor("#0f172a")

plt.tight_layout()
plt.savefig("output.png", dpi=150, bbox_inches="tight", facecolor="#0f172a")
plt.show()
print("\nVisualization saved.")

Click Run to execute the Python code

Code will be executed with Python 3 on the server

Practice Problems

Problem 1:Find all eigenvalues and eigenvectors of $A = \begin{pmatrix} 2 & 1 & 0 \\ 0 & 3 & 1 \\ 0 & 0 & 2 \end{pmatrix}$.

Solution:

1. Since $A$ is upper triangular, the eigenvalues are the diagonal entries: $\lambda_1 = 2$ (algebraic multiplicity 2) and $\lambda_2 = 3$ (multiplicity 1).

2. For $\lambda = 2$: solve $(A - 2I)v = 0$. $A - 2I = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 0 \end{pmatrix}$. Row reduce: $v_2 = 0, v_3 = 0, v_1$ free.

3. Eigenspace for $\lambda = 2$: $E_2 = \text{span}\{(1, 0, 0)^T\}$. Geometric multiplicity = 1 < algebraic multiplicity = 2, so $A$ is defective.

4. For $\lambda = 3$: solve $(A - 3I)v = 0$. $A - 3I = \begin{pmatrix} -1 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & -1 \end{pmatrix}$. Get $v_3 = 0, v_1 = v_2$.

5. Eigenspace for $\lambda = 3$: $E_3 = \text{span}\{(1, 1, 0)^T\}$.

6. Since geometric multiplicity of $\lambda = 2$ is 1, not 2, the matrix $A$ is not diagonalizable. Its Jordan normal form has a $2 \times 2$ Jordan block for $\lambda = 2$.

Problem 2:Diagonalize $B = \begin{pmatrix} 4 & 1 \\ 2 & 3 \end{pmatrix}$ and express $B$ as $PDP^{-1}$.

Solution:

1. Characteristic polynomial: $\lambda^2 - 7\lambda + 10 = (\lambda - 5)(\lambda - 2) = 0$. Eigenvalues: $\lambda_1 = 5, \lambda_2 = 2$.

2. For $\lambda_1 = 5$: $(B - 5I)v = \begin{pmatrix} -1 & 1 \\ 2 & -2 \end{pmatrix}v = 0$, giving $v_1 = (1, 1)^T$.

3. For $\lambda_2 = 2$: $(B - 2I)v = \begin{pmatrix} 2 & 1 \\ 2 & 1 \end{pmatrix}v = 0$, giving $v_2 = (1, -2)^T$.

4. Form $P = \begin{pmatrix} 1 & 1 \\ 1 & -2 \end{pmatrix}$, $D = \begin{pmatrix} 5 & 0 \\ 0 & 2 \end{pmatrix}$.

5. Compute $P^{-1} = \frac{1}{-3}\begin{pmatrix} -2 & -1 \\ -1 & 1 \end{pmatrix} = \begin{pmatrix} 2/3 & 1/3 \\ 1/3 & -1/3 \end{pmatrix}$.

6. Verify: $PDP^{-1} = \begin{pmatrix} 1 & 1 \\ 1 & -2 \end{pmatrix}\begin{pmatrix} 5 & 0 \\ 0 & 2 \end{pmatrix}\begin{pmatrix} 2/3 & 1/3 \\ 1/3 & -1/3 \end{pmatrix} = \begin{pmatrix} 4 & 1 \\ 2 & 3 \end{pmatrix} = B$.

Problem 3:Prove the spectral theorem: every real symmetric matrix has real eigenvalues and orthogonal eigenvectors.

Solution:

1. Real eigenvalues: Let $Av = \lambda v$ with $A = A^T$. Take the complex conjugate: $A\bar{v} = \bar{\lambda}\bar{v}$ (since $A$ is real).

2. Compute $\bar{v}^T(Av) = \bar{v}^T(\lambda v) = \lambda(\bar{v}^T v) = \lambda\|v\|^2$.

3. Also $\bar{v}^T(Av) = (A^T\bar{v})^T v = (A\bar{v})^T v = (\bar{\lambda}\bar{v})^T v = \bar{\lambda}(\bar{v}^T v) = \bar{\lambda}\|v\|^2$.

4. Since $\|v\|^2 \neq 0$: $\lambda = \bar{\lambda}$, so $\lambda \in \mathbb{R}$.

5. Orthogonal eigenvectors: For distinct eigenvalues $\lambda \neq \mu$ with eigenvectors $u, v$: $\lambda(u^T v) = (Au)^T v = u^T A^T v = u^T Av = \mu(u^T v)$.

6. So $(\lambda - \mu)(u^T v) = 0$. Since $\lambda \neq \mu$, we get $u^T v = 0$. For repeated eigenvalues, Gram-Schmidt within each eigenspace produces orthogonal eigenvectors. Therefore $A = Q\Lambda Q^T$ with $Q$ orthogonal. $\square$

Problem 4:Verify the Cayley-Hamilton theorem for $C = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}$.

Solution:

1. The characteristic polynomial is $p(\lambda) = \lambda^2 - \text{tr}(C)\lambda + \det(C) = \lambda^2 - 5\lambda - 2$.

2. Cayley-Hamilton states $p(C) = C^2 - 5C - 2I = 0$.

3. Compute $C^2 = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}\begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} = \begin{pmatrix} 7 & 10 \\ 15 & 22 \end{pmatrix}$.

4. Compute $5C = \begin{pmatrix} 5 & 10 \\ 15 & 20 \end{pmatrix}$ and $2I = \begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix}$.

5. $C^2 - 5C - 2I = \begin{pmatrix} 7-5-2 & 10-10-0 \\ 15-15-0 & 22-20-2 \end{pmatrix} = \begin{pmatrix} 0 & 0 \\ 0 & 0 \end{pmatrix}$. Verified.

6. Consequence: $C^{-1} = \frac{1}{-2}(C - 5I) = \frac{1}{2}\begin{pmatrix} 4 & -2 \\ -3 & 1 \end{pmatrix} = \begin{pmatrix} -2 & 1 \\ 3/2 & -1/2 \end{pmatrix}$.

Problem 5:Compute $B^{10}$ for $B = \begin{pmatrix} 4 & 1 \\ 2 & 3 \end{pmatrix}$ using diagonalization.

Solution:

1. From Problem 2, $B = PDP^{-1}$ with $\lambda_1 = 5, \lambda_2 = 2$.

2. Therefore $B^{10} = PD^{10}P^{-1}$, where $D^{10} = \begin{pmatrix} 5^{10} & 0 \\ 0 & 2^{10} \end{pmatrix} = \begin{pmatrix} 9765625 & 0 \\ 0 & 1024 \end{pmatrix}$.

3. $B^{10} = \begin{pmatrix} 1 & 1 \\ 1 & -2 \end{pmatrix}\begin{pmatrix} 9765625 & 0 \\ 0 & 1024 \end{pmatrix}\begin{pmatrix} 2/3 & 1/3 \\ 1/3 & -1/3 \end{pmatrix}$.

4. Multiply: $PD^{10} = \begin{pmatrix} 9765625 & 1024 \\ 9765625 & -2048 \end{pmatrix}$.

5. $B^{10} = \frac{1}{3}\begin{pmatrix} 2(9765625) + 1024 & 9765625 - 1024 \\ 2(9765625) - 2048 & 9765625 + 2048 \end{pmatrix}$.

6. $B^{10} = \frac{1}{3}\begin{pmatrix} 19532274 & 9764601 \\ 19529202 & 9767673 \end{pmatrix} = \begin{pmatrix} 6510758 & 3254867 \\ 6509734 & 3255891 \end{pmatrix}$. This is vastly more efficient than multiplying $B$ ten times.

← Systems of Equations Diagonalization →