Part II: Spectral Theory | Chapter 3

Inner Product Spaces

Geometry in abstract vector spaces through inner products and orthogonality

Historical Context

The generalization of the dot product to abstract inner products was driven by David Hilbert's work on integral equations (1904-1910). Hilbert recognized that function spaces could be endowed with an inner product $\langle f, g \rangle = \int f(x)\overline{g(x)}\,dx$, creating what we now call Hilbert spaces. This insight unified geometry and analysis and became the mathematical foundation of quantum mechanics, where states are vectors in a Hilbert space.

Erhard Schmidt developed the orthogonalization process (later called Gram-Schmidt) in 1907, building on earlier work by Jørgen Pedersen Gram. The Cauchy-Schwarz inequality, proved independently by Cauchy (1821), Bunyakovsky (1859), and Schwarz (1885), became the cornerstone of inner product space theory, establishing the connection between algebraic inner products and geometric notions of angle and distance.

2.1 Inner Products and Norms

Definition: Inner Product

An inner product on a vector space $V$ over $\mathbb{F}$ ($= \mathbb{R}$ or $\mathbb{C}$) is a function $\langle \cdot, \cdot \rangle: V \times V \to \mathbb{F}$ satisfying:

Conjugate symmetry: $\langle u, v \rangle = \overline{\langle v, u \rangle}$
Linearity: $\langle \alpha u + \beta v, w \rangle = \alpha\langle u, w \rangle + \beta\langle v, w \rangle$
Positive definiteness: $\langle v, v \rangle \geq 0$ with equality iff $v = 0$

Every inner product induces a norm $\|v\| = \sqrt{\langle v, v \rangle}$ and a metric $d(u, v) = \|u - v\|$. The fundamental inequality is:

Theorem: Cauchy-Schwarz Inequality

For all $u, v \in V$: $|\langle u, v \rangle| \leq \|u\| \cdot \|v\|$, with equality if and only if $u$ and $v$ are linearly dependent.

This allows us to define the angle between vectors:$\cos\theta = \frac{\langle u, v \rangle}{\|u\|\|v\|}$, generalizing Euclidean geometry to infinite-dimensional function spaces.

2.2 Orthogonality

Vectors $u$ and $v$ are orthogonal (written $u \perp v$) if$\langle u, v \rangle = 0$. The orthogonal complement of a subspace $W$ is$W^\perp = \{v \in V : \langle v, w \rangle = 0 \text{ for all } w \in W\}$.

Theorem: Orthogonal Decomposition

If $W$ is a finite-dimensional subspace of inner product space $V$, then every$v \in V$ has a unique decomposition $v = w + w^\perp$ where $w \in W$ and$w^\perp \in W^\perp$. The component $w = \text{proj}_W(v)$ is the best approximation to $v$ in $W$: it minimizes $\|v - w\|$.

If $\{e_1, \ldots, e_k\}$ is an orthonormal basis for $W$, the projection formula becomes$\text{proj}_W(v) = \sum_{i=1}^k \langle v, e_i \rangle e_i$. This is the foundation of Fourier analysis, least squares approximation, and signal processing.

2.3 The Gram-Schmidt Process

Given a linearly independent set $\{v_1, \ldots, v_k\}$, the Gram-Schmidt processproduces an orthonormal set $\{e_1, \ldots, e_k\}$ spanning the same subspace:

$$u_i = v_i - \sum_{j=1}^{i-1} \langle v_i, e_j \rangle e_j, \qquad e_i = \frac{u_i}{\|u_i\|}$$

The process is equivalent to computing the QR decomposition: if the columns of$A$ are $v_1, \ldots, v_k$, then $A = QR$ where $Q$ has orthonormal columns$e_1, \ldots, e_k$ and $R$ is upper triangular with $r_{ij} = \langle v_j, e_i \rangle$.

In practice, the classical Gram-Schmidt can suffer from numerical instability. The modified Gram-Schmidt algorithm recomputes projections after each subtraction, and Householder reflections provide even better numerical stability for QR factorization.

2.4 Orthogonal and Unitary Matrices

Definition

A real matrix $Q$ is orthogonal if $Q^TQ = QQ^T = I$. A complex matrix $U$ is unitary if $U^*U = UU^* = I$. Equivalently, the columns form an orthonormal basis.

Key properties of orthogonal/unitary matrices:

They preserve inner products: $\langle Qu, Qv \rangle = \langle u, v \rangle$
They preserve norms: $\|Qv\| = \|v\|$ (isometries)
They have $|\det(Q)| = 1$ and eigenvalues on the unit circle
They form a group: $O(n)$ (orthogonal) or $U(n)$ (unitary)

2.5 Adjoint Operators

Definition: Adjoint

The adjoint of a linear operator $T$ is the unique operator $T^*$ satisfying$\langle Tv, w \rangle = \langle v, T^*w \rangle$ for all $v, w$. For matrices,$A^* = \bar{A}^T$ (conjugate transpose).

Important classes of operators defined via the adjoint:

Self-adjoint ($T = T^*$): real eigenvalues, orthogonal eigenvectors
Normal ($TT^* = T^*T$): unitarily diagonalizable
Positive definite ($T = T^*$ and $\langle Tv, v \rangle > 0$ for $v \neq 0$)
Isometry ($T^*T = I$): preserves norms

The relationship between an operator and its adjoint is the key to the spectral theorem: normal operators admit a complete orthonormal eigenbasis, while self-adjoint operators additionally guarantee real eigenvalues—the mathematical foundation of quantum mechanical observables.

Computational Laboratory

This simulation demonstrates inner products and norms, the Gram-Schmidt process, orthogonal projections, QR decomposition, and properties of self-adjoint and normal operators.

Inner Product Spaces: Orthogonality and Gram-Schmidt

Python

inner_product_spaces.py157 lines

import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

# ============================================================
# Inner Product Spaces: Orthogonality and Gram-Schmidt
# ============================================================

# --- 1. Inner products and norms ---
print("=" * 60)
print("INNER PRODUCTS AND NORMS")
print("=" * 60)

u = np.array([1.0, 2.0, 3.0])
v = np.array([4.0, -1.0, 2.0])

dot_uv = np.dot(u, v)
norm_u = np.linalg.norm(u)
norm_v = np.linalg.norm(v)
cos_angle = dot_uv / (norm_u * norm_v)

print(f"u = {u}, v = {v}")
print(f"<u, v> = {dot_uv}")
print(f"||u|| = {norm_u:.4f}, ||v|| = {norm_v:.4f}")
print(f"cos(angle) = {cos_angle:.4f}")
print(f"angle = {np.arccos(cos_angle) * 180 / np.pi:.2f}°")

# Cauchy-Schwarz verification
print(f"\nCauchy-Schwarz: |<u,v>| = {abs(dot_uv):.4f} <= ||u|| ||v|| = {norm_u * norm_v:.4f}: {abs(dot_uv) <= norm_u * norm_v + 1e-10}")

# --- 2. Gram-Schmidt process ---
print("\n" + "=" * 60)
print("GRAM-SCHMIDT ORTHONORMALIZATION")
print("=" * 60)

vectors = np.array([[1, 1, 0], [1, 0, 1], [0, 1, 1]], dtype=float)
print(f"Input vectors:\n{vectors}")

def gram_schmidt(V):
    n, m = V.shape
    Q = np.zeros_like(V)
    for i in range(n):
        q = V[i].copy()
        for j in range(i):
            q -= np.dot(V[i], Q[j]) * Q[j]
        Q[i] = q / np.linalg.norm(q)
    return Q

Q = gram_schmidt(vectors)
print(f"\nOrthonormal basis:\n{np.round(Q, 6)}")
print(f"\nVerify orthonormality (Q Q^T):\n{np.round(Q @ Q.T, 10)}")

# --- 3. Orthogonal projection ---
print("\n" + "=" * 60)
print("ORTHOGONAL PROJECTION")
print("=" * 60)

b = np.array([1.0, 2.0, 3.0])
A = np.array([[1, 0], [0, 1], [0, 0]], dtype=float)  # xy-plane
proj = A @ np.linalg.lstsq(A, b, rcond=None)[0]
residual = b - proj
print(f"Vector b = {b}")
print(f"Projection onto xy-plane: {proj}")
print(f"Residual: {residual}")
print(f"Residual ⊥ projection: {np.dot(proj, residual):.10f}")

# --- 4. QR decomposition ---
print("\n" + "=" * 60)
print("QR DECOMPOSITION")
print("=" * 60)

M = np.array([[1, 1, 0], [1, 0, 1], [0, 1, 1]], dtype=float)
Q_qr, R_qr = np.linalg.qr(M)
print(f"Matrix M:\n{M}")
print(f"\nQ (orthogonal):\n{np.round(Q_qr, 6)}")
print(f"\nR (upper triangular):\n{np.round(R_qr, 6)}")
print(f"\nQ^T Q = I? {np.allclose(Q_qr.T @ Q_qr, np.eye(3))}")
print(f"Q R = M? {np.allclose(Q_qr @ R_qr, M)}")

# --- 5. Adjoint operator properties ---
print("\n" + "=" * 60)
print("ADJOINT AND SELF-ADJOINT OPERATORS")
print("=" * 60)

A_sym = np.array([[3, 1, 0], [1, 2, 1], [0, 1, 4]], dtype=float)
print(f"Symmetric A:\n{A_sym}")
print(f"A = A^T? {np.allclose(A_sym, A_sym.T)}")

x = np.random.randn(3)
y = np.random.randn(3)
print(f"<Ax, y> = {np.dot(A_sym @ x, y):.6f}")
print(f"<x, Ay> = {np.dot(x, A_sym @ y):.6f}")
print(f"Self-adjoint verified: {np.allclose(np.dot(A_sym @ x, y), np.dot(x, A_sym @ y))}")

# Normal matrix (complex)
N = np.array([[1, -1], [1, 1]], dtype=float)
print(f"\nN:\n{N}")
print(f"N^T N:\n{N.T @ N}")
print(f"N N^T:\n{N @ N.T}")
print(f"Normal (N^TN = NN^T)? {np.allclose(N.T @ N, N @ N.T)}")

# --- 6. Visualization ---
fig, axes = plt.subplots(1, 3, figsize=(16, 5))

# Plot 1: Gram-Schmidt in 2D
ax = axes[0]
v1 = np.array([2, 1])
v2 = np.array([1, 2])
e1 = v1 / np.linalg.norm(v1)
e2_proj = v2 - np.dot(v2, e1) * e1
e2 = e2_proj / np.linalg.norm(e2_proj)

ax.arrow(0, 0, v1[0], v1[1], head_width=0.08, fc="#3b82f6", ec="#3b82f6", lw=2, label="v₁")
ax.arrow(0, 0, v2[0], v2[1], head_width=0.08, fc="#0ea5e9", ec="#0ea5e9", lw=2, label="v₂")
ax.arrow(0, 0, e1[0]*1.5, e1[1]*1.5, head_width=0.06, fc="#06b6d4", ec="#06b6d4", lw=2, label="e₁")
ax.arrow(0, 0, e2[0]*1.5, e2[1]*1.5, head_width=0.06, fc="#38bdf8", ec="#38bdf8", lw=2, label="e₂")
ax.set_xlim(-0.5, 2.5)
ax.set_ylim(-0.5, 2.5)
ax.set_aspect("equal")
ax.legend(fontsize=8)
ax.set_title("Gram-Schmidt Process", fontsize=12, fontweight="bold")
ax.grid(True, alpha=0.3)
ax.set_facecolor("#0f172a")

# Plot 2: Projection
ax = axes[1]
from mpl_toolkits.mplot3d import Axes3D
b_3d = np.array([1, 2, 3])
proj_3d = np.array([1, 2, 0])
ax_3d = fig.add_subplot(132, projection='3d')
ax_3d.quiver(0, 0, 0, b_3d[0], b_3d[1], b_3d[2], color="#3b82f6", arrow_length_ratio=0.1, label="b")
ax_3d.quiver(0, 0, 0, proj_3d[0], proj_3d[1], proj_3d[2], color="#0ea5e9", arrow_length_ratio=0.1, label="proj")
ax_3d.quiver(proj_3d[0], proj_3d[1], proj_3d[2], 0, 0, 3, color="#06b6d4", arrow_length_ratio=0.1, label="residual", linestyle="--")
ax_3d.set_xlabel("x")
ax_3d.set_ylabel("y")
ax_3d.set_zlabel("z")
ax_3d.set_title("Orthogonal Projection", fontsize=10, fontweight="bold")
ax_3d.legend(fontsize=7)
axes[1].set_visible(False)

# Plot 3: Orthogonality check
ax = axes[2]
n_dim = 50
random_vecs = np.random.randn(n_dim, n_dim)
Q_rand, _ = np.linalg.qr(random_vecs)
gram_matrix = Q_rand.T @ Q_rand
im = ax.imshow(np.abs(gram_matrix - np.eye(n_dim)), cmap="Blues", vmin=0, vmax=1e-14)
ax.set_title("||Q^TQ - I|| entries", fontsize=12, fontweight="bold")
plt.colorbar(im, ax=ax, fraction=0.046)
ax.set_facecolor("#0f172a")

plt.tight_layout()
plt.savefig("output.png", dpi=150, bbox_inches="tight", facecolor="#0f172a")
plt.show()
print("\nVisualization saved.")

Click Run to execute the Python code

Code will be executed with Python 3 on the server

← Diagonalization Spectral Theorem →