Part IV: Advanced Topics | Chapter 2

Introduction to Functional Analysis

Linear algebra in infinite dimensions: Banach and Hilbert spaces

Historical Context

Functional analysis emerged from Hilbert's work on integral equations, Banach's thesis on normed spaces (1920), and Riesz's representation theorem. Stefan Banach systematically developed the theory of complete normed spaces in his 1932 monograph, establishing the Hahn-Banach, open mapping, and uniform boundedness theorems. These "big three" theorems, together with the Riesz representation theorem, form the pillars of functional analysis and underpin modern PDE theory, quantum mechanics, and optimization.

4.1 Normed Spaces and Banach Spaces

Definition: Banach Space

A Banach space is a complete normed vector space: every Cauchy sequence converges. Key examples: $\ell^p$ (sequences with finite $p$-norm), $L^p(\Omega)$ (functions with finite $\int|f|^p$), $C[a,b]$ (continuous functions with sup-norm).

The $\ell^p$ norms on $\mathbb{R}^n$: $\|x\|_p = \left(\sum_i |x_i|^p\right)^{1/p}$ for$1 \leq p < \infty$, and $\|x\|_\infty = \max_i |x_i|$. All norms on finite-dimensional spaces are equivalent, but this fails dramatically in infinite dimensions.

4.2 Hilbert Spaces

Definition: Hilbert Space

A Hilbert space is a complete inner product space. The inner product$\langle \cdot, \cdot \rangle$ induces the norm $\|x\| = \sqrt{\langle x, x \rangle}$. The key example is $L^2(\Omega)$ with $\langle f, g \rangle = \int f\bar{g}$.

The Riesz representation theorem states that every continuous linear functional on a Hilbert space $H$ has the form $\phi(x) = \langle x, y \rangle$ for a unique$y \in H$. This identifies $H$ with its dual $H^*$, a property special to Hilbert spaces.

An orthonormal basis $\{e_n\}$ satisfies Parseval's identity:$\|x\|^2 = \sum_n |\langle x, e_n \rangle|^2$. The Fourier basis $\{e^{inx}/\sqrt{2\pi}\}$is the canonical orthonormal basis for $L^2[0, 2\pi]$.

4.3 Bounded Linear Operators

An operator $T: X \to Y$ between normed spaces is bounded if$\|T\| = \sup_{\|x\|=1} \|Tx\| < \infty$. The space $\mathcal{B}(X, Y)$ of bounded operators is itself a Banach space. A bounded operator is compact if it maps bounded sets to precompact sets—compact operators are the "infinite-dimensional analogues of matrices".

4.4 Fundamental Theorems

Hahn-Banach: Every bounded linear functional on a subspace extends to the whole space with the same norm
Open Mapping (Banach): A surjective bounded linear map between Banach spaces is open
Closed Graph: A linear map between Banach spaces is bounded iff its graph is closed
Uniform Boundedness (Banach-Steinhaus): A pointwise bounded family of operators is uniformly bounded

4.5 Spectral Theory in Infinite Dimensions

For compact self-adjoint operators on a Hilbert space, the spectral theorem gives$Tx = \sum_n \lambda_n \langle x, e_n \rangle e_n$ where $\lambda_n \to 0$. The eigenvalues form a countable set accumulating only at 0. This is the infinite-dimensional analogue of matrix diagonalization and underpins the Fredholm theory of integral equations.

For general bounded operators, the spectrum $\sigma(T) = \{\lambda : T - \lambda I \text{ not invertible}\}$can include continuous spectrum and residual spectrum beyond the point spectrum (eigenvalues).

Computational Laboratory

This simulation compares ℓᵖ norms, demonstrates Fourier approximation in L², verifies eigenvalue decay for compact operators, and visualizes unit balls in different norm topologies.

Functional Analysis: Norms, Hilbert Spaces, and Compact Operators

Python

functional_analysis.py130 lines

import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

# ============================================================
# Introduction to Functional Analysis
# ============================================================

# --- 1. ℓ^p norms ---
print("=" * 60)
print("ℓ^p NORMS")
print("=" * 60)

x = np.array([3, -4, 1, -2, 5], dtype=float)
print(f"x = {x}")
for p in [1, 2, 3, np.inf]:
    norm = np.linalg.norm(x, ord=p)
    print(f"||x||_{p} = {norm:.4f}")
print(f"||x||_0 (count nonzero) = {np.count_nonzero(x)}")

# --- 2. Hilbert space: Fourier approximation ---
print("\n" + "=" * 60)
print("HILBERT SPACE: FOURIER APPROXIMATION")
print("=" * 60)

t = np.linspace(0, 2*np.pi, 1000)
f = np.where(t < np.pi, 1.0, -1.0)  # square wave

def fourier_approx(t, f, n_terms):
    approx = np.zeros_like(t)
    for k in range(1, n_terms+1, 2):
        ck = 2 * np.trapezoid(f * np.sin(k * t), t) / (2 * np.pi)
        approx += ck * np.sin(k * t)
    return approx

for n in [1, 3, 5, 11, 51]:
    approx = fourier_approx(t, f, n)
    error = np.sqrt(np.trapezoid((f - approx)**2, t) / (2*np.pi))
    print(f"n={n:2d} terms: L² error = {error:.6f}")

# --- 3. Compact operators and eigenvalue decay ---
print("\n" + "=" * 60)
print("COMPACT OPERATOR EIGENVALUE DECAY")
print("=" * 60)

N = 200
# Integral operator: K(s,t) = min(s,t) (Green's function)
s = np.linspace(0, 1, N)
ds = s[1] - s[0]
K = np.minimum(s[:, None], s[None, :]) * ds

eigenvalues_K = np.sort(np.abs(np.linalg.eigvalsh(K)))[::-1]
print(f"Kernel matrix: {N}×{N}")
print(f"Top 10 eigenvalues: {np.round(eigenvalues_K[:10], 6)}")
print(f"Theoretical: λ_k = 1/((k-0.5)²π²)")
theoretical = [1/((k-0.5)**2 * np.pi**2) for k in range(1, 11)]
print(f"Theoretical:  {[round(t, 6) for t in theoretical]}")

# --- 4. Operator norms ---
print("\n" + "=" * 60)
print("OPERATOR NORMS")
print("=" * 60)

A = np.random.randn(5, 5) / np.sqrt(5)
print(f"||A||_2 (spectral norm) = {np.linalg.norm(A, ord=2):.6f}")
print(f"||A||_F (Frobenius) = {np.linalg.norm(A, 'fro'):.6f}")
print(f"||A||_1 (max col sum) = {np.linalg.norm(A, ord=1):.6f}")
print(f"||A||_∞ (max row sum) = {np.linalg.norm(A, ord=np.inf):.6f}")

# Verify: ||A||_2 = σ_max
svals = np.linalg.svd(A, compute_uv=False)
print(f"σ_max = {svals[0]:.6f} = ||A||_2? {np.isclose(svals[0], np.linalg.norm(A, ord=2))}")

# --- 5. Convergence in different topologies ---
print("\n" + "=" * 60)
print("CONVERGENCE IN DIFFERENT NORMS")
print("=" * 60)

n_vals = [10, 50, 100, 500, 1000]
for n in n_vals:
    x_n = np.zeros(n)
    x_n[:int(np.sqrt(n))] = 1 / np.sqrt(np.sqrt(n))
    l1 = np.linalg.norm(x_n, 1)
    l2 = np.linalg.norm(x_n, 2)
    linf = np.linalg.norm(x_n, np.inf)
    print(f"n={n:4d}: ||x_n||_1={l1:.4f}, ||x_n||_2={l2:.4f}, ||x_n||_∞={linf:.4f}")

# --- 6. Visualization ---
fig, axes = plt.subplots(1, 3, figsize=(16, 5))

# Plot 1: Unit balls in different norms
ax = axes[0]
theta = np.linspace(0, 2*np.pi, 1000)
for p, color, label in [(1, "#3b82f6", "ℓ¹"), (2, "#0ea5e9", "ℓ²"), (5, "#06b6d4", "ℓ⁵"), (100, "#38bdf8", "ℓ∞")]:
    r = (np.abs(np.cos(theta))**p + np.abs(np.sin(theta))**p)**(-1/p)
    ax.plot(r * np.cos(theta), r * np.sin(theta), color=color, linewidth=2, label=label)
ax.set_aspect("equal")
ax.set_title("Unit Balls in ℓᵖ", fontsize=12, fontweight="bold")
ax.legend(fontsize=9)
ax.grid(True, alpha=0.3)
ax.set_facecolor("#0f172a")

# Plot 2: Fourier approximation
ax = axes[1]
ax.plot(t, f, color="#38bdf8", linewidth=1, alpha=0.5, label="f(t)")
for n, color in [(3, "#3b82f6"), (11, "#0ea5e9"), (51, "#06b6d4")]:
    ax.plot(t, fourier_approx(t, f, n), color=color, linewidth=1.5, label=f"n={n}")
ax.set_title("Fourier Approximation", fontsize=12, fontweight="bold")
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3)
ax.set_facecolor("#0f172a")

# Plot 3: Eigenvalue decay
ax = axes[2]
ks = np.arange(1, 51)
ax.semilogy(ks, eigenvalues_K[:50], "o", color="#3b82f6", markersize=4, label="Computed")
ax.semilogy(ks, [1/((k-0.5)**2*np.pi**2) for k in ks], "--", color="#0ea5e9", label="Theory: 1/((k-½)²π²)")
ax.set_xlabel("k", color="gray")
ax.set_ylabel("λ_k", color="gray")
ax.set_title("Compact Operator Eigenvalues", fontsize=12, fontweight="bold")
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3)
ax.set_facecolor("#0f172a")

plt.tight_layout()
plt.savefig("output.png", dpi=150, bbox_inches="tight", facecolor="#0f172a")
plt.show()
print("\nVisualization saved.")

Click Run to execute the Python code

Code will be executed with Python 3 on the server

← Tensors & Multilinear Algebra Numerical Linear Algebra →