For the last two years, the Nonlinear Artificial Intelligence Lab and I have labored to incorporate diversity in machine learning. Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artificial neural networks. In software, we constructed neural networks from neurons that learn their own activation functions (relating inputs to outputs), quickly diversify, and subsequently outperform their homogeneous counterparts on image classification and nonlinear regression tasks. Sub-networks instantiate the neurons, which meta-learn especially efficient sets of nonlinear responses.

Our examples included conventional neural networks classifying digits and forecasting a van der Pol oscillator and physics-informed Hamiltonian neural networks learning Hénon-Heiles stellar orbits. 

As a final real-world example, I video recorded my wall-hanging pendulum clock, ticking beside me as I write this. Engineered to be nearly Hamiltonian, and assembled with the help of a friend, the pendulum’s Graham escapement periodically interrupts the fall of its weight as gravity compensates dissipation. Using software, we tracked the ends of its compound pendulum, and extracted its angles and angular velocities at equally spaced times. We then trained a Hamiltonian neural network to forecast its phase space orbit, as summarized by the figure below. Once again, meta-learning produced especially potent neuronal activation functions that worked best when mixed.

Meta-learning 2 activations for forecasting a real pendulum clock engineered to be almost Hamiltonian. Left: Falling weight (not shown) drives a wall-hanging pendulum clock. Center: State space flow from video data is nearly elliptical. Right: Box plots summarize distribution of neural network mean-square-error validation loss, starting from 50 random initializations of weights and biases, for a fully connected neural networks of sine neurons (blue), type-1 neurons (yellow), type-2 neurons (orange), and a mix of type 1 and type 2 neurons (red).

A particle confined to an impassable box is a paradigmatic and exactly solvable one-dimensional quantum system modeled by an infinite square well potential. Working with Bill Ditto, Elliott Holliday and I recently explored some of its infinitely many generalizations to two dimensions, including particles confined to regions that exhibit integrable, ergodic, or chaotic classical billiard dynamics, using physics-informed neural networks. In particular, we generalized an unsupervised learning algorithm to find the particles’ eigenvalues and eigenfunctions, even in cases where the eigenvalues are degenerate. During training, the neural network adjusts its weights and biases, one of which is the energy eigenvalue, so that its output approximately solves the stationary Schrödinger equation with normalized and mutually orthogonal eigenfunctions.

2D billiard orbits (left) and corresponding quantum eigenfunctions (right).

The motion of one of the simplest dynamical systems, a torqued, damped, nonlinear pendulum, can be infinitely complicated.

Consider a simple pendulum of length l and mass m at a counter-clockwise angle \theta from downward at time t rigidly connected to an axle of radius r wrapped by a rope that hangs down one side with a mass M climbing up and down it, as in the attached animation.

If the climber’s position varies sinusoidally

s = s_0 + A \sin \omega t

relative to the hanging rope, then its downward position

y = s + r \theta

and its acceleration

\ddot y = \ddot s + r \ddot \theta = -\omega^2 A \sin \omega t + r \ddot \theta.

The total downward force on the climber

M\ddot y = \sum_\text{down} f = Mg-T

implies upper rope tension

T = Mg-M \ddot y = M g + M\omega^2 A \sin \omega t-M r \ddot \theta.

If the axle and rope have negligible inertia, then the total torque about the axle

m l^2 \ddot \theta = \sum_\text{CCW}\tau = -mgl \sin\theta + rT-\gamma \dot\theta,

where \gamma is the axle viscosity. The full motion equation

m l^2 \ddot \theta = – mgl \sin\theta + rMg + rM \omega^2 A \sin \omega t-Mr^2\ddot\theta-\gamma \dot\theta

rearranges to

(ml^2+Mr^2)\ddot \theta = -mgl\sin\theta + rMg + rM\omega^2 A \sin \omega t-\gamma \dot\theta.

For parameters that describe the animation’s chaotic motion,

\ddot \theta = – \sin\theta + 0.7155 + 0.4 \sin 0.25 t-0.75 \dot\theta.

Green mass sinusoidally climbs up and down brown rope torquing blue pendulum into chaotic motion.

Quantum field theory predicts that the temperature of empty space should depend on the observer’s motion, increasing proportionally with acceleration. Here I attempt an accessible introduction to this striking effect, related to Hawking radiation and discovered independently by Fulling, Davies, and Unruh, assuming only sophomore-level physics (including hyperbolic functions) with some assistance from Mathematica.

Hyperbolic Motion

Constant acceleration in Newtonian mechanics is parabolic, while constant acceleration in Einsteinian mechanics is hyperbolic and asymptotic to light speed \color{red}c\color{black} = 1 in natural units. For 1+1-dimensional Minkowski spacetime, the difference in squared space and time displacements is the square of proper time displacement,

d\tau^2 = dt^2-dx^2.

For constant proper acceleration, this has the solution

\begin{aligned}dt &= d\tau \cosh a\tau,\\ dx &= d\tau \sinh a\tau,\end{aligned}

with velocity

v = \frac{dx}{dt} = \tanh a\tau \le 1,

where for small times v \sim a \tau \sim a t. If t = 0 and a x = 1 at \tau = 0, then integration gives

a t = \sinh a \tau,\\a x = \cosh a\tau.

Hyperbolic identities then imply

a^2 x^2-a^2 t^2 = 1

and

a x + a t = e^{a\tau}.

Quantum Vacuum

Due to Heisenberg indeterminacy, electromagnetic fluctuations fill the vacuum. Consider a single such sinusoidal wave of angular frequency \omega_0 = k_0\color{red}c in natural units. If you move at constant velocity, you observe the wave doppler-shifted to a different frequency. But if you move at constant acceleration, you observe the wave doppler-shifted to a range of frequencies corresponding to your range of velocities. For an accelerated observer at time \tau,

\phi[x,t] = \exp\left[i(k_0 x + \omega_0 t)\right] = \exp\left[i\omega_0(x + t)\right] \\= \exp\left[{i \frac{\omega_0}{a} e^{a \tau}} \right] = \phi[\tau].

Expand this waveform as a sum of harmonics

\phi[\tau] = \int_{-\infty}^{\infty}\,\frac{d\omega}{2\pi} \Phi[\omega] \exp[-i \omega \tau]

where the Fourier components

\Phi[\omega] = \int_{-\infty}^{\infty}d\tau\, \phi[\tau] \exp[+i \omega \tau].

To regularize this divergent integral, subtract a tiny imaginary part i \epsilon from the angular frequency \omega to incorporate a decaying exponential factor e^{-\epsilon \tau} in the integrand, and zero it after integrating. Find

\Phi[\omega] = \lim_{\epsilon\rightarrow 0} \int_{-\infty}^{\infty}d\tau\, \exp\left[ i \frac{\omega_0}{a} e^{a\tau} \right] \exp[+i (\omega – i \epsilon) \tau]\\~ \\= \lim_{\epsilon\rightarrow 0}\, \exp\left[ i\frac{\pi}{2} \left(\frac{\epsilon + i\omega}{a}\right) \right]\left(\frac{a}{\omega_0}\right)^{(\epsilon + i \omega)/a}\frac{1}{a} \Gamma\left[\frac{\epsilon + i \omega}{a} \right] \\= \exp\left[-\frac{\pi}{2} \frac{\omega}{a}\right]\left(\frac{a}{\omega_0}\right)^{i \omega / a}\frac{1}{a} \Gamma\left[i \frac{\omega}{a} \right],

where \Gamma[n+1] = n! analytically continues the factorial function to the complex plane. The spectrum is the absolute square of the Fourier transform,

S[\omega] = \left|\Phi[\omega]\right|^2 = \frac{\pi}{a\omega} \left(\coth\left[ \pi \frac{\omega}{a} \right]-1 \right) \\= \frac{2\pi}{\omega a} \frac{1}{e^{2\pi \omega/a}-1} \propto \frac{1}{e^{\hbar \omega/k T}-1},

where the Planck factor suggests Bose-Einstein statistics and a thermal photon bath of temperature kT = a \hbar / 2\pi. In SI units,

T = \frac{a \hbar}{2\pi k c} \sim 40~\text{zK}~\left(\frac{a}{g_E} \right),

where g_E is Earth’s surface gravity, and a zeptokelvin is very cool.

Hawking-Unruh Temperature

Just prior to the 1970s work of Fulling, Davies, and Unruh, Stephen Hawking famously predicted that despite their reputations black holes should radiate with an effective temperature

T = \frac{\kappa \hbar}{2\pi k c},

where \kappa is the black hole’s surface gravity (observed at infinity). The Unruh and Hawking results may be linked by the equivalence principle, which equates acceleration and gravity, and by event horizons. In General Relativity, the black hole horizon is a boundary that causally disconnects the interior from the exterior. Similarly, when you accelerate, a Rindler horizon appears a distance c^2/a \sim 1~\text{ly} \left(g_E / a \right) behind you, causally disconnecting you from a region of spacetime whose photons you can outrun (so long as your acceleration continues).