For the last two years, the Nonlinear Artificial Intelligence Lab and I have labored to incorporate diversity in machine learning. Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artificial neural networks. In software, we constructed neural networks from neurons that learn their own activation functions (relating inputs to outputs), quickly diversify, and subsequently outperform their homogeneous counterparts on image classification and nonlinear regression tasks. Sub-networks instantiate the neurons, which meta-learn especially efficient sets of nonlinear responses.

Our examples included conventional neural networks classifying digits and forecasting a van der Pol oscillator and physics-informed Hamiltonian neural networks learning Hénon-Heiles stellar orbits. 

As a final real-world example, I video recorded my wall-hanging pendulum clock, ticking beside me as I write this. Engineered to be nearly Hamiltonian, and assembled with the help of a friend, the pendulum’s Graham escapement periodically interrupts the fall of its weight as gravity compensates dissipation. Using software, we tracked the ends of its compound pendulum, and extracted its angles and angular velocities at equally spaced times. We then trained a Hamiltonian neural network to forecast its phase space orbit, as summarized by the figure below. Once again, meta-learning produced especially potent neuronal activation functions that worked best when mixed.

Meta-learning 2 activations for forecasting a real pendulum clock engineered to be almost Hamiltonian. Left: Falling weight (not shown) drives a wall-hanging pendulum clock. Center: State space flow from video data is nearly elliptical. Right: Box plots summarize distribution of neural network mean-square-error validation loss, starting from 50 random initializations of weights and biases, for a fully connected neural networks of sine neurons (blue), type-1 neurons (yellow), type-2 neurons (orange), and a mix of type 1 and type 2 neurons (red).

A particle confined to an impassable box is a paradigmatic and exactly solvable one-dimensional quantum system modeled by an infinite square well potential. Working with Bill Ditto, Elliott Holliday and I recently explored some of its infinitely many generalizations to two dimensions, including particles confined to regions that exhibit integrable, ergodic, or chaotic classical billiard dynamics, using physics-informed neural networks. In particular, we generalized an unsupervised learning algorithm to find the particles’ eigenvalues and eigenfunctions, even in cases where the eigenvalues are degenerate. During training, the neural network adjusts its weights and biases, one of which is the energy eigenvalue, so that its output approximately solves the stationary Schrödinger equation with normalized and mutually orthogonal eigenfunctions.

2D billiard orbits (left) and corresponding quantum eigenfunctions (right).

The motion of one of the simplest dynamical systems, a torqued, damped, nonlinear pendulum, can be infinitely complicated.

Consider a simple pendulum of length l and mass m rigidly connected to an axle of radius r wrapped by a rope that hangs down one side with a mass M climbing up and down it, as in the attached animation.

If the climber’s height

x = x_0  + \frac{a}{\omega^2} \sin \omega t,

varies sinusoidally (relative to the axle), then its acceleration

\ddot x = – a \sin \omega t,

also varies sinusoidally, so the total force on the climber

M\ddot x = \sum_\text{down} f = Mg-T

implies upper rope tension

T = Mg + ma \sin \omega t,

where 0 < a < g. If the axle and rope have negligible inertia, then the total torque on the axle

m l^2 \ddot \theta = \sum_\text{CCW}\tau = – mgl \sin\theta + rT – \gamma \dot\theta,

where \gamma is the axle viscosity. The full motion equation

m l^2 \ddot \theta = – mgl \sin\theta + rMg + rma \sin \omega t – \gamma \dot\theta

reduces to

\ddot \theta = – \sin\theta + 0.7155 + 0.4 \sin 0.25 t – 0.75 \dot\theta

for parameters that describe the animation’s chaotic motion.

Green mass sinusoidally climbs up and down brown rope torquing blue pendulum into chaotic motion.

Quantum field theory predicts that the temperature of empty space should depend on the observer’s motion, increasing proportionally with acceleration. Here I attempt an accessible introduction to this striking effect, related to Hawking radiation and discovered independently by Fulling, Davies, and Unruh, assuming only sophomore-level physics (including hyperbolic functions) with some assistance from Mathematica.

Hyperbolic Motion

Constant acceleration in Newtonian mechanics is parabolic, while constant acceleration in Einsteinian mechanics is hyperbolic and asymptotic to light speed c = 1 in natural units. For 1+1-dimensional Minkowski spacetime, the difference in squared space and time displacements is the square of proper time displacement,

d\tau^2 = dt^2 – dx^2.

For constant proper acceleration, this has the solution

dt = d\tau \cosh a\tau,vdx = d\tau \sinh a\tau,

with velocity

v = \frac{dx}{dt} = \tanh a\tau \le 1,

where for small times v \sim a \tau \sim a t. If t = 0 and a x = 1 at \tau = 0, then integration gives

a t = \sinh a \tau,\\a x = \cosh a\tau.

Hyperbolic identities then imply

a^2 x^2 – a^2 t^2 = 1

and

a x + a t = e^{a\tau}.

Quantum Vacuum

Due to Heisenberg indeterminacy, electromagnetic fluctuations fill the vacuum. Consider a single such sinusoidal wave of angular frequency \omega_0 = k_0 in natural units. If you move at constant velocity, you observe the wave doppler-shifted to a different frequency. But if you move at constant acceleration, you observe the wave doppler-shifted to a range of frequencies corresponding to your range of velocities. For an accelerated observer at time \tau,

\phi[x,t] = \exp\left[i(k_0 x + \omega_0 t)\right] = \exp\left[i\omega_0(x + t)\right] \\= \exp\left[{i \frac{\omega_0}{a} e^{a \tau}} \right] = \phi[\tau].

Expand this waveform as a sum of harmonics

\phi[\tau] = \int_{-\infty}^{\infty}\,\frac{d\omega}{2\pi} \Phi[\omega] \exp[-i \omega \tau]

where the Fourier components

\Phi[\omega] = \int_{-\infty}^{\infty}d\tau\, \phi[\tau] \exp[+i \omega \tau].

To regularize this divergent integral, subtract a tiny imaginary part i \epsilon from the angular frequency \omega to incorporate a decaying exponential factor e^{-\epsilon \tau} in the integrand, and zero it after integrating. Find

\Phi[\omega] = \lim_{\epsilon\rightarrow 0} \int_{-\infty}^{\infty}d\tau\, \exp\left[ i \frac{\omega_0}{a} e^{a\tau} \right] \exp[+i (\omega – i \epsilon) \tau]\\~ \\= \lim_{\epsilon\rightarrow 0}\, \exp\left[ i\frac{\pi}{2} \left(\frac{\epsilon + i\omega}{a}\right) \right]\left(\frac{a}{\omega_0}\right)^{(\epsilon + i \omega)/a}\frac{1}{a} \Gamma\left[\frac{\epsilon + i \omega}{a} \right] \\= \exp\left[-\frac{\pi}{2} \frac{\omega}{a}\right]\left(\frac{a}{\omega_0}\right)^{i \omega / a}\frac{1}{a} \Gamma\left[i \frac{\omega}{a} \right],

where \Gamma[n+1] = n! analytically continues the factorial function to the complex plane. The spectrum is the absolute square of the Fourier transform,

S[\omega] = \left|\Phi[\omega]\right|^2 = \frac{\pi}{a\omega} \left(\coth\left[ \pi \frac{\omega}{a} \right]-1 \right) \\= \frac{2\pi}{\omega a} \frac{1}{e^{2\pi \omega/a} – 1} \propto \frac{1}{e^{\hbar \omega/k T}-1},

where the Planck factor suggests Bose-Einstein statistics and a thermal photon bath of temperature kT = a \hbar / 2\pi. In SI units,

T = \frac{a \hbar}{2\pi k c} \sim 40~\text{zK}~\left(\frac{a}{g_E} \right),

where g_E is Earth’s surface gravity, and a zeptokelvin is very cool.

Hawking-Unruh Temperature

Just prior to the 1970s work of Fulling, Davies, and Unruh, Stephen Hawking famously predicted that despite their reputations black holes should radiate with an effective temperature

T = \frac{\kappa \hbar}{2\pi k c},

where \kappa is the black hole’s surface gravity (observed at infinity). The Unruh and Hawking results may be linked by the equivalence principle, which equates acceleration and gravity, and by event horizons. In General Relativity, the black hole horizon is a boundary that causally disconnects the interior from the exterior. Similarly, when you accelerate, a Rindler horizon appears a distance c^2/a \sim 1~\text{ly} \left(g_E / a \right) behind you, causally disconnecting you from a region of spacetime whose photons you can outrun (so long as your acceleration continues).

The forces between two electric charges in arbitrary motion are complicated by velocity, acceleration, and time-delay effects. The forces need not even lie along the line joining the two charges!

Suppose a source charge q^\prime is at position \vec r^{\,\prime} with velocity \vec v^{\,\prime} and acceleration \vec{a}^{\,\prime}, and a test charge q is at position \vec r with velocity \vec v and acceleration \vec{a}. Let their separation \vec{\mathcal{R}}= \vec r – \vec r^{\,\prime}. Since electromagnetic “news” travels at light speed \color{blue}c\color{black} = 1 (in natural units), define the earlier retarded time s < t  implicitly by t – s = \mathcal{R}[s] / \color{blue}c\color{black} = \mathcal{R}_{\color{red}s}.

Liénard-Wiechert electric and magnetic potentials at test charge q due to source charge q^\prime are

\varphi_{\color{red}t} =\frac{q^\prime_e}{4\pi \mathcal{R}_{\color{red}s} }=\frac{q^\prime}{4\pi \mathcal{R}_{\color{red}s} } \left( \frac{1}{1 – \hat{\mathcal{R} }_{\color{red}s} \cdot \vec v^{\,\prime}_{\color{red}s}} \right), \hspace{1cm} \vec A_{\color{red}t}= \varphi^\prime_{\color{red}s} \vec v^{\,\prime}_{\color{red}s},

where the source charge q^\prime is “smeared” to an effective charge q^\prime_e by its motion, and the subscripts indicate evaluation times. If the source charge velocity \vec v^{\,\prime}_{\color{red}s} = 0, then the potentials simplify to the electrostatics limit. Differentiate the potentials to find Maxwell’s electric and magnetic fields

\vec{\mathcal{E}} = – \vec \nabla \varphi – \partial_t \vec A \hspace{1cm} \vec{\mathcal{B}} = \vec \nabla \times \vec A,

and the Lorentz force law

\vec F = q \left( \vec{\mathcal E} + \vec v \times \vec{\mathcal B}\right)

generalizes Coulomb’s law to

\vec F_{\color{red}t} = q \frac{ q^\prime } { 4 \pi \mathcal{R}^2_{\color{red}s}} \left( \frac{ 1 } {1 – \hat{\mathcal{R}}_{\color{red}s} \cdot \vec{v}^{\,\prime}_{\color{red}s} }\right)^3 \bigg( \frac{}{} (1-v^{\prime 2}_{\color{red}s})(\hat{\mathcal{R}}_{\color{red}s}-\vec{v}^{\,\prime}_{\color{red}s}) + \vec{\mathcal{R}}_{\color{red}s} \times \left((\hat{\mathcal{R}}_{\color{red}s}-\vec{v}^{\,\prime}_{\color{red}s}) \times \vec{a}^{\,\prime}_{\color{red}s} \right) \hspace{3.4cm} + \,\vec v_{\color{red}t} \times \left( \hat{\mathcal{R}}_{\color{red}s} \times \left( (1-v^{\prime 2}_{\color{red}s})(\hat{\mathcal{R}}_{\color{red}s}-\vec{v}^{\,\prime}_{\color{red}s}) + \vec{\mathcal{R}}_{\color{red}s} \times \left((\hat{\mathcal{R}}_{\color{red}s}-\vec{v}^{\,\prime}_{\color{red}s}) \times \vec{a}^{\,\prime}_{\color{red}s} \right) \right) \right) \, \bigg).

If the test charge velocity \vec v = \vec 0, then the magnetic terms vanish; if the source charge acceleration \vec{a}^{\,\prime} = \vec 0, then the radiation terms vanish; if, in addition, the source charge velocity \vec v\,^\prime = \vec 0, then the generalized Coulomb’s law reduces to the familiar electrostatic limit. Finally, Newton’s second law with Einstein’s momentum

\vec F = \frac{d\vec p}{dt} = \frac{d}{dt} \frac{ m \vec v }{\sqrt{1 – v^2}} = \frac{ m \vec a}{\sqrt{1 – v^2}} + \frac{ m \vec v}{(1 – v^2)^{3/2}} \vec v \cdot \vec a

generates the motion equations using 3-vectors and lab time (instead of 4-vectors and proper time), succinctly summarizing all of electromagnetism.

Using Mathematica, I eliminated present time t from the motion equations with the substitution t = s + \mathcal{R}_{\color{red}s} and numerically integrated them with respect to the retarded time s for various initial conditions, as in the figure below. Positive and negative charge pairs tugged by non-zero net forces spiral into one another as they radiate away energy and momentum, graphically demonstrating the instability of classical atoms.

Mathematica simulations of interacting charged particle including magnetic, radiation, and time-delay effects. Blue dots mark present positions, and red dots mark retarded positions. Arrows represent force pairs, which are typically not “equal but opposite”.