# Mathematical and physical music visualization

Literature review and proposed project direction — 8 October 2026, with two implementation references added on 9 October. The original review contained 61 references. This preserves the broad scoping document; the completed implementation correction and numerical comparison are in the [technical report](research-report.md).

## Project recommendation

Build a browser laboratory for seeing how sound excites geometry. Make a Chladni sand scene the flagship, supported by a small collection of wave, oscillator, and pattern formation models. Keep the black and white aesthetic, but let visitors inspect the equation, boundary conditions, input signal, and chosen audio mapping behind each scene.

For a CS portfolio, the strongest contribution would combine computational modeling, real time signal processing, GPU programming, numerical validation, and a carefully designed evaluation. The proposed research question is:

> How do physically specified dynamics, direct musical control, and hybrid mappings affect visual coherence, perceived synchrony, and computational cost in a browser based Chladni visualizer?

This is a research proposal. No comparative experiment or listener study has been conducted. Implementation checks for the current demonstrator are recorded separately in the release review. Existing cymatics tools and music visualization research establish substantial prior work; novelty would have to come from the particular method, comparison, or evidence we produce. See [CymaSense](https://doi.org/10.1145/3064857.3079159), [Graf et al.](https://arxiv.org/abs/2106.10134), and the [existing project catalogue](annotated-bibliography.md#existing-projects).

The package includes a [mathematical model specification](mathematical-models.md), a [research and publication plan](study-and-publication-plan.md), and an [annotated bibliography](annotated-bibliography.md). The bibliography contains 63 references: 39 research publications or manuscripts, 21 academic or official technical resources, and 3 existing projects. It records access limitations and publication status. This is a substantial scoping review, rather than a claim to have exhaustively searched all literature.

## What the attached photographs show

The photographs are consistent with Chladni figures: grains form patterns associated with nodal regions of a vibrating surface. They show a square plate with a central fastener or exciter. The labels 345, 3835, and 3975 Hz describe that apparatus; they do not identify universal shapes that every plate produces at those frequencies.

A frequency selects a response through the object's geometry, dimensions, elastic properties, supports, damping, and excitation position. A central mounting can change both the admissible motion and the coupling to individual modes. Grain size, collisions, and surrounding air can also affect what becomes visible. Plate vibration literature and newer direct and inverse particle experiments make these dependencies important. [Leissa](https://ntrs.nasa.gov/citations/19700009156), [Bourrud et al.](https://doi.org/10.1016/j.powtec.2025.120867).

We cannot infer all those parameters from three photographs. The first simulator should therefore have its own explicitly specified geometry. A simply supported rectangular plate is a useful analytical benchmark. A free square plate with a central mounting would be a later model requiring more careful numerical treatment.

### Why the plate and the sand are separate problems

A vibration model produces a displacement field and its modes. A grain model determines how particles move through that field. Drawing a nodal contour solves neither the full time dependent vibration problem nor the grain transport problem.

A stretched membrane is governed by tension and a spatial Laplacian. A thin bending plate uses flexural rigidity and a biharmonic operator. A membrane can produce beautiful Chladni-like diagrams, but it should be identified as a membrane. The educational finite element approach in [Müller](https://arxiv.org/abs/1308.5523) and the plate formulation in [Brugnoli et al.](https://arxiv.org/abs/1809.11136) provide complementary foundations.

For grains, a recent experiment supports a spatial diffusion mechanism in a specified membrane regime. It gives us a useful alternative to simply pulling particles toward a zero contour. It also makes a particle versus density solver comparison plausible. Applying that model to the photographed metal plate would be an extrapolation requiring validation. [Abramian et al.](https://doi.org/10.1103/PhysRevResearch.7.L032001).

### Why a song will not necessarily produce one clean sand pattern

A laboratory demonstration often holds a near resonant tone long enough for a mode and particle pattern to develop. A song contains overlapping frequencies, changing attacks, and limited time at any particular excitation. Its response is a superposition with memory.

For the proposed damped mode equation, an amplitude decay time is approximately 1/(ζω). Grain redistribution adds another timescale. Increasing damping can improve response speed while weakening or broadening resonance. Increasing transport speed can make patterns settle faster while changing the grain model. These are parameters with visible consequences, rather than free aesthetic improvements.

This tension is the project's most interesting starting point: a system can preserve a clear physical interpretation and still respond weakly to a rapidly changing song. We should measure that behavior and give users a deliberate choice of presentation.

## Three kinds of scenes

Every scene should state what its output represents.

| Interpretation | What the visualization does | Example |
| --- | --- | --- |
| Signal measurement | Draws a property of the recorded signal | Waveform, spectrogram, stereo phase portrait |
| Specified simulation | Uses audio as an input to stated dynamical equations | Forced plate modes, wave propagation |
| Designed mathematical mapping | Maps musical features into model parameters or graphical choices | Bass controls fluid injection; onsets change Gray–Scott feed rate |

A simulation can also contain an aesthetic modification. That modification should be visible in its explanation: for example, a beat triggered gain boost, an accelerated grain timescale, or a rescaled frequency range. This makes the demo more educational and gives the research comparison well defined conditions.

“Physics inspired” is a useful description when the rules imitate a phenomenon without sufficient evidence to reproduce it. “Validated physical simulation” should be reserved for a stated validation domain, including its errors and assumptions.

## Fifteen visualization families worth considering

The priorities below are our engineering judgment. They are not measured performance or aesthetic rankings.

| Family | What visitors would see | Model and honest interpretation | Priority |
| --- | --- | --- | --- |
| 1. Waveform and spectral field | A fine waveform, frequency ridges, and a moving grayscale spectrogram | Direct signal representations. Useful as an inspection layer and a baseline; a spectrum magnitude loses phase information. [STFT](https://www.audiolabs-erlangen.de/resources/MIR/FMP/C2/C2_STFT-Basic.html) | Foundation |
| 2. Stereo phase portrait and Lissajous curves | A luminous trace folding into ellipses and knots | Plot the left and right channels against each other. A generated pair of sinusoids gives a classical Lissajous figure; arbitrary music need not form a neat closed curve. Mono input plotted against itself collapses to a diagonal. | First collection |
| 3. Harmonograph | Slowly evolving loops with fading trails | Superposed damped oscillators. Can use audio to excite each oscillator, or use extracted pitches to select their frequencies. The latter is an explicit musical mapping. | Small extension |
| 4. Chladni sand | White grains gathering into nodal curves on a black plate | Plate eigenmodes plus a stated grain transport approximation. The central research scene. [Plate mechanics](https://arxiv.org/abs/1809.11136), [stochastic models](https://doi.org/10.1017/S0013091515000139) | Flagship |
| 5. Drum and membrane modes | Circular rings, diameters, and patterns on unusual domains | Tension dominated wave equation with declared supports. Geometry can be explored independently of sand. [Numerical Chladni figures](https://arxiv.org/abs/1308.5523) | First collection |
| 6. Wave propagation and interference | Ripples colliding, reflecting, and revealing standing waves | A wave equation driven at selected locations. Relative phase and boundaries matter. Two sources can reinforce or cancel; energy cannot be reconstructed from FFT magnitudes alone. | First collection |
| 7. Fluid ink and vortices | White dye filaments winding through dark flow | Incompressible flow with audio chosen as forcing and dye injection. Stable numerical graphics methods are practical; musical forcing is our design. [Stam](https://www.dgp.toronto.edu/people/stam/reality/Research/pdf/ns.pdf) | Second collection |
| 8. Faraday surface patterns | Polygonal, cellular, or standing surface waves | Parametric excitation of a fluid interface. A simple additive ripple shader does not reproduce the instability. Full modeling is substantially more involved. [Périnet et al.](https://arxiv.org/abs/0901.0464), [Liu and Wang](https://www.nature.com/articles/s41567-023-02294-y) | Advanced |
| 9. Reaction–diffusion | Labyrinths, spots, divisions, and growing textures | Gray–Scott chemistry supplies a mathematical pattern engine. Music can modulate parameters, but that coupling does not follow from acoustic physics. [Pearson](https://arxiv.org/abs/patt-sol/9304003) | Second collection |
| 10. Coupled oscillator ring | Dots drifting out of phase and then locking together | Kuramoto phase dynamics. Synchronization has an inspectable order parameter. Music controlling coupling or forcing is a proposed mapping. [Strogatz](https://doi.org/10.1016/S0167-2789(00)00094-4) | Distinctive extension |
| 11. Swarmalators | Moving rings, rotating clusters, and spatial phase waves | Coupled position and phase interactions. This offers a less familiar mathematical scene with strong explanatory value. [O’Keeffe et al.](https://www.nature.com/articles/s41467-017-01190-3) | Research extension |
| 12. Chaotic trajectories | A delicate moving attractor with persistent trails | A nonlinear dynamical system such as Lorenz. Audio parameter modulation creates mathematical art; its path is not a literal reconstruction of sound propagation. [Lorenz](https://doi.org/10.1175/1520-0469(1963)020%3C0130:DNF%3E2.0.CO;2) | Optional |
| 13. Fourier epicycles and harmonic ribbons | Rotating vectors assembling a waveform or a closed path | Fourier synthesis with complex coefficients and phase. A geometric drawing reconstructed by Fourier series is a different input from music; their relationship should be demonstrated explicitly. [STFT](https://www.audiolabs-erlangen.de/resources/MIR/FMP/C2/C2_STFT-Basic.html) | Educational extension |
| 14. Geometry and mesh resonance | Nodal sets changing with a shape or mesh | Solve a Laplacian or an appropriate elasticity operator. Mesh modes are not automatically plate modes. Spectral geometry connects shape with eigenvalues without guaranteeing a unique inverse. [Kac](https://www.math.ucdavis.edu/~hunter/m207b/kac.pdf), [Gordon et al.](https://arxiv.org/abs/math/9207215) | Advanced |
| 15. Flocking | Particles separating, aligning, and regrouping to sound | Local behavioral rules with musical control. Visually useful, but its interpretation is generative behavior rather than acoustic mechanics. [Reynolds](https://www.red3d.com/cwr/papers/1987/boids.html) | Optional |

The most coherent first collection would contain Chladni sand, a stereo phase portrait, and wave interference, with a compact waveform/spectrum inspector available everywhere. These scenes expose three different relationships between sound and geometry without requiring several complex solvers at once.

Fluid ink and reaction–diffusion would add contrasting textures after the plate pipeline works. Kuramoto or swarmalators could then become the unusual mathematical extension. Faraday waves are a good longer term investigation, especially if a collaborator can help with fluid mechanics or a physical experiment.

## What the literature suggests for the visual design

Audio driven visualization research supports treating feature extraction, mapping, and rendering as separate components. It also provides precedents for evaluating usability and creative experience, rather than judging a system only by screenshots. [Graf et al.](https://arxiv.org/abs/2106.10134).

A live mixed reality study compared music linked and randomized visuals and found advantages on some perceptual outcomes. Its setting and mappings differ from our browser plate, so it motivates a controlled synchrony comparison rather than proving that our preferred condition will win. [Erdmann et al.](https://www.frontiersin.org/journals/virtual-reality/articles/10.3389/frvir.2025.1552321/full).

Black and white is compatible with a meaningful research direction. Shape and motion can carry structure without a universal sound to color table. Timbre to shape correspondence has been studied using grayscale as well as colored stimuli. [Adeli et al.](https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2014.00352/full).

The proposed visual language is white grain density on black, restrained trails, thin geometry overlays, and a readable time axis. Use intensity for density or energy, stroke width for a separate scalar, and line style or small signed labels when positive and negative displacement need distinguishing. Avoid making one brightness channel represent several different quantities simultaneously.

More visual activity need not mean a better experience. Research on dynamic audiovisual complexity motivates recording motion and image complexity alongside liking, with care about how far abstract stimulus results generalize. [Clemente et al.](https://discovery.ucl.ac.uk/id/eprint/10198080/). Our scene should support quiet passages and silence, not fill every moment with the same level of motion.

## Audio features and the mappings they can support

The analysis layer should expose a small set of interpretable features before introducing learned embeddings or emotion classification.

| Feature | Useful control | Main limitation |
| --- | --- | --- |
| Waveform samples | Physical forcing, oscilloscope, stereo trace | A renderer at display rate cannot show every audio oscillation faithfully |
| RMS or band power | Excitation gain, dye quantity, grain activation | Loudness perception and physical force are not identical to digital RMS |
| Complex spectrum | Resonant response and coherent reconstruction | Windowing creates a time and frequency tradeoff |
| Spectral centroid | A proposed brightness or spatial detail mapping | It is a weighted average, not a complete timbre or emotion description |
| Positive spectral change | Transient markers and brief impulses | An onset is not necessarily a beat |
| Beat and tempo estimates | Repeated pacing or phase cues | Rhythm inference can fail on weak or ambiguous pulses |
| Chroma | Optional pitch class structure | Octaves are combined and dense mixes can be ambiguous |
| Harmonic/percussive decomposition | Separate sustained and transient controls | Approximate spectrogram separation, not isolated instruments |

The [FMP material](https://www.audiolabs-erlangen.de/resources/MIR/FMP/C2/C2_STFT-Basic.html) provides an accessible signal processing foundation. Its [spectral novelty](https://www.audiolabs-erlangen.de/resources/MIR/FMP/C6/C6S1_NoveltySpectral.html) and [chroma](https://www.audiolabs-erlangen.de/resources/MIR/FMP/C3/C3S1_SpecLogFreq-Chromagram.html) explanations support later features. For harmonic/percussive preprocessing, read the actual median filtering method in [FitzGerald](https://dafx.de/paper-archive/2010/DAFx10/DerryFitzGerald_DAFx10_P15.pdf).

Two operating modes should be distinguished. A live microphone pipeline has to be causal. An uploaded track can be analyzed in advance, allowing lookahead and deliberate playback scheduling. Preanalysis is useful for a demo, but its results cannot be reported as live causal performance.

The physical plate branch should use the waveform or phase preserving resonant filters. A bank of FFT band energies selecting attractive patterns is a valid designed mapping, with a different interpretation. MDN's [AnalyserNode](https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode) is convenient for initial measurements; custom complex analysis or filtering may require an [AudioWorklet](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet).

## A feasible browser architecture

The first version can operate entirely on the visitor's device: local audio selection, playback, analysis, simulation, and rendering. Include generated tones and a short licensed example so the laboratory is immediately usable.

~~~mermaid
flowchart LR
    I["Local audio / generated tone"] --> A["Playback and audio clock"]
    A --> F["Waveform and feature analysis"]
    F --> M["Declared model or mapping"]
    M --> S["Simulation state"]
    S --> R["Monochrome renderer"]
    F --> X["Signal inspector"]
    M --> X
~~~

Recommended implementation path:

1. Use TypeScript and Web Audio for the application and signal graph. Start with a readable Canvas reference renderer and an analytically solvable plate.
2. Separate audio time, simulation time, and display time. Timestamp updates against the audio clock; interpolate the state for rendering.
3. Precompute expensive geometry eigenmodes offline and ship a compact mode bank. Integrate a reduced set of mode amplitudes at appropriate time resolution.
4. Use WebGL2 for particle updates and grid fields once the reference is correct. Check floating point texture capabilities and quality limits on actual devices.
5. Treat WebGPU compute as an optional backend, selected by capability detection. Current documentation still identifies availability limitations. [WebGL2](https://developer.mozilla.org/en-US/docs/Web/API/WebGL2RenderingContext), [WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API).
6. Add microphone input and recording later. Permissions, secure contexts, and recording formats affect these flows. [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia), [MediaRecorder](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder).

There is no benchmark yet to justify a particle count, grid resolution, or guarantee of 60 frames per second. Make these adaptive settings and benchmark them. Long tracks also need a memory policy: whole file decoding, streaming playback with incremental analysis, or a documented duration limit.

### Why YouTube links need a separate product decision

The YouTube IFrame API documents playback controls and events, but no raw PCM sample interface for this analysis pipeline. An embedded player is not an audio source that our page can freely inspect. The Web Audio standard also requires silence for a MediaElementAudioSourceNode when the media is CORS cross origin under its specified security conditions. [YouTube IFrame API](https://developers.google.com/youtube/iframe_api_reference), [Web Audio](https://www.w3.org/TR/webaudio/).

YouTube's developer policies additionally constrain downloading and separating audiovisual components in API clients. A general link extraction backend would introduce an entirely different service and policy problem. [Developer policies](https://developers.google.com/youtube/terms/developer-policies).

For the first release, local files and generated tones are the dependable inputs. A permitted direct audio URL could be supported when its server allows the necessary access. A YouTube link can remain an investigated future feature; an embedded video beside a preset animation should not be described as analysis of that video's audio.

### Free hosting

A static client is compatible with GitHub Pages from a public repository, within its published limits, or Cloudflare Pages. Audio processing and graphics then run on visitors' hardware. Hosting documentation does not establish our future runtime performance. [GitHub Pages](https://docs.github.com/en/pages/getting-started-with-github-pages/what-is-github-pages), [limits](https://docs.github.com/en/pages/getting-started-with-github-pages/github-pages-limits), [Cloudflare limits](https://developers.cloudflare.com/pages/platform/limits/).

Large sample collections, uploads stored on a server, account systems, and extraction services would change that simple deployment model. They are unnecessary for the initial research demonstrator.

## What would make this a convincing portfolio project

The project should have two connected outputs.

The website is an accessible laboratory: choose audio or a tone, choose a scene, explore a few meaningful controls, and open a short explanation. The normal interface should stay simple. Technical inspection belongs in an optional panel showing frequency, response, assumptions, and model state.

The research artifact explains one focused contribution with reproducible evidence. For the plate scene, a useful paper could compare physical forcing, direct feature control, and a hybrid; quantify their response and numerical behavior; and evaluate what viewers perceive. The study should use the same scene and renderer so that differences in visual style do not dominate the comparison.

A second possible CS contribution is a comparison between stochastic particles and a continuum grain density solver. That can stand on numerical error, computational cost, and reproducibility even before a human study. It is a more contained route if participant recruitment is unavailable.

Release code, model definitions, benchmark inputs, configurations, seeds, and validation results together. Publish the first completed report with the demo and a versioned archive; consider a scholarly venue only after the evidence supports a clear claim. A DOI identifies an artifact; it does not itself establish peer review. The [study and publication plan](study-and-publication-plan.md) gives concrete routes and thresholds.

## Decisions to carry into prototyping

The recommended starting model is an analytical simply supported plate with a small mode bank. Its supports should be shown in the diagram. It provides a reliable reference before attempting the central mounting seen in the photographs.

The recommended research scope is one plate scene and a declared comparison of three audio couplings. Keep particle versus density transport as either a separate numerical study or a carefully bounded secondary question.

The recommended first product scope is local audio, generated tones, Chladni sand, a stereo trace, wave interference, and a shared signal inspector. Add fluid or reaction–diffusion only after the core pipeline and explanations work. The literature supports exploring more than four methods; the first release can still be small enough to validate.