Astra's favourite songs

GPT-6 Astra can't make sound, but it can paint. We asked it four times to compose an original 30-second piece it loves and paint it as a spectrogram, a picture of sound precise enough to play back. This page turns the paintings into audio, then lets Stable Audio 3 re-perform each one with a little, then a lot of, freedom.

How it was made

Every step below names the exact model, setting and file, so you can redo it. The scripts and prompts are in Download everything. Hardware: an M2 MacBook Pro; Stable Audio 3 ran on Apple's MLX, everything else on the CPU.

  1. Ask Astra for its favourite music

    GPT-6 Astra ran through OpenAI's Codex CLI (0.166), reasoning effort high, four times in parallel, each run in its own empty folder so they couldn't see each other:

    codex exec -m gpt-6-astra -c model_reasoning_effort=high -s workspace-write \
      --skip-git-repo-check -C run1 -o run1/final.md "$(cat prompt_to_astra.md)"

    The prompt asks for an original piece it would love to hear itself, total creative freedom, and a strict picture format. Astra used its built-in image generation tool (no code allowed), saved the image, and replied with a title, a description, a timeline and why it loves the piece.

    The full prompt, word for word
  2. Why the format is pinned down

    A spectrogram is only playable if a program knows exactly what each pixel means. The prompt fixes every axis:

    • Time: 0 to 30.0 s, linear, left to right, so each pixel column is a known moment.
    • Frequency: HTK mel scale from 0 Hz (bottom) to 12 kHz (top), mel = 2595·log10(1 + f/700). Mel gives the bass, where melody lives, more rows than a linear axis would.
    • Loudness: the magma colormap from −80 dB (black) to 0 dB (pale yellow). Magma gets steadily brighter, so a colour maps back to one loudness.
    • Nothing inside the plot but data: ticks, labels and the colourbar sit outside, where the decoder ignores them. The colourbar doubles as a second colour reference.

    Astra asked its tool for 3840 × 1280 px and got 2172 × 724 every time (the tool has no size setting). The plot area came out about 1850 × 500 px: ~16 ms of music per pixel column, and ~500 rows for 0–12 kHz.

  3. Crop the plot area

    decode.py finds the thin light-grey frame Astra drew: rows and columns whose longest run of light, colourless pixels spans most of the image are axis lines; the widest pair of horizontal lines is the plot, the vertical lines refine its sides. Each edge then steps inward past the line and its anti-aliasing, plus one pixel of margin. If no frame is found it falls back to the bounding box of colourful pixels. A narrow framed rectangle right of the plot is taken as the colourbar.

  4. Read colour back as loudness

    A 3×3 median filter removes the image model's grain, then every pixel is matched to its nearest colour (in RGB) on a 1024-step magma ramp, and separately on a ramp sampled from Astra's own colourbar. Whichever ramp fits the plot better wins (it was magma for three images, the painted colourbar for song 4). The match position maps linearly to −80…0 dB, giving a grid of about 500 mel rows × 1850 time columns.

  5. Turn the picture into sound: Griffin-Lim

    A spectrogram stores how loud each frequency is, not the phase of the waves, so the phase has to be invented. specgrid.py converts the grid into a normal linear spectrum: anything below −76 dB counts as silence (the black page itself decodes to about −80), each column is interpolated to the STFT frame times, and each FFT bin takes the power average of the image rows it covers. 0 dB is set to a sine wave of amplitude 0.5.

    invert.py then runs fast Griffin-Lim (momentum 0.99, 400 iterations, 2048-point FFT, hop 256, 44.1 kHz) from random phase. No trained model is involved: it is the most literal reading of the picture, with Griffin-Lim's typical phasey, metallic shimmer. The result is peak-normalised to 0.89.

    In a first round we also tried additive sinusoids, the Vocos vocoder and NVIDIA's BigVGAN-v2. Griffin-Lim stays closest to the picture among the methods that need no hand-built synthesiser, so it is the base for everything below.

  6. Let Stable Audio 3 re-perform it

    cleanup.py feeds the Griffin-Lim audio to Stability AI's Stable Audio 3 (medium model, SAME-L codec) as audio-to-audio, using the MLX command-line tool from the stable-audio-3 repo (optimized/mlx/sa3). The text prompt is Astra's own title plus its "What it sounds like" sentence. One render of 30 s takes ~15 s on the M2.

    sa3 --dit medium --decoder same-l --init-audio raw_gl.wav --init-noise-level 0.2 \
        --seconds 30 --steps 8 --seed 7 --prompt "<title>. <what it sounds like>" \
        --out sa3_raw_gl_s0.2.wav

    Seed 7, 8 sampling steps, classifier-free guidance off (the default 1.0), the same for all four songs at σ = 0.1, 0.2, 0.35 and 0.6. The input is written as 16-bit stereo first (mono copied to both channels). What σ does is explained below.

  7. Measure how close each version stays to the picture

    Each output is measured back onto Astra's exact grid: an STFT (8192-point below 400 Hz for fine bass resolution, 2048-point above, hop 256), power-averaged into the same ~500 mel rows and 1850 columns, in dB relative to its own peak, floored at −80. The match score is the Pearson correlation between that grid and the decoded picture, over pixels where either is louder than −60 dB. 1 means identical, 0 means unrelated, negative means energy where Astra painted silence and silence where it painted sound.

    Match measures faithfulness to the picture, not how good something sounds. The player shows both maps side by side so you can see what it compares.

  8. Prepare for this page

    Every version is loudness-matched to −18 LUFS (two-pass ffmpeg loudnorm, linear mode, true peak −1.5 dB) and encoded as 192 kbps MP3, so switching versions changes the sound, not the volume. The scrolling strips are "dadaspecs", the spectrogram from our Stable Audio 3 studio: 128 mel bands up to 16 kHz, left and right channels folded to one, and tinted by which band leads (red bass, green mids, blue highs). They show more treble and transients than Astra's axes do; the small magma maps under each player use Astra's axes exactly.

    Redo it yourself, end to end
    # Python env: numpy, scipy, torch, pillow, matplotlib (pyproject.toml + uv.lock in the download)
    uv sync
    # 1. Ask Astra, four runs, each in its own empty folder
    for i in 1 2 3 4; do mkdir -p run$i; codex exec -m gpt-6-astra -c model_reasoning_effort=high \
      -s workspace-write --skip-git-repo-check -C run$i -o run$i/final.md "$(cat prompt_to_astra.md)" & done; wait
    # 2. One folder per song: Astra's image + its reply (the reply's **Title:** line names the piece)
    mkdir -p songs/s1-title
    cp run1/spectrogram.png songs/s1-title/astra_spectrogram.png
    cp run1/final.md songs/s1-title/astra_description.md
    # 3. Decode, Griffin-Lim, then SA3 at σ 0.1 / 0.2 / 0.35 / 0.6, plus match scores
    #    (needs Stability's stable-audio-3 repo, MLX runtime installed, on Apple Silicon)
    export SA3=/path/to/stable-audio-3/optimized/mlx/sa3
    uv run python run_all.py songs/s1-title
    #    → songs/s1-title/work/raw_gl.wav, sa3_raw_gl_s0.1.wav … and fidelity.json

    The download also holds Astra's four original PNGs and replies, so you can skip step 1 and decode exactly the images used here.

What Stable Audio 3 and σ do

Stable Audio 3 is a diffusion model: it learned to turn random noise into music, one denoising step at a time, guided by a text prompt. It doesn't work on the waveform directly. Its SAME-L codec first squeezes the audio into a compact sequence of latent vectors (256 numbers for every ~93 ms), and the model works there.

Audio-to-audio (often called SDEdit) starts from existing audio instead of pure noise. The Griffin-Lim render is encoded into latents, then blended with random noise: start = (1 − σ) · audio + σ · noise. The model then denoises from that point back to clean audio, steered by Astra's description. σ is how much of the original you smudge out before SA3 repaints it. At σ = 0 nothing changes; at σ = 1 the input is gone and it is plain text-to-music.

Low σ keeps Astra's structure (timing, pitches, where things are) and only lets SA3 redo the fine texture: Griffin-Lim's shimmer becomes real-sounding struck and plucked notes. Higher σ hands more of the piece to SA3: by 0.35 it keeps the rhythm and register layout but plays its own instruments; by 0.6 it is a different piece.

One subtlety makes SA3 bolder than the numbers suggest. Its "ping-pong" sampler adds fresh noise again after every step, and its noise schedule is warped (a log-SNR shift), so the second step runs at a higher noise level than the σ you asked for. The ladder on the right shows the peak.

Coloured bar: the starting noise σ. White line: the highest noise level the 8-step schedule reaches (step 2). Values from SA3's MLX sampler: build_pingpong_schedule(8, σ, use_logsnr_shift=True).

What we found

Astra follows a technical format precisely. All four images have a clean frame, ticks at the requested mel positions, a magma colourbar and nothing inside the plot. It wrote itself a far more detailed image prompt than ours, with exact frequencies and timings (open "The image prompt Astra wrote for itself" under each song).

Its taste converges. Four independent runs produced four variations on one idea: a low hum, sparse glassy plucks and clicks, gliding tones that meet in one bright harmonic moment around the middle, then dissolve. Two titles are almost the same ("A Common Divisor", "The Almost-Common Divisor"). The pieces are sparse and synthetic: stacked harmonic lines, thin clicks, faint noise clouds.

The pictures play back well with plain phase retrieval. Astra paints perfectly clean lines over digital silence. Griffin-Lim reads them literally; the neural vocoders we tried in the first round, trained on real recordings, smeared them (match 0.24–0.48 for Vocos, 0.26–0.57 for BigVGAN-v2, against 0.51–0.64 for Griffin-Lim).

SA3 rewrites quickly. σ 0.1–0.2 is the faithful range: same events, SA3's timbres. At 0.35 you hear SA3's instruments on Astra's skeleton, at 0.6 a new piece. Song 3 is the exception: SA3 erases most of it even at σ 0.1, as if it does not hear that picture's long gliding chords as music.

Match with Astra's picture, every version

Pearson correlation of each version's re-measured spectrogram with the decoded picture (1 = identical). It tracks faithfulness, not quality: Griffin-Lim scores highest and sounds the most synthetic.

Download everything

Everything needed to redo the experiment or check our numbers.

Download all files (.zip)