any-audio

[Hear the difference](#hear-the-difference) · [Install](#install) · [Sound sources](#sound-sources) · [Speech TTS](#speech-tts) · [中文说明](https://github.com/cloudoyx/any-audio/blob/main/README.zh-CN.md)

---

AnyAudio is an audio production toolkit for AI agents. It turns a sound idea into an actual audio project: write notes and render real instruments, write lines and generate speech, drop in sound effects, arrange the structure, and mix the multitrack result. Video soundtracks are just one use — scenes, sound design, standalone music, audiobooks, podcasts and game audio all come out of the same pipeline.

The agent keeps the creative decisions. AnyAudio supplies the tools and execution needed to turn a sound idea into a finished audio file.

What it solves

Models can already write code that makes sound, but it rarely sounds like an instrument, a recording, or a finished mix. AnyAudio fills that gap:

• Real instrument timbres — rendered from local sample banks instead of hand-written oscillators.
• Notes land directly on the timeline — millisecond accuracy; music, narration and effects share one clock.
• Automatic mix balance — tracks are placed in order with per-track gain, pan and fades; declaring a track as narration makes the plugin duck the other buses under the voice, and the master is peak-normalised so the whole mix does not clip.
• Effects and voices — sound effects either sampled from the installed banks or generated procedurally, plus speech through Edge TTS and MiMo-V2.5-TTS (currently free).

Hear the difference

Both pairs contain identical material: one version made without the plugin, one with it. Listen first, read the numbers after.

> GitHub plays embedded videos muted by default — click the speaker icon in the player to hear the audio.

"Fly Me to the Moon", bossa nova: arranged and adapted by the agent

The same 57-second bossa nova. Version A was produced the way an agent works without AnyAudio, writing NumPy synthesis directly; version B was rendered by AnyAudio from local sample banks. Same arrangement, only the sound source changed.

| | Version A — without AnyAudio | Version B — with AnyAudio |

| --- | --- | --- |

| Peak / average level | -1.5 dBFS / -19.9 dB | -1.5 dBFS / -20.8 dB |

| Crest factor | 18.4 dB | 19.3 dB |

| Spectral centroid | 1,356 Hz | 1,890 Hz |

| Energy above 4 kHz | 6.8% | 11.6% |

Music version A — waveform and spectrogramMusic version B — waveform and spectrogramhttps://github.com/user-attachments/assets/9f496f67-0bd7-496e-96d9-4c2ccb4a28ebhttps://github.com/user-attachments/assets/86ac18e0-f530-452b-9711-7f7471855b80

Version B uses sampled instruments: it sounds more like real players, with more high-frequency detail and more dynamic headroom.

A rainy night in five lines: speech and sound effects

The same 24-second scene, five lines of dialogue, plus rain, footsteps and a door, produced twice. Lines and timing are identical.

| | Version A — without AnyAudio | Version B — with AnyAudio |

| --- | --- | --- |

| Peak / average level | -1.5 dBFS / -27.5 dB | -1.5 dBFS / -26.1 dB |

| Crest factor | 26.0 dB | 24.6 dB |

| Spectral centroid | 5,629 Hz | 4,401 Hz |

| Energy above 4 kHz | 55.3% | 44.0% |

Speech and effects version A — waveform and spectrogramSpeech and effects version B — waveform and spectrogramhttps://github.com/user-attachments/assets/c4ae1a17-d7bc-4db1-b96a-be244fd75358https://github.com/user-attachments/assets/4fca1c9e-b373-4ab7-b18b-afe3e81ad102

Version B's voices match the emotion and pacing of the scene more closely, and its effects are more realistic and more detailed.

Install

Node.js 22.19+ (22.x) or 24+ (bundled with DSH); ffmpeg is only needed for speech and for muxing video.

dsh plugin --profile desktop add dsh-any-audio

If DSH boots a different profile, replace desktop with that profile name. Restart DSH once after installing so it picks up the plugin.

Then open the AnyAudio page in DSH Settings to install a sound source (GeneralUser GS is the default suggestion), switch the speech engine and adjust mixing defaults. No SoundFont is bundled with the package.

From source

For local development, install from a clone instead of the registry:

git clone https://github.com/cloudoyx/any-audio.git
cd any-audio
npm install && npm run font
dsh plugin --profile desktop add ""

The path must be absolute (on Windows, D:/any-audio style), and DSH needs a restart once after adding it. npm run font downloads both suggested banks (about 72 MB) into the shared sound library, which is handy when you want the banks ready before opening Settings.

A clone also gives you the headless renderer, useful for iterating on a score without an agent:

node scripts/anyaudio-cli.mjs render score.json out.wav

Quick start

Describe the work as usual while generating a video; the agent calls the plugin to render audio on its own:

> Make a 30-second ink-wash animation: a seed in a rock crevice grows into a sapling. Add a little Chinese-style music and wind, no narration.

You can adjust musical detail and mixing behaviour as you go:

> Guzheng lead, strings bed, cello on the low end; pull the music down whenever a voice comes in.

For a song, state the brief and call it directly:

> Write me a Japanese RPG battle theme, loopable, 30 to 60 seconds. Drums, electric guitar and strings.

You can even build an audiobook from scratch:

> Turn Chapter 1 of Romance of the Three Kingdoms into an audiobook — deep, rich narration, distinguishable character voices, with simple background music and effects.

Sound sources