224 lines
10 KiB
Markdown
224 lines
10 KiB
Markdown
# Voice Studio
|
|
|
|
A local, browser-based voice-character training workbench for ACE-Step 1.5 XL.
|
|
Maintain any number of characters, each with separate recordings, annotations,
|
|
dataset versions, jobs, logs, and exports.
|
|
|
|
## Delivery status
|
|
|
|
Source implementation only. At the user's request, the application has **not**
|
|
been launched or tested, dependencies have not been installed, and no models
|
|
have been trained. No services or containers have been started.
|
|
|
|
The default paths reflect the existing local installation inspected during
|
|
development.
|
|
|
|
## Start when ready
|
|
|
|
From this project directory in PowerShell:
|
|
|
|
```powershell
|
|
.\Start-VoiceStudio.ps1
|
|
```
|
|
|
|
Open <http://127.0.0.1:9874>. The launcher does not install packages or open
|
|
another ACE-Step instance. It uses:
|
|
|
|
- ACE-Step installation: `D:\ai\ACE-Step-1.5`
|
|
- Python: `D:\ai\ACE-Step-1.5\.venv\Scripts\python.exe`
|
|
- Character library: `..\Voice Characters`
|
|
- Generated data: `.\workspace`
|
|
|
|
To supply another interpreter or installation:
|
|
|
|
```powershell
|
|
.\Start-VoiceStudio.ps1 -AceStepRoot "D:\Tools\ACE-Step-1.5" `
|
|
-Python "D:\Tools\ACE-Step-1.5\.venv\Scripts\python.exe" -Port 9874
|
|
```
|
|
|
|
The browser server requires Python 3.10+ and the small set of packages in
|
|
`requirements.txt`. It does not install the ACE-Step runtime. If those
|
|
packages are missing, install them deliberately into the chosen interpreter:
|
|
|
|
```powershell
|
|
& "D:\ai\ACE-Step-1.5\.venv\Scripts\python.exe" -m pip install -r .\requirements.txt
|
|
```
|
|
|
|
Review changes to a shared environment before installing. ACE-Step itself
|
|
must have its own CUDA-enabled Torch, training dependencies, and downloaded
|
|
model assets. Training requires an NVIDIA CUDA GPU; memory requirements
|
|
depend on the XL variant, clip duration, and upstream implementation.
|
|
|
|
No Docker, Postman, cloud services, paid APIs, or MCP server is required.
|
|
This is a local web application, not an MCP integration. The browser server
|
|
binds only to `127.0.0.1`; do not expose it through a public proxy.
|
|
|
|
## Multiple voice characters
|
|
|
|
Every immediate subfolder of the character library is a separate character.
|
|
The existing `GothMetalMaleV2` folder is discovered, not hard-coded.
|
|
|
|
Use **New character** to create another folder and independent workspace,
|
|
then **Add recordings** to copy WAV, FLAC, or OGG files into it. Uploads receive
|
|
unique filenames; existing files are never overwritten. Alternatively, add
|
|
a character subfolder and audio files directly to the library before opening
|
|
or refreshing the studio.
|
|
|
|
Display names, generation triggers, default captions, and language are
|
|
editable. Use a distinct trigger per voice. Identity changes apply to future
|
|
datasets and do not silently rewrite old annotations.
|
|
|
|
Environment overrides:
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `ACESTEP_ROOT` | Default ACE-Step installation |
|
|
| `VOICE_STUDIO_SOURCE` | Default recording library |
|
|
| `VOICE_STUDIO_WORKSPACE` | Generated datasets, settings, job history, and exports |
|
|
|
|
Saved settings override the default installation and source paths.
|
|
The output workspace is selected before launch, not moved through the UI.
|
|
|
|
## Dataset preparation and review
|
|
|
|
1. Select or create a character.
|
|
2. Add **isolated vocals** that you own or are authorized to use.
|
|
3. Set the clip duration and silence threshold, then prepare a dataset.
|
|
4. Listen to the resulting clips. Correct captions and enter accurate lyrics,
|
|
syllables, or descriptions of the actual non-lexical vocalisation.
|
|
5. Mark each clip reviewed and save.
|
|
|
|
Preparation creates a new version every time:
|
|
|
|
```text
|
|
workspace\
|
|
<character-id>\
|
|
character.json
|
|
datasets\
|
|
<version>\
|
|
ace_audio\ 48 kHz stereo PCM-16 WAV
|
|
svc_audio\ 40 kHz mono PCM-16 WAV
|
|
dataset.json Native ACE-Step samples/metadata
|
|
segments.json Original paths and segment timestamps
|
|
tensors\ Generated only by explicit preprocessing
|
|
ready.json Written only after complete preprocessing
|
|
jobs\
|
|
<job-id>\
|
|
job.json
|
|
output.log
|
|
recipe.json Training jobs only
|
|
adapter\ Training outputs, if produced by the backend
|
|
```
|
|
|
|
Silence detection uses 20 ms energy windows, brief boundary padding, and a
|
|
minimum retained segment length of 1.5 seconds. Long spans are cut near a
|
|
low-energy boundary where possible; audition cuts because this is not a
|
|
phoneme-aware segmenter. Quiet clips below the configured threshold are
|
|
excluded and reported. Very short isolated sounds may be omitted.
|
|
|
|
Audio is not denoised, separated, pitch-corrected, or transcribed automatically.
|
|
Resampling peaks are limited only when needed to prevent output clipping.
|
|
For SVC, a phase-cancelling stereo downmix falls back to the strongest channel
|
|
and produces a warning. Original recordings are never modified.
|
|
|
|
Unknown lyrics remain blank. Vocal data is explicitly **not** marked
|
|
instrumental. ACE-Step preprocessing is blocked until every clip has vocal
|
|
text, a caption, and a listening-review confirmation. This does not guarantee
|
|
that an annotation is correct; it is a human review gate.
|
|
|
|
## Path A: new songs with ACE-Step XL
|
|
|
|
1. Open **Train a voice**.
|
|
2. Select the exact installed XL variant: XL-SFT, XL-Turbo, or XL-Base.
|
|
3. Confirm permission and suitable vocal data.
|
|
4. Run **Preprocess clips**. The studio invokes upstream's two-pass
|
|
`preprocess_audio_files` in a separate process using the ACE-Step interpreter.
|
|
5. Run **Start training** after preprocessing completes.
|
|
|
|
The integration targets the local ACE-Step source interfaces inspected at
|
|
commit `14c0211`:
|
|
|
|
- `acestep.training_v2.preprocess.preprocess_audio_files`
|
|
- `train.py --plain --yes fixed`
|
|
|
|
Commands use the literal checkpoint directory name plus an explicit
|
|
`--base-model xl_*`. This avoids the inspected CLI path validator's missing
|
|
XL aliases while retaining the correct XL family for training configuration.
|
|
No upstream files are modified.
|
|
|
|
Starting settings: batch size 1, BF16, rank 16, learning rate 0.0001,
|
|
gradient accumulation 4, gradient checkpointing, encoder offload, AdamW,
|
|
and zero DataLoader workers. These are starting settings, not a validated
|
|
24 GB memory guarantee. Default 12-second clips reduce activation memory.
|
|
Close other GPU workloads before training. Lower clip length/rank if needed.
|
|
|
|
Every preprocessing attempt gets a separate tensor directory. Completion
|
|
requires all expected samples to succeed. A dataset hash and XL variant are
|
|
recorded; changes invalidate readiness and require preprocessing again.
|
|
|
|
The console displays actual subprocess output, not estimated loss curves or
|
|
invented progress. Nonzero exits and explicit backend `[FAIL]` reports are
|
|
failures. Training completion additionally requires final `.safetensors`
|
|
weights and their adapter configuration. That check does **not** certify voice
|
|
quality: audition checkpoints with prompts and melodies not used in training.
|
|
|
|
Load the resulting adapter **directory**, retaining `adapter_config.json` and
|
|
its weights together, through the matching ACE-Step XL model's adapter loader.
|
|
Include the trigger recorded in that training job's `recipe.json` in the
|
|
generation caption; renaming a character does not rewrite an old model's
|
|
trigger. In ComfyUI, the
|
|
installed nodes must explicitly support the XL variant and the saved adapter
|
|
format; compatibility is not assumed and weights are not silently converted.
|
|
|
|
Syllable runs can provide timbre and pitch coverage, but do not establish good
|
|
lyric pronunciation or full-song generalisation. A voice-only set can also
|
|
bias outputs toward a cappella performances. Add authorized clean sung phrases
|
|
with varied pitches, vowels, consonants, dynamics, and accurately annotated
|
|
lyrics; evaluate on held-out performances before relying on the character.
|
|
|
|
## Path B: singing voice conversion
|
|
|
|
For preserving an existing singer's melody, timing, and performance while
|
|
changing the voice, use a **separate RVC model**, for example through
|
|
[Applio](https://github.com/IAHispano/Applio).
|
|
|
|
1. Prepare and listen to the character's vocal clips.
|
|
2. Open **Models & exports** and build an SVC dataset export.
|
|
3. Download and extract the ZIP. It contains the mono 40 kHz WAVs and local
|
|
training handoff instructions, not a trained conversion model.
|
|
4. In a separately installed Applio environment, select a unique model name
|
|
and the extracted character folder; use 40 kHz, preprocess, extract
|
|
pitch/features, train, and build its retrieval index following the current
|
|
Applio training guide.
|
|
5. Use its `.pth` and `.index` for conversion of isolated source vocals, then
|
|
mix the converted vocals back with the instrumental.
|
|
|
|
Applio installation, downloads, training, and conversion are **not automated**
|
|
by this studio. That boundary avoids guessing an unselected backend version
|
|
or mixing conflicting Torch dependencies. SVC training generally does not
|
|
need lyrics, so the export is available without ACE-Step lyric annotation.
|
|
It still requires human listening and authorized audio.
|
|
|
|
An ACE-Step LoRA is not an RVC/so-vits-svc model. Renaming an adapter does not
|
|
convert it. ACE-Step audio conditioning/cover workflows may be useful, but
|
|
are not a guaranteed drop-in replacement for performance-preserving SVC.
|
|
|
|
## Job and file behaviour
|
|
|
|
- One dataset/GPU/export job runs at a time. Mutations are blocked during jobs.
|
|
- History, logs, and artifacts persist per character. Interrupted jobs are
|
|
marked failed on the next studio startup, never reported as completed.
|
|
- Stopping the studio requests termination of its active backend process.
|
|
There is no browser cancellation/resume UI. A subsequent training request
|
|
starts a new run; native CLI resume remains an upstream advanced workflow.
|
|
- Preparation and uploads write only new files. Earlier datasets and partial
|
|
failed runs are retained for inspection; no automatic deletion is performed.
|
|
- Model weights and audio remain local. The application contains no telemetry,
|
|
remote fonts, external JavaScript, or audio-upload services.
|
|
|
|
### Upstream references
|
|
|
|
- [ACE-Step 1.5](https://github.com/ace-step/ACE-Step-1.5)
|
|
- [ACE-Step LoRA training tutorial](https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/en/LoRA_Training_Tutorial.md)
|
|
- [Applio](https://github.com/IAHispano/Applio)
|