Voice-Model-Training/AceStepXL-Training/README.md

226 lines
10 KiB
Markdown

# Voice Studio
A local, browser-based voice-character training workbench for ACE-Step 1.5 XL.
Maintain any number of characters, each with separate recordings, annotations,
dataset versions, jobs, logs, and exports.
## Delivery status
Source implementation only. At the user's request, the application has **not**
been launched or tested, dependencies have not been installed, and no models
have been trained. No services or containers have been started.
The default paths reflect the existing local installation inspected during
development. Header inspection found six stereo, 48 kHz PCM-16 WAV recordings
in the initial character folder, totaling approximately 17 minutes. The audio
has not been listened to or assessed for vocal quality or training permission.
## Start when ready
From this project directory in PowerShell:
```powershell
.\Start-VoiceStudio.ps1
```
Open <http://127.0.0.1:9874>. The launcher does not install packages or open
another ACE-Step instance. It uses:
- ACE-Step installation: `D:\ai\ACE-Step-1.5`
- Python: `D:\ai\ACE-Step-1.5\.venv\Scripts\python.exe`
- Character library: `..\Voice Characters`
- Generated data: `.\workspace`
To supply another interpreter or installation:
```powershell
.\Start-VoiceStudio.ps1 -AceStepRoot "D:\Tools\ACE-Step-1.5" `
-Python "D:\Tools\ACE-Step-1.5\.venv\Scripts\python.exe" -Port 9874
```
The browser server requires Python 3.10+ and the small set of packages in
`requirements.txt`. It does not install the ACE-Step runtime. If those
packages are missing, install them deliberately into the chosen interpreter:
```powershell
& "D:\ai\ACE-Step-1.5\.venv\Scripts\python.exe" -m pip install -r .\requirements.txt
```
Review changes to a shared environment before installing. ACE-Step itself
must have its own CUDA-enabled Torch, training dependencies, and downloaded
model assets. Training requires an NVIDIA CUDA GPU; memory requirements
depend on the XL variant, clip duration, and upstream implementation.
No Docker, Postman, cloud services, paid APIs, or MCP server is required.
This is a local web application, not an MCP integration. The browser server
binds only to `127.0.0.1`; do not expose it through a public proxy.
## Multiple voice characters
Every immediate subfolder of the character library is a separate character.
The existing `GothMetalMaleV2` folder is discovered, not hard-coded.
Use **New character** to create another folder and independent workspace,
then **Add recordings** to copy WAV, FLAC, or OGG files into it. Uploads receive
unique filenames; existing files are never overwritten. Alternatively, add
a character subfolder and audio files directly to the library before opening
or refreshing the studio.
Display names, generation triggers, default captions, and language are
editable. Use a distinct trigger per voice. Identity changes apply to future
datasets and do not silently rewrite old annotations.
Environment overrides:
| Variable | Meaning |
|---|---|
| `ACESTEP_ROOT` | Default ACE-Step installation |
| `VOICE_STUDIO_SOURCE` | Default recording library |
| `VOICE_STUDIO_WORKSPACE` | Generated datasets, settings, job history, and exports |
Saved settings override the default installation and source paths.
The output workspace is selected before launch, not moved through the UI.
## Dataset preparation and review
1. Select or create a character.
2. Add **isolated vocals** that you own or are authorized to use.
3. Set the clip duration and silence threshold, then prepare a dataset.
4. Listen to the resulting clips. Correct captions and enter accurate lyrics,
syllables, or descriptions of the actual non-lexical vocalisation.
5. Mark each clip reviewed and save.
Preparation creates a new version every time:
```text
workspace\
<character-id>\
character.json
datasets\
<version>\
ace_audio\ 48 kHz stereo PCM-16 WAV
svc_audio\ 40 kHz mono PCM-16 WAV
dataset.json Native ACE-Step samples/metadata
segments.json Original paths and segment timestamps
tensors\ Generated only by explicit preprocessing
ready.json Written only after complete preprocessing
jobs\
<job-id>\
job.json
output.log
recipe.json Training jobs only
adapter\ Training outputs, if produced by the backend
```
Silence detection uses 20 ms energy windows, brief boundary padding, and a
minimum retained segment length of 1.5 seconds. Long spans are cut near a
low-energy boundary where possible; audition cuts because this is not a
phoneme-aware segmenter. Quiet clips below the configured threshold are
excluded and reported. Very short isolated sounds may be omitted.
Audio is not denoised, separated, pitch-corrected, or transcribed automatically.
Resampling peaks are limited only when needed to prevent output clipping.
For SVC, a phase-cancelling stereo downmix falls back to the strongest channel
and produces a warning. Original recordings are never modified.
Unknown lyrics remain blank. Vocal data is explicitly **not** marked
instrumental. ACE-Step preprocessing is blocked until every clip has vocal
text, a caption, and a listening-review confirmation. This does not guarantee
that an annotation is correct; it is a human review gate.
## Path A: new songs with ACE-Step XL
1. Open **Train a voice**.
2. Select the exact installed XL variant: XL-SFT, XL-Turbo, or XL-Base.
3. Confirm permission and suitable vocal data.
4. Run **Preprocess clips**. The studio invokes upstream's two-pass
`preprocess_audio_files` in a separate process using the ACE-Step interpreter.
5. Run **Start training** after preprocessing completes.
The integration targets the local ACE-Step source interfaces inspected at
commit `14c0211`:
- `acestep.training_v2.preprocess.preprocess_audio_files`
- `train.py --plain --yes fixed`
Commands use the literal checkpoint directory name plus an explicit
`--base-model xl_*`. This avoids the inspected CLI path validator's missing
XL aliases while retaining the correct XL family for training configuration.
No upstream files are modified.
Starting settings: batch size 1, BF16, rank 16, learning rate 0.0001,
gradient accumulation 4, gradient checkpointing, encoder offload, AdamW,
and zero DataLoader workers. These are starting settings, not a validated
24 GB memory guarantee. Default 12-second clips reduce activation memory.
Close other GPU workloads before training. Lower clip length/rank if needed.
Every preprocessing attempt gets a separate tensor directory. Completion
requires all expected samples to succeed. A dataset hash and XL variant are
recorded; changes invalidate readiness and require preprocessing again.
The console displays actual subprocess output, not estimated loss curves or
invented progress. Nonzero exits and explicit backend `[FAIL]` reports are
failures. Training completion additionally requires final `.safetensors`
weights and their adapter configuration. That check does **not** certify voice
quality: audition checkpoints with prompts and melodies not used in training.
Load the resulting adapter **directory**, retaining `adapter_config.json` and
its weights together, through the matching ACE-Step XL model's adapter loader.
Include the trigger recorded in that training job's `recipe.json` in the
generation caption; renaming a character does not rewrite an old model's
trigger. In ComfyUI, the
installed nodes must explicitly support the XL variant and the saved adapter
format; compatibility is not assumed and weights are not silently converted.
Syllable runs can provide timbre and pitch coverage, but do not establish good
lyric pronunciation or full-song generalisation. A voice-only set can also
bias outputs toward a cappella performances. Add authorized clean sung phrases
with varied pitches, vowels, consonants, dynamics, and accurately annotated
lyrics; evaluate on held-out performances before relying on the character.
## Path B: singing voice conversion
For preserving an existing singer's melody, timing, and performance while
changing the voice, use a **separate RVC model**, for example through
[Applio](https://github.com/IAHispano/Applio).
1. Prepare and listen to the character's vocal clips.
2. Open **Models & exports** and build an SVC dataset export.
3. Download and extract the ZIP. It contains the mono 40 kHz WAVs and local
training handoff instructions, not a trained conversion model.
4. In a separately installed Applio environment, select a unique model name
and the extracted character folder; use 40 kHz, preprocess, extract
pitch/features, train, and build its retrieval index following the current
Applio training guide.
5. Use its `.pth` and `.index` for conversion of isolated source vocals, then
mix the converted vocals back with the instrumental.
Applio installation, downloads, training, and conversion are **not automated**
by this studio. That boundary avoids guessing an unselected backend version
or mixing conflicting Torch dependencies. SVC training generally does not
need lyrics, so the export is available without ACE-Step lyric annotation.
It still requires human listening and authorized audio.
An ACE-Step LoRA is not an RVC/so-vits-svc model. Renaming an adapter does not
convert it. ACE-Step audio conditioning/cover workflows may be useful, but
are not a guaranteed drop-in replacement for performance-preserving SVC.
## Job and file behaviour
- One dataset/GPU/export job runs at a time. Mutations are blocked during jobs.
- History, logs, and artifacts persist per character. Interrupted jobs are
marked failed on the next studio startup, never reported as completed.
- Stopping the studio requests termination of its active backend process.
There is no browser cancellation/resume UI. A subsequent training request
starts a new run; native CLI resume remains an upstream advanced workflow.
- Preparation and uploads write only new files. Earlier datasets and partial
failed runs are retained for inspection; no automatic deletion is performed.
- Model weights and audio remain local. The application contains no telemetry,
remote fonts, external JavaScript, or audio-upload services.
### Upstream references
- [ACE-Step 1.5](https://github.com/ace-step/ACE-Step-1.5)
- [ACE-Step LoRA training tutorial](https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/en/LoRA_Training_Tutorial.md)
- [Applio](https://github.com/IAHispano/Applio)