AI Science Video Studio — AI科普视频自动化制作技能

Overview

Full pipeline for producing explainer / educational videos that combine a digital human avatar (intro/outro) with animated content slides (body), voice-narrated by a cloned personal voice (F5-TTS MLX), with karaoke-style subtitles throughout.

The pipeline follows an 8-stage workflow:

Script Planning → Digital Human → TTS Voice → Content Slides
    → Subtitles → Audio Repair → Final Compositing → QA Review

Default configuration is tuned for 1280×720 (16:9), 24fps, CRF 20 encoding, a single presenter avatar, and Mandarin Chinese narration. All parameters are adjustable.

When to Use This Skill

Trigger on any of the following intents:

• User asks to create an "AI科普" (AI science explainer) video
• User wants an educational/explainer video with digital human + slides format
• User mentions combining a talking avatar with content slides
• User needs the full pipeline: script → voice clone → slides → subtitles → compositing
• User says "做一个讲解视频", "生成科普视频", "制作AI讲解类视频"

Do NOT use this skill for:

• Pure short films / drama without educational content → use ai-short-film-studio
• Only SadTalker PiP compositing without slides → use sadtalker-pip-compositing
• Only Google Flow video generation → use google-flow-automation

Pipeline Stages

Stage 1: Script Planning

Create a script.json file defining the video structure with exactly 5 paragraphs:

{
  "intro":      { "type": "digital_human", "engine": "google_flow", "duration": 10, "narration": "开场旁白...", "flow_prompt": "..." },
  "content_1":  { "type": "slides",        "engine": "pillow",      "duration": 30, "narration": "正文第一段旁白..." },
  "content_2":  { "type": "slides",        "engine": "pillow",      "duration": 25, "narration": "正文第二段旁白..." },
  "content_3":  { "type": "slides",        "engine": "pillow",      "duration": 29, "narration": "正文第三段旁白..." },
  "outro":      { "type": "digital_human", "engine": "google_flow", "duration": 10, "narration": "结尾旁白...", "flow_prompt": "..." }
}

Rules:

• intro and outro use digital_human type (talking avatar)
• content segments use slides type (animated content screens)
• Each segment must specify: type, engine, duration (seconds), narration text
• Narration text should be ≤15 seconds worth of speech per segment (~60 Chinese characters)
• Duration field is the target video length (not TTS length — TTS naturally sets the pace)

For detailed script format specification, see references/script_format.md.

Stage 2: Digital Human Generation (Intro & Outro)

Two approaches are available. Prefer Google Flow for standalone talking-head segments; use SadTalker for picture-in-picture overlay on content slides.

Option A: Google Flow CDP Automation

Use the google-flow-automation skill to generate intro/outro videos:

1. Launch Chrome with remote debugging on port 9222
2. Navigate to labs.google/fx/tools/flow
3. Upload avatar reference image
4. Enter the Chinese prompt from script.json
5. Wait ~3-5 minutes for 10-second video generation
6. Download as intro.mp4 and outro.mp4

Key parameters:

• Avatar: upload the user's preferred reference image (portrait photo)
• Prompt: in Chinese, describe the scene and delivery style
• Model: Omni Flash, 16:9 aspect ratio, 10s duration
• Account: the user's Google account credentials (handled by Chrome profile)

Option B: SadTalker MPS (for PiP on content)

Use the sadtalker-pip-compositing skill when the digital human should appear as a circular picture-in-picture overlay on content slides.

Steps:

1. Run scripts/fix_sadtalker_numpy.py for numpy 2.x compatibility
2. Extract avatar image + TTS audio
3. Run SadTalker 3-stage inference with device='mps'
4. Create circular mask (120×120) with PIL
5. FFmpeg overlay onto content at bottom-left corner

PiP Parameters:

Stage 3: Voice Generation (TTS)

Primary: F5-TTS MLX Voice Cloning

Use F5-TTS MLX on Apple Silicon for personal voice cloning:

from f5_tts_mlx.generate import generate

# For content narration (MUST use estimate_duration=True!)
audio = generate(
    text="旁白文本...",
    ref_audio_path="/path/to/ref_voice.mp3",
    ref_audio_text="参考音频的文本内容",
    steps=64,
    cfg_strength=2.5,
    speed=1.0,
    estimate_duration=True,  # CRITICAL for Chinese!
)

CRITICAL — estimate_duration=True: Without this parameter, F5-TTS generates extremely short audio for Chinese text (0.5-0.9 seconds per sentence). With it, the model estimates target duration and generates properly-length audio.

Parameter table:

Post-processing: After generation, compute the actual-vs-target duration ratio and apply atempo to fine-tune timing:

# Example: actual 11.98s, target 10.0s → atempo=1.198
ffmpeg -i generated.wav -filter:a "atempo=1.198" output.wav

Fallback: edge-tts

When F5-TTS is unavailable or produces garbled output:

edge-tts --voice zh-CN-YunxiNeural --text "旁白文本" --write-media output.wav

Voice selection:

Stage 4: Content Slide Rendering

Render animated content slides using Pillow frame-by-frame rendering + FFmpeg pipe.

Use scripts/render_slides.py as the template. The script should:

1. Accept narration text split into lines
2. Render each frame with progressively "typed" text (one new line per frame)
3. Use terminal/IDE aesthetic: dark background (#1a1a2e), green/white text, monospace font
4. Output 1280×720, 24fps PNG frames via FFmpeg pipe
5. Sync frame count to the TTS audio duration

Key rendering parameters:

• Resolution: 1280×720
• Frame rate: 24fps
• Background: dark (#1a1a2e or pure black for terminal look)
• Text: green (#00ff41) for code, white for explanatory text
• Font: SF Mono or Menlo for code sections; STHeiti for Chinese text

The script is at scripts/render_slides.py. Customize the content per video topic while keeping the rendering engine intact.

Stage 5: Subtitle Rendering

Generate karaoke-style subtitles as transparent PNG frames overlayed on the final video.

The process:

Audio (.wav)
  → Whisper small/medium transcription
  → Word-level timestamps (segments + words)
  → Text correction mapping (fix Whisper mis-transcriptions)
  → Pillow frame-by-frame PNG rendering (transparent BG)
  → FFmpeg overlay onto video

Use scripts/render_subtitles.py as the rendering engine.

Subtitle style specification (intro and outro MUST match):

Text correction mapping: Always maintain a correction dictionary to fix Whisper mis-transcriptions of technical terms and proper names:

corrections = {
    "材领": "才林",
    "Anthropy": "Anthropic",
    "Cloud Code": "Claude Code",
}

CRITICAL — Consistency rule: Intro and outro subtitles MUST use the exact same rendering engine (Pillow) with identical style properties. Never mix Pillow and ASS/other formats — FFmpeg on macOS lacks libass support.

Stage 6: Audio Repair

Common audio issues and their fixes. Run scripts/audio_analyzer.py for automated detection before proceeding.

Channel fix command:

ffmpeg -i input.wav -af "channelmap=map=FL-FL|FL-FR" -ar 48000 -ac 2 output.wav

Gap smoothing approach: For each audio gap >80ms detected by scripts/audio_analyzer.py:

# Split at gap, apply fade-out/fade-in, re-concatenate
ffmpeg -i audio.wav -af "afade=t=out:st=GAP_START-0.25:d=0.25,afade=t=in:st=GAP_END:d=0.25" patched.wav

Stage 7: Final Compositing

Assemble all segments with professional transitions using FFmpeg.

Compositing order:

intro.mp4 → content_1.mp4 → content_2.mp4 → content_3.mp4 → outro.mp4

Use scripts/compose_final.py for automated assembly.

Encoding parameters:

Transition effects:

• Video: xfade=transition=fade:duration=0.5:offset=<time> (cross-fade, eliminates hard cuts)
• Audio: acrossfade=d=0.5:curve=tri (triangular cross-fade, smooth audio joins)

Pre-compositing checklist:

1. All segments re-encoded to CRF 20 (unified quality)
2. All audio normalized to 48000Hz stereo
3. Subtitle overlays applied to each segment
4. Transitions prepared: 0.5s offset for each segment boundary

Stage 8: Professional QA Review

Run a systematic quality review before delivering the final video.

QA dimensions and inspection methods:

Verification commands:

# Audio per-second analysis
python3 scripts/audio_analyzer.py output.mp4

# Video quality check
ffprobe -v error -select_streams v:0 \
  -show_entries stream=codec_name,width,height,r_frame_rate,bit_rate \
  output.mp4

# Subtitle sync verification
whisper model small --language zh output.mp4

For the complete QA checklist, see references/qa_checklist.md.

File Naming Convention

AI科普第{N}期_{主题}_v{N}.mp4

Example: AI科普第一期_SadTalker画中画_v11.mp4

Intermediate files:

Quality Targets

Critical Pitfalls

For the complete pitfalls reference, see references/pitfalls.md. Key highlights:

1. F5-TTS Chinese too short: Always set estimate_duration=True for Chinese content narration. Without it, audio is only 0.5-0.9s per sentence.

2. Alpha channel compositing: When using alphamerge, the human RGBA video is the color source (first input), and the circular mask PNG is the alpha (second input). Reversing them produces a white circle with no human visible.

3. Concat format mismatch: Different segments may have different sample rates (16000 vs 48000Hz) or channel counts (mono vs stereo). Unify all segments to 48000Hz stereo before concatenation.

4. Subtitle rendering engine inconsistency: Always use Pillow for both intro and outro subtitles. FFmpeg on macOS lacks libass, making ASS-subtitle filters unavailable.

5. AI-generated BGM gaps: Google Flow's AI-generated background music may contain silence gaps (100-400ms). Smooth them with 250ms crossfades at each gap boundary.

Bundled Resources

Scripts

• scripts/render_slides.py — Pillow-based content slide frame renderer (1280×720, dark IDE theme, progressive text reveal)
• scripts/render_subtitles.py — Karaoke-style subtitle PNG renderer (word-by-word orange highlight, transparent BG, STHeiti 44px)
• scripts/compose_final.py — End-to-end FFmpeg compositing (xfade + acrossfade + alimiter + CRF20 unified encoding)
• scripts/audio_analyzer.py — Audio QA analysis tool (second-by-second mean/peak detection, gap finder, clipping detector)

References

• references/script_format.md — Complete script.json format specification and examples
• references/qa_checklist.md — Detailed 8-dimension QA review checklist
• references/pitfalls.md — Comprehensive list of known pitfalls with root causes and fixes