Audio Intelligence

Gladia's audio intelligence features extract structured data and insights from transcripts. They work on top of the base transcription — most are enabled by adding options to the transcribe() call (pre-recorded) or the startSession() config (live).

SDK-first: always use the official SDK — see sdk-integration for policy, setup, and fallback criteria.

When to Use

• User asks about a specific feature: diarization, translation, PII redaction, sentiment, NER, subtitles, summarization, etc.
• Enabling or configuring one or more audio intelligence features on pre-recorded or live transcription
• Understanding which features are available in live vs pre-recorded mode
• Combining multiple features in a single transcription job

When NOT to use: For basic transcription without audio intelligence features, go directly to pre-recorded-transcription or live-transcription. For gotchas and errors related to specific features, see troubleshooting.

References

Consult these resources as needed:

• ./references/live-audio-intelligence.md -- Detailed config and WebSocket responses for all live-mode features
• ./references/pre-recorded-audio-intelligence.md -- Detailed config and response structures for all pre-recorded audio intelligence features
• ../pre-recorded-transcription/SKILL.md -- Pre-recorded transcription workflow and options
• ../live-transcription/SKILL.md -- Live transcription session config and event handling
• ../sdk-integration/SKILL.md -- SDK setup, client initialization, and SDK vs raw API decision guide
• ../troubleshooting/SKILL.md -- Common errors, gotchas, and verification checklist

Feature Availability

Live features split into two groups: real-time (results stream during the session) and post-processing (results arrive after stopRecording()). See ./references/live-audio-intelligence.md for details.

Quick Config Examples

Code examples assume GladiaClient is already initialized — see sdk-integration for setup.

Speaker Diarization (pre-recorded only)

const result = await client.preRecorded().transcribe("audio.mp3", {
  diarization: true,
  diarization_config: { number_of_speakers: 2 },
});
// Each utterance includes a `speaker` field (0-indexed integer)

result = client.prerecorded().transcribe("audio.mp3", {
    "diarization": True,
    "diarization_config": {"number_of_speakers": 2},
})

Translation (pre-recorded and live)

Pre-recorded:

const result = await client.preRecorded().transcribe("audio.mp3", {
  translation: true,
  translation_config: { target_languages: ["fr", "es"] },
});

result = client.prerecorded().transcribe("audio.mp3", {
    "translation": True,
    "translation_config": {"target_languages": ["fr", "es"]},
})

Live (result streams as translation WebSocket events — see live-audio-intelligence.md):

const session = client.liveV2().startSession({
  // ... audio format options ...
  realtime_processing: {
    translation: true,
    translation_config: { target_languages: ["fr"] },
  },
});

from gladiaio_sdk import LiveV2InitRequest, LiveV2RealtimeProcessing

session = client.live().start_session(
    LiveV2InitRequest(
        # ... audio format options ...
        realtime_processing=LiveV2RealtimeProcessing(
            translation=True,
            translation_config={"target_languages": ["fr"]},
        ),
    )
)

Summarization (pre-recorded and live)

Pre-recorded:

const result = await client.preRecorded().transcribe("audio.mp3", {
  summarization: true,
  summarization_config: { type: "bullet_points" },
});

Live (arrives after stopRecording() as post_summarization event):

const session = client.liveV2().startSession({
  // ... audio format options ...
  post_processing: {
    summarization: true,
    summarization_config: { type: "bullet_points" },
  },
});
session.on("message", (msg) => {
  if (msg.type === "post_summarization") console.log(msg.data.results);
});

For full per-feature config options and response structures, see:

• Pre-recorded: ./references/pre-recorded-audio-intelligence.md
• Live: ./references/live-audio-intelligence.md

Common Mistakes

• code_switching: true with empty languages: triggers evaluation across 100+ languages and causes frequent misdetections. Always provide 3-5 expected languages.
• Custom vocabulary intensity above 0.6: values over 0.6 cause false positives where unrelated words get replaced. Keep at 0.4-0.6 and use pronunciations for better results.
• Expecting diarization, PII redaction, subtitles, or audio-to-LLM in live mode: these four features are pre-recorded only.
• Enabling many features simultaneously without considering cost/latency: each enabled feature adds processing time. Enable only what you need; combine diarization + summarization + translation only when all are required.

For the full gotcha list, see troubleshooting.

Further Reading

• Audio intelligence overview
• Speaker diarization
• Translation
• Sentiment analysis
• Named entity recognition
• PII redaction
• Custom vocabulary
• Summarization
• Chapterization
• Audio-to-LLM
• Live audio intelligence