Testing an async FastAPI + Postgres + LLM app

Every commit that touches the API ships a test. Every test runs the same way: pytest, no flags, no setup steps. A fresh checkout goes from git clone to green dots in two commands. That ergonomic is load-bearing — if running tests is friction, tests don't get written, and the development loop slows to a crawl.

The shape of "easy to run" here is Django-equivalent: pytest creates a throwaway app_test database from your schema bootstrap SQL, runs the suite with per-test transactional rollback, and drops the database at the end. You never see it, never clean it up. Set KEEPDB=1 to keep it across runs (Django's --keepdb) when iterating fast on one test.

This skill ships drop-in scaffolding. The first time it runs in a fresh checkout, copy the templates from assets/ into your test tree. After that, every new test imports the same fixtures. The skill's job from then on is to keep patterns consistent, refresh guidance against installed versions, and prevent common drift (mocking the DB, faking the LLM over HTTP, hand-rolling SSE assertions).

When to invoke this skill

Trigger on any of:

• "Write a test for X", "add a regression for Y", "TDD this"
• Any new file under the test tree, or any modification to one
• "How do I test the SSE endpoint?", "how do I test a tool dispatch?"
• "Set up pytest", "bootstrap the test suite", "wire up CI tests"
• Adding pytest, pytest-asyncio, httpx, etc. to [dev-dependencies]
• "Mock the LLM", "fake the Claude / Anthropic client", "test without hitting the API"
• "Truncate the DB between tests", "tests are leaking state", "transaction not rolling back"
• A request mentioning "test plan", "verifiable", "smoke test", "integration test", "end-to-end test", "fixture"

If a request talks about correctness, regression prevention, or the development loop, treat it as a trigger.

First action: refresh against installed versions

The Python testing ecosystem moves. Before writing fixtures or helpers, check the installed versions of the libraries you're about to use. Use the context7 MCP:

mcp__context7__resolve-library-id  libraryName="FastAPI"   # or "pytest", "pytest-asyncio", "httpx"
mcp__context7__query-docs          libraryId="/<resolved-id>"
                                   query="<the exact thing you're doing>"

Good queries: "AsyncClient with ASGITransport for FastAPI app", "pytest-asyncio fixture loop scope session", "asyncpg pool override in FastAPI dependency_overrides". Bad queries: "pytest" or "FastAPI testing" — too broad.

Skip context7 for things that aren't framework-specific: vanilla asyncpg SQL, dataclass usage, Python's unittest.mock.

Stack — recommended decisions

If you change a row, update it here in the same commit.

The three test layers

Every test belongs in exactly one of these directories. The layer determines what the test is allowed to touch and how fast it must run.

Why three? So you can run the fast layer continuously during dev (pytest tests/unit -x) and the slow layers only when you need them. The unit layer is your TDD inner loop; integration and e2e gate the PR.

api/tests/                 # illustrative; mirror your own package layout
├── conftest.py            # all fixtures — copied from assets/conftest.py.tmpl
├── _helpers/
│   ├── __init__.py
│   ├── sse.py             # SSE stream consumer
│   ├── factories.py       # tiny seed factories
│   └── fake_llm.py        # fake LLM client (Anthropic SDK shape)
├── unit/
│   └── test_envelope.py
├── integration/
│   ├── test_health.py
│   └── test_appointment_endpoint.py
└── e2e/
    ├── test_chat_basic.py
    └── test_chat_book_appointment.py

Mirror the source tree where it makes sense, but don't be religious about it — group by feature when that's clearer.

Bootstrapping into a fresh project

If tests/conftest.py doesn't exist yet, copy these templates into place before any test is written (adjust the destination to your package layout):

assets/conftest.py.tmpl                  → tests/conftest.py
assets/_helpers/sse.py.tmpl              → tests/_helpers/sse.py
assets/_helpers/factories.py.tmpl        → tests/_helpers/factories.py
assets/_helpers/fake_llm.py.tmpl         → tests/_helpers/fake_llm.py
assets/example_unit_test.py.tmpl         → tests/unit/test_envelope.py
assets/example_integration_test.py.tmpl  → tests/integration/test_health.py
assets/example_e2e_test.py.tmpl          → tests/e2e/test_chat_basic.py

Add empty __init__.py files under tests/, tests/_helpers/, tests/unit/, tests/integration/, tests/e2e/ so pytest collects cleanly without rootdir games.

Add dev dependencies per assets/pyproject-deps.md (read it). Drop the pytest config block from assets/pytest_config.md into your pyproject.toml.

Minimal mode (no DB, no LLM yet)

The drop-in conftest.py.tmpl assumes the app already imports asyncpg (for get_pool) and an LLM SDK (for get_llm_client). On a fresh checkout that hasn't built those yet, the full template fails at collection time with ModuleNotFoundError. Ship this minimal conftest instead and graduate to the full template the moment the DB pool lands. The contract the tests rely on — a client fixture yielding an AsyncClient — stays the same; only the plumbing behind it changes.

# tests/conftest.py — minimal mode
"""Minimal conftest. App + AsyncClient, no DB, no LLM.
Replace with assets/conftest.py.tmpl when the pool lands."""
from __future__ import annotations

from collections.abc import AsyncIterator

import pytest
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient

from api.main import app as fastapi_app


@pytest.fixture
def app() -> FastAPI:
    return fastapi_app


@pytest.fixture
async def client(app: FastAPI) -> AsyncIterator[AsyncClient]:
    transport = ASGITransport(app=app)
    async with AsyncClient(transport=transport, base_url="http://testserver") as ac:
        yield ac

tests/_helpers/sse.py (the SSE consumer) is safe to ship from day one — it has no DB or LLM dependency. Use it for a mock /chat endpoint's contract test. The graduation trigger is unambiguous: the first PR that adds asyncpg to pyproject.toml should also swap this file for assets/conftest.py.tmpl, copy _helpers/factories.py and _helpers/fake_llm.py, and ensure your schema bootstrap SQL exists.

Writing a test (5 steps)

The full set of patterns lives in references/example-tests.md. The short form for a typical feature:

1. Pick the layer. Pure logic → unit/. Anything that touches a route or the DB → integration/. Anything that asserts on the SSE stream → e2e/.
2. Use the fixtures, don't roll your own. The client fixture yields an AsyncClient already wired to the app with the test DB connection plumbed in. The db_conn fixture gives you a raw asyncpg.Connection inside the same transaction the app is using. The fake_llm fixture replaces the LLM client.
3. Seed via factories, not raw SQL. The factories in _helpers/factories.py know the schema and the FK shape, so a future column addition doesn't shower you with broken tests.
4. Assert on the response envelope. Every API response goes through your response envelope (see writing-fastapi-apis). Assert on success, data, and error.code — never on raw JSON shape that bypasses it.
5. Don't skip the cleanup question. If the test commits explicitly (rare — only when testing commit boundaries), it owes a TRUNCATE. Otherwise the rollback fixture handles it. See references/db-strategy.md § "Escape hatch".

Faking the LLM — the boundary

The LLM client is reached via a single dependency in the application. Tests override that dependency with a fake — they never mock at the HTTP layer. Reason: an LLM SDK does its own streaming/state machine, and HTTP-level mocks (respx, httpx_mock, vcrpy) have been a source of flaky tests for every team that's tried them. Stubbing at the SDK boundary keeps the test surface tiny.

The concrete example throughout this skill uses the Anthropic SDK (AsyncAnthropic) — substitute whichever client your app uses; the boundary pattern is identical.

# The dependency the application defines
from anthropic import AsyncAnthropic

_llm = AsyncAnthropic()  # module-level singleton; reads its key from env

def get_llm_client() -> AsyncAnthropic:
    return _llm

# The test fixture overrides it
@pytest.fixture
def fake_llm(app):
    fake = FakeLLM()
    app.dependency_overrides[get_llm_client] = lambda: fake
    yield fake
    app.dependency_overrides.pop(get_llm_client, None)

The FakeLLM class is a tiny harness that lets a test program the next response: fake.expect_text("Hello"), fake.expect_tool_use("book_appointment", input={...}), fake.expect_pause_turn(). Full pattern in references/llm-mocking.md.

When you genuinely need a real LLM call (e.g. a nightly groundedness regression on a tiny corpus), mark the test:

@pytest.mark.live_llm
async def test_real_search_returns_citation(client, ...):
    ...

live_llm is registered as a marker in pytest_config.md and deselected by default (-m "not live_llm"). CI opts in only on the nightly job. Keep these tests rare; they're slow, costly, and flaky.

SSE assertions

Once /chat starts streaming, you can no longer return an HTTP error — see writing-fastapi-apis. Tests therefore need to consume the stream and assert on parsed events, including the terminal event: error case.

Use the read_sse_events() helper:

from tests._helpers.sse import read_sse_events

async def test_chat_streams_tokens(client, fake_llm):
    fake_llm.expect_text("hi there")

    async with client.stream("POST", "/chat", json={"message": "hello"}) as resp:
        events = await read_sse_events(resp)

    assert resp.status_code == 200
    assert [e.event for e in events][-1] == "end"
    assert "".join(e.data["text"] for e in events if e.event == "token") == "hi there"

Full event types and the error case in references/sse-testing.md.

Common pitfalls (and how the skill prevents each)

CI integration (one-liner)

# .github/workflows/test.yml — abbreviated
- run: docker compose up -d db
- run: uv run pytest -m "not live_llm"

The DB lifecycle fixture handles CREATE DATABASE / DROP DATABASE itself, so CI doesn't need an extra setup step. A nightly job can add -m live_llm against a separate LLM-key secret with a small budget.

For running parallel workers, sharding the DB across workers, and KEEPDB=1 semantics, see references/db-strategy.md § "Parallelism". For the local db service that tests connect to, see dockerizing-fastapi-uv.

Files in this skill

• SKILL.md — this file
• assets/conftest.py.tmpl — the canonical conftest with all fixtures
• assets/_helpers/sse.py.tmpl — SSE stream consumer
• assets/_helpers/factories.py.tmpl — seed factories
• assets/_helpers/fake_llm.py.tmpl — fake LLM client (Anthropic SDK shape)
• assets/example_unit_test.py.tmpl — minimal unit test
• assets/example_integration_test.py.tmpl — minimal integration test
• assets/example_e2e_test.py.tmpl — minimal SSE end-to-end test
• assets/pyproject-deps.md — uv add --dev commands
• assets/pytest_config.md — pytest config block for pyproject.toml
• references/db-strategy.md — full Django-equivalent DB lifecycle, rollback, escape hatch, parallelism
• references/llm-mocking.md — FakeLLM patterns including server-tool blocks and pause_turn
• references/sse-testing.md — SSE event types, terminal error case, full assertion examples
• references/example-tests.md — one worked example per layer

Cross-skill notes

This skill is the how. The what lives elsewhere:

• For the route shell, error envelope, and PII rules the tests assert against — see writing-fastapi-apis.
• For the LLM loop's invariants (stable tools[], prompt-cache placement, pause_turn handling) that tests exercise via fake_llm — see building-llm-agent-loops.
• For containers, docker-compose, and the local db service tests connect to — see dockerizing-fastapi-uv.
• For managing Python dependencies with uv — see the built-in uv skill.