AnyChunker Skill

Language: English · 简体中文

Split any text for LLM / RAG / Agent pipelines using the anychunker library.

The killer feature is structure-preserving block-level Markdown splitting: code fences, tables, and HTML blocks stay intact, and every chunk carries parent/child index trees so you can reconstruct hierarchical context later — ideal for high-quality RAG retrieval layers.

Workflow

1. Verify install — run python scripts/verify_install.py to confirm anychunker is importable and print installed extras (see Environment check).
2. Pick the right chunker — cross-check the user's intent against the Chunker selection matrix. If unsure, ask the user one clarifying question ("Do you need to preserve Markdown code blocks / tables?" is usually the deciding factor).
3. Load only the needed API section — references/api-reference.md is long; grep for the class name and read only that section.
4. Write the smallest working example — do NOT dump every parameter. Start with chunk_size + chunk_overlap and defaults; add tokenizer / language / embedding only when the user asks.
5. Verify with a real run — actually execute the script and print len(doc.chunks), first/last chunk content preview, and any metadata the user cares about. Never fabricate output.
6. Hand off cleanly — for RAG ingestion, use document.batchIterator(batch_size=N) to feed downstream (embedding API, vector DB); see Batch iteration.

Installation

anychunker is on PyPI. Requires Python ≥ 3.7.

pip install anychunker

Optional extras (install only when the user's use case requires them):

Same command works on macOS / Linux / Windows. On Windows use PowerShell or cmd; the pip install line is identical.

Environment check

Run the bundled verification script — it's pure Python and works on macOS / Linux / Windows without modification:

python scripts/verify_install.py

It prints:

• anychunker version + install path
• Whether optional deps (transformers, sentence-transformers, jieba) are importable
• A tiny end-to-end smoke test that splits a 3-line Markdown snippet

If anychunker is missing, the script exits with code 1 and prints the exact install command — do not pip install on the user's behalf; show them the command and wait.

Chunker selection matrix

Use this table to pick a class, then read only that section of references/api-reference.md.

Core imports

Import everything from the top-level package. __init__.py uses lazy loading, so importing anychunker alone does not pull transformers / torch; the heavy deps only load when you touch a class that needs them.

from anychunker import (
    AnyTextChunker,             # recursive text split
    AnyMarkdownChunker,         # header-based Markdown split
    AnyMarkdownBlockChunker,    # block-level Markdown split (★ RAG go-to)
    AnyCodeChunker,             # code split by language
    AnySentenceChunker,         # sentence split
    AnySemanticsChunker,        # semantic split
    Language,                   # enum: PYTHON / JS / GO / RUST / MARKDOWN / HTML / ...
    Chunker,                    # schema: one chunk
    DocumentMetadata,           # schema: doc-level metadata
    ChunkBatcher,               # schema: a batch of chunks
)

Minimal usage — the four most common paths

1. Structure-preserving Markdown split (recommended default for RAG)

from anychunker import AnyMarkdownBlockChunker

chunker = AnyMarkdownBlockChunker(chunk_size=500, chunk_overlap=50)
doc = chunker.invoke(markdown_text)

for c in doc.chunks:
    print(c.chunk_id, c.metadata.get("title"), c.content[:60])

Each chunk's metadata carries block_id, title, headings, and full block_parent_id / block_child_id arrays — use these to rebuild hierarchical context in your vector store.

2. Recursive text split (fast, no external deps)

from anychunker import AnyTextChunker

chunker = AnyTextChunker(chunk_size=500, chunk_overlap=50)
doc = chunker.invoke(long_text)

Default separators are tuned for mixed CJK + Latin text (\n\n, \n, 。, , ""). Override with separators=[...] if needed.

3. Token-accurate split for a specific LLM

chunker = AnyTextChunker.from_tokenizer(
    "Qwen/Qwen3-8B",   # any HuggingFace tokenizer id or local path
    chunk_size=1024,
    chunk_overlap=100,
)
doc = chunker.invoke(text)

chunk_size is now measured in tokens (not characters).

4. Semantic split

from anychunker import AnySemanticsChunker
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
def emb(sentences): return model.encode(sentences).tolist()

chunker = AnySemanticsChunker(embedding_model=emb)
doc = chunker.invoke(text)

Any callable that returns List[List[float]] works — you're not locked into sentence-transformers.

Batch iteration

Every invoke() returns a Document. To feed downstream in batches (embedding API rate limits, DB bulk inserts):

for batch in doc.batchIterator(batch_size=16):
    # batch is a ChunkBatcher with:
    #   batch.chunks               → List[Chunker]
    #   batch.get_content_text()   → joined content
    #   batch.get_chunk_ids()      → [chunk_id, ...]
    #   batch.total_content_length → int
    embeddings = my_embed_api([c.content for c in batch.chunks])
    my_vector_db.upsert(batch.get_chunk_ids(), embeddings)

Quick demo

Run the bundled demo to see AnyMarkdownBlockChunker in action end-to-end (works on macOS / Linux / Windows):

python scripts/quick_demo.py

It splits a mixed Markdown sample (headings + code fence + HTML table) and prints each chunk's chunk_id, title, content preview, and parent/child block ids so you can see the hierarchy tree.

Parameter cheat sheet

Common to every chunker unless noted otherwise:

For AnyMarkdownBlockChunker specific params, see references/api-reference.md#anymarkdownblockchunker.

Common pitfalls

Recipes

More end-to-end recipes (RAG ingestion, jieba length, code splitting, semantic tuning) live in references/recipes.md. Grep for the section that matches the user's ask and read only that section.

Links

• PyPI: https://pypi.org/project/anychunker/
• GitHub: https://github.com/anyforge/anychunker
• DeepWiki: https://deepwiki.com/anyforge/anychunker/