LLM Flow

An AI toolkit for configuring, benchmarking, comparing, prompting, evaluating, fine-tuning, analyzing, and optimizing LLM workflows. Each command logs timestamped entries to local files with full export, search, and statistics support.

Commands

Core AI Operations

Command	Description
`llm-flow configure <input>`	Record a configuration change (or view recent configs with no args)
`llm-flow benchmark <input>`	Log a benchmark run and its results
`llm-flow compare <input>`	Record a model or output comparison
`llm-flow prompt <input>`	Log a prompt template or prompt engineering note
`llm-flow evaluate <input>`	Record an evaluation result or metric
`llm-flow fine-tune <input>`	Log a fine-tuning session or parameters
`llm-flow analyze <input>`	Record an analysis observation
`llm-flow cost <input>`	Log cost tracking data (tokens, dollars, etc.)
`llm-flow usage <input>`	Record API usage metrics
`llm-flow optimize <input>`	Log an optimization attempt and outcome
`llm-flow test <input>`	Record a test case or test result
`llm-flow report <input>`	Log a report entry or summary

Utility Commands

Command	Description
`llm-flow stats`	Show summary statistics across all log files
`llm-flow export <fmt>`	Export all data in `json`, `csv`, or `txt` format
`llm-flow search <term>`	Search all entries for a keyword (case-insensitive)
`llm-flow recent`	Show the 20 most recent activity log entries
`llm-flow status`	Health check: version, entry count, disk usage, last activity
`llm-flow help`	Display full command reference
`llm-flow version`	Print current version (v2.0.0)

How It Works

Every core command accepts free-text input. When called with arguments, LLM Flow:

Timestamps the entry (YYYY-MM-DD HH:MM)
Appends it to the command-specific log file (e.g. benchmark.log, cost.log)
Records the action in a central history.log
Reports the saved entry and running total

When called with no arguments, each command displays the 20 most recent entries from its log file.

Data Storage

All data is stored locally in plain-text log files:

~/.local/share/llm-flow/
├── configure.log     # Configuration changes
├── benchmark.log     # Benchmark results
├── compare.log       # Model comparisons
├── prompt.log        # Prompt templates & notes
├── evaluate.log      # Evaluation metrics
├── fine-tune.log     # Fine-tuning sessions
├── analyze.log       # Analysis observations
├── cost.log          # Cost tracking
├── usage.log         # API usage metrics
├── optimize.log      # Optimization attempts
├── test.log          # Test cases & results
├── report.log        # Report entries
├── history.log       # Central activity log
└── export.{json,csv,txt}  # Exported snapshots

Each log uses pipe-delimited format: timestamp|value.

Requirements

Bash 4.0+ with set -euo pipefail
Standard Unix utilities: wc, du, grep, tail, date, sed
No external dependencies — pure bash

When to Use

Building AI agent workflows — log each step of your agent pipeline (configure → prompt → evaluate → optimize) with full traceability
Tracking LLM costs and usage — record per-request costs, token counts, and API usage to monitor spending across providers
Benchmarking and comparing models — log benchmark metrics side-by-side to make data-driven model selection decisions
Fine-tuning experiment tracking — capture hyperparameters, dataset details, and evaluation scores for every fine-tuning run
Generating compliance reports — export all logged activity to JSON/CSV for audits, SOC reviews, or stakeholder reporting

Examples

# Configure a new workflow
llm-flow configure "workflow: summarize → classify → respond, model=claude-3.5"

# Benchmark a model
llm-flow benchmark "claude-3.5-sonnet: 94% accuracy, 0.8s p50 latency, $0.003/req"

# Log a prompt template
llm-flow prompt "system: You are a helpful assistant. Always cite sources."

# Track API costs
llm-flow cost "March week 3: 890k tokens in, 210k tokens out, $12.40 total"

# Evaluate output quality
llm-flow evaluate "human eval score: 4.2/5.0 across 50 samples"

# Search across all logs
llm-flow search "claude"

# Export to CSV for analysis
llm-flow export csv

# Quick health check
llm-flow status

Configuration

Set the DATA_DIR variable in the script or modify the default path to change storage location. Default: ~/.local/share/llm-flow/

Llm Flow

安装技能