CAFE — Compound-AI Factorial Evaluation

What is CAFE — Compound-AI Factorial Evaluation?

CAFE treats every knob in your AI pipeline - retrieval, reranking, prompts, models, and tools - as an experimental factor. It runs factorial experiments, evaluates outputs using a configurable LLM (and optionally human reviewers), and applies mixed-effects models to determine: - Which techniques actually improve quality - How much each technique contributes - Whether the observed differences are statistically significant Open source and self-hostable.