Braintrust

LLM eval and experiment platform

Free tier + paidModel APIs, routers & observabilitybraintrust.dev

What it is

An eval-first evaluation platform: datasets, scoring functions, experiment comparison, a prompt playground and production logging in one place. It aims to answer 'did this prompt change make things better or worse' with numbers, rather than only showing traces.

Why it earns a slot

Turns prompt tweaking from vibes into a regression suite — pin a dataset, run the eval, see which model or prompt scores higher before you ship. Included model credits mean the graders themselves are covered.

Know before you commit

Billing on processed data volume and score counts punishes verbose traces — logging whole documents in every span pushes you past 1GB fast. Free retention is only 14 days.

Getting started

npm i braintrust (or pip install braintrust)

Field notes

0 from the community

Loading notes…

Field check

Unverified

Pricing snapshot

Starter free ($10/mo model credits, 1GB data, 10k scores, 14-day retention), Pro $249/mo ($249 credits, 5GB, 50k scores), Enterprise custom. Overage $3-4/GB and $1.50-$2.50 per 1k scores.

Snapshots move fast — verify at checkout, and post a field note if it changed.

Drop into your Lovable chat

Evaluate Braintrust (https://www.braintrust.dev) for this project, then use it only if it fits what we are already building.
Context: LLM eval and experiment platform. An eval-first evaluation platform: datasets, scoring functions, experiment comparison, a prompt playground and production logging in one place. It aims to answer 'did this prompt change make things better or worse' with numbers, rather than only showing traces.
Reviewer notes (general, not written for this codebase): Turns prompt tweaking from vibes into a regression suite — pin a dataset, run the eval, see which model or prompt scores higher before you ship. Included model credits mean the graders themselves are covered.
Watch out for: Billing on processed data volume and score counts punishes verbose traces — logging whole documents in every span pushes you past 1GB fast. Free retention is only 14 days.
Pricing: Starter free ($10/mo model credits, 1GB data, 10k scores, 14-day retention), Pro $249/mo ($249 credits, 5GB, 50k scores), Enterprise custom. Overage $3-4/GB and $1.50-$2.50 per 1k scores.
Setup: npm i braintrust (or pip install braintrust)
Rules: treat the notes above as background only. Do not assume a theme, style, layout or feature that is not already in this project — inspect the existing code first and ask me before changing visual direction.

/api/public/tools/braintrust

Also in Model APIs, routers & observability

Anthropic API / Claude Platform

platform.claude.com

Frontier model provider API

PaidModel APIs, routers & observability

Per token. Opus 5 $5/$25 per M in/out; Sonnet 5 $2/$10; Haiku 4.5 $1/$5; Fable 5 and Mythos 5 $10/$50. Batch API 50% off; cache reads 0.1x input.

Open dossier

Fal.ai

fal.ai

Generative media inference host

PaidModel APIs, routers & observability

Per output or per second for media models: images ~$0.02-$0.04 at 1MP, video $0.05-$0.40 per second. Serverless GPU rates $1.10-$8.50/hr with committed-use discounts.

Open dossier

Fireworks AI

fireworks.ai

Open-model inference host

Free tier + paidModel APIs, routers & observability

Per token serverless across Standard/Priority/Fast tiers; embeddings $0.008-$0.10/M. On-demand GPUs billed per second: H100/H200 ~$7-8/hr, B200 ~$10-13/hr. Fine-tuning $0.50-$40 per M training tokens. $1 free credit.

Open dossier

Google AI Studio / Gemini API

ai.google.dev

Model provider API with free tier

Free tier + paidModel APIs, routers & observability

Free tier in AI Studio with rate limits. Paid: Gemini 3.7 Flash $0.75/$3.75 per M in/out; Gemini 3.5 Flash-Lite $0.30/$2.50; Gemini 2.5 Pro $1.25/$10.

Open dossier