sofe-dbt — FinOps for data pipelines
SOFE is ESLint for your cloud. sofe-dbt is ESLint for your dbt data pipelines — with the real Athena bill attached to every model.
Every dollar shown comes from real Athena DataScannedInBytes— never a heuristic. If real bytes aren't available, sofe-dbt reports cost: unknown.
Install & run
pip install sofe-dbt # 1. generate dbt artifacts (writes real Athena scan stats) dbt build # 2. evaluate FinOps policies with real cost per model sofe-dbt check . # 3. gate a PR in CI (exit 1 on high/critical findings) sofe-dbt check . --fail-on high
How it works
sofe-dbt reads three standard dbt artifacts from target/ and feeds them to the SOFE engine as resources:
manifest.json— DAG, materializations, tests, dependenciesrun_results.json— execution time + real bytes scanned (dbt-athena writes it intoadapter_response)catalog.json— table sizes and row counts
Real cost math
cost_per_run = max(bytes_scanned, 10 MB) / 1e12 x $5.00/TB # us-east-1 on-demand monthly_cost = cost_per_run x runs_per_month # runs from meta.schedule
Offline (CI) the bytes come from run_results.json; live, from Athena GetQueryExecution (with --profile).
The 10 policies
| Policy | Severity | What it catches |
|---|---|---|
| dbt-full-refresh-in-prod | critical | incremental model re-scanning everything (the #1 hidden Athena bill) |
| dbt-high-blast-radius-no-tests | critical | 10+ downstream dependents, no tests |
| dbt-long-run-not-incremental | high | >30min run without incremental materialization |
| dbt-large-table-no-partition | high | >1GB table without partition_by (full scans) |
| dbt-model-last-run-failed | high | last run failed — burns compute, blocks downstream |
| dbt-zombie-model-no-consumers | medium | no downstream dependents and no exposure |
| dbt-small-table-should-be-view | medium | <1000 rows materialized as table |
| dbt-ephemeral-over-referenced | medium | ephemeral model inlined 5+ times |
| dbt-freshness-test-missing | medium | source with no freshness monitoring |
| dbt-model-no-description | low | consumed model with no documentation |
CI/CD gate (GitHub Actions)
- run: pip install dbt-athena-community sofe-dbt - run: dbt deps && dbt build && dbt docs generate - run: sofe-dbt check . --fail-on high # blocks the PR with the cost shown
A full reproducible walkthrough (CDK stack + golden data + dbt + check) lives in the repo's GUIDE.md.