Using the output

CI/CD recipes

SpecTracer is a CLI that reads files and writes files. There's no agent, no service, and no state — which makes CI integration almost boring. These are working pipelines you can paste and adjust.

The shape of it

Every integration, on every platform, is the same four moves:

1

Run your suites with machine-readable output

JUnit XML for unit and integration, Cucumber JSON for E2E. Most runners already do this, or need one flag.

2

Make sure all the files land in one workspace

If your suites run in parallel jobs, that means artifacts up and artifacts down. This is the only genuinely fiddly part.

3

Run spec-tracer

One command. It exits 1 if a gate you configured is breached.

4

Publish the report

Upload the HTML as an artifact; optionally read the JSON to comment on the PR or post metrics.

Don't skip step 2's implications Tests that fail to run produce no results file, and a missing results file is silently treated as "zero tests for that layer" — not as an error. A crashed E2E job can therefore make coverage drop without failing the report step. Make the test jobs themselves fail loudly.

GitHub Actions

Single-job version — simplest, and correct for most repos.

.github/workflows/ci.yml
name: CI
on: [push, pull_request]

jobs:
  test-and-report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: astral-sh/setup-uv@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: uv sync

      - name: Unit tests
        run: uv run pytest tests/unit --junitxml=reports/unit.xml

      - name: Integration tests
        run: uv run pytest tests/integration --junitxml=reports/int.xml

      - name: E2E tests
        run: uv run behave features/ --format json -o reports/e2e.json

      - name: Generate SpecTracer report
        run: uv run spec-tracer

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: spectracer-report
          path: |
            reports/spectracer-report.html
            reports/spectracer-report.json
Note the if: always() on the upload Without it, a gated spec-tracer run that exits 1 skips the upload — and you lose the report exactly when you most need to read it.
spectracer.config.json to go with it
{
  "features": ["./features"],
  "unit":        { "": ["./reports/unit.xml"] },
  "integration": { "": ["./reports/int.xml"] },
  "e2e":         { "": ["./reports/e2e.json"] },
  "output":      "./reports/spectracer-report.html",
  "output_json": "./reports/spectracer-report.json"
}

Parallel test jobs

If unit, integration, and E2E run as separate jobs, each uploads its results and a final job downloads them all into one workspace before running SpecTracer.

.github/workflows/ci.yml — report job
  report:
    needs: [unit, integration, e2e]
    if: always()          # still report when a suite failed
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5

      - name: Collect every suite's results
        uses: actions/download-artifact@v4
        with:
          path: reports
          merge-multiple: true

      - run: uv sync && uv run spec-tracer

Point the config at the directories rather than individual files so it doesn't need to know how many artifacts arrived:

spectracer.config.json
  "unit": { "": ["./reports"] },   // every .xml found, recursively
  "e2e":  { "": ["./reports"] },   // every .json found, recursively
Careful with a shared directory Scanning the same directory for both unit and integration registers every XML file under both layers and double-counts your pyramid. Give each layer its own subdirectory (reports/unit, reports/integration) whenever more than one layer emits the same file extension.

PR comments

Two levels. Link to the artifact, or read the JSON and post the actual numbers.

Link to the report

.github/workflows/ci.yml
      - name: Comment PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## SpecTracer coverage report\n\n[Download the report](${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})`
            })

Post the numbers

Better, because it shows up in the timeline without anyone downloading anything. Requires output_json.

.github/workflows/ci.yml
      - name: Comment PR with completion
        if: always() && github.event_name == 'pull_request'
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          REPORT=reports/spectracer-report.json
          PERCENT=$(jq -r '.summary.completion.percent' "$REPORT")
          HEALTH=$(jq -r '.summary.health.status'       "$REPORT")
          REASONS=$(jq -r '.summary.health.reasons[]? | "- " + .' "$REPORT")

          gh pr comment ${{ github.event.pull_request.number }} --body "$(cat <<EOF
          **SpecTracer** · declared tests matched: **${PERCENT}%** · health: **${HEALTH}**

          ${REASONS}
          EOF
          )"

Gating the build

SpecTracer ships two gates, and you can build any third one you need on top of the JSON.

GateConfigFails when
Test failures"error_on_failure": trueAny collected result is a failure.
Coverage floor"fail_on": ["progress"]Declared-tests-matched drops below progress_threshold_amber.
Inverted pyramid"fail_on": ["pyramid"]Unit count is below integration + E2E combined.
Slow E2E suite"fail_on": ["e2e_runtime"]Total E2E time exceeds e2e_duration_red_seconds.

Amber never gates — only red does. All gates are additive: any one of them exiting non-zero fails the step.

A custom gate

For anything the built-ins don't cover — an absolute coverage floor, a no-regression rule — read the JSON:

fail if completion is under 75%
      - name: Enforce coverage floor
        run: |
          PERCENT=$(jq -r '.summary.completion.percent' reports/spectracer-report.json)
          if (( $(echo "$PERCENT < 75" | bc -l) )); then
            echo "::error::Scenario coverage ${PERCENT}% is below the 75% floor"
            exit 1
          fi
Don't scrape the HTML The HTML report's markup is not a stable interface and will change between versions. The JSON is governed by a published JSON Schema; script against that.

SpecTracer stores nothing between runs — deliberately. Every run emits a summary.completion / summary.pyramid / summary.health snapshot, and whatever metrics system you already run owns the history, retention, and charting.

.github/workflows/ci.yml
      - name: Generate report
        run: uv run spec-tracer

      - name: Post coverage metrics
        env:
          METRICS_ENDPOINT: ${{ secrets.METRICS_ENDPOINT }}
        run: |
          curl -X POST "$METRICS_ENDPOINT" \
            -H "Content-Type: application/json" \
            -d @reports/spectracer-report.json

That's the whole integration: one request reading a file that already exists. Swap the curl for your dashboard's SDK if it has one — Datadog, Grafana Cloud, an internal service, even a spreadsheet webhook. The only requirement on the far end is something that can store a time series and draw it.

GitLab CI

.gitlab-ci.yml
stages:
  - test
  - report

variables:
  UV_PYTHON: "3.12"

unit-tests:
  stage: test
  script:
    - uv run pytest tests/unit --junitxml=reports/unit.xml
  artifacts:
    when: always
    paths: [reports/unit.xml]

integration-tests:
  stage: test
  script:
    - uv run pytest tests/integration --junitxml=reports/int.xml
  artifacts:
    when: always
    paths: [reports/int.xml]

e2e-tests:
  stage: test
  script:
    - uv run behave features/ --format json -o reports/e2e.json
  artifacts:
    when: always
    paths: [reports/e2e.json]

spectracer-report:
  stage: report
  when: always
  needs: [unit-tests, integration-tests, e2e-tests]
  script:
    - uv sync
    - uv run spec-tracer
  artifacts:
    when: always
    paths: [reports/spectracer-report.html]
    expose_as: Coverage Report

expose_as puts a direct link to the report on the merge request page.

Jenkins

Jenkinsfile (declarative)
pipeline {
    agent any

    stages {
        stage('Install') {
            steps { sh 'uv sync' }
        }
        stage('Run tests') {
            parallel {
                stage('Unit') {
                    steps { sh 'uv run pytest tests/unit --junitxml=reports/unit.xml' }
                }
                stage('Integration') {
                    steps { sh 'uv run pytest tests/integration --junitxml=reports/int.xml' }
                }
                stage('E2E') {
                    steps { sh 'uv run behave features/ --format json -o reports/e2e.json' }
                }
            }
        }
        stage('SpecTracer report') {
            steps { sh 'uv run spec-tracer' }
        }
    }
    post {
        always {
            archiveArtifacts artifacts: 'reports/spectracer-report.*',
                             allowEmptyArchive: true,
                             fingerprint: true
        }
    }
}

Rollout playbook

Getting the pipeline right is the easy half. Getting a team to care about the number is the other half.

1

Write the feature files first

Whoever owns requirements — QA, product, tech lead — writes the .feature files and assigns each scenario an @id:. This is the scope definition, and it should happen before any tagging of tests.

2

Tag tests opportunistically

Don't schedule a tagging sprint. Add @scenario: tags as people touch tests anyway, and require them on new tests in code review. The number climbs on its own.

3

Publish the report before gating it

Upload the artifact on every build for a sprint or two. Let people see their own number move before it can block them.

4

Gate, then raise the bar

Turn on error_on_failure and the health checks you care about. Ratchet progress_threshold_amber upward as the real number climbs past it.

Commit the config, ignore the report spectracer.config.json is your coverage contract and belongs in version control. The generated HTML and JSON are build artifacts — add them to .gitignore.
Microservices in one repo Give each service a module key and have its scenarios declare @require-unit:<service>. Only that service's results will satisfy the requirement, so one team's coverage can't paper over another's. See module scope.