Pebellslearn by building
0%
RoadmapsMatrixCoursesSandboxSign in
CI/CD for Machine Learning  ›  Capstone Project

CML: Model Reports in Pull Requests

CML 13 minCML is a tool that runs inside CI to post metrics, plots and comparisons as a comment on the pull request, so reviewers judge a model change by its results.

CML: Model Reports in Pull Requests

A green tick tells a reviewer that the churn pipeline ran. It does not tell them whether the new model is better. They could download the logs and dig for numbers, but they will not. CML (Continuous Machine Learning) brings the numbers to them: a comment on the pull request with metrics, the change against main, and plots.

The rule

Put the evidence where the decision is made.

The decision to merge happens on the pull request page, so the model's results should appear there. CML is a command-line tool, usually run inside GitHub Actions or GitLab CI, that turns a Markdown file into a pull request comment:

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # full history, so origin/main can be compared
      - uses: iterative/setup-cml@v2
      # ... setup-python, pip install, dvc pull as before ...
      - run: dvc repro
      - name: Report
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          echo "## Churn model report" > report.md
          dvc metrics diff origin/main --md >> report.md
          echo "![Confusion matrix](./confusion.png)" >> report.md
          cml comment create report.md

Step by step:

  • dvc repro trains and writes metrics.json and confusion.png.
  • dvc metrics diff origin/main --md prints a Markdown table of each metric on main, on this branch, and the difference. It needs main's history, which is why checkout uses fetch-depth: 0.
  • cml comment create posts report.md to the pull request, uploading the image it references. REPO_TOKEN gives it permission.

CML can also start cloud machines as temporary CI runners (cml runner launch), so a heavy training job can use a GPU that GitHub's standard runners do not have.

The shortcut

A good model report answers three questions, in this order: How good? Better than main? Where does it fail?

The report on "add playlist_count feature"Step 1 of 4

  1. How good?

    Recall 0.74, precision 0.66 on the frozen test set.

The trap is a report with only the new number. "Recall 0.74" means little until you see what main scored on the same data.

Drills

Answer fast — trust the rule.

quick drill

What does cml comment create report.md do?

quick drill

Why does the checkout step use fetch-depth: 0 in this workflow?

quick drill

A report says only "Recall: 0.74". What is the most important thing missing?

quick drill

The churn pipeline needs a GPU that standard GitHub runners lack. Which CML feature helps?

Remember

  • Put the evidence where the decision is made: on the pull request.
  • A report answers: how good, better than main, where it fails.
  • Trap: a lone metric with no comparison against main.

Practice — level up

0/1 solved

Write your answer, then compare it with a rubric and a model answer.

ExplainWhy a report in the PR

Argue for adding a CML report to the churn repo's CI. Explain what the report would contain, how it gets onto the pull request, and what decision it helps the reviewer make.