AI-Driven PR Reviews

Overview

AI-assisted PR review adds a layer between the author and the human reviewer — an automated pass that catches common issues before a person spends time on them. Done right, it reduces the volume of obvious comments human reviewers make (style violations, missing null checks, obvious logic errors), freeing human attention for the things AI cannot evaluate: intent, architecture, team norms, and long-term maintainability.

AI review is not a substitute for human review. It is a pre-screening pass that makes human review faster and more focused. A PR that passes an AI review is not approved — it is ready for a human to review meaningfully.

For how code review is structured more broadly — author responsibilities, reviewer responsibilities, and turnaround SLAs — see Code Review Best Practices.

Why It Matters

Surface-level issues waste reviewer time. A human reviewer who spends 10 minutes commenting on null checks, error handling, and missing edge cases has 10 fewer minutes for architectural feedback. AI catches the surface layer reliably; humans should be reserved for the deeper layer.

Feedback is faster when it comes from automation. AI review runs the moment a PR opens — not when a reviewer gets to it. An author who receives immediate feedback on obvious issues can fix them before any human has looked, reducing review round-trips.

Review quality is inconsistent without a baseline. Some reviewers are thorough; others move quickly. An automated first pass creates a consistent baseline — the same set of checks runs on every PR regardless of who reviews it or how much time they have.

AI finds patterns humans skim over. In a 200-line diff, a human reviewer may miss an inconsistent error handling pattern on line 174. AI tools scan the entire diff without attention fatigue.

Standards & Best Practices

AI review augments human review — it does not replace it

An AI approval is not an approval. Every PR still requires at least one human review before merge. The AI review is a pre-screening layer, not a gate. Configure your repository to require human approval regardless of AI review status.

AI review handles the surface layer; humans handle the substance

Set the AI reviewer's scope explicitly

AI reviewers produce noise when asked to review everything at maximum sensitivity. Configure what the tool should and should not flag:

• Enable: security issues, missing error handling, code smells, documentation gaps
• Disable or lower sensitivity on: style comments already covered by your linter, comments on auto-generated files, trivial naming suggestions

Most AI review tools support path-based exclusions — exclude generated files, migration files, and vendor code from review.

Human reviewers acknowledge AI comments, not re-review them

When an AI review flags an issue that a human reviewer would also flag, the human acknowledges it rather than repeating it. This avoids double-commenting and keeps the review thread clean. If the AI comment is wrong or irrelevant, the human reviewer dismisses it with a note — which also trains calibration over time.

Don't merge based on AI review alone

Never configure branch protection rules that allow merge on AI review approval without human approval. The risk is not theoretical — AI tools miss context, misunderstand intent, and produce false positives. A human must be in the loop before merge.

How to Implement

Step 1 — Choose an AI review tool

Three approaches, each with different integration models:

CodeRabbit — dedicated AI review service. Connects to GitHub/GitLab and posts inline PR comments automatically. Understands the whole diff, not just individual files.

Claude Code — run a review prompt against the PR diff from the terminal or as a CI step. Flexible but requires prompt engineering and CI setup.

GitHub Copilot PR review — available to GitHub Copilot Enterprise subscribers. Integrated natively into the GitHub PR interface.

Step 2 — Install and configure CodeRabbit (or equivalent)

If using CodeRabbit:

1. Install the GitHub App at github.com/apps/coderabbitai
2. Grant access to the repositories you want reviewed
3. Add .coderabbit.yaml to the repository root:

# .coderabbit.yaml
language: 'en-US'

reviews:
  profile: 'chill' # Options: chill, assertive
  request_changes_workflow: false # Never block merge — surface issues only
  high_level_summary: true
  poem: false
  review_status: true
  collapse_walkthrough: false

  path_filters:
    - '!**/*.generated.ts'
    - '!**/migrations/**'
    - '!**/vendor/**'
    - '!**/*.lock'

  auto_review:
    enabled: true
    drafts: false # Skip draft PRs
    base_branches:
      - 'main'

The request_changes_workflow: false setting is important — AI should never block a merge through the changes-requested state. It surfaces issues; humans decide whether they block merge.

Step 3 — Set up Claude Code as a CI review step (alternative)

For teams that prefer a script-based approach over a SaaS tool:

# .github/workflows/ai-review.yml
name: AI PR Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    if: github.event.pull_request.draft == false
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get PR diff
        id: diff
        run: |
          git diff origin/${{ github.base_ref }}..HEAD > pr_diff.txt
          echo "lines=$(wc -l < pr_diff.txt)" >> $GITHUB_OUTPUT

      - name: Run AI review
        if: steps.diff.outputs.lines < 1000
        run: |
          claude --print "Review this PR diff for: security issues, missing error handling, logic errors, and missing edge cases. Do not comment on style or formatting — that is handled by the linter. For each issue, cite the file and line. Be concise." < pr_diff.txt > review_output.txt

      - name: Post review comment
        if: steps.diff.outputs.lines < 1000
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const review = fs.readFileSync('review_output.txt', 'utf8');
            if (review.trim()) {
              github.rest.issues.createComment({
                issue_number: context.issue.number,
                owner: context.repo.owner,
                repo: context.repo.repo,
                body: `## AI Pre-Review\n\n${review}\n\n*This is an automated pre-screening. A human review is still required before merge.*`
              });
            }
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The lines < 1000 guard skips AI review on oversized diffs — AI review produces noise on very large changes and encourages authors to break them up instead.

Step 4 — Establish how the team interacts with AI review comments

Decide and document:

1. Authors resolve clearly-wrong AI comments before requesting human review. If the AI flagged something incorrect, add a brief response explaining why — this also helps calibrate the tool over time.
2. Authors acknowledge valid AI comments before requesting human review. "Fixed in latest commit" or "Handled — see line 47" clears the thread.
3. Human reviewers do not re-comment on things the AI already flagged correctly. A thumbs-up emoji on the AI comment is sufficient acknowledgement.
4. Human reviewers dismiss irrelevant AI comments with a note. This creates a feedback signal for tuning the tool's sensitivity.

Step 5 — Review calibration quarterly

AI review tools improve with configuration. Every quarter, review:

• What categories of AI comments are consistently valid? (Raise their priority)
• What categories are consistently noise? (Suppress or lower sensitivity)
• Are there file types or paths that should be excluded?
• Is the tool flagging things the linter should be catching instead?

Tools & Templates

AI review tool comparison

Prompt template for Claude Code PR review

Use this prompt when reviewing your own diff before opening a PR, or as the input for a CI step:

You are reviewing a pull request diff. Your job is to surface issues a human reviewer should be aware of — not to approve or reject the PR.

Review scope:
- Security: hardcoded secrets, injection risks, insecure defaults
- Correctness: logic errors, off-by-one errors, unhandled null/undefined
- Error handling: uncaught exceptions, silent failures, missing error types
- Edge cases: empty input, zero, negative values, concurrent access

Do not comment on:
- Formatting, indentation, or whitespace
- Style preferences already enforced by a linter
- Variable naming unless critically misleading

Format: For each issue, state the file path, a one-line description of the problem, and a one-line suggestion. If there are no issues, say "No issues found."

.coderabbit.yaml for a TypeScript Next.js project

language: 'en-US'

reviews:
  profile: 'chill'
  request_changes_workflow: false
  high_level_summary: true
  poem: false

  path_filters:
    - '!**/*.generated.ts'
    - '!**/.next/**'
    - '!**/node_modules/**'
    - '!**/*.lock'
    - '!**/migrations/**'

  auto_review:
    enabled: true
    drafts: false

  tools:
    github-checks:
      enabled: true
    languagetool:
      enabled: false # Disable prose checks — this is a code repo
    ruff:
      enabled: false # Not a Python project
    eslint:
      enabled: true

Common Pitfalls

Treating AI approval as human approval. Never configure branch protection to allow merge on AI review alone. AI tools lack the context to make merge decisions. They surface issues; humans decide.

Ignoring AI review comments entirely. If the team treats AI comments as background noise from the start, the tool provides no value and creates friction. Establish the norm that AI comments are acknowledged before human review begins.

Configuring AI review to block merges. Setting the AI tool to request_changes_workflow: true means a single AI false positive can block a PR until dismissed. AI review should always be advisory, not blocking.

Running AI review on oversized diffs. AI review on a 2,000-line diff produces a long list of comments with low signal-to-noise ratio, encourages the team to ignore all of them, and rewards large PRs over small ones. Set a line limit; skip AI review for diffs that exceed it and ask the author to split the PR.

Double-commenting. When human reviewers repeat issues the AI already flagged, the PR thread becomes cluttered and authors have to respond to the same issue twice. Agree that human reviewers acknowledge AI comments rather than restating them.

Not tuning sensitivity over time. A tool configured at defaults will produce noise in categories specific to your stack. Review the signal-to-noise ratio quarterly and suppress categories that are consistently irrelevant to your codebase.