Script-Based Validation & Reporting

Overview

Manual quality checks are skipped under deadline pressure, forgotten when the person who ran them leaves the team, and inconsistent between engineers. Scripts make checks repeatable, schedulable, and auditable — the same check runs the same way every time, regardless of who triggered it or what the schedule looks like.

Script-based validation is broader than linting or testing. It covers any quality check that can be expressed as a script: checking for security vulnerabilities in dependencies, verifying that all environment variables are documented, measuring coverage trends, auditing API response times, or generating reports that surface metrics no single dashboard captures.

This page covers how we write validation scripts, how we integrate them into CI and scheduled workflows, and how we produce reports that are actionable rather than just informative.

Why It Matters

Manual checks are forgotten. A check that exists only in someone's memory or a document nobody reads is a check that doesn't run. Scripts that are committed to the repository and run in CI or on a schedule run every time, without anyone remembering to run them.

Reports make the invisible visible. A codebase's test coverage trend, its dependency vulnerability count, and its bundle size over time are all invisible without measurement. Scripts that produce these reports turn invisible metrics into visible ones — and visible metrics get acted on.

Automation scales where manual processes don't. A manual dependency audit on a repository with 200 dependencies takes hours. A script that runs pnpm audit and posts the results to Slack takes seconds. The automation does not get tired, does not miss packages, and does not depend on someone having the time to run it.

Consistent reporting supports engineering conversations. When a technical debt discussion is grounded in a script-generated report — "we have 47 files with complexity scores above 15, and they account for 63% of our bugs this quarter" — it is more persuasive and more actionable than an assertion backed by gut feel.

Standards & Best Practices

Scripts are production code

A validation script that is fragile, undocumented, or exits with the wrong code does more harm than no script. Apply the same standards as application code: handle errors, exit with meaningful codes, document what the script does and what it expects.

#!/usr/bin/env bash
set -euo pipefail   # Exit on error, undefined variables, pipe failures

set -euo pipefail should be the first line of every bash script. It makes scripts fail loudly rather than continuing silently after an error.

Scripts must be idempotent

Running a script twice should produce the same result as running it once. Scripts that have side effects (creating files, posting to external services) should be safe to re-run. If a script creates a report file, it should overwrite it, not append to it.

Exit codes are the API

A script's exit code is its interface with CI. Exit 0 means success; anything else means failure. Never exit 0 unconditionally. If the check fails, the exit code must be non-zero so CI can block or alert appropriately.

# Bad: always exits 0, CI cannot detect failure
pnpm audit || true

# Good: exits non-zero if high or critical vulnerabilities exist
pnpm audit --audit-level=high

Separate reporting scripts from enforcing scripts

Some scripts enforce a standard and block CI on failure. Others collect data and report without blocking. Keep these separate — combining them creates scripts that are too blunt (block on anything) or too permissive (report but never block).

Scripts belong in scripts/ and are documented

Every script in scripts/ has a comment block at the top: what it does, what it requires, and how to run it. A script without documentation is a script nobody will maintain.

#!/usr/bin/env bash
# scripts/check-env-docs.sh
#
# Checks that every environment variable in .env.example has a comment explaining it.
# Fails if any variable is undocumented.
#
# Usage: bash scripts/check-env-docs.sh
# Requires: .env.example in the repository root

How to Implement

Step 1 — Identify checks that are currently manual or inconsistent

Walk through the team's informal quality practices and ask: what checks are run by some engineers but not others? What checks are run manually before a release but might be forgotten? What metrics does the team talk about but never actually measures?

Common candidates:

• Dependency vulnerability audit
• Bundle size compared to previous release
• Dead code detection (exports that are never imported)
• Coverage trend (not just current coverage, but direction)
• API contract validation (does the implementation match the OpenAPI spec?)
• Environment variable documentation completeness
• Stale branches (branches with no commits in 30+ days)

Step 2 — Write the script with proper error handling

Start from a working bash or Node.js script before wiring it to CI. Apply the principles from the Standards section: set -euo pipefail at the top, a meaningful exit code on failure, and a documentation header describing what the script checks, what it requires, and how to run it.

Keep scripts focused: one check per script. A single script that audits dependencies, checks bundle size, and runs coverage does too many things — it fails for unclear reasons and is harder to skip selectively.

Ready-to-use script templates for common checks are maintained in the Validation Script Starter Kit (link to be published when available).

Step 3 — Add enforcing scripts to CI

Enforcing scripts block merge on failure. Add each enforcing script as a required job in the CI pipeline, triggered on pull requests to the main branch. Required jobs should be fast — if a check takes more than a minute, consider whether it belongs in CI on every PR or in a scheduled run instead.

Keep enforcing and reporting jobs in the same workflow file so the team has a single place to see all quality gates.

Step 4 — Add reporting scripts to CI as non-blocking steps

Reporting scripts collect data and post it somewhere useful without blocking merge. Mark reporting jobs with continue-on-error: true so a transient failure in report generation (network, rate limit) does not block a PR. Only enforcing checks should block.

Step 5 — Schedule overnight or weekly reports

Some metrics are better tracked on a cadence than on every PR: coverage trends, stale branch counts, weekly vulnerability summaries. Use scheduled CI runs (or Claude Code Routines for teams using Claude Code) for reports that surface trends rather than gate individual changes.

Tools & Templates

Common validation scripts by category

Script templates

Ready-to-use bash and Node.js starter scripts for the most common checks (dependency audit, bundle size, coverage reporting, stale branches) — along with CI wiring examples — are maintained in the Validation Script Starter Kit (link to be published when available). These are not reproduced here because they change as tooling evolves and keeping them in the playbook creates a maintenance burden. The principles in this page are stable; the templates are not.

Common Pitfalls

Scripts that exit 0 on failure. The most common mistake: a script that catches errors internally and always exits successfully. CI cannot distinguish a passing check from a suppressed failure. Use set -e and never unconditionally exit 0.

Scripts without documentation. A script with no comment explaining what it does, what it requires, and how to interpret its output will be deleted or worked around when it starts failing. Document every script at the top.

Reporting scripts that block merges. A reporting script that occasionally fails on transient issues (flaky network, rate limit) should never block a merge. Mark reporting steps with continue-on-error: true. Only enforcing checks should block.

Manual scripts that are "run before release." If a script is not in CI, it will eventually not be run before a release. Either put it in CI or accept that it will be skipped. There is no in-between that reliably works.

Reports nobody reads. A weekly Slack report that nobody opens for three weeks is noise. Before building a report, confirm there is a person or process that will act on it. Reports that drive no action should be removed.

Overly strict enforcement on day one. Adding a bundle size check that enforces the current limit plus zero tolerance will fail immediately on the next legitimate feature. Set thresholds with headroom; tighten them deliberately over time.