Testing

Run these checks before you say a change is green. They are the same commands CI runs, so a local pass predicts a CI pass. For a tester walking through a lesson rather than running suites, see TESTING.md.

Before you start

pnpm install
python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt
pnpm build:libs                # some packages resolve through ./dist, absent on a fresh checkout
export STUDIUM_EPHEMERAL=1     # keep learner state in memory, as CI does

Without STUDIUM_EPHEMERAL=1, anything that starts the stdio MCP server writes learner state under ~/.studium. Always use the virtualenv's Python, never the system one.

The gates

Command What it covers
pnpm lint ESLint over every .ts and .tsx file; zero errors expected
pnpm build the whole workspace compiles
pnpm test every TypeScript package and app (pnpm -r test), including the VS Code drivers, then the status-line script and the release tools (pnpm test:tools; no network, nothing is sent)
.venv/bin/python tools/pytest_all.py every Python suite, one pytest process each
bash tools/bundle_lint.test.sh contract tests of the lesson bundle linter
node apps/lesson-mcp/scripts/headless_smoke.mjs a bare MCP client drives a real lesson over stdio

A run of tools/pytest_all.py succeeds with exit code 0 and a last line of the form N suites, 0 failing.

TypeScript suites

Each package and app has a test script; pnpm test runs them all. Most use Vitest, with tests next to the code (*.test.ts). To run one unit:

pnpm --filter @studium/lesson-session-host test
pnpm --filter @studium/lesson-mcp test
pnpm --filter @studium/lesson-mcp exec tsc --noEmit

Notable suites:

Suite What it proves
apps/lesson-mcp/src/learn-and-review.e2e.test.ts from "learn X" to the first question in six Studium tool calls, then a review a week later after a server restart
packages/lesson-session-host (host.*.test.ts, public-scene.test.ts) redaction of answer keys, release of passages on leave, study plan, durable state
apps/lesson-server (server.redaction.test.ts) the HTTP server never sends an answer key
apps/teacher-dashboard dashboard pages and chart text alternatives (Vitest); pnpm --filter @studium/teacher-dashboard test:e2e runs Playwright integration tests
packages/activity-presentation the viewer language helpers (node --test on ui/activity-language.test.mjs)

VS Code drivers

The extension is tested by drivers: plain TypeScript scripts under apps/studium-vscode/src/*.driver.ts, run with tsx, that exercise the extension without launching VS Code. Some render the lesson pane in jsdom and click its controls; extension-host.driver.ts runs the extension against a fake vscode module and the real MCP server. The list of drivers that pnpm test runs is the test script of apps/studium-vscode/package.json. To run one:

pnpm --filter studium-vscode test                                   # typecheck, then every driver
node_modules/.pnpm/node_modules/.bin/tsx apps/studium-vscode/src/webview-pane.driver.ts

Python suites

tools/pytest_all.py exists because two services both import a module named tests.conftest, so a single pytest process over services/ aborts at collection. The wrapper runs one process per suite. Suites are discovered, not listed: a top-level folder that holds test files directly is one suite (tools/); otherwise each child folder holding tests is a suite (services/planner, packages/lesson-bundle-schema, eval/tests, and so on). The run fails if any test file is reached by no suite.

.venv/bin/python tools/pytest_all.py              # all suites
.venv/bin/python tools/pytest_all.py -- -x        # extra arguments go to every pytest process
.venv/bin/python -m pytest services/planner -q    # one suite

A bare pytest at the root only collects packages/ (see pytest.ini): do not use it as a gate.

Other checks

Command When
node tools/bundle_lint.mjs <lesson.json> after editing a lesson by hand
node_modules/.pnpm/node_modules/.bin/tsx fixtures/atomistique-l1/drive_check.mjs (and the other *_check.mjs there) diagnostics of the reference lesson; they always exit 0, so read their output
node design/scripts/build-tokens.mjs --check and node design/scripts/check-contrast.mjs after changing design tokens
node apps/lesson-mcp/scripts/http-smoke.mjs <new folder> after changing the HTTP adapter; writes its report into a folder that must not exist yet
tools/audit-*.mjs UI audits on synthetic fixtures; see the audit record

The evaluation harness under eval/ measures lessons and runs; it is not a gate.

CI

.github/workflows/ci.yml runs on every push and on pull requests to main, with STUDIUM_EPHEMERAL=1 for every job.

Job Runs Steps
Lint + Test every push install with --frozen-lockfile, blocking audit of critical production advisories, lint, build, TypeScript tests, bundle lint, headless MCP smoke, Python suites, ASTRA validator, planner graph build, Observatory tests
Tester onboarding path every push ./setup.sh from a fresh clone, then the VS Code driver suite
VSIX + standalone MCP server every push package and check the VSIX, start the bundled lesson-mcp.cjs over stdio, upload both (see release)
Tauri build, Demo M6 manual dispatch only the frozen desktop app on macOS and Linux, then the end-to-end demo

Dependabot opens grouped weekly update pull requests for npm, pip and GitHub Actions.

Rules for tests

Tests are part of the contract. Changing an existing test is a decision you state in the commit message, never a side effect of a fix. A new behaviour comes with a test that fails without it.

View the source of this page on GitHub