Testing

Civix Testing Guide

This document describes the automated testing setup for the Civix monorepo (web app, Interview API, and mobile app), what was implemented, and how to run and extend tests.

Overview

AreaToolWhat it coversWhere
Web API (health, interview routes)VitestRoute handlers: health, interview start/chat/evaluate/complete; status codes, response shape, validationapps/web
Web UI + API from browserPlaywrightLanding page, GET /api/health, docs page loadapps/web
Mobile API clientJestapi-client.ts: isAPIAvailable, startInterview, sendMessage, evaluateAnswer, completeInterview, error handling (429, 404, 500, network)apps/mobile
Mobile E2E (optional)MaestroInterview tab → Mock Interview → disclaimer → difficulty selectionapps/mobile

No real network or API keys are required for unit/integration tests; E2E tests run against a running app or dev server.

What was implemented

1. Web app – API integration tests (Vitest)

  • Config: apps/web/vitest.config.ts (path alias @/ for imports).
  • Files:
    • app/api/health/route.test.ts – GET health returns status: "ok", endpoints list includes interview routes.
    • app/api/openapi/route.test.ts – validates OpenAPI spec shape, supported interview mode enums, and phase/timing metadata fields.
    • app/api/interview/start/route.test.ts – POST start returns 200 with sessionId, phase, message; 400 for invalid mode or invalid language; accepts en, es, fa, ps, so; GET returns ok.
      • Includes signed-request checks in enforce mode (missing signature headers rejected, valid v2 signatures accepted, replay nonce rejected, legacy v1 rejected in enforce mode).
    • app/api/interview/chat/route.test.ts – 400 for missing fields, 404 for unknown session, phase-lock metadata (phaseLocked, timers), deterministic no-repeat N-400 sequencing, minimum required N-400 count before phase advance, and enforce-mode signed/unsigned checks.
    • app/api/interview/evaluate/route.test.ts – 400/404 handling, phase-locked evaluation with OUT_OF_ORDER protection, civics active question/token sequencing, threshold checks for standard vs exemption modes, and enforce-mode signed/unsigned checks.
    • app/api/interview/complete/route.test.ts – 400/404, 200 with passed/scores/duration/recommendations, 403 on device mismatch, plus enforce-mode signed/unsigned checks.
    • lib/openai-budget.test.ts – validates daily request cap and monthly budget guardrails for OpenAI usage policy.
  • How it works: Tests import and invoke Next.js route handlers (GET/POST) with NextRequest; no server start. Sessions use in-memory store; rate limiting uses Redis when configured and in-memory fallback otherwise.
  • Run: From apps/web: npm run test (or npm run test:watch).

2. Web app – E2E tests (Playwright)

  • Config: apps/web/playwright.config.ts – testDir: ./tests/e2e, baseURL: http://localhost:3000, webServer runs npm run dev so the dev server starts automatically in CI.
  • Files:
    • tests/e2e/landing.spec.ts – Home page loads, title and hero/CTA visible, no critical JS errors.
    • tests/e2e/health.spec.ts – GET /api/health returns 200 and JSON with status: "ok".
    • tests/e2e/docs.spec.ts – /docs and /docs/how-to-use load, main content visible.
  • How it works: Playwright starts the Next.js dev server (or reuses it when not in CI), then runs browser tests against that URL.
  • Run: From apps/web: npm run test:e2e. First run installs Chromium if needed (npx playwright install chromium).

3. Mobile app – API client unit tests (Jest)

  • Files: apps/mobile/__tests__/api-client.test.ts, apps/mobile/__tests__/algorithm.test.ts, apps/mobile/__tests__/i18n-parity.test.ts, apps/mobile/__tests__/multilang-parity.test.ts, and others.
  • What it tests:
    • algorithm: SM-2 (calculateQuality, sm2Calculate), selectQuestions (review/weak/learn/coverage/senior/difficulty/stateProfile), selectExamQuestions (20 standard, 10 senior), checkExamPassed, updateQuestionStats (including skipped), formatNextReview; calculateMasteryProgress returns correct separate counts for learning, due, overdue (no double-count); review mode selects only due questions when present; coverage mode includes unseen when they exist.
    • competency milestones: computeCompetencySnapshot(...) classifies learner tier (foundation, developing, proficient, interview_ready) from mastery/accuracy/overdue/risk signals.
    • deterministic shuffle: shuffleArraySeeded(...) provides reproducible ordering for tests/simulation while preserving randomized production behavior.
    • pass-risk layer: Priority scoring now includes a risk component that amplifies weak/overdue/high-error questions to improve pass-rate outcomes.
    • senior regression coverage: Senior mode excludes state-dependent questions without profile context, includes them when profile exists, prioritizes evaluatable senior exam items, and keeps algorithm SENIOR_QUESTION_IDS synchronized with senior65 flags in the canonical EN bank.
    • See docs/ALGORITHM-REPORT.md for algorithm behavior.
    • i18n-parity: Every locale (en, es, fa, ps, so) has the same string keys as English; catches missing translations.
    • multilang-parity: Question banks for es, fa, ps, so keep canonical ID parity with EN; unapproved overlays fall back to EN; translation progress is bounded. Somali has full 128 approved overlays (so-overlays.ts), so getBank('so') returns questions and answers in Somali.
    • api-client:
    • isAPIAvailable: false when EXPO_PUBLIC_API_URL is empty (no fetch); true when GET /api/interview/start returns 200 or 405; false when fetch throws.
    • startInterview: POST to /api/interview/start with body { mode, language, weakTopics }, headers Content-Type, X-Device-ID; response shape.
    • sendMessage / evaluateAnswer / completeInterview: Correct URL, method, body, and response shape.
    • Error handling: 429 → APIClientError RATE_LIMITED; 404 → SESSION_NOT_FOUND; security codes (MISSING_SIGNATURE_HEADERS, ABUSE_THROTTLED, etc.) map to explicit client guidance; 500 and network reject → appropriate message and code.
    • canUseAIInterview: false when lastAIInterviewDate is today; otherwise delegates to isAPIAvailable.
  • Practice/quiz design: Answer choices are shuffled per question (shuffleArray in lib/algorithm.ts) in Practice and Quiz so the correct answer is not always in the same position—reducing position bias and supporting retention.
  • How it works: fetch and env are mocked in the test file; AsyncStorage is already mocked in jest.setup.ts. No real API or network.
  • Run: From apps/mobile: npm test (all tests) or npm test -- --testPathPattern=api-client.

4. Mobile app – E2E (Maestro, optional)

  • Flow: apps/mobile/.maestro/flows/interview-smoke.yaml.
  • Steps: Launch app → tap "Interview" → tap "Mock Interview" → tap "I Understand" (disclaimer) → assert "Choose Difficulty" visible.
  • How it works: Maestro drives the app on a device/simulator. Requires Maestro CLI and a built/installed app. On first run the disclaimer appears; after accepting once, subsequent runs may skip that step (flow still taps "I Understand" if present).
  • Run: Install Maestro, then from apps/mobile: npm run test:e2e or maestro test .maestro/flows/interview-smoke.yaml.
  • Docs: apps/mobile/.maestro/README.md.

How to run (quick reference)

CommandWhereDescription
npm run testapps/webVitest: API route tests
npm run test:watchapps/webVitest watch mode
npm run test:e2eapps/webPlaywright E2E (starts dev server)
npm run test:interview-apiapps/webLive API smoke flow (health -> start -> chat (conversation) -> evaluate (test phases) -> complete)
npm testapps/mobileJest: all unit tests (includes api-client)
npm test -- --testPathPattern=api-clientapps/mobileJest: api-client only
npm run test:e2eapps/mobileMaestro Interview smoke (app must be installed)

For signed-request enforce deployments, run smoke test with the signing secret:

cd apps/web
INTERVIEW_SIGNING_SECRET=your-secret npm run test:interview-api

CI and pre-release

  • Web: Run npm run test and npm run test:e2e from apps/web before deploy. E2E will start the dev server; for CI, set CI=true so Playwright uses one worker and retries.
  • Mobile: Run npm test (and optionally npm run preflight) from apps/mobile. Maestro E2E is optional and typically run on a device/simulator or in a separate mobile CI job.
  • Production readiness: See docs/08-PRODUCTION-READINESS.md; automated tests are part of the quality gates.

Extending tests

  • New API route (web): Add app/api/<route>/route.test.ts; import the GET/POST handler and call it with a NextRequest. Use path alias @/ (already in vitest.config).
  • New E2E (web): Add a spec under apps/web/tests/e2e/; use page.goto('/path') and request.get('/api/...') with baseURL.
  • New api-client usage (mobile): Add cases in __tests__/api-client.test.ts with mocked fetch and env.
  • New Maestro flow (mobile): Add a YAML file under apps/mobile/.maestro/flows/ and run with maestro test <file>.

Summary

Automated tests cover: (1) Web Interview API (health, start, chat, evaluate, complete) via Vitest; (2) Web landing, health endpoint, and docs via Playwright; (3) Mobile algorithm + API client behavior and errors via Jest; (4) Optional mobile Interview flow via Maestro. All of this supports deployment confidence and regression detection without requiring production API keys or live services for unit/integration runs.