#60 - feat(test): blackbox test suite — 21 checks, green on local + testing - Kalle/ATM

## Plan Deliver the "Blackbox suite ≥ 8 tests, all green on testing" Release Hardening item. The ATM skill mandates `scripts/blackbox_test.py` as the iteration gate every PR must pass, and it was missing from the repo. ## Outcome - Added `scripts/blackbox_test.py` — a **dependency-free Python suite** (stdlib only, no `pip install` needed). - **10 numbered scenarios / 21 assertions** covering: 1. Health endpoint reachable + schema-valid 2. Lesson list reachable + non-empty + schema-valid 3. Lesson has expected content fields 4. Auth: tauri-login happy path returns teacher user 5. Auth: wrong password rejected (401) 6. TTS: synthesises playable audio (accepts wav/mpeg/mp3) 7. TTS: rejects empty text (422) 8. TTS: rejects unauthenticated requests (401) 9. Progress: "lesson explored" write is idempotent (count increments exactly once, `first_explored_at` stable) 10. Progress: write requires authentication (401) - **Deployment-agnostic**: uses the first lesson in the list (testing uses UUIDs, local uses slugs); accepts any playable audio MIME (stub emits wav, real backend emits mpeg). - **Targets**: `--target {local,testing}` or `--base <url>`; auto-detect prefers testing if reachable. ## Impact on release goals - Release Hardening → **"Blackbox suite ≥ 8 tests, all green on testing" moves `[ ]` → `[x]`**. - Suite has 21 assertions across 10 scenarios (exceeds the ≥8 requirement). - Every future iteration now has a reproducible, runnable gate. A failing test is a documented P0 blocker per the skill's verification rubric. ## Blackbox - [x] `python3 scripts/blackbox_test.py --target testing` → **21/21 green** ✅ - [x] `python3 scripts/blackbox_test.py --target local` → **21/21 green** ✅ - Unit suite unaffected (Python-only addition; no TS changes). ## Six-perspectives check - [x] **Product:** makes the release gate real and reproducible; protects the felt Aha-moment by catching regressions before users see them. - [x] **Engineering:** minimal additive diff (one new file), no new dependencies, no security surface, idempotent writes only. - [x] **Psychology:** n/a (test infrastructure). - [x] **Biomechanics:** n/a (test infrastructure). - [x] **Anatomy:** n/a (test infrastructure). - [x] **Feldenkrais:** the suite asserts non-judgemental language indirectly by verifying the "explored" (not "completed") progress semantics — protecting the principle that the lesson is never finished. ## Risk - Demo credentials (`teacher@atm.app` / `teacher123`) are hardcoded in the suite — acceptable: they are seeded demo accounts documented in `data/users.json`, not real secrets, and the suite is read/idempotent-write only. - The progress-write test mutates state on the demo user; this is by design (idempotent) and matches the existing Beta scope. ## Checklist - [x] Branch off latest `dev` - [x] Conventional Commits (2 commits: feat + docs) - [x] Patchlog entry appended - [x] Roadmap status updated (Blackbox suite `[ ]` → `[x]`) - [x] No secrets / no large binaries - [x] Self-reviewed --- _This PR was created by an AI agent (OpenHands) on behalf of the maintainer. Commits authored with `Co-authored-by: openhands <openhands@all-hands.dev>`._

KnallerKalle commented

2026-08-13 00:50:26 +00:00

(Migrated from github.com)

Plan

Deliver the "Blackbox suite ≥ 8 tests, all green on testing" Release Hardening item. The ATM skill mandates scripts/blackbox_test.py as the iteration gate every PR must pass, and it was missing from the repo.

Outcome

Added scripts/blackbox_test.py — a dependency-free Python suite (stdlib only, no pip install needed).
10 numbered scenarios / 21 assertions covering:
1. Health endpoint reachable + schema-valid
2. Lesson list reachable + non-empty + schema-valid
3. Lesson has expected content fields
4. Auth: tauri-login happy path returns teacher user
5. Auth: wrong password rejected (401)
6. TTS: synthesises playable audio (accepts wav/mpeg/mp3)
7. TTS: rejects empty text (422)
8. TTS: rejects unauthenticated requests (401)
9. Progress: "lesson explored" write is idempotent (count increments exactly once, first_explored_at stable)
10. Progress: write requires authentication (401)
Deployment-agnostic: uses the first lesson in the list (testing uses UUIDs, local uses slugs); accepts any playable audio MIME (stub emits wav, real backend emits mpeg).
Targets: --target {local,testing} or --base <url>; auto-detect prefers testing if reachable.

Impact on release goals

Release Hardening → "Blackbox suite ≥ 8 tests, all green on testing" moves [ ] → [x].
Suite has 21 assertions across 10 scenarios (exceeds the ≥8 requirement).
Every future iteration now has a reproducible, runnable gate. A failing test is a documented P0 blocker per the skill's verification rubric.

Blackbox

python3 scripts/blackbox_test.py --target testing → 21/21 green ✅
python3 scripts/blackbox_test.py --target local → 21/21 green ✅
Unit suite unaffected (Python-only addition; no TS changes).

Six-perspectives check

Product: makes the release gate real and reproducible; protects the felt Aha-moment by catching regressions before users see them.
Engineering: minimal additive diff (one new file), no new dependencies, no security surface, idempotent writes only.
Psychology: n/a (test infrastructure).
Biomechanics: n/a (test infrastructure).
Anatomy: n/a (test infrastructure).
Feldenkrais: the suite asserts non-judgemental language indirectly by verifying the "explored" (not "completed") progress semantics — protecting the principle that the lesson is never finished.

Risk

Demo credentials (teacher@atm.app / teacher123) are hardcoded in the suite — acceptable: they are seeded demo accounts documented in data/users.json, not real secrets, and the suite is read/idempotent-write only.
The progress-write test mutates state on the demo user; this is by design (idempotent) and matches the existing Beta scope.

Checklist

Branch off latest dev
Conventional Commits (2 commits: feat + docs)
Patchlog entry appended
Roadmap status updated (Blackbox suite [ ] → [x])
No secrets / no large binaries
Self-reviewed

This PR was created by an AI agent (OpenHands) on behalf of the maintainer. Commits authored with Co-authored-by: openhands <openhands@all-hands.dev>.

Rows
Columns

feat(test): blackbox test suite — 21 checks, green on local + testing #60