Skip to content
AITesting|novetest
GitHub Get started →
Main For Agents Products · novetest Products · Test Case Generator Products · novetest Console soon Docs · novetest Docs · novetest Console soon Blog Contact ★ Star on GitHub Get started →
Docs/Cheat Sheet

Cheat Sheet

The one-page operational companion. This is the card to keep open while you work the develop-with-novetest loop — one screen of setup, the loop, the decision flow, and every routing table. It compresses the full docs; it does not replace them (depth lives in Quick Start, Understanding Results, and Troubleshooting).

Everything below is verbatim-true to the novetest/v1 contract; all sample values are real CLI captures.

Session setup (once)

bash
# Install (Linux / macOS)
curl -fsSL https://ailovestesting.com/products/novetest/install.sh | sh
powershell
# Install (Windows PowerShell)
irm https://ailovestesting.com/products/novetest/install.ps1 | iex
bash
# Pin JSON output for the whole session (agents: always do this)
export NOVETEST_OUTPUT=json

Health probe — "is novetest usable right now":

bash
command -v novetest >/dev/null || { echo "novetest missing"; exit 1; }
NOVETEST_OUTPUT=json novetest --version | \
  jq -e '.ok == true and .schema == "novetest/v1"' || { echo "version envelope malformed"; exit 1; }
if [ -d ./.novetest ]; then
  NOVETEST_OUTPUT=json novetest status | jq -e '.ok == true' || { echo "status envelope malformed"; exit 1; }
fi

Every command returns the same envelope frame — exactly six top-level keys, JSON-sorted: command, data, errors, ok, schema, warnings (schema is always "novetest/v1"; there is no top-level version, verb, or exit_code field — the exit code is the process status).

The loop (every cycle)

  1. novetest init — once per project: pins the engine, creates .novetest/. Safe to re-run (no-op on an existing store).
  2. novetest test — runs the suite, stores the run, derives coverage / regression / localization, returns ranked data.recommendations[].
  3. Route on the exit code, then on the top recommendation's category (decision flow below).
  4. Act, then re-run novetest test. Optional drill-down: novetest inspect <run_id>.

The decision flow

novetest test
└─ exit code?
   ├─ 0 → done — all green. (Targeted run? verify it collected tests:
   │        inspect → data.run_summary.summary_counts.total > 0.)
   ├─ 3 → tests FAILED or ERRORED. ok stays true — NOT a crash.
   │      └─ top category (lowest priority number)?
   │         ├─ regression_with_localization / investigate_location / coverage_gap
   │         │     → pick the best finding by slots.rank ASC, then
   │         │       slots.score_normalized DESC (the array is NOT score-ordered)
   │         │     → open slots.file @ slots.primary_line → fix → re-run
   │         ├─ investigate_regression → slots.test_id is the newly-failing
   │         │       test → fix → re-run
   │         ├─ flaky_suspected → only fires with --reruns N (N ≥ 1);
   │         │       treat as flaky, stabilize the test
   │         ├─ unavailable_analysis → informational (slots.reason_per_stage);
   │         │       fix the failures, re-run
   │         └─ all_green → exclusive; never coexists (exit 0 case)
   ├─ 2 → usage error — fix the call and retry:
   │        uninitialized → novetest init · not-found → novetest memory list
   │        invalid-flag → read errors[0].message · adapter-invalid-target →
   │        fix the target expression · engine-ambiguous → init --engine <name>
   │        confirm-required → --confirm ONLY with operator approval
   ├─ 4 → engine problem:
   │        engine-missing / engine-misconfigured → install from
   │        data.engine_readiness.issues[] if policy permits, else surface
   │        no-engine-detected → cd into a real project, novetest init
   │        adapter-<kind> → engine-level failure; surface
   ├─ 5 → store-corrupt / store-wipe-failed → surface to operator
   │        (do NOT wipe .novetest/ on your own — it destroys history)
   └─ 1 → cli-error → capture the envelope, report; do not retry blindly

The same flow as code:

python
env = json.loads(stdout)
if exit_code == 0:
    pass                                      # all green
elif exit_code == 3:                          # ok: true — data, not a crash
    recs = sorted(env["data"]["recommendations"], key=lambda r: r["priority"])
    top = recs[0]; cat = top["category"]
    if cat in {"investigate_location", "regression_with_localization", "coverage_gap"}:
        # NOT score-ordered: pick by rank (asc), then score_normalized (desc)
        best = min((r for r in recs if r["category"] == cat),
                   key=lambda r: (r["slots"]["rank"], -r["slots"]["score_normalized"]))
        target = (best["slots"]["file"], best["slots"]["primary_line"])
    elif cat == "investigate_regression":
        test_id = top["slots"]["test_id"]     # newly-failing test
    elif cat == "unavailable_analysis":
        reasons = top["slots"]["reason_per_stage"]   # informational
elif exit_code in (2, 4, 5, 1):
    handle_error(env["errors"][0]["code"])    # table below

Exit codes (route on this FIRST)

ExitokMeaning
0trueCLI succeeded; tests passed. Data-level unavailable outcomes also land here.
1falseUncaught internal error (cli-error). Capture + report; don't retry blindly.
2falseBad input — fix the call and retry.
3trueCLI succeeded; your tests failed or errored. Product data, not a tool error.
4falseNative engine not ready, or an adapter invocation failed.
5falseProject Store corrupt/unreadable. Surface.

The invariant that bites: ok: true does NOT imply exit 0 — exit 3 is the normal failed-tests path. Never treat exit 3 as a crash.

Recommendation categories (route on category, sort by priority asc)

prioritycategoryAction
1regression_with_localizationNewly-failing test ∩ ranked location — the strongest fix target.
2investigate_locationSBFL-ranked suspicious location. Open slots.file @ slots.primary_line.
3investigate_regressionNewly-failing transition vs baseline (slots.test_id).
4coverage_gapUncovered lines overlap a suspect location.
5flaky_suspectedOnly via novetest test --reruns N (N ≥ 1) when a failed run's replay diverges.
6unavailable_analysisTests failed but a stage was unavailable (slots.reason_per_stage). Informational.
7all_greenNo failures, no regressions. Exclusive — never coexists with another.

There is no severity field; priority (int, lower = higher) is the only ranking. Within a location-bearing category, pick by slots.rank (ascending) then slots.score_normalized (descending) — never by array position. Never parse summary.

errors[0].code → one-line recovery

Exiterrors[0].codeRecovery
2uninitializednovetest init, retry.
2not-foundnovetest memory list, pick a real 26-char ULID.
2invalid-flagAllowed values are listed in errors[0].message.
2adapter-invalid-targetFix the target expression (drop/quote the leading dash).
2engine-ambiguousnovetest init --engine <name> from data.candidates[].
2confirm-required--confirm ONLY with operator approval (destructive).
4engine-missing / engine-misconfiguredInstall hints are in data.engine_readiness.issues[] (NOT errors[0].details).
4no-engine-detectedcd into a real project (see data.candidates[]), novetest init there.
4adapter-<kind>Engine-level failure — read the stderr tail in errors[0].message; surface.
5store-corrupt / store-wipe-failedSurface. Do NOT auto-recover (destroys history).
1cli-errorCapture the envelope, report. Do not retry blindly.

Pin against code, never the message. The engine-readiness codes are the state verbatim (engine-missing — no doubled prefix, no engine-not-ready).

Micro-reference

  • Retry safety: every verb is retry-safe; test/run create a new run_id per call; reset --confirm is the only destructive verb. novetest does no network I/O at invocation time.
  • stage_eligibility.localization carries the SBFL mode string (sbfl_per_test / sbfl_aggregate / failure_proximity) when a finding exists — NOT the word "available".
  • unavailable ≠ error: an analysis stage that can't produce facts reports it as data on a successful (exit 0) command.
  • Output modes: --output flag > NOVETEST_OUTPUT env > TTY autodetect. ndjson = one envelope per line. Only JSON/NDJSON shapes are stable — never parse text mode.