# Trace v0.4.0: test your staging app

Trace's public website is a demo and download page. Real checks run on your computer. The local dashboard must remain on loopback.

## Start

Install Python 3.11 or newer, extract the ZIP, and double-click `Start_Trace.bat` on Windows, or run `bash start_trace.sh` on macOS/Linux. These launchers open your browser after the server binds successfully. Terminal users can run:

```text
python scripts/launch.py serve --open-browser
```

Open the printed local URL if your desktop does not support automatic browser opening. Use **Try the reference lab** to learn the result types, or **Test my staging app** for your configured adapter.

## Connect → review → run

1. Start the supplied Mem0 staging gateway, or your own implementation of the Trace HTTP contract. Mem0 is an optional dependency installed in a separate environment; setup commands appear in the wizard.
2. In **My staging app**, select the integration and enter its adapter URL and a unique connection name. Remote staging requires HTTPS. This is an adapter endpoint, not an arbitrary database or hosted Mem0 account URL.
3. Choose Quick (2s), Standard (15s), or Extended (120s). Advanced settings expose the synthetic document count, timeout, environment-variable name for a token, and cache coverage declaration. Never enter the actual token in the dashboard.
4. Click **Test connection and preview scope**. This sends only an inventory request for a fresh random namespace. It never ingests, warms query caches, or deletes records. A successful connection probe is not a completed verification.
5. Review source, vectors, cache, history, and workers. **Checked** means configured API observations are planned; excluded and unavailable components have no verified zero count. Inventory completeness remains the adapter's assertion, not independent discovery.
6. Acknowledge the synthetic writes, then run. Trace records the issued IDs before network writes, observes the complete window, checks the unrelated control, and attempts cleanup.

The supplied Mem0 configuration observes active memories and vector points as two views of the same store. It has no answer cache. Audit history and adapter ID mappings remain outside the deletion contract. Worker completion and copies appearing after the declared window are not proven absent.

## Understand the receipt

The result separates verification, coverage, and cleanup. A remaining vector is an observed failure. Missing or malformed evidence is inconclusive. A verification observation can pass while cleanup remains unresolved; the final receipt is then inconclusive. Inspect issued IDs before rerunning unresolved cleanup.

Results contain next actions, inventory samples, events, the contract hash, and a JSON download. Event text is explanatory; recorded observations determine the verdict.

## Save → fix → replay → compare

Save a **completed FAIL** as a regression. Fix your staging application while preserving its adapter URL and contract. Select **Rerun saved contract**, then compare the original failure with the replay. Only the same target and contract can be compared. A URL, observation window, coverage, or token-variable change needs a new connection name; immutable saved assertions are never silently weakened.

Completed reports and regressions survive restart. Refresh the ledger or resume progress using the run ID. Interrupted checkpoints remain inconclusive and cannot be saved as confirmed regressions.

## CI gate

Register the staging target and save a failing regression once in the local data directory. Provision that registry and saved contract for your CI runner using your own controlled artifact process; retain the same URL and contract. Set any token in the CI environment. The runtime history database must not be included in a public release.

```text
python scripts/launch.py --data-dir .trace-ci external-ci --regression SAVED_REGRESSION_ID --output reports/trace-ci.json
```

Repeat `--regression` for multiple checks. Exit codes: **0 PASS; 1 FAIL; 2 INCONCLUSIVE/configuration or missing-contract error**. Preserve the JSON and generated local evidence as CI artifacts. No entry means no success: a missing saved regression exits 2.

## Distributed validation

`tests/run_distributed_qdrant.py` is opt-in. It creates three uniquely labelled Docker Qdrant v1.18.0 peers, an independent SQLite source store and separate cache, then drives the real Trace lifecycle, a vector-only failure, replay/comparison, and an actual peer restart. It uses `wait=true`, strong write ordering, and read consistency `all`. It accepts no production cluster or collection. Its owned containers and network are removed after the test.

```text
docker pull qdrant/qdrant:v1.18.0
python tests/run_distributed_qdrant.py --work-dir scratch-cluster --output reports/distributed-qdrant.json
```

This verifies the specified local topology. It is not evidence for partition tolerance, leader elections, every consistency setting, hosted/customer deployments, physical erasure, or model unlearning. See [Qdrant's consistency documentation](https://qdrant.tech/documentation/scaling/consistency-guarantees/).