# Trace v0.3.2 — adversarial and real-system stress report

Executed 3 November 2026 on Windows/Python 3.13.12. Full final suite: **101 passed, 0 skipped**, 118.99s. Environment: pytest 8.4.1, Mem0 2.2.1, embedded Qdrant client 1.18.0, MCP SDK 1.29.0. No paid services or hosted models were used.

## What failed before hardening

The initial 21-check adversarial run had 15 passes and six failing checks, revealing five distinct weaknesses. This is an observed initial result, not a claim that all earlier tests were exhaustive:

| Weakness | Observed original behavior | Correction |
| --- | --- | --- |
| Duplicate JSON keys | An ambiguous inventory produced PASS | Reject duplicate keys and non-finite JSON constants |
| Truncated chunked HTTP | IncompleteRead escaped and could abort verification/cleanup | Normalize HTTP framing errors to minimized INCONCLUSIVE evidence |
| Slow-drip body | Peer extended the body read beyond its configured budget, ending PASS | Stream with monotonic body budget checks |
| Delayed cleanup propagation | Healthy deletion became INCONCLUSIVE because cleanup was checked only once | Poll each cleanup inventory through the configured deletion window |
| Process interruption | A durable checkpoint was not available through external_get after restart | Surface orphan checkpoints as interrupted/INCONCLUSIVE, never completed evidence |

The retained adversarial set was extended to 22 passing regressions, including killing an actual CLI process during a request, restarting the workflow, recovering all four issued IDs, and explicitly cleaning their records. The suite also verifies one-active-job backpressure and a running state before completion. SQLite final evidence remains separate from untrusted orphan checkpoints; a checkpoint cannot be saved as a confirmed regression.

## Seeded HTTP fault benchmark

Seed 20261103; 17 labelled families × 10 repetitions, with varied 3/7/25 document counts and 120/160/200ms observation windows. Expected verdicts were defined by the independent test application's behavior, not inferred from Trace's output.

| Ground-truth category | Cases | Observed verdict |
| --- | --- | --- |
| Healthy immediate or valid propagation delay | 20 | 20 PASS |
| Observable vector/cache/retrieval/partial-delete/resurrection fault | 50 | 50 FAIL |
| Missing/malformed/ambiguous/error/timeout/control-loss evidence | 100 | 100 INCONCLUSIVE |

All 170 matched their expected in-contract label. Observed false PASS: 0; false FAIL on healthy cases: 0; unnecessary INCONCLUSIVE on healthy cases: 0. These are repeated local cases from known fault families, not 170 independent applications or an unbiased production sample. No production accuracy percentage or confidence interval is claimed.

Full-lifecycle HTTP-fixture latency: median 0.427s; p95 (nearest rank) 0.883s; maximum 1.029s. Eight concurrent runs × 25 target records all passed and cleaned up, in 1.168s total wall time. These lightweight fixtures are not comparable to Mem0 inference/service latency.

## Actual Mem0 load, pagination and restart

Unmodified Mem0/Qdrant was run in a separate process. Seeded 137 unrelated background memories, requiring at least three raw 64-item scroll pages. Hard-stopped and restarted the service against the same persistent data: both inventories remained 137 and retrieval succeeded.

Then ran four simultaneous Trace lifecycles and six sustained lifecycles, each with 25 target documents plus one unrelated control. All ten passed and cleaned their active records; the 137 background memories survived. Background records were finally deleted in bounded batches and both active inventories were zero. Total records created: **397**. Trace's per-run target cap remains 25; the 137-record test exercises background isolation, enumeration, persistence and batched cleanup rather than claiming a new 137-document Trace scenario.

Full-lifecycle latency, including ingestion, verification and cleanup: median **2.98s**, p95/maximum **7.95s** across ten runs. Under concurrency, the staging gateway serializes Mem0 operations with a lock; queued work is included in those timings. Cold seeding took 13.15s including initialization. This is a bounded local workload, not a long-duration soak, distributed benchmark or hard-real-time SLA.

The first load attempt passed its application assertions but failed temporary-directory teardown on a Windows file lock. Added bounded handle-release retries for the verified owned scratch path. The final load run exited successfully with filesystem teardown complete. The earlier scratch directory was also removed after verifying its exact owned path. No staging processes were left running.

Mem0 retained 794 audit-history rows during the workload; these are outside the active-memory deletion contract. Mem0-visible memories and raw Qdrant points are two views of one backing store, and no answer cache exists. Lexical local embeddings and zero LLM calls mean this does not test neural retrieval quality or extraction.

## Explicit blind-spot demonstrations

Both challenges returned PASS under the configured observable contract while the independent harness could still find a copy:

1. A vector appeared after the observation window ended. A finite sample window cannot promise absence forever.
2. A copy remained in a store that the adapter never exposed. Declared inventories cannot prove the absence of undisclosed stores.

These are two real limits of what this test protocol can establish, not successes silently folded into the 170-case result. Earlier Mem0 testing also found retained canary text in audit history. Reports must keep all three limits visible. Next coverage expansion should prioritize independently observed stores/history and longer declared soak windows.

## Reproduce

```text
python -m pytest -q
python tests/run_stress_benchmark.py --repeats 10 --seed 20261103 --output stress-metrics.json
```

In the optional Mem0 environment with mem0ai==2.2.1 and qdrant-client==1.18.0:

```text
python tests/run_mem0_load.py --work-dir scratch-load --output mem0-load.json
```

The source ZIP contains these drivers, the independent adversarial HTTP application, retained regression tests, JSON metrics and final JUnit receipt. Tests mutate only disposable synthetic data in local test apps. Remote customer deployments, packet loss/TLS/DNS behavior, slow response headers, graph memory, long-duration soak and production availability remain unvalidated. Socket connection/header timeouts are not an absolute whole-request deadline; body streaming is the hardened path measured here.