Final-year systems project

A self-healing loop for small web products — detect, diagnose, patch, canary, verify, or roll back.

Fuses behavioral and error telemetry to catch three classes of production issues, investigates root cause on an Aurora-adapted agent, patches and previews the fix, then closes the loop through a human-gated canary release with automatic rollback — the one part of this pipeline nobody has published closed end to end.

incident_pipeline.trace simulated run
Detect
Investigate
Diagnose
Patch
Preview
Approve
Canary
Verify
Promote
booting trace…
Illustrative walkthrough of the state machine — not live production telemetry.
System design

A flow, not a stack

The earlier “layers” mixed together tools, pipeline stages, and infrastructure. This map separates the three jobs AcesoLoop performs during one incident.

How to read this

Follow the solid arrows left to right. Observe finds the problem, investigate proves the cause and prepares a fix, then release safely decides whether the fix stays or rolls back.

one incident · three jobs · explicit handoffs
replaying incident trace
External / shared infra
Adapted from Aurora
AcesoLoop build
01 / OBSERVE & FUSE

Find a real incident

Combine what users did with what the system reported, then decide whether the change is abnormal.

PostHog

Funnels + session replay

Sentry / logs

Errors + stack traces

Playwright

Synthetic user sessions

normalize + correlate
Signal fusion

One comparable signal shape for every source.

Anomaly detector

Classifies crashes, slowdowns, or silent flow breakage.

incident + evidence
02 / INVESTIGATE & REPAIR

Prove the cause and fix it

Use Aurora to investigate, then make the patch earn its way through tests and a real preview.

Aurora investigation agent

Gather evidence, rank hypotheses, explain the root cause.

Patch Agent

Generate a scoped code change with an open-weight model.

Test-suite gate

Reject and retry if the existing tests fail.

Preview verification

Replay synthetic traffic and confirm the original anomaly clears.

verified patch
03 / RELEASE SAFELY

Let it reach users carefully

A person approves the evidence; a small canary proves the release before full promotion.

Human approval

Review diagnosis, diff, and preview evidence.

5% canary

Send a weighted slice of traffic to the new version.

Same anomaly detector, live

Monitor recovery and catch any new anomaly.

HEALTHYPromote to 100%
ANOMALOUSRollback automatically
Target app & traffic
Productioncurrent version
Previewpatched version
Canary5% traffic
Promote / rollback100% or last known good
Key distinction: Aurora is one component inside job 02. Jobs 01 and 03 are the AcesoLoop contribution around it.
Pipeline detail

The nine states, end to end

The map above shows the three jobs. These nine states are the exact Postgres-backed state machine inside those jobs.

01Signal ingestion & fusion
Behavioral events (PostHog) and error events (Sentry/logs) are normalized into one signals table with a common shape, plus a rolling signal_windows aggregate polled every 1–5 minutes. This is the single highest-leverage piece of engineering — every downstream stage depends on signals being comparable regardless of source.
02Anomaly detection
Deliberately not an LLM call. A rules-plus-z-score classifier routes each window into one of three classes: frontend runtime error, latency regression, or a broken user flow — the third defined specifically by a funnel drop with no matching error, which is exactly what fusion catches and error-only tools miss.
03Root-cause investigation
The Aurora-adapted agent gathers evidence — stack traces, recent diffs, PostHog session replays — through multi-step tool use and produces a ranked, justified diagnosis rather than a single-shot guess.
04Repair generation
The Patch Agent writes a scoped fix and runs the existing test suite. No patch proceeds if tests fail — reject and retry, not a soft warning.
05Preview verification
The patch deploys to an isolated preview environment; synthetic traffic re-runs to confirm the original anomaly signal actually clears before any human sees it.
06Human approval
The single mandatory checkpoint. A reviewer sees the evidence ledger — diagnosis, diff, preview results — and approves or rejects in under a minute.
07Canary release
Approved patches ship to a small weighted slice of traffic. The same fused anomaly detector from stage 02 keeps monitoring — no separate canary-specific logic to drift out of sync.
08Recovery verification & rollback
Promote to full release only if the original signal clears and no new anomaly appears. Otherwise, automatic rollback, logged with the reason — this is the part of the loop with the thinnest prior art anywhere.
09Evidence ledger
Every stage's decision and justification is recorded and attached to the PR — the primary research artifact, and the dataset every ablation in this project is measured from.
Build accounting

How much of this is already Aurora

A reasoned estimate, weighted by effort share per stage — not a measured number, since this combination hasn't been built before.

~28%
of total engineering effort, covered by adapting Aurora — concentrated almost entirely in investigation
~72%
genuinely new build — signal fusion, patch validation, canary, and verified rollback
Signal ingestion
35 / 65
Anomaly detection
5 / 95
Investigation agent
75 / 25
Patch + test gate
35 / 65
Preview deploy + verify
0 / 100
Human approval
15 / 85
Canary release
0 / 100
Recovery + rollback
0 / 100
Evidence ledger
30 / 70
Covered by Aurora
New build
Zero paid infrastructure

Tech stack — free or self-hosted, no exceptions

Reasoning model
Qwen3.6-27B
Dense, Apache 2.0, runs on a single consumer GPU via Ollama. Diagnosis synthesis and the harder "broken flow" class.
self-hosted
Coding model
Qwen3-Coder-Next
80B total / ~3B active MoE, Apache 2.0. 70.6% SWE-bench Verified on ~46GB unified memory.
self-hosted
Fallback inference
OpenRouter / Groq free tier
Rate-limited, zero-cost hosted access to open-weight models for demo and eval bursts.
free tier
Investigation substrate
Aurora
Self-hosted, Apache 2.0 agentic RCA platform — LangGraph orchestration, sandboxed execution, guardrails.
open source
Behavioral telemetry
PostHog
Session replay, funnels, feature flags — self-hosted or free cloud tier.
free tier
Error telemetry
Sentry
Free developer tier, or plain structured application logs.
free tier
Target app
Next.js + TS + Postgres
Seeded with 8–12 deliberate bugs across all three issue classes.
open source
Preview / canary
Vercel Hobby + Argo Rollouts
Vercel free tier for preview deploys; kind/minikube + Argo Rollouts if deeper canary control is needed.
free tier
Orchestration
XState / Postgres FSM + BullMQ
State machine plus Redis-backed worker queue so long-running steps never block a request thread.
open source
What to measure against

Real published comparators, not invented targets

80–85%
ARGUS root-cause accuracy, ~94% faster time-to-insight — primary Investigation Agent comparator
2026, open-source-stack AIOps deployment
59–69%
SWE-bench Pro frontier range as of June 2026 — JS/TS specifically lower and more variable
Standardized public set / top active vendor
41.2%
Best system's PassRate on long-horizon, cross-interface tasks — justifies verifying real outcomes, not self-reports
WeaveBench, 2026
new
Canary promote/rollback correctness and human-intervention trend — no published comparator exists yet
AcesoLoop's own first-of-kind result