Tuning Cipher Pol: From Noise to Signal in Scope-Drift Detection

My scope-drift detector flagged every new component file. I ignored 90% of its alerts within a week.

What Cipher Pol is

Cipher Pol is Stella Protocol’s scope-drift monitor — the skill whose one job is to catch surface-area changes that don’t appear in the approved PRD. Named after the One Piece secret intelligence agency. It fires when an agent is about to add a route, an endpoint, a schema, or a third-party dependency. It classifies the drift as INTEL (noted, not blocking), ALERT (review required), or INTERCEPT (block until PM approves). It logs everything to brain/scope-changes.md so the PM has a full audit trail come review time.

In theory this is the cleanest primitive in the framework. In practice, v0.5.0 shipped with a definition of “scope drift” that made it nearly useless.

The v0.5.0 design

Original rule in the skill file:

Trigger Cipher Pol when the agent is about to create any
new file or modify the import graph in a way that introduces
new module boundaries.

On paper: comprehensive. If new code appears, flag it. PM reviews. If it’s legit, PM approves. If not, INTERCEPT.

The noise problem

I built House of Riddle on Stella v0.5.0. 21 routes, 15+ components, roughly 40 utility and hook files across the run of the project. Cipher Pol fired on essentially every one of them.

A sample hour of brain/scope-changes.md:

- ALERT: new file components/ui/Card.tsx
- ALERT: new file components/ui/CardHeader.tsx
- ALERT: new file components/ui/CardBody.tsx
- ALERT: new file hooks/useToast.ts
- ALERT: new file lib/format-date.ts
- ALERT: new file types/riddle.ts

None of those are scope drift. Card.tsx is a presentational primitive inside an approved page. useToast is plumbing. format-date is a three-line utility. The PRD did not need to list them because they are not user-facing surface area.

By day three, brain/scope-changes.md was 400 lines of refactor noise. I stopped reading it. When a real scope drift landed — the agent wanted to add a /marketplace route that was not in the PRD — the alert was buried at line 287 between two component splits. I approved the route mentally during a session, never logged the rationale, and caught it during review two weeks later as an unexplained live route.

Signal-to-noise was maybe 1:8. Alert fatigue desensitized both me and the agent.

Diagnosis

The rule was measuring code structure churn, not scope drift. Those are different things. A refactor that splits one 800-line component into six files is zero scope drift and high structural churn. A single new page that reads from a new DB table is one file of churn and a significant scope drift.

Cipher Pol was firing on the wrong signal.

The tuning

I redefined scope drift in the skill file as user-facing or externally-observable surface area. Specifically:

• Pages and routes (new URL a user can reach)
• API endpoints (new shape exposed to clients)
• Database tables, columns, or migrations
• External integrations (new third-party SDK, API, or webhook)
• New environment variables the app requires to run

Explicitly not scope drift:

• Component files
• Hooks, utilities, helpers
• Type definitions
• Tests
• Refactors that move code between files without changing the set of routes / endpoints / tables

The one-paragraph change in protocol/governance/cipher-pol.md:

Scope drift is a change to the product's external surface:
routes, endpoints, schema, integrations, required env.
Internal restructuring — components, utilities, types,
tests — is not scope drift and must not trigger Cipher Pol.
Refactors that leave the route/endpoint/schema set identical
are structural churn, not drift.

The skill file trigger logic now checks whether the file being created is in an app/ route segment, a pages/api/ or app/api/ handler, a migration file, or a known integration directory. Component files under components/ and utilities under lib/ no longer fire.

Measured improvement

Next project after the fix: I re-ran the same kind of build on top of tuned Cipher Pol. Over a two-week run, Cipher Pol fired 11 times total. Nine were real scope drifts I had to consciously approve or reject. Two were edge cases (a webhook handler that looked like a utility; an integration file in an unusual path) that I added to the trigger rules.

Signal-to-noise went from roughly 1:8 to roughly 1:1. More importantly, I started reading the log again. Alerts got attention because alerts were rare.

The second-order effect I didn’t predict: the agent behaved differently too. Under v0.5.0, Cipher Pol alerts were so frequent that the agent had effectively learned to race past them — “flagged, noted, continuing.” Under the tuned version, a Cipher Pol alert started functioning as a genuine checkpoint. The agent would pause, surface the drift in chat, and wait for an explicit decision. The rule did not change what the agent could do; the density of alerts changed what the alerts meant.

The same dynamic shows up in monitoring generally. A pager that goes off every hour is a broken pager. Nobody tunes Cipher Pol’s definitions in their head — they just stop reading the logs. Tightening the definition was not a restriction, it was a restoration of meaning.

I write up what the tuned Cipher Pol caught on the next project in the pre-launch audit post.

Key Takeaways

1. Governance rules written on paper break on first contact with real data. A rule that sounds rigorous can still be useless if rigor points at the wrong thing.
2. Tune on production, not theory. If your governance layer produces alerts you ignore, the alerts are not the problem — the definition is. Fix the definition before you add more alerts.
3. Rarity is what gives an alert meaning. A pager that fires every hour gets muted. A pager that fires once a week gets attention. The density of alerts matters as much as their content.

Satellite: Morgans (this post) · Cipher Pol (subject) · Pipeline: ITERATE — Observation → Morgans