The masked diff is the only diff — privacy by construction, and where it stops

The finding I trust most in Membrane AI isn't a vulnerability. It's an info-level note, local-heuristic:masked, whose message starts a secret was masked out of this diff upstream. Membrane reviews a code change in stages, and only one stage passes it on for model review. By default the reviewer behind that stage is a plain heuristic, and it raises this note when the text it received contains a [MASKED: placeholder where a credential used to be. Next to the scanner's own secret findings, the note reads like a receipt: the model-facing stage reviewed the change without ever holding that key.

The repo's handover records that receipt from the all-container end-to-end run: a webhook accepted with 202, and a rejected verdict with two blocking secret findings, a SQL-concatenation warning and local-heuristic:masked beside them. I didn't re-run the container stack for this post, but I did re-run every unit and regression test mentioned below. The post covers the design behind that receipt, a contract bug that graceful degradation turned into one warning line, and three edges of the guarantee I found while writing it up. One of them takes the receipt apart.

Membrane AI is my attempt at guardrails for AI-generated code. A diff arrives through a webhook or a gRPC stream, crosses Kafka, and an orchestrator runs it through a chain of stages: a deterministic analyzer that flags secrets and risky patterns and masks the secrets, then an advisory semantic stage that posts the change to a Python semantic service for model review. A reporter turns the verdict into a GitHub commit status and a PR comment. That pipeline, a membrane CLI and an HTTP gateway for MCP tool calls are built and tested; a VS Code extension that shells out to the CLI compiles but has no tests yet.

The models are pluggable and, by default, stand-ins. The semantic service has a local tier, a transparent heuristic until a vLLM endpoint is configured, and a premium tier, Claude and Gemini in consensus, which stays off until someone enables it and supplies keys. The code is MIT-licensed on GitHub.

The obvious way to keep secrets away from a model is to mask before every model call: a mask(diff) in the local-model adapter, another in the premium adapter, one more in whatever tier comes next. That holds until someone, possibly me months later, writes an adapter that assumes the diff was cleaned upstream. The guarantee is then only as strong as the least careful adapter, and a human reviewing each new one is all that stands between a key and two model vendors.

Membrane's orchestrator takes the other route, recorded as decision D-024. A stage doesn't only return findings; it can return a rewritten artifact, which the orchestrator swaps in before the next stage runs. The port in internal/ports/ports.go states the contract in capitals: a non-empty MaskedDiff "REPLACES the diff seen by all later stages". The loop in internal/app/process.go is what makes that true:

var findings []domain.Finding
current := sub // local copy: stages may redact the diff for later stages
for _, stage := range uc.stages {
	out, err := stage.Analyze(stageCtx, current)
	if err != nil {
		return uc.runFallback(ctx, sub)
	}
	findings = append(findings, out.Findings...)
	if out.MaskedDiff != "" {
		current.Diff = out.MaskedDiff
	}
}

In the shipped wiring, the analyzer runs first and is the only stage that returns a masked diff: every credential match becomes [MASKED:<rule>]. The semantic stage runs after it and forwards sub.Diff unchanged, into a wire field named masked_diff. It has no masking code and needs none: by the time it runs, there is no raw diff in its hands to forget about. The vLLM and premium adapters landed later on the day the masked diff was threaded through, and they needed no redaction code either. TestHandle_MaskedDiffFlowsToLaterStages pins this down: the stage after the masker must see the placeholder, or the test fails.

That's the whole difference. Convention asks every consumer to behave. Construction changes what they're handed.

One reader keeps the raw diff by design: the cache key. A cache hit, keyed by a Blake3 hash of the ruleset version and the diff, skips the whole pipeline. That lookup happens before any stage runs, when the mask doesn't exist yet; producing it takes a gRPC call to the analyzer. So Handle computes the key from the submission as it arrived, and the loop above works on a copy. A resubmission of the same raw diff hits the cache. The key is a one-way digest, and the value is a verdict, which carries findings rather than the diff.

Before a verdict goes into the cache, the orchestrator strips everything that identifies one submission (ID, organization, repository, commit, PR number) and re-stamps it from the live submission on every hit. Findings are cached; identity never is. That also makes the cache organization-agnostic, which comes back further down.

The second raw reader is intentional too. When a required stage fails, including running out the 1,200 ms budget the whole chain shares while it is still working, the orchestrator falls back to a deterministic, in-process secret scan of the original submission. The scan has to see raw text to find a secret, and it isn't a model. Its verdicts are never cached, and TestHandle_FallbackVerdictIsNotCached says why in its failure message: "a degraded answer would stick for 72h".

Masking is only as good as detection, and detection lives in one package, pkg/scan, which the analyzer, the orchestrator's fallback and the CLI all call. Its patterns are few and high-precision, because false positives erode developer trust: private-key blocks, AWS key IDs, GitHub and Slack tokens, and one generic assignment rule. A detector finding carries a rule name and a line number, never the value ("potential credential detected (aws-access-key-id); remove and rotate it"), and the verdict keeps only the rule and that message, so what the detectors contribute to a PR comment carries no secret.

It's all line scanning, and that's a decision rather than a shortcut: D-019 defers AST detectors until the resolver can supply full files, because "a diff alone cannot be parsed into a meaningful AST". I'd rather ship a regex that says what it is than a parser that pretends a hunk is a file. A regex only knows the syntax someone wrote into it, though, and that shows up below.

The analyzer is required; the semantic stage is advisory. It's wrapped in app.Optional, which turns any error into a stage-unavailable warning instead of triggering the fallback, because the fallback would throw away the findings the analyzer already produced. A warning on its own turns a clean verdict into needs_review, so a degraded run leans toward "a human looks", never toward a silent approval. The semantic service degrades the same way inside: a vLLM outage falls back to the heuristic, one premium vendor's outage becomes an info finding while the other's findings stand, and only a failure of every premium reviewer returns 503, which Optional turns into a warning again.

That design absorbed a real bug. The semantic stage can attach "gold context", snippets of the organization's best code that the resolver retrieves from pgvector, but retrieval is best-effort: no resolver, a non-UUID organization or a resolver error all mean "no context". In Go, "no context" was a nil slice, and encoding/json writes a nil slice as null; the Python side declared the field as a list. In the all-container run, the orchestrator sent "gold_context": null, pydantic answered 422, and the handover records that this run is what found the bug.

Follow that 422 through the code. The semantic stage reports any non-200 as unavailable, and Optional turns it into the same stage-unavailable warning a timeout would produce, with this as its entire message:

advisory stage failed; verdict produced without it: orchestrator.adapters.semanticstage: semantic service returned 422

The analyzer's two credential findings block on their own, so by the code's rules the decision reads rejected with or without the semantic stage; the decision couldn't have shown this bug. In the verdict, a missing local-heuristic:masked says something failed, if you knew to expect it; only the last three digits of that line say it wasn't a vendor outage. The handover doesn't record what gave it away.

The bug had a second cost. An advisory failure still counts as a pipeline verdict, so the degraded answer is cached like any other: for 72 hours by default, a resubmission of that diff would get the verdict without the semantic stage's findings. That's the "degraded answer would stick for 72h" outcome the fallback test guards against, arriving by a path the test doesn't cover. A probe against a copy of the repo confirmed the cache write.

The fix went in on both sides, each with a regression test that passes today (TestAnalyze_NonUUIDOrgSkipsRAG in Go, test_evaluate_tolerates_null_gold_context in Python). The Go half is one struct tag:

type evaluateRequest struct {
	SubmissionID   string        `json:"submission_id"`
	OrganizationID string        `json:"organization_id"`
	Language       string        `json:"language"`
	MaskedDiff     string        `json:"masked_diff"`
	GoldContext    []GoldContext `json:"gold_context,omitempty"`
}

On the Python side, the field became list[GoldContextIn] | None = None, with a docstring that says it plainly: be liberal in what we accept.

Putting the whole promise in one place makes that place worth attacking. I did that while writing this post and found three edges, in order of how live they are.

Detection. Start here, because it's the only edge that's live in the shipped wiring. The orchestrator guarantees that the analyzer's output is the only diff; it can't guarantee that the analyzer caught everything. This is the generic rule in pkg/scan:

(?i)(?:password|passwd|secret|api[_-]?key|token)\s*[:=]\s*["'][^"']{8,}["']

password = "…" and password: "…" match. Go's short declaration, password := "…", doesn't: the character class takes the colon, and the pattern then wants a quote where the equals sign sits. (The AWS key in Figure 1 is matched by its value, so := doesn't matter there.)

Follow that line through the code and both halves of the promise fail at once. No match means no finding and no mask: the semantic stage receives the password verbatim, the default heuristic files only an info-level local-heuristic:password, which never changes a decision, and the change is approved.

Put the same line in a diff that also holds an AWS key and the verdict is rejected, carrying the receipt from the top of this post, local-heuristic:masked, next to a password the semantic stage read in plain text. If the premium tier were enabled with both keys, those two notes alone would score 0.4 + 0.3 = 0.7 against the default escalation threshold of 0.5, and that diff, raw password included, would go to Claude and Gemini.

The eval harness reports precision and recall of 1.000 on its golden corpus, and it isn't wrong: among its 14 cases, the only generic-secret positive is a Python assignment. When I added one Go short declaration locally, recall fell to 0.889 and the harness failed its own 0.90 floor. The gate was fine; the corpus was missing the case.

Wiring. The loop guarantees that a masked diff propagates; it can't guarantee that a masking stage exists. cmd/orchestrator/main.go builds the chain from config: the analyzer when ANALYZER_ADDR is set, otherwise the in-process scan, then the semantic stage when SEMANTIC_URL is set (both names carry the MEMBRANE_ORCHESTRATOR_ prefix). The in-process scan returns findings but no masked diff, though pkg/scan could compute one, so an empty analyzer address plus a semantic URL hands the semantic stage the raw diff. A second probe confirmed it: the verdict still said rejected, and the semantic stage received AKIAIOSFODNN7EXAMPLE verbatim. No shipped configuration does this, since the all-container compose file and the Helm values set both addresses, but config shouldn't be able to express it at all.

Context. The invariant covers the diff under review, not the context retrieved to review it. Gold snippets go into both model prompts, the local vLLM one and the two-vendor premium one, exactly as stored, and both prompts then introduce the diff as "Masked diff under review (secrets already redacted)". That label is a promise made by a string, and "curated code rarely holds a key" is a convention again.

The cache inherits the gap: its key ignores the organization, so findings a model wrote with one organization's snippets in its prompt would be served to any organization that submits the same diff. Both are latent while nothing outside the tests writes gold rows. And the raw diff still travels the submission topic: the guarantee is about model stages, not about Kafka.

Construction doesn't make a guarantee unbreakable. It gives the guarantee an address, and that address deserves its own tests.

One smaller gap is in my own test. TestHandle_MaskedDiffFlowsToLaterStages has a comment promising that the cache key still comes from the raw diff, then asserts only that the verdict came from the pipeline. The code holds (the cache probe found the entry under the raw-diff key), but a comment isn't an assertion.

The fixes are small because detection already lives in one package. Return the masked copy from the in-process stage, or refuse to start a semantic stage with no masker ahead of it. Fix the regex once, and the analyzer, the fallback and the CLI all get it in the same commit. Add the Go case to the corpus, mask gold snippets before they reach a prompt, key the cache per organization before gold context goes live, and make that test assert the cache key. None of that is done yet.

• Real models. No real vLLM endpoint or vendor keys have been wired and tuned yet; the handover lists that as the next step.
• Real retrieval. Embeddings come from a deterministic stub, and nothing outside the tests writes gold rows yet.
• Target architecture. AST analysis is deferred, and the Envoy gateway and AWS deployment in the architecture documents are design, not code.
• Automated end-to-end runs. GitHub Actions never runs on this account (D-036), so the gates are a local task lint and task test, and the container run is documented rather than scripted.

You don't need Membrane to use the pattern:

• Make the redacted artifact a return value and let the orchestrator substitute it. Don't ask consumers to redact.
• Key caches on what arrived and on whose context shaped the answer; send downstream only what you'd be comfortable leaking.
• Name wire fields after the invariant (masked_diff), then enforce it somewhere that isn't the name.
• Let advisory stages fail toward review, and assert that they spoke.
• Refuse to start a chain without a masker at its head, and grow the test corpus with every miss.

Don't make every stage promise not to look. Hand each one something with nothing to see, then spend your paranoia on the one place that does the handing.