Skip to content

ADR-0006 — Empirical-only discipline: no guessing in research or planning

Status

accepted

Context

This is a meta-decision — a rule about how we make decisions rather than a specific technical choice. It is also the rule that the rest of the project's research and planning discipline rests on, so capturing it as an ADR matters even though there is no single file whose decision it is.

The project has been burned multiple times by plausible-but-wrong assumptions that were accepted without evidence:

  • Stage S1 ran for several sub-stages on the assumption that "force-writing DeviceInfo fields is fine because the real values will eventually show up", which produced a working devicectl list devices but masked the actual pair-flow mismatch. Stage S2.A reverted four of those force-writes and found the real problem underneath.
  • Stage S2 was originally planned around Option α ("let the real pair flow run naturally over the real tunnel"), which was an assumption about pymobiledevice3's tunnel being speakable by CoreDeviceService. Stage S2.B's smoke test falsified that assumption in one click.
  • Earlier attempts at protocol research proposed "just reply with a reasonable guess and see what CDS does next" as a way to bisect dialog state. This was rejected, correctly, because a single lucky guess would produce false positive results that poison every subsequent step.

Each of these cost days of work that, with a stricter evidentiary rule, would have been weeks shorter.

Decision

Every claim in research and planning documents must be either (a) backed by a raw artifact or a direct primary-source quote, or (b) explicitly labeled as a gap / hypothesis. Plausible invention is forbidden, even as a temporary placeholder.

Operational consequences of this rule:

  • Research documents under research/ carry status: verified only when a raw artifact is cited in the evidence list. Otherwise they are hypothesis or draft.
  • Plans under plans/ that depend on an unverified assumption mark that assumption with !!! hypothesis, name the research task that would confirm or refute it, and do not act on it until resolved.
  • Protocol dialog bisection (like the Stage 2 Phase C Q1-Q5 queue) never replies to the other side with bytes we cannot source from either a capture or a primary reference. When the next step's reply is unknown, the bisection halts on a !!! gap admonition.
  • The friend-capture fallback (asking somebody with a real Mac+iPhone pair to run a capture script) is an escape valve of last resort, not an alternative to local empirical work. It is only permitted after all local empirical avenues have been exhausted.

Consequences

Wanted:

  • Downstream decisions are trustworthy because their premises are individually trustworthy.
  • "Unknown" becomes a first-class state that halts progress instead of being silently papered over with a plausible-looking assumption.
  • Future sessions can read the research tree without having to re-verify every claim.

Accepted as cost:

  • Research sometimes stops at a clean gap instead of producing a "complete" dialog reconstruction. This feels slower.
  • Some plans contain explicit hypothesis sections that make the reader pause instead of handing them a smooth story.
  • The rule applies to us as well — the temptation to push through with "this is probably right, let's keep going" is real and has to be resisted in the moment.

Evidence

This ADR does not cite a single decision artifact; it cites the pattern of cases where the rule was violated and fixed, and where it was followed and paid off.

References

  • ADR-0004 — the decision that followed from applying this rule to Phase B's findings.
  • memory/feedback_correct_solutions_only.md — user-stated version of the same discipline.