Sketch frame verdict fix — Codex implementation handover

Scope and root cause

Implemented on /Users/gerard/Projects/studio-backend-calibrate, branch gerard/pregen-fix-target, from a03589f3.

I agree with both anchors in SKETCH_FRAME_VERDICT_NOT_BLIND.md:

1. Anchor A is a root cause. The upload-time connectedTo label was present in the exact first-turn context used to produce readsAsFrame. The instruction saying the label could be wrong did not make the verdict blind.
2. Anchor B is a root cause. The backend question quoted only the model-preferred frame. A user could not compare start, middle and end before choosing a letter.

There were two additional enforcement failures:

• frameRelabelAsk.answer was write-once, so a later explicit named-frame correction could not replace the first answer.
• Finalization derived several values from resolvedConnectedTo, but did not require the compiled frame field to target that same moment and did not verify the stored result after the atomic write.

The secondary polygon-placement problem is deliberately not fixed in this change. The user confirmed that it should remain separate. This change does not alter polygon geometry, character mapping, the compiler's interpretation of left/right positions, or frame prose derived from those polygons.

Files changed — literal before and after

1. src/workbench/sketchChat/sketchChat.service.ts

The first-turn context now withholds the upload label. Resumed turns restore it after the blind verdict has already been persisted.

BEFORE:

const shotMetadataContext = SketchChatService.buildShotMetadataContext(storyData, shotData, state.request.sketchImageUrl);

AFTER:

// The upload label is deliberately withheld on the only turn that can see the drawing and emits
// readsAsFrame. It is restored on every resumed turn, after the blind verdict is already persisted,
// so it can phrase the comparison without anchoring the verdict.
const shotMetadataContext = SketchChatService.buildShotMetadataContext(storyData, shotData, state.request.sketchImageUrl, !!state.existingChatSession);

BEFORE:

public static buildShotMetadataContext(storyData: StoryVideoType, shotData?: StoryVideoShot, sketchImageUrl?: string): string {

AFTER:

public static buildShotMetadataContext(storyData: StoryVideoType, shotData?: StoryVideoShot, sketchImageUrl?: string, includeSketchConnectedTo = true): string {

BEFORE:

if (sketchImageUrl) {

AFTER:

if (sketchImageUrl && includeSketchConnectedTo) {

Acceptance criterion: 1.

2. src/promptTemplates/agenticGuruSystemPrompts/sketchAnalysisGuruPrompt.ts

BEFORE:

export const sketchAnalysisSystemPromptVersion = 'v20260817.2';

AFTER:

export const sketchAnalysisSystemPromptVersion = 'v20260817.3';

BEFORE:

On that same FIRST response you MUST also include "readsAsFrame": "start" | "middle" | "end" — the frame of this shot the drawing actually depicts, judged by comparing it against ALL THREE frame descriptions, not only the one the metadata claims.

AFTER:

On that same FIRST response you MUST also include "readsAsFrame": "start" | "middle" | "end" — the frame of this shot the drawing actually depicts, judged by comparing it against ALL THREE frame descriptions, not only the one the metadata claims.

The upload-time connectedTo label is deliberately withheld from the first-turn metadata so this verdict is blind. Do not infer a prior label from the conversation or assume that the first frame listed is preferred. Judge only the drawing against the three frame descriptions.

BEFORE:

The shot has three frame descriptions (initial/middle/final). The sketch metadata may include "Sketch connected to: start|middle|end" — this is the frame the user picked at upload time, before anything had looked at the drawing, so it can be wrong.

AFTER:

The shot has three frame descriptions (initial/middle/final). On the image-reading first turn, the backend deliberately withholds the upload-time "Sketch connected to" label so it cannot anchor readsAsFrame. On later turns the label may be present only so the backend can explain what would change; it is not evidence about what the drawing depicts.

Acceptance criterion: 1.

3. src/agenticGuru/sketchAnalysisGuru.ts

Persisted contradiction/override state

BEFORE:

displacedQuestion?: string;

AFTER:

displacedQuestion?: string;
/** A deterministic sketch-description/frame-text contradiction awaiting an explicit override. */
contradictionWarning?: {
  frame: SketchConnectedToFrame;
  reason: string;
  question: string;
};
/** The frame for which the user explicitly acknowledged a contradiction with "anyway"/"override". */
contradictionOverride?: SketchConnectedToFrame;

Correctable answer and warning gate

BEFORE:

const frameRelabelAnswer = this.parseFrameRelabelAnswer(userMessage);
const askBeingAnswered = this.analysisState.frameRelabelAsk;
if (frameRelabelAnswer && askBeingAnswered) {
  this.updateAnalysisState({ frameRelabelAsk: { ...askBeingAnswered, answer: frameRelabelAnswer } });

AFTER:

const frameRelabelChoice = this.parseFrameRelabelAnswer(userMessage);
const askBeingAnswered = this.analysisState.frameRelabelAsk;
if (frameRelabelChoice && askBeingAnswered) {
  const warningTurn = await this.frameContradictionWarningIfNeeded(frameRelabelChoice, askBeingAnswered);
  if (warningTurn) return warningTurn;

  const previousAnswer = askBeingAnswered.answer;
  const frameRelabelAnswer = frameRelabelChoice.frame;
  const answeredAsk = { ...askBeingAnswered, answer: frameRelabelAnswer };
  delete answeredAsk.contradictionWarning;
  delete answeredAsk.contradictionOverride;
  if (frameRelabelChoice.overrideContradiction) answeredAsk.contradictionOverride = frameRelabelAnswer;
  this.updateAnalysisState({
    frameRelabelAsk: answeredAsk,
    ...(previousAnswer && previousAnswer !== frameRelabelAnswer && { phase: SketchAnalysisPhase.DISCUSSING, finalReviewPresented: false }),
  });

Acceptance criteria: 2 and 4.

First-turn observation is available to the deterministic check

BEFORE:

if (this.analysisState.readsAsFrame || !isFirstTurn) return;
let value: unknown;
try {
  value = (JSON.parse(outputText) as { readsAsFrame?: unknown }).readsAsFrame;
} catch {
  return;
}
const frame = Object.values(SketchConnectedToFrame).find((candidate) => candidate === String(value).trim().toLowerCase());
if (!frame) return;
this.updateAnalysisState({ readsAsFrame: frame });

AFTER:

if (!isFirstTurn) return;
let parsed: { sketchDescription?: unknown; readsAsFrame?: unknown };
try {
  parsed = JSON.parse(outputText) as { sketchDescription?: unknown; readsAsFrame?: unknown };
} catch {
  return;
}
const frame = Object.values(SketchConnectedToFrame).find((candidate) => candidate === String(parsed.readsAsFrame).trim().toLowerCase());
this.updateAnalysisState({
  ...(typeof parsed.sketchDescription === 'string' && parsed.sketchDescription.trim() && !this.analysisState.sketchDescription
    ? { sketchDescription: parsed.sketchDescription.trim() }
    : {}),
  ...(frame && !this.analysisState.readsAsFrame ? { readsAsFrame: frame } : {}),
});

Acceptance criterion: 2.

Every frame option now shows its own stored description

BEFORE:

const field = SketchAnalysisGuru.FRAME_INSTRUCTION_FIELD[to];
const description = field ? (shot as unknown as Record<string, unknown>)[field] : undefined;
const target =
  typeof description === 'string' && description.trim() ? `: '${SketchAnalysisGuru.frameQuoteExcerpt(description)}'` : ' (no description written for that frame yet)';
const opening = !from
  ? `This sketch is not connected to a frame yet — it reads most like your ${to} frame${target}.`
  : from === to
    ? `This sketch reads like the ${to} frame it is connected to${target}.`
    : `This sketch is connected to your ${from} frame, but reads more like your ${to} frame${target}.`;
const options = SketchAnalysisGuru.frameChoiceOptions(to).map((frame, index) => {
  const letter = String.fromCharCode(97 + index);
  if (frame === to) return `${letter}) ${from === to ? `Yes, keep it as ${frame} frame` : `Yes, connect to ${frame} frame`}`;
  return `${letter}) ${frame === from ? `Keep it as ${frame} frame` : `Connect to ${frame} frame instead`}`;
});
return [`${opening} Which frame should it be connected to?`, ...options].join('\n');

AFTER:

const opening = !from
  ? `This sketch is not connected to a frame yet — it reads most like your ${to} frame.`
  : from === to
    ? `This sketch reads like the ${to} frame it is connected to.`
    : `This sketch is connected to your ${from} frame, but reads more like your ${to} frame.`;
const options = SketchAnalysisGuru.frameChoiceOptions(to).map((frame, index) => {
  const letter = String.fromCharCode(97 + index);
  const choice =
    frame === to
      ? from === to
        ? `Yes, keep it as ${frame} frame`
        : `Yes, connect to ${frame} frame`
      : frame === from
        ? `Keep it as ${frame} frame`
        : `Connect to ${frame} frame instead`;
  return `${letter}) ${choice} — ${SketchAnalysisGuru.frameOptionDescription(frame, shot)}`;
});
return [`${opening} Compare all three frame descriptions. Which frame should it be connected to?`, ...options].join('\n');

frameOptionDescription() reads and excerpts the corresponding initial, middle or final frame field for each option.

Acceptance criterion: 3.

Deterministic contradiction detection and explicit override

BEFORE:

// No backend comparison existed between sketchDescription and the selected frame text.

AFTER:

private static readonly OBSERVABLE_POSTURES: Array<{ name: string; pattern: RegExp }> = [
  { name: 'seated/settled', pattern: /\b(?:seated|sitting|sits?|sat|settled\s+(?:into|in|on)\s+(?:the\s+|a\s+)?(?:chair|seat))\b/i },
  { name: 'standing', pattern: /\b(?:stands?|standing)\b/i },
  {
    name: 'mid-transition into a seat',
    pattern: /\b(?:mid[- ]?(?:drop|sit)|about\s+to\s+sit|lowering[^.!?]{0,24}(?:chair|seat)|hips?\s+(?:just\s+)?contacting[^.!?]{0,20}(?:chair|seat))\b/i,
  },
  { name: 'kneeling', pattern: /\b(?:kneels?|kneeling)\b/i },
  { name: 'lying down', pattern: /\b(?:lies|lying|prone|supine)\b/i },
];

private frameDescriptionContradiction(frame: SketchConnectedToFrame, shot: StoryVideoShot): string | null {
  const sketchDescription = this.analysisState.sketchDescription;
  const frameDescription = SketchAnalysisGuru.frameInstruction(frame, shot);
  if (!sketchDescription || !frameDescription) return null;

  const sketchPostures = SketchAnalysisGuru.OBSERVABLE_POSTURES.filter(({ pattern }) => pattern.test(sketchDescription));
  if (sketchPostures.length !== 1) return null;

  const framePostures = SketchAnalysisGuru.OBSERVABLE_POSTURES.filter(({ pattern }) => pattern.test(frameDescription));
  const incompatible = framePostures.find(({ name }) => name !== sketchPostures[0].name);
  if (!incompatible && !(sketchPostures[0].name === 'seated/settled' && /\bempty\s+chair\b/i.test(frameDescription))) return null;

  const frameClaim = incompatible?.name ?? 'an empty chair';
  return `The saved sketch description reads as ${sketchPostures[0].name}, but the ${frame} frame explicitly describes ${frameClaim}.`;
}

return `${reason}\nThat is a direct visual contradiction. Choose another frame by name, or reply "use ${frame} frame anyway" to override it deliberately.`;

The check is deliberately narrow. It warns only when the persisted drawing description expresses one unambiguous posture family and the chosen frame explicitly contains an incompatible family (including the reproduced seated/standing/empty-chair case). Mixed-posture sketches are not guessed at.

Acceptance criterion: 2.

An explicit named frame can replace an existing answer

BEFORE:

const ask = this.analysisState.frameRelabelAsk;
if (!ask || ask.answer) return null;

AFTER:

const ask = this.analysisState.frameRelabelAsk;
if (!ask) return null;

const bareFrame = normalized.match(/^(start|middle|mid|end)(?:\s+frame)?$/)?.[1];
const instructedFrame =
  !/[?]$/.test(raw) && /\b(?:label|connect|set|change|switch|move|make|treat|use|confirm|override)\b[^.!?]{0,60}\b(start|middle|mid|end)\s+frame\b/i.exec(raw)?.[1];
const explicitValue = bareFrame ?? instructedFrame;
if (explicitValue) {
  const frame = (explicitValue.toLowerCase() === 'mid' ? SketchConnectedToFrame.MIDDLE : explicitValue.toLowerCase()) as SketchConnectedToFrame;
  return { frame, overrideContradiction: /\b(?:anyway|override|despite)\b/i.test(raw) };
}

if (ask.answer || ask.contradictionWarning) return null;

Bare a/b/c parsing remains below that guard and therefore cannot reopen an answered question.

Acceptance criterion: 4.

Prepare/finalize/store invariants

BEFORE:

// No assertion tied a frame-specific patch field to resolvedConnectedTo.

AFTER:

private static assertFramePatchTargetsResolved(decision: Pick<PrepareSketchAnalysisReviewPayload, 'resolvedConnectedTo'>, patch: SketchAnalysisMetadataPatch): void {
  const resolved = decision.resolvedConnectedTo;
  if (!resolved) throw new SharedErrors.InvalidOperationError('A completed sketch analysis must have a confirmed connected frame.');

  const frameFields = Object.values(SketchAnalysisGuru.FRAME_INSTRUCTION_FIELD);
  const touched = frameFields.filter((field) => patch[field] !== undefined);
  if (touched.length === 0) return;

  const expected = SketchAnalysisGuru.FRAME_INSTRUCTION_FIELD[resolved];
  if (!touched.includes(expected)) {
    throw new SharedErrors.InvalidOperationError(
      `The sketch analysis resolved to ${resolved}, but its patch targets ${touched.join(', ')} instead of ${expected}. Refusing to write a split-frame result.`
    );
  }
}

This assertion runs after compilation, again before applying a pending review, and again immediately before persistence. Off-moment fields needed for a shot-wide camera/background cascade remain allowed only when the patch also contains the frame field corresponding to the confirmed moment. The corrupt shape from the report — resolvedConnectedTo: start with only finalFrameInstructions changed — is rejected.

BEFORE:

if (!params.resolvedConnectedTo && params.analysisOutcome !== SketchAnalysisOutcome.FAILED_FRAME_MISMATCH) {
  logger.warn('finalize_sketch_analysis: resolvedConnectedTo missing on successful analysis', {
    storyId: this.storyId,
    sceneId: this.sceneId,
    shotId: this.shotId,
  });
}

AFTER:

if (!params.resolvedConnectedTo && params.analysisOutcome !== SketchAnalysisOutcome.FAILED_FRAME_MISMATCH) {
  return { success: false, error: 'A completed sketch analysis must have a confirmed connected frame.' };
}

BEFORE:

const frame = params.resolvedConnectedTo.toUpperCase();
params.generationNotes.push(
  `SKETCH MOMENT: This sketch depicts the ${frame} frame. Use ${params.resolvedConnectedTo}-frame pose, gaze, prop state, character relationships, and camera state when resolving visual instructions.`
);

AFTER:

params.generationNotes = [
  ...(params.generationNotes ?? []).filter((note) => !/^SKETCH MOMENT:/i.test(note)),
  SketchAnalysisGuru.canonicalSketchMomentNote(params.resolvedConnectedTo),
];

The atomic write result is then checked literally:

const sketch = updatedShot.sketches?.find((candidate) => candidate.imageURL === this.sketchImageUrl);
if (sketch?.connectedTo !== frame) {
  throw new Error(`Stored sketch connectedTo "${sketch?.connectedTo ?? 'missing'}" did not match resolvedConnectedTo "${frame}".`);
}

const storedAnalysis = updatedShot.sketchAnalyses?.find((candidate) => candidate.sketchImageUrl === this.sketchImageUrl);
if (storedAnalysis?.resolvedConnectedTo !== frame) {
  throw new Error(`Stored sketch analysis resolvedConnectedTo "${storedAnalysis?.resolvedConnectedTo ?? 'missing'}" did not match "${frame}".`);
}

const canonicalNote = SketchAnalysisGuru.canonicalSketchMomentNote(frame);
const momentNotes = storedAnalysis.generationNotes?.filter((note) => /^SKETCH MOMENT:/i.test(note)) ?? [];
if (momentNotes.length !== 1 || momentNotes[0] !== canonicalNote) {
  throw new Error(`Stored sketch analysis did not contain exactly one canonical ${frame} SKETCH MOMENT note.`);
}

Acceptance criterion: 5.

4. tests/workbench/sketchChat/sketchChat.service.test.ts

BEFORE:

// No test exercised a metadata context with the upload-time connectedTo label deliberately hidden.

AFTER:

it('withholds the upload-time frame label from the first image-reading context', () => {
  const context = SketchChatService.buildShotMetadataContext(
    { storyId: 'story-1', characters: [] } as StoryVideoType,
    {
      shotId: 'shot-1',
      updatedAt: new Date(),
      initialFrameInstructions: 'A character stands beside an empty chair.',
      middleFrameInstructions: 'The character lowers into the chair.',
      finalFrameInstructions: 'The character is seated.',
      sketches: [{ imageURL: 'https://example.com/sketch.jpg', connectedTo: 'start' }],
    } as StoryVideoShot,
    'https://example.com/sketch.jpg',
    false
  );

  expect(context).toContain('Initial frame (start):');
  expect(context).toContain('Middle frame (middle):');
  expect(context).toContain('End frame (end):');
  expect(context).not.toContain('Sketch connected to:');
});

Acceptance criterion: 1.

5. tests/agenticGuru/sketchAnalysisGuru.test.ts

Existing later-stage tests now call the pre-existing frameCheckSettled() helper because successful reviews are no longer permitted to have no confirmed frame.

BEFORE:

const guru = createGuru();
const prepared = await guru.executeTool({

AFTER:

const guru = createGuru();
frameCheckSettled(guru, SketchConnectedToFrame.MIDDLE);
const prepared = await guru.executeTool({

The frame-question assertions changed from checking one quoted preferred description to checking all three literal option descriptions.

BEFORE:

expect(asked.question).toContain("your middle frame: 'Drone crosses in front of the residences.'");

AFTER:

expect(asked.question).toContain("a) Yes, connect to middle frame — 'Drone crosses in front of the residences.'");
expect(asked.question).toContain("b) Keep it as start frame — 'Drone starts at frame left.'");
expect(asked.question).toContain("c) Connect to end frame instead — 'Drone exits at frame right.'");

New regression tests were added with these literal test names:

it('warns on a direct posture contradiction and requires an explicit override', async () => { ... });
it('lets an explicit named-frame instruction correct an answer, but a later bare letter cannot reopen it', async () => { ... });
it('uses a later explicit frame correction instead of forcing the first answer back at prepare time', async () => { ... });
it('refuses a review whose only frame patch targets a different moment than the confirmed frame', async () => { ... });
it('replaces stale SKETCH MOMENT notes with exactly one note for the confirmed frame', async () => { ... });

Their literal core assertions are:

expect(warning.at(-1)).toContain('saved sketch description reads as seated/settled');
expect(warning.at(-1)).toContain('use start frame anyway');
expect(guru.getAnalysisState().frameRelabelAsk?.answer).toBeUndefined();
expect(guru.getAnalysisState().frameRelabelAsk?.contradictionWarning?.frame).toBe(SketchConnectedToFrame.START);

expect(guru.getAnalysisState().frameRelabelAsk?.answer).toBe(SketchConnectedToFrame.END);
expect(guru.getAnalysisState().pendingReview?.finalizeData.resolvedConnectedTo).toBe(SketchConnectedToFrame.END);

expect(prepared.success).toBe(false);
expect(guru.getAnalysisState().pendingReview).toBeUndefined();

expect(result.analysisResult.generationNotes.filter((note: string) => note.startsWith('SKETCH MOMENT:'))).toEqual([
  'SKETCH MOMENT: This sketch depicts the END frame. Use end-frame pose, gaze, prop state, character relationships, and camera state when resolving visual instructions.',
]);

Acceptance criteria: 2, 3, 4 and 5.

Acceptance criteria result

1. Blind readsAsFrame: satisfied. The actual upload label value is absent from first-turn shot metadata and returns only on resumed turns.
2. Mechanically check explicit contradictions: satisfied for deterministic posture-state facts. The reproduced seated versus standing/mid-drop/empty-chair contradiction warns and cannot be stored until the user explicitly writes an override such as use start frame anyway. The backend does not auto-select another frame.
3. Let the user compare: satisfied. All three option lines include the corresponding stored frame description excerpt.
4. Later correction: satisfied. A named instruction can change an existing answer; bare letters cannot reopen it. A changed answer disarms any pending review, and prepare uses the latest answer.
5. Write invariant: satisfied. Successful analysis requires a frame; a frame patch must include the matching frame field; stale SKETCH MOMENT notes are replaced with one canonical note; and the returned stored shot is checked for matching sketches[].connectedTo, analysis resolvedConnectedTo, and canonical generation note.

Verification performed

TypeScript

Command: npx tsc --noEmit
Result: passed, 0 errors

Focused tests

Command: npx vitest run tests/agenticGuru/sketchAnalysisGuru.test.ts tests/workbench/sketchChat/sketchChat.service.test.ts tests/workbench/sketchChat/sketchAnalysisValidator.test.ts tests/workbench/sketchChat/sketchAnalysisPersistence.test.ts
Result after the final code cleanup: 4 files passed, 117 tests passed

An earlier two-file focused run also passed:

Command: npx vitest run tests/agenticGuru/sketchAnalysisGuru.test.ts tests/workbench/sketchChat/sketchChat.service.test.ts
Result: 2 files passed, 68 tests passed

Full suite

Command: npm test
Result: 100 files passed; 1412 passed, 1 skipped (1413 total)

Formatting and lint

Command: npx biome format --write <five changed TypeScript files>
Result: formatted successfully

Command: npx biome lint <five changed TypeScript files>
Result: checked 5 files; no fixes or errors

Live replay

not verified live

I did not replay story 7d2b0c4e-33d1-4bac-833f-a55f27990c43, shot 1973f948-a4ed-4980-a1a0-4613ba6146ec through the live sketch-analysis flow. Therefore the tests prove the exact context and deterministic backend behavior, but do not prove what GPT-5.1 will return for that image under prompt version v20260817.3.

Still unverified or still broken

• The exact live image has not been replayed, so a live readsAsFrame: "end" and final resolvedConnectedTo: "end" have not been observed.
• The contradiction detector is intentionally conservative and lexical. It covers explicit posture families (seated/settled, standing, mid-transition, kneeling, lying) and empty chair. It does not claim to solve arbitrary semantic contradictions.
• The secondary polygon-placement bug remains unfixed by agreement. The compiler can still produce frame prose whose left/right character placement contradicts the user's C1/C2 polygon mapping. That requires a separate compiler/spatial-grounding change and separate acceptance tests.

Behavior changed beyond the narrow anchors

• A completed analysis with no confirmed frame now fails instead of logging a warning and writing an unscoped result. This is necessary for acceptance criterion 5.
• If neither the model verdict nor a prior upload label exists, the backend uses the already-documented ambiguous default (middle) as the suggested frame and still asks the user. It does not silently write that default.
• Existing model-authored SKETCH MOMENT: notes are removed before the one canonical note is generated. This prevents an old START note and a new END note from coexisting.
• A direct contradiction adds one deliberate user confirmation round-trip. This is the requested warn-and-require-override behavior.