Skip to content

Magic Select (Image Tweak)

Magic Select lets a user pick an object in a shot or sketch frame, move it (with rotation and flip), or crop the image. The backend identifies objects, segments them, optionally inpaints the hole left behind, and composites the cutout onto a filled plate when Apply runs.

The product name in UI is Magic Select. Routes live under /workbench/magicSelect/*; most internal code names still use tweak (e.g. TweakService).

This doc describes the backend pipeline as it exists today. When you change a stage, model default, or cache key, update this file.

Table of contents

  1. End-to-end flow
  2. API endpoints
  3. Pipeline stages
  4. Identification (sketch-merged)
  5. Fill prep and inpaint
  6. Apply (move and crop)
  7. Configuration
  8. Caching and invalidation
  9. Realtime events
  10. Module layout
  11. Related docs and tests

End-to-end flow

User opens Magic Select on a shot/sketch frame
        │
        ▼
POST /identifyAndFillObjects  ──► 202 + operationId (or 200 if cached)
        │
        ▼ (background)
┌───────────────────────────────────────────────────────────────┐
│ IDENTIFY — YOLO + SAM + vocabulary + prompted segmentation   │
│  • Provisional objects streamed via tweakIdentificationProgress│
│  • Authoritative list persisted to tweakMediaArtifacts         │
└───────────────────────────────────────────────────────────────┘
        │
        ▼
User selects one object, drags destination box, optionally rotates/flips
        │
        ▼
POST /fillObjects (optional, per object)  ──► 200
        │
        ▼
┌───────────────────────────────────────────────────────────────┐
│ FILL (one object per session)                                  │
│  1. prepareFillPixels — mask refine, cutout, hole, flat frame │
│  2. Inpaint — fal-ai/flux-pro/v1/erase (source + hole mask)    │
│  3. Plate = model output (no rectangular collage)             │
│  4. Persist plate, holeMask, cutout on the object             │
└───────────────────────────────────────────────────────────────┘
        │
        ▼
User presses Apply
        │
        ▼
POST /applyTweak
        │
        ├─ cropRegion ──► sync 200, Sharp extract, append new frame
        │
        └─ objects[0] ──► 202 + operationId
                │
                ▼ (background)
           Grey holes on source → flux-pro erase (source + mask) → restamp cutouts
           (or Sharp composite only when destination covers the hole)
                │
                ▼
           Upload → append shot image/sketch
                │
                ▼
           magicSelectApplyReady notification + SHOT_UPDATED

One fill per object; multi-move/remove Apply. Identification returns many candidates. /fillObjects still fills one objectId at a time. Apply accepts 1–12 entries: union hole masks, flux-pro erase on the original source, then restamp cutouts for moves only (remove: true skips restamp).

Fill timing. Identification does not run fill anymore. Fill runs when the frontend calls /fillObjects for the selected object, or lazily during Apply when the moved destination would expose the source hole and no valid fill cache exists.

API endpoints

All routes live under /workbench/magicSelect (magicSelect.router.ts).

Method Path Sync / async Purpose
POST /identifyAndFillObjects 202 (or 200 if identification cached) Start object identification on imageURL
POST /maskObject 200 On-demand SAM mask for one object (if identification left it bbox-only)
POST /fillObjects 200 Inpaint hole + build plate/cutout for one objectId
POST /applyTweak 202 for move, 200 for crop Commit move or crop as a new shot/sketch frame

Common request fields

Every endpoint expects:

  • type: 'image' (rendered shot) or 'sketch'
  • storyId, sceneId, shotId
  • imageURL: must match exactly one entry in shot.images or shot.sketches

Object-scoped calls also require objectId. Fill and identification accept forceInvalidateCache (default false).

Apply request shape

Apply is mutually exclusive:

  • Move / remove: objects: [{ objectId, destination?, remove?, rotationDegrees?, flipHorizontal?, flipVertical? }, ...] — 1–12 entries. destination is [x1, y1, x2, y2] in source pixels for moves. remove: true erases the hole only (no destination).
  • Crop: cropRegion: { x, y, width, height } — normalized 0–1 coordinates relative to the source frame.

Object removal (apply without a destination) is not supported; the validator requires a positive destination box for moves.

Pipeline stages

Step Name Where Models / libs
1 Detect identificationPipeline.ts YOLO-World XL on Replicate
2 Segment segmentation.ts SAM 3.1 on Fal (fal-ai/sam-3-1/image)
3 Merge & dedupe identificationPipeline.ts Deterministic mask/bbox overlap rules
4 Mask normalize pixels.ts Sharp — binary mask, bbox clip
5 Fill prep pixels.ts, maskAlternatives.ts Contour stroke refine, hole/cutout/window mats
6 Inpaint edit magicSelect.service.ts → AssetGen fal-ai/flux-pro/v1/erase (buildFluxEraseFillModelConfig)
7 Tone match toneMatch.ts Deterministic edge-band color fit (no model)
8 Composite pixels.ts Sharp — cutout onto plate at destination

Pipeline version string (fill cache): flux-pro-erase-v2 in magicSelect.service.ts.

Identification (sketch-merged)

Default pipeline for both shot images and sketches (SharedConstants.TWEAK_*_OBJECT_DETECTION_PIPELINE = 'sketch-merged').

Three identification lanes run concurrently:

                    ┌── YOLO detect ──► SAM per label group ──► detector candidates
                    │
source imageURL ────┼── SAM prompt "person" ─────────────────► prompted candidates
                    │
                    └── SAM prompt "mannequin" ────────────────► prompted candidates
                    │
                    └── SAM per vocabulary concept ──────────► vocabulary candidates
                              (from shot/env/props text, capped at 7)

Vocabulary concepts come from promptVocabulary.ts — shot description, environment signature objects, and prop titles from the story. Vocabulary SAM calls are concurrency-limited (same cap as vocabulary limit).

Prompted person and mannequin prompts always run; neither material reliably implies the other is absent.

Merge order (mergeIdentificationCandidates):

  1. Prompted candidates are accepted first (figures anchor dedupe).
  2. Vocabulary candidates drop if they duplicate an accepted mask/bbox.
  3. Detector candidates drop if duplicated by prompted or vocabulary.

Duplicate thresholds: 60% mask union for general objects, 20% for figure labels (person, teddy bear, dog).

Streaming provisional objects

While identification runs, each finished segmentation batch publishes tweakIdentificationProgress with authoritative: false and a monotonic revision. The frontend can render highlights before the final list is persisted.

When identification completes, objects and ImageIdentification metadata are stored on tweakMediaArtifacts (kind: magicSelectObjectPrep), keyed by (storyId, sceneId, shotId, mediaId).

Legacy pipeline kinds

identifyTweakCandidates still supports sketch (prompted-only) and shot-image (detector-only) for old persisted pipelineKind values. New runs always use sketch-merged.

Fill prep and inpaint

Fill prep (prepareFillPixels)

Given source RGBA + SAM mask + bbox:

  1. Enclose mask holes (fillEnclosed).
  2. Contour stroke refine (refineMaskWithContours in maskAlternatives.ts) — pulls thin dark ink outlines into the mask when SAM stops at inner fill edges (important for sketches).
  3. Build hole matte (dilated), cutout matte (eroded), and stitch window:
  4. mask window: composite reveals exactly the inpaint hole.
  5. box-pad-tone-matched window (default): padded box around the hole for tone matching.
  6. Produce artifacts:
  7. flatFrame — source with hole pixels greyed (UI / debug; model uses source + holeMask)
  8. holeMask — alpha PNG of inpaint region
  9. plateWindow — alpha PNG of region kept from model output
  10. cutout — cropped RGBA PNG of the object

Source alpha is preserved through the pixel pipeline.

Step 4 — inpaint model

Magic Select fill uses Fal flux-pro erase on the original source + hole mask (buildFluxEraseFillModelConfig).

Contract:

  1. Upload the source frame at native WxH as image_url.
  2. Upload a black/white mask (inpaintModelMaskPng(holeMask)) as mask_url — white = inpaint region.
  3. Plate = model output resized to source WxH when dimensions differ — no padded-hole embed and no tone-match collage.

/fillObjects produces this plate for MagSelect UI (background under the floating cutout).

On Apply, MagSelect unions hole masks, runs flux-pro erase on the source (cutouts are not in the model input), then stamps the cutouts onto the model plate. Result imageParams stores assetGenJobId, durationMs, stageDurationsMs, and fillPath; createdAt is the Apply-start placeholder time.

Apply (move and crop)

Crop

Synchronous. Sharp extracts cropRegion from the source, uploads PNG, appends to shot.images or shot.sketches. No identification or fill required.

Move

Asynchronous (202). Background task (runApplyObjectMove):

  1. Resolve object; ensure mask + fill cache (runFill if needed for cutout / UI plate).
  2. If destinationCoversWindow(...) — skip inpaint, Sharp-composite cutout onto source.
  3. Else hole-then-restamp (one or many moves):
  4. Union hole masks; erase on the original source (cutouts are not placed before the model)
  5. fal-ai/flux-pro/v1/erase full-frame backfill
  6. applyCutoutToPlate for each move onto the model plate (restamp)
  7. Upload result (with imageParams.assetGenJobId when erase ran, plus durationMs / stageDurationsMs / fillPath), append tweaked frame, publish notifications.

ImageMedia.createdAt on the result keeps the PROCESSING placeholder timestamp (Apply start). Finish time ≈ createdAt + imageParams.durationMs.

reconcile is locked to false (no post-move Gemini shadow pass).

Configuration

Constants in src/shared/sharedConstants.ts, exposed via config helpers:

Constant Default Effect
TWEAK_SHOT/SKETCH_OBJECT_DETECTION_PIPELINE sketch-merged Identification lanes
TWEAK_*_VOCABULARY_LIMIT 7 Max vocabulary SAM prompts
TWEAK_FILL_STITCH_WINDOW box-pad-tone-matched Stitch + tone-match behavior

Model IDs: src/assetGenModelConfigs/workbenchV2/tweakModelConfigs.ts. Fill default is fal-ai/flux-pro/v1/erase (source + hole mask).

Caching and invalidation

Identification cache

Stored on tweakMediaArtifacts per source mediaId. Reused when objects + identification exist and forceInvalidateCache is false.

A concurrent identification run is deduplicated: if status === 'processing' and younger than 10 minutes, /identifyAndFillObjects returns the existing operationId without starting a second run.

Fill cache (per object)

An object’s fill is reusable when all of the following match the current pipeline:

  • fillCacheKey (derived from edit model, stitch window, tone-match version, prompt version)
  • plateURL, holeMaskURL, cutoutURL, cutout crop offsets

fillCacheKey format:

{modelKey}:{stitchWindow}:{pipelineVersion}:{promptVersion}

Changing any constant above invalidates durable plates without deleting identification.

On-demand mask

If identification produced a bbox-only object (mask upload failed or SAM returned no paired mask), /maskObject runs a single SAM call for that detection and persists maskURL.

Realtime events

Event When Payload highlights
tweakIdentificationProgress Each identification batch completes operationId, revision, objects[], authoritative: false
magicSelectPrepReady (user notification) Identification run finishes or fails operationId, success/error
magicSelectApplyReady (user notification) Apply background task finishes operationId, optional preview action
shotUpdated Apply succeeds Updated shot document

Frontend flow diagram: studio-frontend → app/_shared/schema-segmentation.md.

Module layout

src/workbench/magicSelect/
├── README.md                 ← this file
├── magicSelect.router.ts     route definitions
├── magicSelect.controller.ts HTTP handlers (202 vs sync split)
├── magicSelect.validator.ts  Zod request schemas
├── magicSelect.service.ts    orchestration, caching, apply/fill/identify
├── identificationConfig.ts   pipeline + vocabulary limits per material
├── identificationPipeline.ts YOLO + SAM stages, merge/dedupe
├── segmentation.ts           Replicate YOLO + Fal SAM job wrappers (SAM masks → ViMatte cutouts)
├── viMatte.service.ts        ViMatte /v1/matte client (SAM mask → cutout URL)
├── promptVocabulary.ts       shot/env/prop → SAM prompt concepts
├── pixels.ts                 mask math, fill prep, composite
├── maskAlternatives.ts       contour + stroke refine for sketch masks
├── fillConfig.ts             stitch window, tone-match flag, edit model
└── toneMatch.ts              deterministic plate tone fit

Persistence: src/models/tweakMediaArtifacts.model.ts (magicSelectObjectPrep artifacts on the source image’s mediaId).

Frontend

  • app/_shared/schema-segmentation.md — provisional vs authoritative object flow (frontend repo)
  • docs/image-tweak-ui-plan.md — UI integration notes (if present)

Backend tests

  • tests/workbench/magicSelect/service.test.ts — identification, gpt padded-hole fill, apply, async notifications
  • tests/workbench/magicSelect/identificationPipeline.test.ts — merge/dedupe, concurrency
  • tests/workbench/magicSelect/pixels.test.ts — fill prep, destination coverage
  • tests/workbench/magicSelect/maskAlternatives.test.ts — contour stroke refine
  • tests/workbench/magicSelect/toneMatch.test.ts — tone stitch safety

Eval / fixtures

  • python_eval/tests/fixtures/image_tweak_shot_fixture.json
  • python_eval/tests/fixtures/image_tweak_variant_shot_fixture.json

When adding a pipeline stage or changing defaults, add or update a unit test and bump TWEAK_FILL_PIPELINE_VERSION if fill output semantics change.