Mervin Praison

This article documents a repeatable image-model test harness you can reuse whenever mer.vin evaluates a new generator—applied here to Ideogram 4.0 open weights (June 2026) against GPT Image 2 and closed Ideogram on the same dystopian-ad briefs.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  BRIEF[Locked prompt pack dystopian ads] --> P1[Run GPT Image 2]
  BRIEF --> P2[Run Ideogram closed API]
  BRIEF --> P3[Run Ideogram open ComfyUI]
  P1 --> SCORE[Score rubric per output]
  P2 --> SCORE
  P3 --> SCORE
  SCORE --> LOG[Archive images plus notes]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class SCORE,LOG agent
  class BRIEF,P1,P2,P3 hook

Copy this flow for the next open-weight or API image launch.

Why this brief is a good benchmark

Independent testers used cynical post-apocalyptic advertising: brand copy that should not exist, selling products nobody should want, in places nobody should live. That stresses typography, tone, layout, and safety filters—not pretty landscapes. Ideogram ships open weights claiming performance near GPT Image 2; this run checks that claim on real design work.

Test matrix (frozen for reuse)

Scoring rubric (use on every future image test)

Model results from the comparison run

Outputs below are from the public three-way test (GPT Image 2 vs Ideogram closed vs Ideogram open). Panels are ordered as shared in the source post—use them as ground truth when reproducing the harness.

Panel 1: legible THE ROAD AHEAD copy. Panels 2–3: mangled headline and grey safety block.

Layout holds; watch for invented spellings on product name and body copy.

Tests long-form marketing copy and vertical type treatments.

Note SERENNEX / SEERENEX spelling drift between panels.

Typos include EVERYTHIING and missing IS in title strings.

Brand drift MESTREEM and headline garbage TTVH / THING STING on middle panel.

Right panel shows Image blocked by safety filter on plain-language run.

Comparison outputs from tests documented on LinkedIn (Luka Tisler).

What the outputs show

• GPT Image 2 — Strongest on spelling and legible body copy at ad scale; dystopian satire passed safety without a blank frame in this run.
• Ideogram closed — Often layout-aware: text sits in the right regions with professional ad composition; words still misspell or invent (SERENNEX, EVERYTHIING, garbled headlines).
• Ideogram open (ComfyUI) — Similar layout strength after workflow tuning; spelling still weak; extra tuning time did not close the gap to GPT Image 2. One concept hit safety filter on plain language while JSON/box mode is the documented path.

JSON prompting vs plain language (open weights)

Ideogram 4.0’s headline feature is structured JSON with bounding boxes and palettes (technical blog). The tester also ran natural language only—how most users work outside Comfy power-users. Result on at least one dystopian water ad: no image, only Image blocked by safety filter. GPT Image 2 produced the same thematic brief without that block. For future tests, always log both JSON and plain prompts on open weights.

Checklist before you add a model to mer.vin tests

• Freeze 5–10 prompts spanning text-heavy ads, posters, and one safety-edge satire brief.
• Run the same aspect ratio (16:9 recommended for Ideogram → video handoff).
• Record backend, seed/workflow version, and iteration count.
• Score with the rubric above; store PNGs under /tmp/image-model-tests/<model>/.
• Note license: Ideogram 4 weights are non-commercial on Hugging Face—production tests need a separate commercial path.
• Re-test when Comfy nodes or ideogram-oss/ideogram4 inference updates ship.

Verdict for Ideogram 4 open weights (this run)

Open weights are strategically important (local 9.3B DiT, JSON layout, 2K)—but this comparison did not show open weights matching GPT Image 2 on typography or beating closed Ideogram on spelling after real Comfy tuning time. Worth revisiting when tooling matures; not yet a drop-in replacement for text-critical ad QA.

Builder takeaway

GPT Image 2 can lock a chaotic 3×3 animation storyboard in one frame—then Seedance 2.0 reads panel order, camera notes, and destruction beats to deliver a 15-second continuous cartoon chase with far better continuity than prompt-only video.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  BRIEF[15s chase beat sheet] --> GRID[GPT Image 2 3x3 storyboard sheet]
  GRID --> QA[Annotate camera motion timing]
  QA --> SD[Seedance 2.0 image to video]
  SD --> OUT[Continuous cinematic clip]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class GRID,OUT agent
  class BRIEF,QA,SD hook

One grid locks character and layout before you spend credits on motion.

What this workflow proves

A May 2026 demo (viral on LinkedIn) pushed storyboard-to-video consistency with a full cat vs mouse chase through a wrecked house: nine pre-production panels in one sheet, each sketched like real animation layout—with camera direction, motion cues, timing notes, and escalating destruction—then animated as one flowing shot in Seedance 2.0.

Video: demonstration from Beginners Blog on LinkedIn.

Why a 3×3 grid beats nine separate stills

Treat the sheet as an animation layout, not a pretty still.

Step-by-step pipeline

• 1 — Beat the 15 seconds — Outline intro, chase peaks, and payoff (e.g. nine ~1.5–2s beats). Note what breaks in the set: lamps, plaster, furniture.
• 2 — Generate the sheet in GPT Image 2 — Ask for a 3×3 cinematic storyboard in 16:9 (community guides warn Seedance may crop odd aspect ratios). Style: animation pre-vis / storyboard sketch, not final render. Per OpenAI’s image guide, use quality: high and explicit panel numbering.
• 3 — Annotate like pre-production — Per panel: shot size (WS/MS/CU), arrow for camera move, squiggle for character motion, short timing label. Escalate destruction left-to-right or along the chase path.
• 4 — Seedance 2.0 image-to-video — Upload the grid as @image1. Prompt: follow panel order, preserve cartoon style, continuous take, match debris and motion blur between beats. Hosts include Higgsfield, Replicate, or API wrappers.
• 5 — Iterate motion first — If pacing fails, rewrite the motion prompt before regenerating the board (video runs cost far more than image edits).

Seedance motion prompt skeleton

Generate a cinematic cartoon chase using @image1 as the storyboard.
Follow panels 1→9 in order. One continuous 15-second flow, no random cuts.

Maintain: same cat and mouse designs, destroyed house layout, 2D animation
storyboard style, flying debris, motion blur on fast jumps.

Camera: motivated moves per panel arrows—low chase along floor, whip pan
through doorway, brief CU on impact, wide reveal of wrecked room.

No new characters, no photoreal shift, no panel borders visible in final.

What reviewers noticed

Strengths from the demo: Seedance 2.0 held visual continuity across panels—debris, exaggerated jumps, and camera flow felt cohesive. Critiques in thread comments flag occasional physics glitches (props moving without contact, scale drift)—worth a proofwatch pass and tighter negatives before shipping client work.

Related grid patterns

Community repos such as GPT-Image-2-Seedance2-Workflow document variants: 3×3 for 15s beats, 4×4 for denser trailers, and 5×3 for product montages. Match grid size to target length; cartoon chases benefit from fewer, readable panels with loud motion annotations.

Builder takeaway

Frontier models now score alike on benchmarks—what changes output quality is the harness around them: global rules, project context, tools, durable memory, and delegated specialists, not another clever one-shot prompt.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  P[Prompt era one message] --> C[Context era curate window]
  C --> H[Harness era model plus five parts]
  H --> P1[Personalisation]
  H --> P2[Context rules]
  H --> P3[Action tools]
  H --> P4[Memory files]
  H --> P5[Delegation agents]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class H agent
  class P,C,P1,P2,P3,P4,P5 hook

Each era wraps the last; benchmarks converge while harness depth differentiates outputs.

Three eras (each wraps the last)

Context engineering (popularised by Andrej Karpathy in 2025) reframes the job: the model has a finite attention budget, so filling the window with the right tokens beats polishing adjectives. Anthropic’s engineering write-up treats it as the natural evolution of prompt design—selecting, compressing, and isolating context across agent steps.

Harness engineering (Mitchell Hashimoto, February 2026) adds durable structure: every agent mistake becomes a rule or tool so it cannot repeat. His formula: Agent = Model + Harness. See My AI Adoption Journey for the original “engineer the harness” discipline—AGENTS.md lines from real failures, plus scripts for verification.

The moat is markdown rules, memory files, skills, and agents—not prompt wording alone.

The five harness components

Strip all five and you have a chatbot. Bolt them on and the same model can run multi-step jobs while you focus elsewhere. Practitioner guides (e.g. Wait, your Claude doesn’t have a harness?!) map the same architecture to Claude Code and Cowork: global vs project rules, connector write access, and optional Skills / Routines on top.

Folder layout at a glance

~/.claude/                   # global harness
├── CLAUDE.md                # personalisation
├── memory/                  # durable corrections
├── skills/                  # slash-command workflows
└── agents/                  # delegation specialists

your-project/
├── CLAUDE.md                # project context
├── skills/                  # project workflows
└── memory/                  # project memory

The compounding discipline

• Agent does something wrong → stop → write a permanent rule → save in the right file.
• Prune stale memory quarterly so contradictions do not pollute context.
• Prefer five well-used connectors over twenty unused MCP installs.
• Use cheaper models for execution, stronger models for strategy in delegated pipelines.
• Skills bundle context + action (+ sometimes delegation) behind one repeatable trigger.

OpenAI’s harness engineering writing and agent-first Codex workflows echo the same shift: most of the “app” is environment design, not raw model pick. When benchmarks cluster, the moat is files, memory, and verification loops—not prompt trivia.

Builder takeaway

Draw a flight path on a still image and let Seedance (via a FLORA node canvas) turn it into a continuous FPV drone take—pink lines act as motion guides the model must follow, then strip from the render.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  STILL[Illustrated still Midjourney class] --> DRAW[Pink path overlay in editor]
  DRAW --> FLORA[FLORA image node]
  FLORA --> SD[Seedance image to video]
  SD --> FPV[FPV clip guides removed]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class STILL,FPV agent
  class DRAW,FLORA,SD hook

The path colour is a layout hint; the prompt must strip guides from the final clip.

What the technique solves

A June 2026 workflow (viral alongside the Seedance drone FPV trend) shows how to steer camera motion without keyframes in a 3D package: paint a pink path on a single illustrated frame, feed that composite into FLORA, connect a Seedance video node, and describe the shot so the model follows the line while removing all guide markings from the output.

Video: demonstration from Ross Symons on LinkedIn.

Step-by-step (creator pipeline)

If pink bleeds through, tighten removal language and regenerate.

Why pink guide lines work

Video models read the uploaded frame as both appearance and layout hint. A high-contrast path colour (pink in public examples) separates “where to fly” from the artwork. The prompt must state twice that guides are for motion only and must be erased—otherwise artefacts bleed into the clip.

• One continuous take — Ask for no cuts, smooth inertia, natural banking.
• Style lock — Name painterly / watercolor / linework so Seedance does not photoreal-shift the village.
• Beat-by-beat geography — Describe start (rail tracks, low altitude), middle (narrow corridor, clock building left), end (sweep revealing valley + bridge).
• Negative constraints — No duplicated rooftops, warped tracks, watermarks, or leftover pink.

Prompt skeleton (FPV village)

Remove all pink guide lines, arrows, and drawn markings from the final video.
The pink markings are camera motion guidance only.

Create a cinematic first-person FPV drone shot in one continuous take using the
uploaded illustrated hillside village as the starting frame. The drone strictly
follows the pink path on the image. Preserve painterly storybook/anime style.

[Describe start: low along railway → village corridor → climb past smokestack →
final reveal of full valley, bridge, warm daylight.]

No cuts, no teleporting, no visible guide line, no pink markings, no warped
architecture, no text, no watermark.

FLORA canvas notes

FLORA’s video models include several Seedance variants (1.0 Pro, 2.0, 2.0 Fast) for image-to-video with optional start/end frames on some tiers. The node graph lets you iterate prompts without rebuilding the still—swap only the Seedance node or re-run after editing the pink path in Photoshop.

Alternatives mentioned in community replies: replicate the idea in other hosts (e.g. Magnific-class tools) by keeping the same pattern—marked still + explicit path-following prompt + guide removal.

Builder takeaway

Narrative TV spots with two characters, dialogue, and an emotional arc are now feasible on a GPT Image 2 → Seedance 2.0 stack when you front-load the brief into a single seven-section production board—character, wardrobe, environment, eight-shot storyboard, mood, audio, and cinematography—before animating a 15-second film.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  CONCEPT[Concept plus product hero asset] --> BOARD[GPT Image 2 one-page 7-section board]
  BOARD --> QA[Continuity QA cast wardrobe env]
  QA --> SD[Seedance 2.0 narrative plus dialogue]
  SD --> OUT[15s cinematic ad]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class BOARD,OUT agent
  class CONCEPT,QA,SD hook

Front-load the brief in a single high-res board so Seedance does not invent cast or dialogue.

What this workflow targets

A June 2026 creator workflow (shared widely on LinkedIn) argues that product-hero storyboards collapse when you need story: models default to pack shots and skip dialogue beats. The fix is not more panels—it is a structured one-image brief that forces cast, world, shot order, sound, and camera grammar before Seedance runs.

Reported outcomes from testers (treat as anecdotal): ~30 minutes from concept to a 15-second narrative ad, costs compared to $10k–$30k commercial shoots, and boards that resist “single product on white” drift when two characters exchange lines.

Example narrative spot from a board-first GPT Image → Seedance pipeline.

The seven-section production board

Pack everything GPT Image 2 must honour into one composite layout (often generated at 2K–4K widescreen). Typical zones:

Match the scaffold to the brief—mini-film with two characters or product hero montage.

Step-by-step pipeline

• 1 — Lock the product hero — Start from your pack shot or a fast render from a separate image model (creators often use Grok-class tools for cheap, controlled product stills). Upload as a reference in GPT Image 2 so the pack geometry stays stable.
• 2 — Generate the master board in GPT Image 2 — Prompt for an editorial layout with all seven sections labelled. Include the narrative script beat-by-beat inside the storyboard band. Use quality: high and a wide size such as 3840x2160 or 2560x1440 per OpenAI’s image guide.
• 3 — QA before video — Check shot order, dialogue placement, wardrobe match across frames, and that both characters stay distinct. Fix the still board only—image iterations are far cheaper than video.
• 4 — Animate in Seedance 2.0 — Upload the board as the primary reference. Use multimodal mode on platforms such as Higgsfield or API hosts like Replicate: up to ~9 images, optional audio refs, @Image tags in the prompt.
• 5 — Dialogue discipline — Put spoken lines in double quotes. For two speakers, split into separate shots (“Shot 1: @Image1 says … Shot 2: @Image2 replies …”) to reduce lip-sync swap bugs.
• 6 — Polish — Grade, music, and legal review in your NLE; reverse or trim beats in CapCut if motion direction drifts.

8-shot narrative vs 15-panel product grids

Earlier viral pipelines used 5×3 grids (15 frames) optimised for montage and product beauty. Narrative commercials need fewer, story-weighted frames—typically eight—with explicit dialogue and emotional progression. Community repos such as GPT-Image-2-Seedance2-Workflow document both patterns; pick the scaffold to match the brief (montage vs mini-film).

Vertical templates (luxury categories)

Creators packaging this system describe reusable boards for luxury beauty, fashion, fragrance, premium beverage, jewellery, and luxury automotive—same seven-section skeleton, swapped mood keywords, wardrobe, and environment maps. Treat templates as starting briefs; re-specify cast and legal lines per brand.

Seedance narrative prompt skeleton

STYLE: [match board mood — e.g. soft 35mm, warm interior]
REFERENCES: Use @[board_image] as master plan. Follow the 8 storyboard panels in order.

CONSTRAINTS:
- Two characters max on screen per shot
- No camera crew or drones visible
- Dialogue only as written in board audio section

Shot 1: @Image1 [action]. Says: "..."
Shot 2: Cut to @Image2 [action]. Says: "..."
...
Shot 8: Product hero + brand end card beat

Seedance 2.0 supports native audio sync and multi-shot output in one generation on some hosts; label every uploaded asset in the prompt so roles do not mix.

Builder takeaway

GPT Image 2 plus Seedance 2.0 can replace a traditional cinematic ad shoot when you treat the image model as pre-production and the video model as motion—locking cast, lighting, and shot order in one 5×3 (15-frame) production board before any pixels move.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph LR
  BRIEF[Creative brief scenes SFX music] --> GI[GPT Image 2 5x3 board]
  GI --> QA[Frame QA continuity cast product]
  QA --> SD[Seedance 2.0 15-scene prompt]
  SD --> EDIT[CapCut polish reverse audio]
  EDIT --> ADS[15s 4K ad units]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class GI,ADS agent
  class QA,SD,EDIT hook

Iterate on the still board first—video generations cost far more per retry.

The three-step production board workflow

A widely shared creator pipeline (popularised on LinkedIn in June 2026) breaks cinematic spots into three cheap-to-iterate stages:

• Step 1 — Build the 15-frame board in GPT Image 2 — Define 15 scenes with camera angles, SFX, and music direction. Append a cinematic storyboard prompt that requests a structured 5 columns × 3 rows grid. One image should carry cast, locations, lighting, and audio notes together.
• Step 2 — Verify before Seedance — Check scene continuity and shot order frame by frame. Confirm product and character consistency across all 15 panels. Lock music and camera-movement guidance so the video prompt does not drift.
• Step 3 — Animate in Seedance 2.0 — Upload the board as the main reference. Repeat every scene description verbatim in the Seedance prompt. Add hard constraints: no camera gear, no drone in shot, no talking. Fix reverse-motion glitches in CapCut (extract audio, reverse clip) if needed.

One grid image gives Seedance a timeline instead of fifteen disconnected frames.

Example spot from a storyboard-first GPT Image → Seedance pipeline.

Why a single grid beats 15 separate images

Community workflows collected in repos such as GPT-Image-2-Seedance2-Workflow show lower failure rates when Seedance sees one multi-panel storyboard instead of isolated frames. The model reads panel position as timeline order—similar to 3×3 grids, but a 5×3 board maps cleanly to ~15 seconds at roughly one second per beat (or 3–4 seconds per panel when you cut fewer panels).

GPT Image 2 settings for ad boards

OpenAI’s image prompting guide recommends gpt-image-2 for text-heavy, composited, identity-sensitive work. For widescreen boards:

• Size — 3840x2160 (4K landscape) or 2560x1440 (more reliable “2K” ceiling); custom sizes must use edges ≤3840 px, multiples of 16, aspect ratio ≤3:1, 655k–8.3M total pixels
• Quality — high when panels contain small labels, product logos, or dense layout; medium for faster iteration while blocking shots
• References — up to five image[] inputs for product pack shots, talent refs, or style frames
• Prompt shape — write like a creative brief: background → subject → shot list → constraints (“no watermark”, “verbatim tagline in panel 15”)

curl https://api.openai.com/v1/images/generations \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-image-2",
    "prompt": "Cinematic 5x3 storyboard grid, 15 panels left-to-right top-to-bottom. Brand: [PRODUCT]. Scenes: [SHOT LIST]. Consistent talent and lighting. Panel labels for camera + SFX. Photorealistic, 16:9.",
    "size": "3840x2160",
    "quality": "high"
  }'

Outputs above 2560×1440 are documented as experimental—budget extra retries for 4K boards.

Seedance 2.0 prompt pattern

After the board is locked, Seedance should do motion—not redesign. A template used across community case studies:

STYLE: [match board — film stock, grade, lens feel]
WORLD: [lighting, weather, colour temperature]

REFERENCES:
Use @[storyboard_image] as the main reference.
Treat each panel as sequential keyframes; expand into one coherent 15s spot.
Preserve character and product identity from the board.

CONSTRAINTS:
- Dynamic but controlled camera movement
- No camera gear visible in frame
- No drone in shot
- No talking / dialogue on camera

Scenes (repeat storyboard verbatim):
1. [scene 1]
2. [scene 2]
...
15. [scene 15]

Economics and metrics (creator-reported)

Creators running this stack have publicly claimed production cost falling from roughly $5k–$10k per shoot to about $5 per finished ad, output rising to 12 cinematic 15-second spots per week, and ~6× higher hold rate on winning variants versus their old quarterly cadence. Treat those figures as anecdotal until you replicate with your brand, offers, and distribution—image iterations are far cheaper than video generations, so the “storyboard-first” pattern is the durable lesson even if your absolute dollars differ.

Builder takeaway

SAM 3D Body (3DB) is Meta’s promptable single-image model for full-body 3D human mesh recovery—body, feet, and hands—from one RGB photo, built on the open Momentum Human Rig (MHR) representation and shipping with inference code, datasets, and CVPR 2026 oral presentation.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  IMG[Single RGB image] --> ENC[Image encoder DINOv3-H+ or ViT-H]
  PROMPT[Optional mask or 2D keypoints] --> ENC
  ENC --> BODY[Body mesh decoder]
  ENC --> HAND[Hand mesh decoder]
  BODY --> MHR[MHR parameters]
  HAND --> MHR
  MHR --> MESH[Full-body mesh body feet hands]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class ENC,MHR agent
  class PROMPT,BODY,HAND hook

Splitting hand supervision from global body pose reduces learning conflicts on fine fingers.

What it does

Given one in-the-wild image, 3DB predicts a watertight full-body mesh with articulated pose—including fine hand and foot detail—without multi-view capture or a depth sensor. Unlike classic SMPL-only pipelines, it targets the same “segment anything” interaction pattern as SAM: run fully automatic reconstruction, or steer output with segmentation masks and 2D keypoint prompts when the subject is occluded, cropped, or oddly posed.

Single RGB input to full-body mesh with body, feet, and hands.

Momentum Human Rig (MHR)

MHR separates skeletal structure from surface shape in the parametric mesh. That split makes poses easier to inspect, edit, and reuse in downstream rigs (avatars, biomechanics visualisation, AR try-on) compared with entangled body models. Meta positions MHR as the mesh backbone for 3DB and related avatar work, released under a permissive commercial license for the rig itself.

Promptable inference mirrors the SAM family when automatic mesh recovery fails.

Architecture and training

• Encoder–decoder transformer stack with a multi-input image encoder for high-resolution body regions
• Dual decoders — shared encoder feeds separate body and hand heads so global pose learning does not fight fine finger supervision
• Backbones — DINOv3-H+ (840M) and ViT-H (631M) checkpoints on Hugging Face (facebook/sam-3d-body-dinov3, sam-3d-body-vith)
• Annotations — multi-stage pipeline: manual keypoints, differentiable fitting, multi-view geometry, dense keypoint detectors, plus a data engine biased toward rare poses and hard viewpoints
• Evaluation — category-stratified benchmark set (pose and appearance buckets) for behaviour analysis beyond aggregate MPJPE

Reported benchmarks (Nov 2025 checkpoints)

Qualitative comparisons in the official repo pit 3DB against CameraHMR, NLF, and HMR2.0b on challenging occlusions and viewpoints; the paper reports gains in user-preference studies and standard HMR metrics. Treat leaderboard numbers as checkpoint-specific—re-run on your domain (sports, clinical gait, fashion) before production bets.

CVPR 2026 and SAM 3D family

3DB is accepted at CVPR 2026 (oral, pages 7209–7219) alongside SAM 3D Objects for object/scene reconstruction. A joint notebook aligns human and object meshes into one frame—useful for mixed human–object scenes in robotics or VFX previz.

Curated conference reading lists (for example the community top-cvpr-2026-papers repo) flag 3DB under pose estimation with paper, code, video, and demo links—handy when navigating 4,090 accepted papers from 16,092 submissions.

Quick start

# Hugging Face checkpoint (see INSTALL.md for access)
hf download facebook/sam-3d-body-dinov3 --local-dir checkpoints/sam-3d-body-dinov3

python demo.py \
  --image_folder ./images \
  --output_folder ./out \
  --checkpoint_path ./checkpoints/sam-3d-body-dinov3/model.ckpt \
  --mhr_path ./checkpoints/sam-3d-body-dinov3/assets/mhr_model.pt

Python API: setup_sam_3d_body(hf_repo_id="facebook/sam-3d-body-dinov3") then process_one_image. Optional --detector_name sam3 matches the public playground detector. Training data: facebook/sam-3d-body-dataset on Hugging Face.

Builder takeaway

Ideogram 4.0 is a 9.3B open-weight text-to-image model trained from scratch for design-grade output: native 2K resolution, structured JSON prompts with bounding boxes and colour palettes, and weights you can download, fine-tune, and run locally—while the hosted app, API, and MCP stay on the same model.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph LR
  JSON[JSON prompt validated] --> ENC[Qwen3-VL-8B text encoder frozen]
  ENC --> DIT[9.3B single-stream DiT 34 layers]
  NOISE[Flow-matching noise] --> DIT
  DIT --> SAM[Euler sampler asymmetric CFG]
  SAM --> VAE[KL VAE decode frozen]
  VAE --> IMG[Up to 2048 px RGB]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff

  class DIT,IMG agent
  class ENC,VAE,SAM hook

Only the 9.3B DiT is trained; encoder and VAE stay frozen at inference.

What shipped on 3 June 2026

• Open weights on Hugging Face (ideogram-ai/ideogram-4-nf4, ideogram-4-fp8)—gated; accept the Ideogram 4 Non-Commercial Model Agreement to download
• Inference code Apache 2.0 on github.com/ideogram-oss/ideogram4—nf4 fits a single 24 GB GPU; fp8 for broader hardware
• Every Ideogram plan plus the API and MCP for agent workflows—same visual model, different surfaces
• Post-training editing stack in the product: prompt edit, native transparency, layerised text, extend, reframe, upscale, remix, magic fill

Launch reel: open weights, JSON prompting, and product surfaces.

Architecture (technical blog)

Training and inference share one schema—the pipeline rejects prompts that do not parse.

Structured JSON prompting

Training and inference both use the same JSON caption schema. The reference pipeline validates every input and rejects non-conforming JSON—plain strings are expanded via an optional magic prompt (hosted API with IDEOGRAM_API_KEY, or local LLM) into the structured format.

• Bounding boxes — [y_min, x_min, y_max, x_max] in 0–1000 normalised coords (origin top-left)
• Colour palettes — up to 16 hex colours per image, 5 per element
• Typed text — text elements carry the literal string plus a styling description for multi-font posters
• Composable elements — obj and text entries under compositional_deconstruction

python run_inference.py \
  --prompt "campaign poster with clean type" \
  --output out.png \
  --quantization nf4 \
  --magic-prompt-key "$IDEOGRAM_API_KEY"

Benchmark claims (treat as directional)

Independent smoke tests on single prompts can disagree with vendor charts—run your own evals for brand, type, and layout tasks that matter to you.

Three ways in (API, MCP, app)

Partner integrations mentioned at launch include ComfyUI, fal, Replicate, Krea, Leonardo, Cloudflare, and others—check each host for rollout status.

License reality check

Code: Apache 2.0. Weights: Ideogram 4 Non-Commercial Model Agreement (gated on Hugging Face). You can inspect, run locally, and fine-tune for research and non-commercial work; commercial deployment needs a separate commercial path via Ideogram. That split is common in “open weight” image releases—open code ≠ unrestricted commercial weights.

Builder takeaway

Generative UI lets agents render real interface widgets—not only chat text—so a user who asks for a table gets a table, a budget gets cards, and tool output streams inline over a standard agent↔frontend wire.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  USER[User in React app] <-->|SSE AG-UI events| RUN[CopilotKit runtime proxy]
  RUN <-->|tool calls state deltas| AGENT[Agent backend ADK LangGraph etc]
  AGENT --> MCP[MCP tools and data]
  AGENT --> A2A[A2A other agents]
  AGENT --> A2UI[A2UI schema ops on AG-UI stream]
  A2UI --> RENDER[Catalog maps JSON to components]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  classDef decision fill:#444,color:#fff

  class USER,AGENT agent
  class RUN,A2UI hook
  class RENDER decision

Pick the pattern on purpose—most drift into Controlled because the framework default does.

The protocol stack (three jobs)

AG-UI is an open, event-based protocol (MIT, ag-ui-protocol) born from CopilotKit’s agent↔UI work with LangGraph and CrewAI. During a run the backend emits typed events—text chunks, TOOL_CALL_START/END, STATE_DELTA patches—over a single HTTP POST plus SSE stream. State can flow both ways on the same channel: user edits surface to the agent; agent mutations surface to the UI without a second model call when you wire shared state.

A2UI (Apache 2.0, a2ui.org) is Google’s declarative spec for agents emitting UI as JSON schema. It rides on AG-UI; CopilotKit ships production renderers. v0.9 moves to a prompt–generate–validate loop: catalog rules live in the system prompt, the model generates freely, validators catch errors, and the agent self-corrects before the client sees bad JSON.

Past about 15 render tools, declarative A2UI usually pays for itself in context window alone.

Three patterns (not three frameworks)

Most teams confuse “Generative UI” with whichever pattern their framework defaults to. In practice there are three architectural choices on a control→flexibility spectrum:

Pattern 1 — Controlled (frontend owns UI)

You register a React component against a tool name (e.g. CopilotKit’s frontend action hook). The runtime advertises the tool over AG-UI; when the agent calls it, args stream in as props and the component renders inline. No Python tool required for the happy path—design tokens stay yours.

Token tax: every registered component sits in context before the user speaks. ~400 tokens per tool description × 25 components ≈ 10,000 tokens per turn. Past ~15 tools, descriptions overlap (“pie chart” vs “donut chart”) and mispicks rise. Fix descriptions around user intent (“compare proportions of a whole”) not widget names.

Shared state exception: when the agent must pin a metric or append a table row and other panes must update without another LLM call, add an agent-side tool that writes session state; the UI subscribes via the shared-state hook while chat still uses the same frontend tool name.

Pattern 2 — Declarative (A2UI schema)

The agent returns an ordered list of operations—typically create_surface, update_components, update_data_model—with a catalogId your frontend registered. One function can power dozens of card types; token cost stays flat as the library grows.

def search_flights(flights: list[Flight]) -> dict:
    return {
        "a2ui_operations": [
            {"type": "create_surface", "surfaceId": SURFACE_ID, "catalogId": CATALOG_ID},
            {"type": "update_components", "surfaceId": SURFACE_ID, "components": FLIGHT_SCHEMA},
            {"type": "update_data_model", "surfaceId": SURFACE_ID, "data": {"flights": flights}},
        ]
    }

Fixed vs dynamic schema: in fixed mode you author flights.json and the agent only supplies data; in dynamic mode a secondary LLM drafts the component tree per turn but still emits the same a2ui_operations envelope. The catalog is the contract—Zod (or JSON Schema) for allowed components; renderers map types to React. A common production bug: CATALOG_ID on the agent and catalogId in createCatalog on the client differ by one character—UI silently falls back to the basic catalog with no console error.

Trade-off: the model owns layout within the catalog; runs vary. Not for legal copy or marketing surfaces that need pixel lock.

Pattern 3 — Open-ended (MCP Apps and sandboxed HTML)

• MCP Apps — MCP servers expose UI surfaces (e.g. diagram canvases). CopilotKit’s MCP Apps middleware attaches servers without hand-rolling the client protocol.
• Sandboxed HTML — the runtime injects an HTML render tool; the agent returns markup inside an iframe with sandbox allowing scripts + forms, never allow-same-origin.

Open-ended shines for disposable answers (“visualise this API response”) and fails as a primary product surface—brand and layout drift run to run even with style rules in the system prompt. Typical iframe failure: buttons dead because sandbox flags omit allow-forms.

How to choose (decision tree)

• Designer shipped pixel-perfect mocks for this flow? → Controlled
• Dozens of card types or widgets? → Declarative
• One-shot chart the user will never see again? → Open-ended
• Unsure? → default Declarative; promote top 3 flows to Controlled; never default Open-ended
• Already shipping >15 render tools? → you are in Controlled territory; start A2UI this week

Open templates (awesome-llm-apps)

The generative_ui_agents folder ships runnable references across all three patterns:

Builder takeaway

Gemma 4 12B is Google DeepMind’s new encoder-free multimodal open model: text, image, video, and native audio flow through one decoder-only transformer under Apache 2.0, aimed at laptops with about 16 GB VRAM or unified memory and reasoning that approaches the larger 26B MoE sibling.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  IMG[48x48 image patches] --> LLM[Gemma 4 12B decoder]
  AUD[16 kHz audio frames] --> LLM
  TXT[Text tokens] --> LLM
  LLM --> OUT[Text plus tool calls]

  ENC[Separate vision and audio encoders] -.->|not used in 12B| LLM

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  class LLM agent
  class ENC hook

Vision and audio enter the same transformer as text—no separate vision or audio encoder stacks.

What encoder-free changes

Earlier Gemma 4 sizes (E2B/E4B and 31B) attach dedicated vision transformers (~150M–550M) and audio conformers (~300M). Gemma 4 12B Unified removes those stacks:

• Vision (~35M embedder) — 48×48 pixel patches projected with one matmul plus factorised X/Y positional lookups; the LLM backbone does the heavy visual reasoning.
• Audio — no conformer encoder; 16 kHz audio is sliced into 40 ms frames (640 floats) and linearly projected into token space.
• Fine-tuning — LoRA or full tuning updates vision, audio, and text in one pass (Hugging Face, Unsloth) instead of co-tuning frozen encoders.

Google positions this as lower multimodal latency, a smaller memory footprint than medium models with separate encoders, and the first mid-sized Gemma with onboard audio (audio was previously limited to small edge variants).

Google targets ~16 GB VRAM or unified memory with Q4 weights around 6.7 GB plus KV cache.

Model specs and memory

Benchmark snapshot (instruction-tuned)

Figures below come from the official Gemma 4 12B model card (June 2026), comparing instruction-tuned variants:

On several agentic and multimodal scores, 12B sits close to the 26B active-MoE line while using less than half the memory footprint of the full 26B weight load—matching Google’s “laptop-class agent” positioning.

Latency tricks: MTP and LiteRT-LM

Gemma 4 12B ships with a multi-token prediction (MTP) draft model for speculative decoding—higher tokens/sec without changing output quality. For local apps, Google highlights:

• Google AI Edge Gallery on macOS (offline on Apple Silicon, sandboxed Python in-chat)
• Google AI Edge Eloquent — voice-edit style input with Gemma 12B
• litert-lm serve — OpenAI-compatible local API for Continue, Aider, OpenCode, etc.
• Ollama, LM Studio, llama.cpp, MLX, vLLM, SGLang — community inference paths

pip install -U transformers torch accelerate

# Instruction-tuned checkpoint
MODEL_ID = "google/gemma-4-12B-it"

# Local OpenAI-style server (LiteRT-LM)
litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm \
  gemma-4-12B-it.litertlm gemma4-12b
litert-lm serve

Multimodal usage notes

• Put images before text; put audio after text for best results.
• Visual token budgets: 70, 140, 280, 560, 1120 — trade speed vs OCR/detail.
• Audio clips up to 30 s; video up to 60 s at ~1 FPS (per model card).
• Enable chain-of-thought with <|think|> in the system prompt; omit prior thoughts from chat history on follow-up turns.

Where it sits in the Gemma 4 family

Gemma 4 downloads crossed 150 million (Google, June 2026). Weights and notebooks live on Hugging Face and Kaggle; production deploy paths include Model Garden, Cloud Run, and GKE.

Builder takeaway