CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation

Nguyen, Long-Bao; Le, Quang-Khai; Nguyen, Tam V.; Tran, Minh-Triet; Le, Trung-Nghia

Benchmark

CogCanvas: A Benchmark for Evaluating
Multi-Subject Reference-Based Image Generation

Long-Bao Nguyen^1,2,† , Quang-Khai Le^1,2,† , Tam V. Nguyen³ , Minh-Triet Tran^1,2 , Trung-Nghia Le^1,2,‡

¹University of Science, Ho Chi Minh City, Vietnam · ²Vietnam National University, Ho Chi Minh City, Vietnam · ³University of Dayton, Ohio, United States
^†Equal Contribution · ^‡Corresponding Author

📄 Paper 📝 arXiv 💻 Code 🤗 Dataset

Dataset at a Glance

100

Celebrity Identities
(CelebA-HQ)

115

Unique Items
(80 objects + 35 fashion)

29

Background Scenes
(incl. Vietnamese landmarks)

1,952

Reference Images
(total curated pool)

1,361

Compositional Prompts
(2–5 persons)

6

Evaluation Axes
(ID, DINO, CLIP, BG, Aesth., VQA)

Abstract

We introduce CogCanvas, a benchmark for evaluating multi-subject, reference-based image generation under complex spatial and interaction constraints. While existing benchmarks address either single-subject personalization or simple compositional text prompts, none simultaneously evaluates multi-identity scenes with explicit human–object interaction, background scene grounding, and spatial plausibility.

CogCanvas provides a curated reference pool of 1,952 images spanning 100 celebrity identities, 115 unique objects and fashion items, and 29 real-world background scenes, from which we construct 1,361 compositional prompts covering group sizes of 2–5 persons. Each prompt is annotated with structured interaction and position graphs as ground-truth supervision.

We define a six-axis evaluation protocol: identity preservation, object/fashion attribute binding, text–image alignment, background consistency, aesthetic quality, and attribute/interaction evaluation. Benchmarking five representative state-of-the-art methods reveals that all models degrade substantially as group size increases from 2 to 5, with near-complete failure on attribute-binding metrics beyond 3 subjects.

How CogCanvas Compares

CogCanvas is the only benchmark covering all five evaluation dimensions simultaneously.

Benchmark	Venue	#Prompts	Human Refs	Object Refs	HOI	BG	Spatial
T2I-CompBench	NeurIPS'23	6,000	0	0	–	–	Partial
DreamBench	CVPR'23	750	0	30	–	–	–
DreamBench++	ICLR'25	1,350	20	110	–	–	–
MultiHuman-TB	NeurIPS'25	1,800	5,550	0	Partial	–	–
XVerseBench	NeurIPS'25	300	20	119	Partial	–	–
CogCanvas (Ours)	–	1,361	100	115	✓	✓	✓

HOI = Human–Object Interaction evaluation · BG = Background scene reference · Spatial = spatial plausibility evaluation

Six-Axis Evaluation Protocol

ID-Sim (Identity Similarity)

ArcFace cosine similarity between reference and generated faces. Uses Hungarian matching to align detected faces to reference identities in multi-person scenes.

DINO-Sim (Object/Fashion)

DINOv2 cosine similarity for object/fashion regions. Captures holistic semantic similarity for non-face entities such as specific product models and clothing.

CLIP-T (Text Alignment)

CLIP similarity between the input prompt and the generated image. Measures overall text–image semantic alignment but cannot alone distinguish correct identity assignments.

BG-Sim (Background)

DINOv3 similarity on SAM-masked background regions. Directly penalizes hallucinated or semantically inconsistent backgrounds without manual annotation.

Aesth. (Aesthetic Quality)

LAION aesthetic predictor score. Evaluates perceptual quality independent of content accuracy — necessary but not sufficient for benchmark success.

Attr-VQA (Attribute + Interaction)

Gemma4 MLLM binary VQA for attribute binding ("Does person at [pos] wear a [color] outfit?") and interaction consistency from ground-truth interaction graphs.

Quantitative Results

All methods degrade substantially as group size grows from 2 to 5. Underline = best among evaluated methods. ↑ higher is better.

N	Method	ID-Sim ↑	DINO-Sim ↑	CLIP-T ↑	BG-Sim ↑	Aesth. ↑	Attr-VQA ↑
N = 2 (558 prompts)
2	OmniGen2	13.07	33.16	62.23	—	51.63	33.04
2	UNO	23.23	28.03	62.11	—	55.24	35.12
2	DreamO	3.07	5.98	58.79	—	44.42	14.45
2	XVerse	42.33	32.96	61.60	—	39.73	7.80
2	MOSAIC	9.93	4.77	57.29	—	44.01	11.22
N = 3 (382 prompts)
3	OmniGen2	0.62	1.74	57.57	—	37.11	3.68
3	UNO	19.60	17.35	61.02	—	56.01	14.65
3	DreamO	0.39	1.20	57.47	—	41.48	2.99
3	XVerse	27.21	15.62	60.08	—	38.46	4.03
3	MOSAIC	11.45	5.87	57.52	—	42.66	5.67
N = 4 (225 prompts)
4	OmniGen2	0.39	1.60	57.63	—	37.36	5.78
4	UNO	19.60	17.35	61.02	—	56.01	14.65
4	DreamO	0.19	0.75	57.39	—	42.34	4.63
4	XVerse	2.67	2.63	57.71	—	36.54	3.17
4	MOSAIC	2.46	1.99	57.34	—	37.73	2.61
N = 5 (196 prompts)
5	OmniGen2	0.08	0.43	57.38	—	36.85	2.49
5	UNO	13.47	8.58	59.26	—	56.40	7.27
5	DreamO	0.03	0.17	57.24	—	37.35	1.79
5	XVerse	0.52	0.46	57.15	—	36.66	1.28
5	MOSAIC	0.18	0.24	57.24	—	36.79	1.92

BG-Sim results pending. OmniGen2, DreamO, XVerse, MOSAIC score near zero on DINO-Sim and Attr-VQA at N ≥ 4.

BibTeX

@article{Nguyen2026CogCanvas,
  title={CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation},
  author={Long-Bao Nguyen and Quang-Khai Le and Tam V. Nguyen and Minh-Triet Tran and Trung-Nghia Le },
  journal={arXiv preprint arXiv:2606.15867},
  year={2026},
}