Global Graph-Validated Optimization for VLM-Based 3D Indoor Scene Generation
We study open-vocabulary 3D indoor layout generation, which synthesizes diverse and physically plausible indoor scenes from unlabeled 3D assets given free-form language instructions. Recent text-guided layout generation methods leverage large language models (LLMs) and vision-language models (VLMs) to synthesize structured scenes directly from text descriptions. However, most existing approaches model inter-asset relations implicitly or rely on local pairwise constraints during local optimization. Such formulations are misaligned with the global and highly non-convex feasible layout space, often producing locally plausible but globally inconsistent or physically infeasible scenes. To address these limitations, we introduce a graph-based intermediate representation that decouples semantic coherence and physical feasibility, and propose a hybrid search-and-refinement strategy to generate indoor layouts with global semantic consistency and physical feasibility. First, we propose Global Semantic Verification (GSV), which represents scenes as structured scene graphs and enforces semantic constraints through rule-based graph verification. This explicit structural validation prunes contradictory configurations and produces a globally consistent semantic scaffold for scene generation. Second, we introduce Global Physical Feasibility Search (GPFS), a hybrid optimization framework that combines evolutionary search for global exploration with gradient-based refinement for local exploitation. GPFS reduces dependence on VLM-proposed initialization and improves robustness in highly non-convex and discontinuous feasible spaces. Together, GSV and GPFS shift layout generation from local relational modeling and initialization-sensitive optimization toward globally consistent reasoning and exploration. Experiments demonstrate that our method achieves state-of-the-art performance on open-vocabulary 3D indoor layout generation, improving both semantic consistency and physical plausibility.
Cite this item
Huang, J., You, Y., Wang, F., & Dang, Z. (2026). Global Graph-Validated Optimization for VLM-Based 3D Indoor Scene Generation. In P. Favaro, Z. Kukelova, A. Maki, A. Rohrbach, K. Schindler, & F. Tombari (Editors), Computer Vision – ECCV 2026. https://doi.org/10.1007/978-3-032-37461-5_21
Click Copy to copy the citation, or select the text manually.
10.1007_978-3-032-37461-5_21.pdf
Main Document
Submitted version (Preprint)
openaccess
CC BY-NC-SA
15.55 MB
Adobe PDF
4613483ff654c2a06b89fa3c15c4c867