AWSM

01 / OVERVIEW

A scene can look plausible and still be spatially wrong.

Agentic reconstruction uses a large model to inspect observations, call modeling tools, and iteratively build and check an editable scene. The challenge is not only to produce a convincing room, but to preserve the scale, shape, and spatial relationships that make it useful beyond rendering. AWSM asks how geometric evidence can constrain this process so that its outputs become more faithful spatial references for downstream agents.

We reconstruct World Lobby, a simulated scene in NVIDIA Isaac Sim, through four complete engineering routes, from RGB-only modeling to ground-truth-pose-conditioned modeling. Each route produces a frozen Blender scene, not merely a point cloud or a novel-view renderer. The experiment asks what additional geometric evidence buys, where it fails to help, and which errors survive object-centric reconstruction.

Single-scene M1 → M4 diagnostic comparison: M1 uses GT-assisted Sim(3) alignment; M4 uses GT camera poses. Relative reductions compare complete pipelines, not IMU alone.

02 / MOTIVATION

From plausible reconstruction to a usable spatial reference

A reconstruction agent can assemble a plausible room while getting a corner, distance, or passage wrong. For an embodied agent, those are not merely visual defects: they change where a destination lies and which route a body can follow. Geometry grounding means constraining the modeling process with pose, depth, and metric evidence where available, rather than relying on visual plausibility alone. The question is how those constraints survive the conversion from observations into editable objects.

The reception-desk task makes this connection concrete: mark a destination in the reconstructed map, specify routes, and execute them with robot controllers. We use ‘scene representation’ for the editable objects and geometry, ‘map’ for their navigation role, and ‘simulation assets’ for their integration into a simulator. The ‘world’ in AWSM refers to this reusable spatial environment, not a learned dynamics predictor. A persistent spatial foundation connecting navigation, memory, interaction, and simulation is the longer-term goal; the experiments and demo examine its geometric basis and one downstream use.

Contributions

1. A four-route study of geometry-grounded agentic reconstruction, keeping the scene, 180 RGB identities, modeling objective, and editable output format fixed while comparing complete pipelines with different geometric evidence.
2. An evaluation protocol that separates trajectory error, native depth, final-scene depth, surface geometry, and rendered appearance, with frozen Blend/GLB assets, reproducible evaluation subsets, public hashes, and interactive comparisons.
3. A map-based multi-robot simulation example connecting a reconstructed scene to destination marking, predefined routes, and controller execution. It illustrates downstream use rather than a controlled navigation comparison across reconstruction routes.

03 / WORKFLOW

One agentic workflow, four reconstruction routes

The reconstruction agent follows a shared observe–build–verify loop: inspect the available evidence, write Blender Python to construct objects, render review views, and revise the scene before freezing it for evaluation. This tool-using modeling process is what ‘agentic’ describes here; the downstream robot controllers have a separate role.

All routes receive the same 180 RGB identities and the same Astra–Blender modeling objective. M1 tests how far visual priors can go without metric measurements. M2 adds RGB-only ViPE poses and pose-conditioned DA3 depth. M3 uses monocular-inertial ORB-SLAM3 poses with the same class of DA3 observations. M4 supplies ground-truth camera poses to diagnose the remaining modeling error; it does not receive the GT mesh or GT depth as modeling input.

Agentic reconstruction loop

Ground truth is introduced only after the scenes are frozen. M2 and M3 are evaluated under one global SE(3) alignment with scale fixed to one; no mesh ICP or per-view fitting is used. M1 has no native metric trajectory, so its model-space numbers use one frozen GT-assisted Sim(3) from five manual camera associations and remain diagnostic rather than evidence of metric recovery.

04 / RESULTS AND ANALYSIS

Three findings matter more than a leaderboard

Inspect the reconstructed scenes, not just aggregate scores

Use the shared camera to compare each reconstruction with GT. Start with M1 to see how a semantically plausible room can drift in metric layout; compare M2 and M3 to find regions where geometry improves without a matching gain in RGB similarity; then inspect M4 to isolate errors that remain even when camera pose is no longer the bottleneck.

Rotate, zoom, and inspect the frozen scenes

Reconstruction / GTM1M2M3M4GTDisplaySplit comparison with GTView selected scene aloneReset view

M4GT

Scroll here to load the 3D comparison.

Retry

MethodM1M2M3M4Fixed view03672108144

M1 / VIEW 0

INPUT / GT RGB

The browser hides ceilings and surrounding walls only for cutaway inspection. Runtime registration and display choices do not modify the downloadable frozen GLB or Blend files.

What geometric grounding changes—and what it does not

Modeling behaves like a structured—and lossy—regularizer

For M2–M4, the final scene depth is substantially closer to GT than native DA3 depth on the shared 180-view protocol: AbsRel falls by roughly 51–60%. One plausible interpretation is that object-level aggregation suppresses inconsistent local predictions. This is a hypothesis, not a controlled causal result: the same abstraction can also erase real surfaces and fine geometry.

Measurement constrains generation; it does not solve appearance

The strongest geometric route does not dominate PSNR or SSIM. Pose removes one source of uncertainty, but appearance still depends on material estimation, lighting, object detail, and renderer mismatch. Evaluating only RGB would miss spatial progress; evaluating only geometry would miss perceptual failure.

The 100-view check measures consistency, not unseen-view generalization

Table 7 uses a seeded set of 100 cameras drawn from the 175 modeling frames outside the fixed-five check. These are additional evaluation views relative to that check, but their RGB images were available during modeling. On this set, M4 has the lowest depth AbsRel (7.47%), while M3 has the lowest RMSE (1.090 m).

Beyond the lobby: what survives in a phone-captured space?

Office Café is not a second run of the World Lobby benchmark and is not quantitative validation under the same protocol. It is an external, real-capture case study showing how an object-centric scene, a solved source-camera trajectory, video-to-model comparison, and physics replays can coexist in one inspectable artifact. An author-observed failure—a curved real-world corner simplified into a square one—motivates separating local shape from global scale. A correct scale alone cannot repair that shape error; this is a qualitative observation, not another measured benchmark.

Open full-screen model ↗Open shake experiment ↗

Explore the original interactive apps

Links request English with lang=en; the external host controls language and availability. These local content previews are not screenshots of the external website. If embedding is unavailable, use the full-screen links or the local models and videos below.

Both models open in an elevated cutaway view: upper geometry is clipped only for display to expose the interior. Toggle Cutaway to inspect the complete model; Reset view restores the overview. Authored coordinates, materials, and model files are unchanged. Linked orbit controls compare layouts, not registered geometry.

▶

00:00 / 01:10

Shared playback · 70.1 seconds · 30 fps

05 / RELATED WORK

From geometric measurement to editable Real2Sim

AWSM connects two roles: an agent that constructs an editable scene, and downstream agents that use it as a spatial reference. SLAM and visual-inertial estimation supply pose and metric constraints; ViPE and Depth Anything 3 contribute geometric observations. Concurrent work AHa-3D explores video-driven, tool-based Real2Sim, including camera and 3D-reference estimation from video. Our emphasis is geometric grounding: bringing additional physical measurements, especially IMU-informed visual-inertial pose and metric constraints, into agentic scene construction rather than relying on video-derived evidence alone. The four-route comparison tests reconstruction systems, not the isolated effect of IMU; the demo illustrates a map-based application, not a complete autonomous embodied system.

Our evaluation therefore spans both sides of the interface: upstream systems are judged as measurement pipelines, while the Blender outputs are judged as persistent spatial artifacts. This differs from evaluating pose, depth, or novel-view synthesis in isolation.

06 / LIMITATIONS

The boundaries are part of the result

• One synthetic scene and one engineering run per route do not establish general superiority or statistical significance.
• The four routes are complete systems, not a strict single-variable ablation; M2 versus M3 cannot be attributed to IMU alone.
• M4 uses GT camera poses and is a diagnostic route, not a deployable baseline or a theoretical upper bound.
• M1 model-space metrics rely on GT-assisted Sim(3) and must not be ranked as native metric recovery.
• The 100-view appearance and depth sets contain modeling RGB inputs; they are not a held-out novel-view benchmark.
• Appearance metrics mix geometry, material, illumination, and rendering error. M4 revision 4 has not received a complete independent visual re-review.
• The reception-desk demo is map-based simulation execution, not a controlled comparison of navigation across M1–M4. Routes are predefined and localization uses simulator pose; see the demo protocol for scope.
• Multimodal memory, long-term map maintenance, and reduced policy sim-to-real gap are research goals, not measured outcomes of the four-robot demo. The Office Café corner observation has no matched-view quantitative GT measurement here.
• Historical downstream documentation describes the same M4 Blend hash using a different depth frontend. The frozen tables and assets are retained unchanged; their upstream naming needs reconciliation before attributing historical task outcomes to DA3. See the provenance note.

Every table is generated from frozen assets. The public manifests expose model hashes, registrations, fixed views, and the exact seeded evaluation indices.

Tables JSON · Scene manifest · Fixed views · Publication manifest · SHA256

Version and provenance note ↗

07 / CITATION

Cite this work

If this controlled study, its frozen assets, or its evaluation protocol supports your work, please cite the interactive article below.

@misc{awsm_2026,
  title        = {{AWSM}: Agentic World Simulation and Mapping},
  author       = {{Phygital AI}},
  year         = {2026},
  howpublished = {Interactive research article},
  url          = {https://phygital-ai.github.io/agentic-world-simulation-and-mapping/},
  note         = {Reconstruction benchmarks, editable scene assets, and map-based embodied demonstration}
}

08 / REFERENCES

References

1. AHa-3D: Agentic Tool Use for Real2Sim with GPT-6 Astra.
2. ViPE: Video Pose Engine for camera motion and near-metric geometry.
3. Campos et al. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM. IEEE Transactions on Robotics, 2021.
4. Depth Anything 3: Recovering the Visual Space from Any Views.
5. Wang et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004.
6. Zhang et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR, 2018.
7. Blender Foundation. Blender: Free and Open Source 3D Creation Suite.
8. Qin et al. VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator. TRO, 2018.
9. Keetha et al. MapAnything: Universal Feed-Forward Metric 3D Reconstruction. 2025.
10. Zhou et al. Memory Over Maps: 3D Object Localization Without Reconstruction. 2026.
11. Yang et al. 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning. 2024.
12. Khanna et al. GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation. CVPR, 2024.
13. ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning. ICRA, 2024.
14. Clio: Real-time Task-Driven Open-Set 3D Scene Graphs. RA-L, 2024.