ADR-platform-005
Hybrid data-layout unification (`data/habitat` + `data/mp3d` + `data/opennav` + `data/outputs`)
⚠ Partially superseded (2026-05-15). The
data/outputs/{eval, eval_mock, eval_native}consolidation decided below is no longer in effect: ADR-eval-004 re-rooted all eval, canvas, and design-run outputs at the top-leveloutputs/tree (outputs/eval_runs/{run_id}/,outputs/runs/,outputs/design_runs/), and no live code writes todata/outputs/anymore. Thedata/habitat/data/mp3d/data/opennavportions of this ADR remain in force.
Context
The pre-2026-04-17 data/ tree was a flat mix of framework-shaped children (scene_datasets/, datasets/, checkpoints/, ddppo-models/) that VLN-CE and habitat-sim both hardcode relative to data/, plus tool-specific dirs (opennav/ for RAM + SpatialBot + waypoint predictor + Open-Nav DDPPO), plus empty eval output dirs (eval_output/, eval_mock_output/, eval_native_output/), plus an MP3D skybox bucket that had to exist at data/v1/scans/ (another hardcoded VLN-CE/MatterSim location) but was undownloaded. Two classes of pain: (1) future env nodesets (RxR-CE, REVERIE, AI2-THOR) had no obvious slot and would force a second migration; (2) three files under workspace/nodesets/server/opennav_*.py hardcoded absolute /path/to/vlnworkspace/data/opennav/... paths, breaking any clone on a different host. A separate MatterSim skybox preprocessing step (merged <vp>_skybox_small.jpg 3072×512 files) was also missing, which blocked panorama rendering once scans were downloaded.
Decision
Adopt Option-C hybrid layout: data/habitat/{scene_datasets, datasets, checkpoints, ddppo-models} as a VLN-CE-native framework mirror; data/mp3d/v1/scans/<scan>/matterport_skybox_images.zip as an asset-kind namespace consumed today by MatterSim and open to any future simulator reading the same asset; data/opennav/ as a tool-agnostic perception namespace (RAM + SpatialBot + waypoint + DDPPO) kept in place to avoid moving 6.4 GB of weights; data/outputs/{eval, eval_mock, eval_native} consolidation. VLN-CE back-compat via 4 child symlinks (third_party/VLN-CE/data/{datasets,scene_datasets,checkpoints,ddppo-models} → ../../../data/habitat/<name>) inside a real parent directory so submodule-tracked siblings connectivity_graphs.pkl + res/ are preserved. opennav nodesets refactored to _REPO_ROOT + env-var-first defaults (no more absolute paths). New scripts/data/migrate_data_layout.sh runs the whole thing atomically via same-filesystem rename(2) with a --dry-run gate, idempotent guards on every step, and a preflight that checks stat -c %m, free space, the .gitignore /data anchor, and the submodule-artifact preservation invariant. A separate scripts/data/gen_skybox_rgb_mp3d.sh handles the MatterSim per-scan merge step (unzip → inline cv2 merge of 6 faces into 3072×512 → cleanup raw faces), avoiding the upstream downsize_skybox.downsizeWithMerge helper because its camera_parameters() dependency requires undistorted_camera_parameters/*.conf from a different MP3D bundle we don't download. workspace/nodesets/server/matterport3d.py fixed to pass the scans dir (not its parent) to sim.setDatasetPath, and to anchor MATTERPORT_DATA_DIR suffix-stripping on os.sep + "scans".
Alternatives
(a) Asset-kind-only layout (data/scenes/mp3d_glb, data/episodes/r2r_ce, etc.) — rejected because VLN-CE's hardcoded data/scene_datasets, data/datasets, data/checkpoints, data/ddppo-models would then need one symlink per asset class, multiplying the symlink surface. (b) Pure-framework layout with data/habitat, data/mattersim, data/shared — rejected because data/shared/ would hold only opennav/ today, i.e. gratuitous wrapping. (c) Single parent-level VLN-CE symlink third_party/VLN-CE/data → data/habitat — rejected because it would mask submodule-tracked connectivity_graphs.pkl + res/ (consumed by TOP_DOWN_MAP_VLNCE codepath and submodule READMEs). (d) Naming the asset namespace data/mattersim/ instead of data/mp3d/ — rejected because MatterSim is an archived upstream toolkit; MP3D is the durable asset class, and future consumers of the same skybox archive shouldn't force another rename. (e) Physical relocation of data/opennav/ into a new data/shared/perception/ — rejected to avoid moving 6.4 GB of already-downloaded weights for zero correctness gain. (f) Reusing downsize_skybox.downsizeWithMerge from the Matterport3DSimulator submodule for the merge step — rejected because it needs undistorted_camera_parameters/*.conf, a separate MP3D release bundle not in our download set; the inline cv2 merger derives pano ids from *_skybox0_sami.jpg filenames post-extraction instead.
Rationale
The hybrid is honest about what the shapes actually are — habitat/ is a framework mirror (VLN-CE hardcodes data/..., and the 4 existing install_ac_vlnce.sh-created symlinks already have this shape; the retarget is the minimum diff), mp3d/ is an asset-kind namespace (consumer-agnostic — survives a MatterSim replacement), opennav/ is a tool namespace. Child-level symlinks preserve the submodule artifacts at the parent level; a parent-level symlink would delete them. Same-filesystem atomic rename(2) is O(1) and needs no copy / no duplicate disk space — the whole migration is a bounded script with idempotent guards. New env nodesets (RxR-CE, REVERIE, AI2-THOR) slot in as sibling top-level dirs without a second migration. The _REPO_ROOT-relative opennav defaults make the clone portable; env-var-first precedence (OPENNAV_RAM_CKPT, OPENNAV_SPATIALBOT_PATH, OPENNAV_WAYPOINT_CKPT, OPENNAV_DDPPO_CKPT) retains operator flexibility. Splitting into 3 PRs (PR-1 opennav cleanup, PR-2 migration+rewrites, PR-3 17 GB download) bounded blast radius; the skybox preprocessing was a follow-up exposed only after rendering was exercised.
Affected docs
core/roadmap.md (Done entry + TODO #43 closure), core/blueprint.md (no change — data layout is deployment/install, not canvas/executor), core/architecture.md (no change — framework import boundary already documented in ADR-platform-004)