Photogrammetry, NeRF and 3D Gaussian Splatting

Three impressive outputs can still be three different things

A small room has been captured with overlapping photographs. The team wants both an interactive walkthrough and geometry that can be edited later. A sparse/dense SfM-MVS result, a NeRF, and a 3D Gaussian Splatting representation all look plausible in selected views. If the decision rule is only “which image looks best?”, a representation optimized for view synthesis can easily be mistaken for an editable production asset.

In this lesson you will build Capture-to-Reconstruction Method Comparison and Evidence Pack v1. The pack keeps capture quality, pose source, SfM/MVS status, representation type, reconstruction/training cost, novel-view quality, geometry semantics, editability, metric scale, hardware, provenance, failure modes, and downstream target visible before a method is selected.

The three method families are compared without being collapsed into one output type. COLMAP is a general-purpose SfM/MVS pipeline that recovers camera poses and sparse structure before dense reconstruction. The original NeRF models a scene as a continuous volumetric function for novel-view synthesis. The original 3D Gaussian Splatting work represents a scene with optimized 3D Gaussians and targets high-quality real-time novel-view rendering. None of those facts independently determines the best production path. [COLMAP] [COLMAPTUTORIAL] [NERF] [3DGS]

Lesson artifact: an auditable bounded decision using SELECTED, TARGET_SPECIFIC, REWORK_REQUIRED, HOLD, EVIDENCE_INCOMPLETE, PROVENANCE_UNKNOWN, or METHOD_UNSUITABLE without claiming more than the evidence supports.

Planned active time: 60 minutes.

Scope boundary: D3D-003 owns reconstruction-method comparison. It does not absorb the UV/PBR/texturing outcome of D3D-002 or the final optimization/interoperability outcome of D3D-004.

HARD prerequisite — D3D-001

Carry forward this boundary: a convincing viewport result does not replace evidence about geometry intent, transforms, scale, topology/normal limitations, and provenance. If that distinction is fragile, rebuild equivalent prerequisite evidence first; do not fabricate prerequisite completion.

SOFT prerequisite — D3D-002

When capture or reconstructed output depends on texture/material evidence, carry forward only the evidence D3D-003 needs. This lesson does not reteach the texturing workflow.

Choose your route

Answer four questions:

1. Does strong novel-view quality prove that the result is a conventional editable mesh?
2. If SfM poses are unstable, can you record a positive state just because you prefer NeRF or 3DGS?
3. If the downstream target requires a metric-scale editable mesh, is a view-synthesis-only path sufficient?
4. If the subject changes from a textured room to a glossy weak-texture object, should the old capture strategy be copied?

4/4 → Fast Path. 2–3/4 → Repair Path. 0–1/4 → Foundation Path. Every route converges on the same evidence bar.

Route self-check: Q1 should identify capture/overlap and pose evidence; Q2 should distinguish representation semantics from an editable mesh; Q3 should include editability/metric scale and the downstream target; Q4 should keep provenance and hardware constraints separate from visual quality. Score a question as incorrect if its answer misses that boundary.

Mental model: capture, representation, then target

Use this order:

capture constraints → overlap/lighting → pose evidence → SfM/MVS baseline ↔ NeRF ↔ 3DGS → representation/quality/compute/editability/scale trade-off → downstream target → SELECT / RECAPTURE / HOLD

Do not collapse these questions:

The lesson does not compute a single weighted score. One critical failed gate can block a positive state.

Shared foundations

1. Photogrammetry here means evidence-first SfM/MVS

COLMAP documents SfM as recovery of camera parameters and sparse 3D structure from overlapping views, with dense reconstruction as a subsequent stage. Its capture guidance emphasizes texture, reasonably consistent illumination, high overlap, and different viewpoints. Those practices improve reconstruction conditions but do not guarantee metric accuracy or production suitability. [COLMAP] [COLMAPTUTORIAL]

2. Treat NeRF as a volumetric novel-view representation

The original NeRF optimizes a continuous volumetric scene function from images with known camera poses. Strong view synthesis must not be silently translated into “conventional editable mesh.” [NERF]

3. Treat 3DGS as a Gaussian scene representation

The authors' reference implementation starts from SfM inputs, optimizes 3D Gaussians, and provides training and real-time viewing components. Gaussian count and rendered appearance matter, but neither proves mesh editability, physical scale, or downstream interoperability. [3DGS]

4. Method selection is a DESIGN decision

External sources establish method facts. MAKNOW owns the evidence thresholds, state vocabulary, and acceptance logic. A paper or tool manual cannot substitute for the target-specific decision.

Building the Method Comparison and Evidence Pack

Step 1 — record capture before choosing a method

Write capture_id, subject type, camera_count, image_count, overlap quality, lighting consistency, and rights/provenance before evaluating reconstruction output. After SfM, also record sparse_point_count as a non-governing diagnostic observation; point count alone is never an acceptance rule. Weak texture, specularity, or poor overlap must remain visible as capture risk rather than disappearing behind a good render.

Step 2 — keep pose source independent

Reference NeRF and 3DGS workflows also depend on pose/calibration evidence. If camera_pose_source is unknown or SfM has failed, a positive reconstruction path is not justified merely by method preference. A known external pose source can create a different valid path, but it must be explicit.

Step 3 — separate SfM from MVS

sfm_status records camera/structure recovery; mvs_status records the dense stage. A successful sparse reconstruction does not automatically mean a usable dense mesh or watertight geometry. Type the failure mode instead of hiding it.

Step 4 — name the representation

For each method, state what the output is: sparse/dense geometry, a continuous volumetric radiance field, or optimized 3D Gaussians. This prevents the vague statement “we have three 3D models.”

Step 5 — place quality next to semantics

novel_view_quality=PASS means the defined view-synthesis observation passed. If the target requires editable geometry, geometry semantics and editability remain separate gates. A method may be excellent at view synthesis and still be unsuitable for mesh-centric editing.

Step 6 — record compute without inventing universal budgets

Training and reconstruction cost depends on implementation, dataset, hardware, and target. Lab values are synthetic fixture evidence, not universal timing claims.

Step 7 — make metric scale explicit

When metric scale is REQUIRED, metric_scale_evidence must be PASS before a positive state is defensible. A visually coherent scene can still lack proven absolute scale.

Step 8 — separate provenance from quality

Accuracy and beauty do not grant permission to use a capture. license_provenance_ref must point to synthetic, self-created, public-domain, or rights-cleared evidence. AI may not infer ownership or consent.

Step 9 — derive the path from the target

• PHOTOGRAMMETRY_MESH: geometry/editability is central and MVS evidence is sufficient.
• NERF_VIEW_SYNTHESIS: novel-view representation is the target and editable mesh is not required.
• 3DGS_VIEW_SYNTHESIS: realtime-oriented novel-view delivery is desired, hardware fits, and mesh semantics are not the primary requirement.
• HYBRID_MESH_PLUS_VIEW_SYNTHESIS: both editable geometry and a view-synthesis experience are required and the team accepts the cost of maintaining multiple representations.
• RECAPTURE/HOLD: capture or pose evidence is not reliable enough.

Step 10 — bound the final state

SELECTED is valid only when all decisive gates pass and decision.target_constraint_scope=STANDARD_DECLARED_TARGET. TARGET_SPECIFIC requires the same passed gates plus CONSTRAINED_NAMED_TARGET_ONLY. If target authority is still pending, HOLD pairs with PENDING_TARGET_AUTHORITY. Unresolved provenance, missing decisive evidence, repairable capture/pose failure, and method mismatch map to their typed non-success states by precedence; identical visible evidence must not support arbitrary primary states.

Worked example: a small room with two downstream needs

The SYNTH-ROOM-01 baseline has 96 images, high overlap, consistent lighting, COLMAP poses, and successful SfM and MVS. The methods expose different evidence:

Because the target requests both a walkthrough and editable geometry, the evidence allows the learner to evaluate a hybrid path. The template does not prefill that answer; the learner must write the path and state, and the checker only tests consistency.

Invalid shortcut — low overlap

In mutation_low_overlap.json, overlap is low, SfM fails, and stable camera poses are missing. PHOTOGRAMMETRY_MESH + SELECTED is contradictory. NeRF/3DGS cannot use method preference as a replacement for explicit pose evidence either.

Counterexample — novel views are not enough

In mutation_metric_editable_mesh.json, the target changes to a metric-scale editable mesh and metric scale remains UNKNOWN. A NeRF- or 3DGS-only view-synthesis path cannot close the explicit editability/scale requirement even if its novel views look strong.

Important failure modes

1. “They all make 3D from photos, so they are equivalent.”

Repair: record output representation and downstream semantics separately.

2. “The best render is the best production method.”

Repair: include editability, scale, provenance, compute, and target next to quality.

3. “SfM failed, but NeRF/3DGS will automatically rescue us.”

Reference workflows still depend on pose/calibration evidence. Repair: identify an explicit alternate pose source or recapture.

4. “We will fix metric scale later.”

If the current target requires metric scale, unknown scale is a blocker. Repair: add scale-control evidence and revalidate.

5. “We have a GPU, therefore 3DGS wins.”

Hardware is one dimension, not the decision. Repair: start from representation need and downstream target.

Experienced bridge: a representation portfolio

A mature pipeline can produce several representations from one capture lineage: a photogrammetry mesh for asset work, a NeRF for research/view synthesis, and 3DGS for interactive visualization. That is not automatically wasteful if each representation has an explicit owner, purpose, provenance trail, and revalidation boundary.

A bounded pattern:

1. lock capture identity and provenance once;
2. version the pose/calibration lineage;
3. keep representations purpose-bound;
4. validate quality independently for each representation;
5. never assume conversion or mesh extraction is lossless or automatic;
6. recompute method choice when the target changes.

Trade-offs without a permanent winner

Photogrammetry/SfM-MVS naturally supports geometry-centric workflows, but capture quality, texture, overlap, specularity, and reconstruction cleanup remain important. Dense output is not the same as guaranteed metric or production-ready geometry. [COLMAPTUTORIAL]

NeRF is a strong novel-view representation; do not generalize that strength into a universal mesh-replacement claim. Camera poses and training cost remain relevant. [NERF]

3DGS targets high-quality fast novel-view rendering and its reference implementation consumes COLMAP/SfM-style input. Gaussian representation is not conventional mesh semantics. [3DGS]

A hybrid can satisfy dual needs but increases compute, storage, versioning, and consistency cost.

The defensible choice is always target-bounded.

Guided and independent practice

Guided baseline

Open lab://D3D-003/D3D-003-LAB1. Copy evidence/evidence_pack.template.json and populate it only from fixtures/baseline_observations.json. Then write decision.selected_path, decision.final_state, a rationale, and learner-authored review, migration, escalation, stop, and pivot triggers yourself.

From the learner directory run:

python run_checks.py evidence/my_pack.json

The checker is not an answer key. It only catches deterministic contradictions such as a positive state with missing decisive evidence, required metric scale without proof, or a view-only path when editable mesh is explicitly required.

Mutation 1 — lower overlap

Inspect mutation_low_overlap.json. Identify the failed evidence gate and the smallest repair. Do not copy the baseline decision.

Mutation 2 — metric editable mesh

Inspect mutation_metric_editable_mesh.json. If metric_scale_evidence remains UNKNOWN, do not force a positive state.

Mutation 3 — constrained hardware

Inspect mutation_constrained_hardware.json. This fixture changes the hardware constraint while preserving unrelated baseline method evidence. If a reference NeRF/3DGS path requires a compatible accelerator, visual preference cannot replace hardware-fit evidence.

Independent transfer

Use transfer/transfer_fixture.json as the changed-context observations and submit S4 through a copy of transfer/transfer_evidence_pack.template.json. Rebuild capture strategy, pose strategy, method suitability, and the five decision triggers from scratch; then run the same checker on the transfer pack. Preserve uncertainty when evidence is still missing.

Verification: a reviewer must be able to reconstruct the decision

Before submission, close this checklist:

• capture identity and provenance are explicit;
• camera_count and sparse_point_count (when SfM ran) are recorded;
• every method_evidence row has evidence_ref, claim_ref, and authority_class;
• overlap/lighting and pose source are recorded;
• SfM and MVS are separate states;
• each method's output representation is named;
• view quality is not substituted for geometry/editability;
• metric-scale requirement and evidence are both visible;
• compute/hardware observations are fixture-specific;
• downstream target is explicit;
• missing decisive evidence remains listed until truly closed;
• decision rationale points to evidence rather than tool preference;
• review trigger, migration trigger, escalation trigger, stop trigger, and pivot trigger are explicit and evidence-bounded;
• transfer is recomputed rather than copied from the baseline and is submitted through the decision-bearing transfer Evidence Pack.

If a reviewer cannot reproduce the state without hidden AI reasoning, the pack is not ready.

Assessment and transfer

S1 — Retrieval: explain the owned D3D-003 boundary and name one adjacent non-goal.

S2 — Application + decision checkpoint: complete the baseline Evidence Pack, choose a path/state, and record review, migration, escalation, stop, and pivot triggers. Explain why “best novel view” cannot be the decision policy by itself.

S3 — Diagnosis: diagnose the low-overlap mutation: record the invalidated field/state, consequence, and minimal repair.

S4 — Transfer: rebuild capture and method choice for the glossy weak-texture object with a metric-scale requirement in the decision-bearing transfer Evidence Pack, record all five decision triggers, and run the checker. The baseline answer is not an oracle.

Passing requires auditable identity/contract, exact evidence binding, negative evidence, bounded decision, and changed-context transfer.

References, freshness, and claim boundaries

• [COLMAP] official COLMAP documentation for the current SfM/MVS pipeline and capabilities. Live rechecked 2026-09-06; evolving source.
• [COLMAPTUTORIAL] official COLMAP tutorial for SfM stages and capture guidance including texture, illumination, overlap, and viewpoints. Live rechecked 2026-09-06.
• [NERF] original NeRF project / ECCV 2020 method description: a continuous volumetric scene function for novel-view synthesis from posed views. Foundational stable.
• [3DGS] authors' official 3D Gaussian Splatting reference repository: optimized 3D Gaussians, SfM-derived inputs, optimizer, and real-time viewer. Live rechecked 2026-09-06.

These sources support method FACTS. MAKNOW owns the state vocabulary, acceptance logic, and evidence thresholds. This lesson does not claim universal performance or accuracy across every dataset, GPU, capture, or downstream tool.

Next evolving-source review: 2026-10-18, or earlier if source semantics materially change.