Your Phone Can Map a Grocery Store — and Then Guide a Blind Shopper Through It

A single consumer iPhone LiDAR scan of a 3,500 sq ft international grocery store becomes a fully navigable, semantically searchable map — no beacons, no server-grade compute, no manual annotation.

~6 min read · University of Colorado Boulder · April 21, 2026 · Multimodal

TL;DR GIST converts a consumer-grade mobile point cloud (iPhone LiDAR + RGB-D) into a semantically annotated navigation topology, then uses that shared representation to power four downstream tasks: intent-aware product search, one-shot localization, zone classification, and landmark-rich verbal routing. In real-world grocery store tests, it achieved an 80% navigation success rate using generated verbal instructions alone — no visual interface required.

Picture the navigation problem inside a dense international grocery store: thousands of visually similar products, aisle signage in multiple languages, shelves that change weekly, and a long tail of culturally specific items that no standard object detector has ever seen. Standard robot navigation gives you a geometric map — free space, obstacles, corridors. What it doesn't give you is the semantic scaffolding that a person actually uses: "walk past the lentils, turn right at the spice section."

That gap is what GIST (Grounded Intelligent Semantic Topology) is designed to close.

💡 The Core Idea Instead of treating geometry and semantics as separate systems, GIST extracts a shared intermediate representation — a semantic topology — that encodes both where things are and what they mean, from nothing more than a single consumer mobile scan. Every downstream task (search, localization, routing) draws from the same structure, so they stay spatially consistent with each other.

From Raw Scan to Semantic Map

The pipeline starts with a 14.5-minute walk through a 3,500 sq ft store using an iPhone with LiDAR. This yields 52,006 RGB-D frames and 6-DoF odometry. From there, GIST does three things:

Keyframe selection. All frames are embedded via DINOv2 (vitb16). A cosine similarity filter with a 0.85 threshold retains only visually distinct frames, compressing 52,006 frames down to 660 — a 92.4% reduction.

Object extraction and VLM labeling. Each retained keyframe is passed through a YOLOv9 model fine-tuned on the SKU-110K dense shelf dataset. For each frame, GIST picks the sharpest crop and the most visually distinct crop (by DINOv2 cosine distance), then batches these into 5×5 mosaic grids for Gemini processing. The VLM extracts product name, brand, packaging type, and category.

Topology construction. A 2D occupancy map (0.05 m/pixel) is derived from a height-sliced projection of the point cloud. Morphological skeletonization (Zhang-Suen thinning) extracts a navigation graph over walkable space. Junctions and turn points become graph nodes; line-of-sight checks ensure all edges are actually traversable. Products are then snapped to their nearest edge via perpendicular projection, with virtual nodes inserted dynamically during pathfinding to route the agent to within 0.1–0.5 m of the target.

flowchart LR
    A[iPhone RGB-D + LiDAR] --> B[Keyframe Filter\nDINOv2 cosine ≤ 0.85]
    B --> C[YOLOv9 Product Detection\nSKU-110K fine-tune]
    C --> D[VLM Labeling\nGemini mosaic grids]
    D --> E[Semantic Topology\n2D map + nav graph + product labels]
    E --> F1[Intent-Aware Search]
    E --> F2[One-Shot Localizer]
    E --> F3[Zone Classification]
    E --> F4[Verbal Routing]

Four Tasks, One Representation

Zone Classification. All mapped products are classified into 18 semantic zones via a single VLM prompt. A KD-tree assigns every walkable pixel to its nearest zone via inverse-distance-squared voting over the 5 nearest products. One emergent finding: the system autonomously identified a dense cluster of heterogeneous items in one aisle corner that turned out to be the store's dedicated "Organic" section — a macro-structure it discovered from micro-level product distributions without being told it existed.

Intent-Aware Search. Standard keyword matching fails on long-tail, culturally specific inventory. GIST instead passes queries to Gemini against the structured product map. An abstract query like "vegan protein" returns not just exact matches but items categorized as Alternative or Related, with generative reasoning. A recipe query like "biryani ingredients" decomposes into biryani spices, basmati rice, garlic paste, and ghee — each located or estimated on the map simultaneously. When a product is entirely missing from the map (a real problem: FOV clipping means bottom-shelf items may not have been scanned), the system falls back to routing the user to the spatial centroid of the most relevant semantic zone rather than returning an empty result.

One-Shot Semantic Localization. This is the most technically novel piece. GIST builds an offline pose map by discretizing the floor into 0.5 m cells with 8 orientation bins. For each cell, it raycasts 20 rays through the semantic map, collects product labels along each ray, sorts them alphabetically (for permutation invariance), and encodes the resulting string via DistilBERT into a 768-d vector. At query time, a user takes a single smartphone photo; GIST runs YOLOv9, labels the visible products via VLM, encodes them the same way, and retrieves the top-k pose hypotheses by cosine similarity. No visual feature matching. No infrastructure. No server-grade GPU.

The 1.04 m top-5 mean error for correctly-zoned frames is encouraging for a zero-shot text-only method. The paper is appropriately honest that semantic aliasing — two mirrored shelf positions with nearly identical product layouts — inflates Top-1 error and makes this better suited as a global initialization seed for a particle filter than as a standalone tracker.

Visually-Grounded Verbal Routing. A* over the topology graph produces a path. Path segments are chunked at angle changes > 30°. Rather than just sending coordinates to the LLM, GIST renders an annotated occupancy map image (graph, path, start/goal markers) and performs Bresenham line-of-sight checks from path midpoints to identify which products are actually visible along the route. These validated landmarks, with their left/right side computed via cross-product against the heading vector, are passed to Gemini alongside the route segments to generate egocentric, cardinal-direction-free instructions.

Does It Actually Work?

The instruction quality was evaluated with an LLM-as-judge protocol across five criteria (Egocentric Clarity, Landmark Utility, Cognitive Load, Safety & Completeness, Universal Design) over 15 real-world navigation scenarios ranging from 2.9 m / 0 turns to 19.4 m / 2 turns. GIST (Visual) scored 4.43/5 overall, with a perfect 5/5 on Egocentric Clarity. NavComposer — the closest baseline, which generates instructions from RGB sequences paired with discrete action traces — scored 1.80/5 on Safety & Completeness, and the qualitative examples show why: it references "the shopping cart" and "the yellow wall" rather than stable semantic landmarks.

Critically, performance scales with route complexity. On long routes (>10 m), GIST scores 4.59 while the coordinate-only Naive Gemini baseline drops to 2.73. The topology encodes turn structure explicitly; the VLM doesn't have to infer it from images.

The ablation study surfaces a useful practical insight: providing the VLM with either the topology graph alone (4.55) or the occupancy map alone (4.49) slightly outperforms the full composite image (4.43). Reducing visual clutter helps attention. What doesn't work is visual structure without semantic grounding: giving Gemini a bare occupancy map with only start/goal markers collapses Landmark Utility to 2.89.

The real-world test involved 5 graduate student evaluators navigating the grocery store using only generated verbal instructions — no visual path overlay, no map shown. 8 of 10 tasks succeeded (80%), with an average time-to-find of 48.4 seconds for successful trials. Both failures traced to the same root cause: products on the absolute bottom shelf fell below the chest-level scan's field of view and were never mapped.

⚠️ Watch out for

• Bottom and top shelf items are systematically missed by chest-level mobile scans — zone-level fallback can't help if those items are isolated from their neighbors
• Semantic aliasing (mirrored shelf positions with identical product distributions) makes one-shot localization unreliable as a standalone tracker; it needs fusion with odometry or depth
• The formative evaluation used N=5 graduate students in one store — not target users (blind/low-vision individuals), and not across multiple environments
• "Quasi-static" environments still change: maps need periodic re-scanning to stay valid

Why It Matters

The practical target for GIST is assistive technology for blind and low-vision users navigating large, unstructured retail and warehouse spaces — the kind of environment where Bluetooth beacon deployments are expensive, barcode scanning is impractical at scale, and existing navigation systems assume you're already standing in front of the right shelf.

The deeper contribution is architectural. By separating deterministic geometric structure (the topology graph) from semantic AI reasoning (the VLM layer), GIST contains hallucination risk: the path planner can't hallucinate a shortcut through a wall because it operates on verified free space. The VLM operates only on the semantic annotation layer, where errors degrade instruction quality but don't create physical safety hazards.

The single-scan, infrastructure-free approach also means this is deployable anywhere a person can walk with an iPhone — no store cooperation required, no RFID retrofitting, no fixed camera network.

Source: GIST: Multimodal Knowledge Extraction and Spatial Grounding via Intelligent Semantic Topology Authors: Shivendra Agrawal, Bradley Hayes Published: 2026-04-16 PDF: https://arxiv.org/pdf/2604.15495