Speaker
Description
Service robots operating in dynamic human environments must reliably locate objects that are moved, concealed, or completely novel. Current frameworks often assume static environments, failing to address these real-world uncertainties.
We present an open-vocabulary framework that combines spatial, semantic, and geometric reasoning to overcome these challenges. By unifying spatial cues about proximity and topology, semantic priors on typical placements, and geometric constraints that rule out infeasible locations, especially within concealed spaces, our approach finds objects even when they are relocated, hidden in drawers or cabinets, or first encountered through open-vocabulary queries. It also performs in-situ viewpoint planning to model relocated or unseen objects for manipulation and global scene-graph updates.
We validate our framework through extensive real-world trials on the Stretch SE3 mobile manipulator, evaluating search and retrieval in various conditions. Results demonstrate robust navigation (100%) and open-space detection (100%), with semantic-geometric reasoning reducing concealed space search time by 68% versus semantic-only approaches. The system achieves 80% detection in drawers, successfully capturing relocated objects through multi-view integration. Reasoning failed only for novel objects, requiring minimal user hints for correction. Although manipulation is constrained by the Stretch hardware, the cognitive stack consistently locates targets, demonstrating robust reasoning.
This work advances embodied AI by demonstrating how open-vocabulary, multi-modal reasoning enables robust object retrieval in dynamic and occluded environments. Implemented on a low-cost, compact mobile manipulator, our solution combines sophisticated cognitive capabilities with practical deployability, representing a significant step toward accessible service robots for everyday homes.
