CoMind - Be My EyeLearning to See Beyond a Robot’s Own Perspective
Technical ReportBe My Eye is a task-oriented active context sharing framework for multi-robot collaboration. Rather than simply transmitting raw images or generic visual features, each robot distills its local observations and task instructions into compact, task-relevant representations that capture the information most useful to its partners. These shared representations allow robots to leverage complementary perspectives to inform their own decisions and actions, bridging the gap between distributed perception and coordinated behavior.
Don't just share what you see. Share what your partner needs to know.
One World. Many Perspectives.
Partial observability is often seen as a limitation of individual robots. But in a team, it becomes an opportunity: what one robot cannot see may be exactly what another can.
Consider two robots jointly supporting and assembling a box. A panel may occlude one robot's view, leaving it unable to tell how far its partner has progressed, such as whether the base panel has been bonded in place or when the two should flip the box together. Yet these very details happen to lie entirely within the other robot's field of view. Whether the occluded robot should fine-tune its supporting posture or slow its assembly motions so that the parts achieve precise alignment and proper bonding therefore depends on information that it cannot observe but its partner can.
The real challenge is not simply enabling robots to see more, but allowing them to perceive through one another's perspectives and turn distributed observations into coordinated actions. This is a fundamental step toward more tightly coupled collaboration, where each robot's decisions are informed not only by what it sees, but also by what its partners see.
From Local Observations to Collaborative Decisions
Robots operating in the real world are inherently subject to partial observability. Occlusions, spatial separation, and differences in task roles mean that information perceived by one robot may be precisely what another lacks to complete its task. Be My Eye aims to transform these naturally complementary observations into shared context that directly supports collaborative decision-making and action generation. To this end, we propose a two-stage learning framework: first, compressing each robot's local multi-view observations into compact, task-relevant context representations; then, integrating this shared context into partners' action generation through bidirectional information sharing and joint policy optimization.
Active Context Construction
We introduce two complementary learning objectives to guide robots in extracting visual information that is genuinely useful for collaboration:
Task-Oriented Future Prediction
Encourages shared context representations to preserve information predictive of future visual state changes, while weighting visual regions according to their relevance to the current task. This guides the representations to focus on key objects, environmental states, and their changes over time.
Task-Progress Correspondence Learning
Constructs soft labels based on task identity and execution progress to structure the representation space, promoting both discriminability across different tasks and continuity across execution stages within the same task.
Learning to Collaborate Through Shared Context
We establish symmetric context encoding and action policy interfaces across robots, allowing each robot to independently extract and exchange its own task-relevant context in a bidirectional manner. Each receiving robot integrates its partner's shared context with its local visual observations, task instruction, and proprioceptive state, and feeds them into a pretrained vision-language-action (VLA) policy to generate continuous actions. More importantly, during joint optimization, action supervision from the receiving robot can propagate backward through the shared information pathway to update the sending robot's context encoder. This allows information extraction to move beyond standalone representation learning objectives and become directly optimized for the partner's actual action requirements and collaborative decision-making.
Real-World Tasks
We evaluate active context sharing on real robotic platforms across a range of dual-robot collaborative tasks, demonstrating its effectiveness in diverse collaboration scenarios.
Restock the refrigerator with the matching drinks from the kitchen island, following the order of the drinks inside the refrigerator from the top shelf to the bottom shelf. Close the refrigerator door when finished
Efficient Information Exchange
Robot A observes the spatial layout and placement requirements inside the refrigerator, while Robot B handles the drinks waiting to be restocked. By sharing compact, task-relevant context, the two robots exchange essential information about target selection, object handoff, and restocking progress, enabling cross-view collaboration without transmitting redundant visual information.
Collaboration Begins When Another Perspective Becomes Part of Your Own.
Be My Eye advances cross-robot information exchange beyond the passive transmission of visual content toward active context sharing driven by task objectives and collaborative decision-making. Through compact representation learning, bidirectional information exchange, and joint policy optimization, robots can integrate their partners' complementary observations into their own action generation. This provides a reusable learning framework for multi-robot collaboration under visual occlusion, spatial separation, and tightly coupled interactions.
This mechanism naturally complements high-level task planning. Looking ahead, we aim to explore richer forms of shared information, more flexible multi-robot interactions, and scalable collaborative capabilities across diverse tasks and robotic platforms.
