NEBULA: A Unified Ecosystem for Emobodied AI Agent Development & Evaluation

Examples of NEBULA Capability Test task across six core capabilities (Control, Perception, Dynamic Adaptation, Language, Spatial Reasoning, and Robustness) organized into three difficulty levels. Tasks isolate specific skills with controlled complexity. Green marks objects, red marks targets, and blue indicates contextual cues. Bold underlined text shows actions; italic underlined text gives clarifications.

The core impact of this design is its diagnostic power. By isolating a single capability per task, it moves beyond ambiguous success rates to pinpoint the precise cause of failure. This transforms a simple performance metric into a clear, interpretable signal, revealing specific weaknesses (e.g., poor spatial reasoning) that traditional benchmarks would otherwise obscure.

This figure provides examples from the NEBULA Stress Test suite, illustrating the Stability and Adaptability tasks across three progressive difficulty levels. The Stability Score tasks evaluate the smoothness of an agent's actions by increasing the precision required, moving from a simple stack at Level 1 to a more complex, multi-object arrangement at Level 3. The Adaptability tasks assess an agent's capacity to adjust to dynamic changes during execution, beginning with a sudden object movement (Level 1), advancing to a mid-task instruction change (Level 2), and culminating in a command abort that requires rapid re-planning (Level 3).

The true impact of our Stress Tests is revealing an agent's breaking point. By applying targeted pressure, they move beyond success rates to uncover hidden bottlenecks and predict an agent's readiness for the real world. This diagnostic approach allows us to understand why an agent fails in different scenarios by directly linking its system responsiveness (e.g., inference speed) to its ability to handle different difficulty levels.

NEBULA introduces a unified data platform designed to streamline research and foster collaboration. Our platform ingests these varied data formats into a single, standardized NEBULA format, providing the core infrastructure for both unified training and reproducible evaluation. This allows researchers to focus on innovation rather than data engineering.

• Unified & Structured Data Format: Introducing a standardized data schema for consistent representation of robot interactions in an episode structure. Consolidating observations, actions, and metadata for plug-and-play compatibility across datasets.
• Extensible, Robot-Agnostic Architecture: Our platform is not hardcoded to specific hardware. Instead, robot properties are defined in a centralized configuration system, allowing support for various arm setups, gripper types, and sensor configurations.
• Powerful and Intuitive Python SDK: Includes a high-level Python API that abstracts low-level data loading and indexing. This enables researchers to easily query, filter, sample data, and perform machine learning tasks like train-test splits with minimal code.
• Seamless Model Integration: Provides model input adapters for widely used VLA architectures. These adapters bridge the gap between the unified data format and a model's expected input, allowing for immediate benchmarking with minimal integration overhead.

This figure presents two radar charts summarizing model performance across six capability task families.
Left Chart: the mean and standard deviation of success rates across all models for each task family, broken down by three difficulty levels.
Right Chart: displays the average performance of individual models on Easy and Medium tasks.

=> Key Weaknesses: Spatial Reasoning remains a key bottleneck for most models, while nearly all fail at Dynamic Adaptation and Robustness.

This figure shows the stress test evaluations for four different models, comparing their performance across three increasing stress levels. The evaluation consists of four distinct tests: inference frequency, measured in Hertz; latency, measured in milliseconds; a stability score, on a scale from 0 to 1; and adaptability, measured by success rate.

=> Key Weaknesses: Under pressure, most VLAs fail to handle dynamic changes due to critical drops in inference speed and control stability, indicating they are not ready for real-world deployment.