How the refinement layer works.

Our platform fuses egocentric RGB video with LiDAR depth to create synchronized, richly annotated datasets suitable for imitation learning, behavior cloning, and multi‑modal perception. The system emphasizes natural, life‑like motion in demos, smooth time alignment, and research‑grade export formats to accelerate robotics development across home, retail, and industrial workflows.

High‑quality real‑world task data is scarce, especially data that combines hand‑object interactions with accurate 3D scene geometry. Existing datasets often lack time synchronization, calibration, or standardized schemas—slowing down reproducibility and transfer. We address these gaps with a capture and export pipeline purpose‑built for robotics.

• Egocentric stream: Smart‑glasses or spatial cameras capture first‑person RGB (hands, tools, objects).
• LiDAR stream: iPhone/iPad Pro or supported headsets produce point clouds, range images, and optional meshes.
• Sync service: Timestamp alignment, drift handling, and clock skew mitigation across streams.
• Calibration service: Intrinsics/extrinsics and rigid transforms to unify modalities.
• Export layer: Standards‑aligned writers for HDF5, ROS Bag, RLDS, and Open X‑Embodiment.

• Tasks: Household (dishwashing, sweeping), retail (stocking), and industrial (packaging) exemplars.
• Sampling: Configurable FPS (RGB), scan rates (LiDAR), and exposure/gain presets for repeatability.
• Annotations: Optional 2D/3D labels, action segments, hand keypoints, and trajectories.
• Quality: Smooth, life‑like motion profiles for demos to mirror human kinetics and ergonomics.

We apply precise timestamping, drift compensation, and verified transforms to maintain spatial/temporal coherence. This enables reprojection between camera frames, depth scans, and reconstructions for consistent multi‑modal supervision.

• HDF5: Hierarchical multi‑modal containers with metadata, timestamps, and calibration.
• ROS Bag compatibility: Time‑indexed topics for robotics pipelines and playback.
• RLDS: TensorFlow‑friendly structuring for research datasets.
• Open X‑Embodiment: Cross‑dataset standardization for embodiment research.

root:
  session_id: string
  created_at: ISO8601
  devices:
    - name: "egocentric_cam" | "lidar"
      intrinsics: {...}
      extrinsics: {...}
  streams:
    rgb:
      fps: number
      frames: [ { t: number, uri: "..." } ]
    lidar:
      rate_hz: number
      scans: [ { t: number, pcd_uri: "...", intensity: bool } ]
  annotations:
    actions: [ { start_t, end_t, label, confidence } ]
    keypoints: [ { t, hand: "left|right", joints: [...] } ]
  calibration:
    transforms: [ { from, to, matrix4x4 } ]

• Privacy‑by‑design: Optional redaction/blur, local review before export, scoped access controls.
• Data minimization: Capture only what is required for the task; configurable retention.
• Transport & storage: TLS in transit; encrypted storage options depending on deployment.

The platform is designed to support compliance‑minded deployments. Actual compliance depends on your configuration and processes.

• GDPR/UK GDPR: Support for data subject requests, export, and deletion workflows.
• CCPA/CPRA: Transparency, access, and deletion mechanisms can be configured.
• Data residency: Choose storage regions where supported to align with local requirements.

• Use context: Not for safety‑critical decision‑making without additional controls and validation.
• Bias: Encourage diverse capture scenarios; document known limitations and edge cases.
• Accessibility: Interfaces favor clear typography, color contrast, and keyboard navigation.

• Demo motion latency: tuned for smoothness and stability in modern browsers.
• LiDAR scan ingest: optimized for typical mobile LiDAR rates with graceful degradation on low‑end devices.
• Export throughput: dependent on browser/hardware; batch writers chunk data to prevent UI stalls.

• Consumer LiDAR has range/lighting constraints; recommended indoor usage for highest fidelity.
• WebGL contexts may be limited by device/OS; fallback views are provided when possible.
• Annotations may require human review for gold‑standard quality in research settings.

• On‑device action hints and auto‑segmentation assistance.
• Pose/hand keypoint tracks as first‑class export channels.
• Scenario presets with tuned motion profiles (home, retail, warehouse).

Disclaimer

This white paper is provided for informational purposes only and does not constitute legal, compliance, safety, or engineering advice. The platform is provided “as is” without warranties of any kind. Evaluate suitability for your use case and seek professional advice where appropriate.

Intellectual Property & License

Unless otherwise agreed in writing, all content and software are © their respective owners. No license to patents, copyrights, or trademarks is granted by this document. Any sample schema or snippets herein are illustrative.

Data Ownership

You retain ownership of your captured data. You are responsible for obtaining all necessary consents and complying with applicable laws when capturing, processing, or sharing data.

Acceptable Use

Do not use the platform to infringe rights, violate law, or create safety‑critical systems without appropriate safeguards and validation.

Third‑Party Services

Integrations and third‑party tools are subject to their own terms. Ensure you have authorization and comply with their policies.

Export Control

You are responsible for complying with applicable export control and sanctions regulations when exporting or providing access to software, technology, or data.

Limitation of Liability

To the maximum extent permitted by law, no party shall be liable for indirect, incidental, special, consequential, or punitive damages arising from use of this platform or documentation.

These notices are summaries only and do not replace full terms or privacy notices that may apply to your deployment.