XR-Objects: Augmented Interaction

Description:

XR-Objects introduces a novel interaction paradigm for mixed reality (XR) environments. It focuses on enhancing real-world analog object interactions by generating actionable, auto-anchored context menus in augmented reality (AR). The project leverages cutting-edge technologies like ARCore, MediaPipe, and the Gemini multimodal LLM to facilitate seamless object recognition and interaction without pre-registration or markers.

This repository includes:

1. Unity project files for Android implementation, utilizing ARCore and MediaPipe.
2. A project template for future development on XR headsets (Quest 3, Vision Pro, etc.).

Features:

Object Interaction:

• Auto-generated Context Menus: Contextual menus allow users to take actions such as querying metadata, adding notes, or setting timers for detected objects.
• Supported Object Classes: Includes everyday objects like bottles, cups, bowls, laptops, and more. Classes can be expanded via configuration.

Integration with Gemini LLM:

• Facilitates metadata retrieval about objects, enabling tasks like:
  • Asking real-time questions.
  • Comparing objects in the scene.

Expandable Features:

• Developers can add custom context menu actions via Unity Editor.

Device Compatibility:

• Android app implementation with ARCore.
• XR headset compatibility via OpenXR with external webcam integration for object detection.

Requirements:

Android Implementation:

• Unity 2020.3 or higher (specific version requirements in the repository).
• ARCore and MediaPipe dependencies.
• Gemini API key for object metadata queries.
• Android 7.0 (API level 24) or higher.

XR Headset Implementation:

• Unity 2020.3 or higher with OpenXR support.
• External webcam for object detection on headsets like Quest 3.
• ADB installed for APK installation on the headset.

Instructions:

For Android Implementation:

1. Clone the Repository: Pull or clone the repository to your system.
2. Open in Unity Hub:
   • Navigate to the ARCoreMP directory.
   • Open it as a project in Unity Hub. Install the required Unity version if prompted.
3. Set Target Platform:
   • Set the target platform to Android in Build Settings.
   • Configure project settings:
     • Scripting Backend: IL2CPP.
     • Minimum API Level: Android 7.0 (API level 24).
     • Target Architectures: ARMv7 and ARM64.
4. Select the Scene:
   • Use the SceneXRObjects in the Assets/XRObjects/ folder.
   • Ensure it's the only selected scene in Build Settings.
5. Configure Gemini API Key:
   • Obtain an API key from the Gemini developer page.
   • Set this key in the ImageQuery.cs script (located in Assets/XRObjects/Scripts/).
6. Build and Run:
   • Build the project for Android.
   • Deploy and run the app on your Android device.

For XR Headset Implementation:

1. Clone the Repository: Pull or clone the repository.
2. Open in Unity Hub:
   • Navigate to the DemoHeadset directory.
   • Open it as a project in Unity Hub. Install the required Unity version if prompted.
3. Set Target Platform:
   • Set the target platform to Android in Build Settings.
   • Configure project settings:
     • Scripting Backend: IL2CPP.
     • Minimum API Level: Android 7.0 (API level 24).
     • Target Architectures: ARM64.
4. Select the Scene:
   • Use the SceneDemoHeadset in the Assets/XRObjects/SceneDemoHeadset/ folder.
   • Ensure it's the only selected scene in Build Settings.
5. Build the Project:
   • Locate the compiled APK on your computer.
   • Use the command adb install -r -d -g name_of_the_file.apk to install it on the headset.
6. Use External Webcam:
   • Plug in an external webcam to the headset.
   • Grant camera access when prompted.