HoloLens & Mixed Reality

What You’ll Learn

You will leave this workshop understanding:

How to use the HoloLens tools to supercharge development, implement state of the art features and create seamless mixed reality experiences that interact with the real world;
How to interact with holograms to create interactive experiences that come to life and feel as natural as interacting with real objects;
How to blend and incorporate holograms into the real world.

Course Takeaways & Goals

How to set up the HoloLens developer environment
How to set up HoloLens 2 Development with MRTK
Learn about Features in MRTK that can be used for Application development
How to Anchor an Object with Azure Spatial Anchors
How to convert Text to Speech and Speech to Text in Azure Cloud Services
How to Create a Complete Cooking Experience Application for HoloLens 2

Who Should Take This Master Class?

Unity and VR developers & freelancers & solution providers who want to expand their skill set to Mixed Reality and Hololens;
Mixed Reality Solution Providers who want to upskill and boost the experience of their junior developers, designers in a fast-track industry project based format;
Enterprise companies (end users) who are deploying Mixed Reality powered solutions and using no-code MR creator tools and would like to learn more in depth so they can create custom features on existing MR training creator tools and editors;
VR/AR enthusiasts who want to truly explore what it means to interact with and even make use of the physical world in a mixed reality platform for various industry use cases.

Prerequisites

A fundamental understanding of the Unity Engine
Basic C# skills
A Hololens 2 device (discounts for leasing or purchase options available)

Capstone Project

Together, we’ll create an interactive recipe application that walks people through the process of cooking a good meal of spaghetti. The project will involve recipe steps, holographic indicators for ingredients’ locations, read-out-loud instructions, and more!

At the end of the experience, you’ll have the satisfaction of having created an amazing training-based HoloLens 2 application and as a bonus, you’ll also eat a good serving of Spaghetti!

*We will be adding small features of the cooking project each week and completing on the 6th week

Curriculum Outline

1. Developing for HoloLens 2 and Intro to MRTK

In the first week of this course, we’ll be diving into how to get started with HoloLens 2 development. We’ll cover prerequisites, Unity and Visual Studio installation, setting up your HoloLens 2 and setting up your development environment. By the end of Week 1, you’ll build and deploy your first HoloLens 2 application using the OpenXR pathway within Unity.

1.1. Prepare the device for Development: Enable Developer Mode and Device Portal
Accessing Device Portal
Understanding the features of the Device Portal
1.2. Setup the Development Environment on Windows 10
Install Visual Studio
Setting up Unity Build Settings with OpenXR
Creating a simple scene “Hello World”
Testing in the Editor
Building with Visual Studio
Deploying to the device

using Visual Studio directly
using the Device Portal

We will also be covering Microsoft’s Mixed Reality Toolkit (MRTK.) The MRTK is Microsoft’s premiere toolkit for streamlining mixed reality development. The MRTK is a vast resource that needs several weeks to cover. However, we’ll be covering the top features over 2 weeks during this course. This week, we’ll be covering various input methods such as hand, voice, and eye tracking. We’ll also cover spatial awareness and touch on scene understanding. As part of learning these concepts, you will also begin building out what will eventually be the capstone project’s cooking application.

1.3. Hand Interaction
1.4. Voice Interaction
1.5. Eye Tracking
1.6. Spatial Awareness
1.7. Scene Understanding

2. Diving Deeper with MRTK

During Week 2, we’ll continue with the MRTK. We’ll cover MRTK’s Grid Object Collection, Tool Tips, and also show you how you can leverage the MRTK to add interactive objects such as videos into your scene. We will also touch on spatial audio and various solvers included with the MRTK.

2.1. Grid Object Collection
2.2. Tool Tips
2.3. Adding Videos
2.4. Spatial Audio
2.5. Solvers

3. Working with Spatial Anchors

Learn how to use Spatial Anchors. We’ll learn how to anchor the objects in mixed reality to experience the real world; you will also explore the various steps required to start and stop an Spatial Anchors session and create, upload, and download Spatial Anchors on a HoloLens 2.

3.1. Explanation of Spatial Anchors
Concept. Why spatial anchors are important
Local Spatial Anchors and World Anchors
Azure Spatial Anchors
3.2. Exploring the use of spatial anchors / hands-on demo and exercise
Preparing Unity scene
Importing necessary Unity packages
Importing necessary packages
Configuring buttons to test spatial anchors
Connection to Azure Spatial Anchor resources
Testing application on Hololens 2

4. Mixed Reality Azure Speech SDK

This week will focus on using the Azure Speech SDK as part of mixed reality application development. The Azure Speech SDK leverages the power of cognitive services and artificial intelligence on the cloud for speech processing. There’s a wide array of possible capabilities that the SDK provides, including natural language processing, intent understanding, keyword recognition, speech-to-text services, text-to-speech services, translation, and more! This week, we’ll focus on text-to-speech services that the SDK offers in order to bring our cooking recipe instructions to life with a realistic voice that dictates the contents of the recipe cards.

4.1. Azure Speech SDK Introduction
4.2. Building a simple experience that utilizes the speech SDK for text-to-speech functionality

5. Preparing for MRTK3

During Week 5, students will be introduced to the next big version of MRTK, which is expected to be released at the end of 2022 or early 2023. MRTK3 integrates deeper with OpenXR, is more modular, and includes an enhanced set of UI and UX elements to streamline development.

5.1. Introduction to MRTK3
5.2. New features and changes in MRTK3
5.3. MRTK3 and OpenXR
5.4. MRTK3’s new UI capabilities

6. Capstone Project

During Week 6, students will dive deeper in the concepts and methods to begin creating the kitchen recipe HoloLens experience. Students will create tool-tips for ingredients and key locations, edit the labels of the tool-tips, and create a “pin” button to place spatial anchors for tool-tips. Students will place the finishing touches of the kitchen recipe Ho-loLens experience. Students will add cooking steps and timers, and create the ability for users to navigate through the recipe.

6.1. Capstone project setup
6.2. XR training steps best practices
6.3. Interaction and navigation between training steps
6.4. Saving and loading states and anchors in the HoloLens 2
6.5. Deploying, testing, and debugging your HoloLens 2 application