In-device First Experience for HoloLens I
Project Brief
The onboarding, or ‘First Experience’, automatically launches when the HoloLens is powered on for the first time. It walks us through the multi-modal input model of using gestures and voice, as well as covering device fit, device calibration, connecting to WiFi and account sign-in.
While teaching the basics of interacting with holograms was the main goal, it also represented the most anticipated point of the customer journey that began with pre-release expectations, to purchase and product unboxing, to this first contact with holograms.
Additionally, for enterprise and 3rd party customers, the design and mechanics of this first experience would serve as a model for training applications, which would prove to be one of devices’ most popular use cases.
Role: End-to-end Experience Designer
Create 3D assets and spatial prototypes sketches.
Provide design specifications for artists and developers.
Interpret user study results into UX initiatives.
Partner with research, prototyping, PM and engineering.
Bring design clarity to early concepts.
Detail the experience flow and interactions.
Establish visual and motion design.
Draft voice-over dialog scripts.
Insights and Process
Early sketches
Developing a method of constant reactive feedback to eye gaze changes.
End-to-end flow
Connecting to wifi required learning eye gaze and the airtap gesture to operate the holographic keyboard, so over-the-air updates to the experience were not possible; it all had to be complete and written to the device at the factory, before shipping to customers. Additionally, personalization of the device and full Cortana access would not be available until signing in to a user account, which couldn’t happen until after having the skill to use the holographic keyboard.
Using a mobile device or PC was not an option from a product identity perspective; HoloLens must be standalone.
As a self-contained headset, people need to be able to put on the device and learn how to use it without any outside help; no phones, no PC’s, no outside guidance whatsoever.
Ergonomics and the Navigation Ring
If not designed with consideration for human cognitive and physical ergonomics, a person’s first time in mixed reality, where content can be anywhere around you, has the potential to overwhelm our senses, disorient us, and be physically uncomfortable.
The ‘navigation ring’ is a grounding element, always keeping the interface in view, while not being intrusive.
Cortana is your guide
I believe it is essential for people to learn at their own pace, feeling free to make mistakes and to try things again. Research in embodied cognition, task analysis, and ergonomics coupled with usability testing, resulted in a step-by-step, gated learning approach.
The voice of Cortana guides us from start to finish, calling upon embodied coaching elements throughout to provide demonstration.
Through a combination of spoken and demonstrated guidance, people learn by doing, following a simple pattern repeated throughout the experience:
Watch. Try. Practice.
The ‘coach hand’ demonstrates the ‘Bloom’ gesture, as seen through the HoloLens.
Learning head targeting with an array of musical, lighted gems.
Holographic gems for practicing the select gesture and voice command.
An embodied coach performs the gestures in concert with spoken text.
Design for everyone
All people should have an equal and effective experience
While the voice-over narration provides relatable coaching, the entire experience works equally well for individuals who are deaf or situationally deaf. Additionally, it’s equally effective whether you’re sitting, standing or moving about.
In addition to instructional phrases that appear on the ring, time and gaze-based triggers display visual hints on interactive elements; this provides multiple levels of input instruction and navigation options through the entire experience.
Gaze-based and timer-based hints reinforce learning and provide accessibility.
Customers’ emotional journey
Data from research studies, along with direct observation of participants in the UR lab provided insights into people’s emotional journey. From this data, I created visualizations that map valence, or emotional quality, across time and tasks in the flow. This was crucial to understanding and improving the experience, and further helping unite design and engineering teams around measurable outcomes.
Visual styling
Integrating holograms with the physical world
I was particularly inspired by the ‘retro-futuristic’ illustration style with its saturated hue’s, crisp light and stippled textures.
Applying a stippling to their surface materials helped integrate the holograms with the physical world, which is replete with micro-textures and imperfections; tuning the the texture to avoid visual artifacts was paramount.
Coach embodiment
We tested a range of styles for the coach, from human-like to highly stylized. As it was vital that the coach demonstrate precise hand and body positions, the final design is physically relatable, yet avoids the ‘uncanny valley’.
Material explorations for the gems: soft noise function helps integrate into physical world.
A range of faceted coach styles were explored, from highly stylized faceting to detailed.
Specifications
Working through exhaustive iterations in Unity3D, I created spatial layout documents detailing the exact position, orientation and scale of objects. Additionally, interaction flows specify trigger points for voice-over dialog, embodied coach animations, and contextual hints.
Below, In-device image captures from the final experience. If you have a HoloLens 1, open the “Learn Mixed Reality” app to try it yourself.
Impact
Successful in making an unfamiliar, multi-modal interaction model easy to understand and to retain.
The stand-alone app version, focused on the multi-modal tutorial, is among the most used inbox apps on HoloLens 1.
The design influenced concepts, UX patterns, and visual language for future 1st party experiences.
Resulted in greater awareness and customer engagement with the Cortana virtual assistant.
