GAIA: MR Musem Experience
Reimagine a mixed reality experience that speculates the future of museums via implicit learning.
Client
Carnegie Museum of Natural History (CMNH) - Hillman Hall of Minerals and Gems
Team & Role
Designers: Selene Du, Carol Auh, Semina Yi, Aishwarya Rane.
Duration
7 Weeks
Tools
Microsoft HoloLens, MRTK 3, Figma, Unity 3D, Blender 3D.
Overview
Active Learning & Personalization
Learn & Collect
Visitors can explore the colors, patterns, structures, and textures of minerals at the exhibition through hands-on interaction with digital models and collecting them into their Gem Library.
Foster Creativity & Engagement
Create Gem Artworks
Visitors can craft innovative generative gem artwork using the mineral properties stored in their Gem Library. The underlying system enable visitors to effortlessly create stunning gem artwork, ensuring accessibility for all skill levels.
Cultivate Community
Share Creations
Both MR and non-MR users can share their experiences in the collective space. MR users can display their generated artwork through interactive projection mapping, allowing every visitor to appreciate the beauty of gems.
Hillman Hall of Minerals and Gems focuses on displaying the beauty of natural gems and educating visitors about their basic properties.
As a key exhibition hall at the Carnegie Museum of Natural History, Hillman Hall of Minerals and Gems showcases over 1,300 specimens from around the world. The display features a wide variety of minerals and gems in striking colors, fascinating forms, and dramatic shapes.
Young adults and college students are the target users to the MR program.
We targeted Gen-Z as the initial user group to experience the MR program at Hillman Hall due to the client's objectives, the museum's location, and user preferences.
Children
Gen-Z, Millennials
Guardians
Client's Goal
Museum’s endeavor to attract young adults.
Existing programs target children and their parents—besides, lack of programs for younger generations, like college students.
Location
Geographical location of CMNH near colleges.
Pittsburgh is a college town with more than 100,000 students come to study.
User's Preference
Born and growing up with tech, actively embracing new tech products. Can be the pioneer for using next-generation of tech.
Step-by-step structural experience to leverage the personalized visiting experience and preserving the ambience of the space.
01
Onboarding
Users get a brief introduction of this Mixed Reality journey and guidance on wearing the headset at the entrance of Mineral Hall.
02
Collect Colors
Users can find the lighting hint as wayfinding to tell them the interactive gems, and collect the color of gems into their "collection box" by pinching outside the glass box and previewing the color on the sample ball.
03
Collect Patterns & Structures
Users can learn patterns and structures of gems through exhibition and add interested elements into personal libaray.
04
Create an Artwork
Users can create their own generative artwork based on the mineral properties they collected.
05
Share to the community
The wall DPM is where HoloLens and non-HoloLens users can share their experiences. HoloLens users can send their artwork on the wall DPM that traces anyone's body movements.
Design the spatial UI inspired by playing card
The spatial UI, designed based on human factors and rational scales for the MR experience, uses playing cards as a metaphor for personal collections. Each UI element represents a “card,” making the interface intuitive and user-friendly, encouraging easy exploration and management of collections.
Procedural generation makes user's artwork creation easily without any threshold.
Procedural generation simplifies the creation of user artwork, allowing anyone to generate unique pieces without any skill barriers. This approach democratizes art creation, making it accessible to all users.
Research
Through field on-site interview, we talked with:
Interviewees
Design the experience based on the evidence we heard and saw.
We designed the experience based on the evidence we gathered from interviews, focusing on three key aspects: information, space, and wayfinding. We analyzed our findings and translated them into meaningful design insights.
Information:
“it was interesting learning from the info”
“A lot of information to learn”
“Visually attractive, not the information”
Space:
“I love the calm and relaxing space”
“Some parts remains the same, so I feel nostalgic in the space”
“I visited multiple times”
Wayfinding:
“Signage and the map could be placed better.”
“I turn off the learning part of my brain when I enter a museum”
Information:
There is rich information, but the appreciation and digestion depends on the motivation and understanding level of the visitor.
Space:
People like the current ambience of the space and some may not always seeking for new experience even when visiting multiple times.
Wayfinding:
People get confused about the direction of the space layout, but also intentionally explore the space in non-linear manner.
Information:
How might we bridge the gap between existing information and visitor’s attention?
Information:
How might we preserve the current ambience of the physical space and experience but add new layer of experiences?
Wayfinding:
How might we frame a general flow of the space while not restricting?
Set "active learning" as the goal of multi-media museum experience with MR.
By analyzing the gaps between the current experience and visitor expectations, we identified design opportunities and specific directions to guide future development: "creating", "personalization", and "circular route" as the answers to the issues we learned from interviews.
1. Implicit Learning
Assisting visitors retain information sub-consciously
2. Creativity
Inspiring creativity by using the natural beauty of minerals
3. Calm
Enhancing the already peaceful ambience & space
Ideation
Improved the ambiguous routes to a structural circular one.
01.Mixed reality experience with haptic interactions through HoloLens
The mixed reality experience with HoloLens lets visitors touch gem samples, enhancing engagement through haptic interactions. Vision classification technology recognizes the gems, allowing visitors to collect gem properties for their personal collections.
02.Digital Projection Mapping (DPM) for the final phase of sharing to the community.
The DPM includes three separate layers of projections and a body tracking system to create a depth effect for gems on the 2D screen. This setup enhances the visual experience, making the gems appear more dynamic and lifelike to the community.
Design
Design the hand gestures based on basic interaction modules in HoloLens.
We designed six hand gesture interactions with the digital content, incorporating original HoloLens gestures and customizing tap and slide to fit our spatial UI.
Overall structure - "Playing Card" metaphor
01.Why "Cards"?
Deciding the Position and location of interface through "paper prototyping"
02.Why bottom of FoV?
Minimize the body movement while using
Indicate users can hide the UI easily
03.Why "looking down" & "swipe"?
Folding/Unfolding the UI by intuitive head tracking - “look down/up”
Cards has affordance to change by swiping.
Implementation
Prototyped the "Create Artwork" with selecting two properties from four
Rapid prototyping was crucial in our design process. We tested the MR interface for Artwork Creation in Unity, including gripping, spinning, and moving sample artwork, as well as the interactive collection box and its gestures.
Goal for the low-fi prototype
Test the scale and position of UI
Test the “press to select” gesture usability
Test the usability of creating artwork scenario and user flow
Build the procedural generation pipeline for effortless artwork creation.
We embedded generative art into our experience, making it easy for users to produce high-quality gem artwork. Users can apply properties from their collection and see changes on a sample gem model, enhancing learning and review of the exhibition content.
Reflection
Evaluated the design from the initial goal through to further user testing.
A full-stack design & developing process is challenged for Mixed Reality experience.
Designing for the MR experience needs a full-stack developing process, from storytelling to implementation. Because a fast-prototyping is always required for in-process evaluation and innovative exploration, designers need skillsets including 3D modeling, animation, sensor application, and AR/VR/MR interaction development. etc. Although we found that paper prototyping was available for an efficient design process, designers still need a sense of space and a motion-based interaction mindset.
© Yuehui Du 2024