Oak Story
Augmented reality + AI for learning
Timeline
10 weeks, Jun - Aug 2024
Role
UI/UX Designer, Illustrator, Front End Engineer
Process
Ideation, prototyping, iteration, usability testing, user studies
Tools
Figma, Unity, C#, Yarn Spinner, Niantic Lightship (AR)
The Problem
Without place-based context, students struggle to ground their learning in the world around them.
This project is part of ongoing research in the Stanford HCI Group on immersive interfaces for narrative-based, intelligent tutoring. I was responsible for the UI/UX design and front-end implementation in Unity, working on an iteration of the project during my 2024 summer research internship.
Design Process
Low and Med-Fi Prototyping
Over several weeks, the UI design was constantly evolving as content and modalities of interaction were explored. I created several versions to play with different art styles, including cute vector characters and lineless art, before settling on a more realistic, hand-illustrated style to accompany the content about real-world ecology and wildlife.
Design Inspiration
I compiled inspiration from apps and games that utilized narrative, innovative interactions, or educational content in compelling ways. I looked especially for how certain elements were designed for the modalities we wanted to use, including text input, choice menus, and augmented reality/camera activities.
Pilot Testing
I conducted a few pilot tests using a functioning medium-fidelity prototype with other interns and researchers to iterate on the design before building the final product in Unity.
Final Design
Key Screens
I implemented the final design in Unity with C# and Yarn Spinner for the dialogue system. Key features included input modalities including text input, photo-taking, and choice menus for user interactions with the OpenAI backend, in addition to a photo album and dialogue history tool. Dynamically generated dialogue in response to user input was parsed by sentence to be delivered line-by-line, with character detection to animate characters on screen to match. Visual aid cards were also dynamically shown when necessary.
Reflections
Working on this project over the summer was an amazing opportunity to learn about the world of HCI research. I learned how to better work in a team as I worked alongside engineers and researchers, and I got to experience firsthand how user studies are carried out to understand the effect of interfaces, for instance, on learning outcomes. I learned how to iterate quickly, work off of constructive feedback, and take designs to implementation over an incredibly rewarding 10 weeks.