Narratron is an interactive projector that augments hand shadow puppetry with AI-generated storytelling. Designed for all ages, this product transforms traditional physical shadow plays into a truly immersive and intelligent phygital storytelling experience.
Context
The history of hand shadow play is nearly untraceable which was prevalently practiced long before the existence of Greek shadow show Karagiozis or Chinese shadow puppetry Pi Ying Xi. It is a prelinguistic and transcultural form of storytelling that entertains, educates, and instills moral values in the younger generation; it is also a stimuli of creative production, by mimicking the things we see, and by telling the stories we relate. Narratron, in that sense, has deeply embedded AI into this intelligent collective effort of hands, eyes, and brains as a true “fairytale copilot”. Its nature of multimodality that combines visual, auditory, tactile, and textual I/O, supported by the collaboration of LLM, image classifier, speech synthesizer and diffusion models, demonstrates how seamlessly we are able to make bodily interactions with AI. Through bridging the digital and the physical, we are now bridging the ancient and the future.
User Experience
The user's experience with Narratron begins with the startup screen, which creates a serene and focused ambiance, setting the stage for an immersive journey. As the user turns on Narratron, they are greeted by the instructions, preparing them for the captivating experience that lies ahead.Once the startup screen fades away, the user is free to explore and play with their hand shadows in any way they wish. They can experiment with different shapes, sizes, and movements, allowing their creativity to flow unrestricted. Narratron's camera captures the intricate hand shadow shapes created by the user.
The captured hand shadow shapes are then analyzed by AI algorithms integrated into Narratron. These algorithms interpret the hand shadows and translate them into animal keywords. The keywords serve as the foundation for the next step of the process: generating a complete story.
To generate the story, Narratron employs the powerful GPT-3 language model. The animal keyword identified from the user's hand shadow is processed by GPT-3, which harnesses its natural language processing capabilities to create a cohesive and imaginative narrative. The AI-generated story seamlessly combines plotlines, dialogues, and descriptive elements to deliver a captivating storytelling experience.
While the story is being generated, Narratron simultaneously generates a corresponding image using Stable Diffusion that represents the animal associated with the user's hand shadow. This image is then projected onto the surface, adding a visually immersive component to the experience. The vivid and dynamic images enhance the user's connection to the narrative, further immersing them in the story world.
As the story is projected onto the surface, the user is enveloped in an audio-visual storytelling experience. The projector synchronizes the visuals with an audio narration, bringing the story to life through voice acting, sound effects, and music. The combined audio-visual elements create a multi-sensory experience that engages and captivates the user's attention.
To initiate the storytelling experience and progress to the next chapter, the user interacts with Narratron by spinning the knob, reminiscent of vintage movie projectors. This physical interaction adds a sense of nostalgia and tactile engagement to the overall user experience.The spinning of the knob serves as a natural and intuitive way for users to navigate through the story. Each rotation of the knob signifies a progression, unlocking a new chapter and revealing fresh elements of the narrative. The user's active participation in this process creates a sense of agency, allowing them to dictate the pacing and flow of the storytelling experience.
Industrial Design
Narratron embraces a minimalist design approach that exemplifies clean lines, functional simplicity, and a focus on user experience. The overall form of Narratron exudes a sleek and understated aesthetic. Its silhouette is defined by smooth curves and geometric shapes, creating a visually pleasing and harmonious appearance. The product surface of Narratron showcases a restrained yet elegant design. It features a limited number of inputs to maintain a clean and unobtrusive aesthetic. At the top of the device, there is a singular button, which serves as the primary control for capturing the user's hand shadow and initiating the storytelling experience.
In the front of Narratron, a prominent and tactile knob is positioned. This carefully crafted knob serves a dual function, allowing users to initiate the storytelling process and effortlessly transition between different chapters of the narrative. The vintage movie projector-inspired knob adds a touch of aesthetic charm to Narratron. Its design pays homage to classic cinematic devices, evoking a sense of wonder and nostalgia for storytelling traditions of the past. This blending of traditional and modern elements further enhances the immersive nature of the overall user experience.
Narratron's industrial design prioritizes user-centered functionality and ergonomic considerations. The placement of the button and knob is carefully optimized for intuitive operation, ensuring a seamless and enjoyable user experience. The product's clean and uncluttered design eliminates unnecessary distractions, allowing users to focus on the immersive storytelling journey.
Teachable Machine: To enable hand gesture recognition, Narratron utilizes Teachable Machine, an online platform that allows for the training of custom machine learning models. The hand gesture recognition model is trained using Teachable Machine's interface, which captures and analyzes various hand shadow shapes to recognize and interpret user movements accurately.
GPT-3.5-turbo: Narratron employs the powerful GPT-3.5-turbo model developed by OpenAI for generating dynamic and engaging stories. The model is trained on a vast corpus of text, including literary works, articles, and other narrative sources. To specifically train the story generation model for Narratron, The Ugly Duckling story might have been used as a training dataset, providing a foundation for generating narratives based on user inputs and animal keywords.
Stable Diffusion: For image generation, Narratron utilizes the Stable Diffusion algorithm. This algorithm leverages deep generative models to produce visually appealing and contextually relevant images. The image generation model has been trained using a custom dataset created specifically for Narratron. The training data likely includes various animal images and visual elements related to storytelling.
Custom Model Training
Replicate: In order to train the image generation model, Narratron employs the Replicate framework. This framework allows for the development and training of custom deep learning models. Using Replicate, the image generation model is trained on the custom dataset, incorporating techniques such as conditional image generation to ensure that the generated images correspond to the user's hand shadow animal keywords.
Backend
Data Processing and Integration: The backend of Narratron handles data processing and integration tasks. It receives and processes user input from the hand gesture recognition model and passes it to the story generation and image generation components. It also manages the integration between the different components of the system to deliver a seamless and synchronized user experience.
