News

3D smart glasses help students build multimodal literacy

Three dimensional (3D) smart glasses help students to create and communicate narratives while blending virtual and material worlds in meaningful ways, new Australian Catholic University research shows.

A study into the revolutionary technology involving 27 Year six students found the use of 3D virtual objects overlayed on the real world, known as holograms, enhanced print-based two dimensional (2D) written and illustrated texts through the creation of digital multimedia versions.

Lead researcher Professor Kathy Mills, of ACU’s Institute for Learning Sciences and Teacher Education (ILSTE), said the study showed the benefits of transmedia storytelling using a variety of platforms to boost students’ storytelling, communication, and creativity.

“This new generation of wearable technology is not a passing fad. It is the way of the future,” Professor Mills said.

“Using mixed reality technology supports students to produce truly multimedia texts, improves their digital literacy skills, and enhances their storytelling across different modes. Using such technologies targets multimodal literacy learning that is an important part of the national curriculum.”

As part of the study, published in the journal Learning, Media and Technology, the students, aged 10-11 years, first produced traditional 2D written and illustrated narratives set in different biomes.

Next, they used Microsoft HoloLens 2 devices to bring their stories to life by using a range of hologram images superimposed over the real classroom to illustrate the setting, characters, and events of their texts.

The students used voice activated video recording to tell their stories supported by the 3D holograms. Researchers were able to monitor their work by livestreaming the students’ view through their 3D mixed reality smart glasses to a computer.

Professor Mills said researchers were consequently able to understand the students’ experiences with the novel storytelling process and its similarities and distinctions to conventional forms of writing a story.

“The students were absolutely mesmerised. They were totally immersed in this experience that turned their 2D drawings and print-based writing into 3D interactive holographic scenes that they could talk to,” she said.

“They were able to provide greater visual detail to their stories, move holograms around to provide different perspectives and angles, and focus viewer attention in ways they could not do with conventional 2D media.”

Professor Mills said the students’ interactions with the virtual environment using hand and finger movements, known as haptics, to carry out functions such as grabbing, dragging, pinching, pointing, and touching holograms supported and embedded their learning.

“The in-air haptic gestures allow for deeper meaning-making by activating the brain in a more stimulating and interactive way,” Professor Mills said.

Despite the excitement over the new media, Professor Mills said the students themselves noted that the art of storytelling was at the heart of the activity.

“This technology will have a significant impact on education in the future, particular as the cost of these resources drops,” Professor Mills said. “But it is not at the expense of traditional forms of literacy, rather it’s complementary and gives students the best of the material and virtual reality worlds.”

Link to study: https://www.tandfonline.com/doi/full/10.1080/17439884.2023.2207142?src=