Tristan Braud is an assistant professor and director of the extended reality and immersive media lab (XRIM) at the Hong Kong University of Science and Technology. His work focuses on immersive environments, VR/AR technology, and gaming. We had the pleasure of interviewing Tristan about his work, research, and academia’s future in conjunction with web3.
Hi Tristan, can you tell us about yourself and how you got involved in the metaverse space as an academic?
“Hi, I am currently an assistant professor at the Division of Integrative Systems and Design and director of the extended reality and immersive media lab (XRIM) at the Hong Kong University of Science and Technology. My work primarily focuses on immersive environments, augmented reality (AR), virtual reality (VR), and more traditional virtual worlds and gaming.”
“For me, researching the metaverse has been a fairly natural evolution. It started with my Ph.D., where I focused on minimizing latency at the network and transport layers (TCP/IP). However, this was very technical work with little concern for user experience, which led me to question the actual usefulness of the projects I was working on.”
“Thus, I decided to move to more user-centric topics for my postdoc. I focused on augmented and virtual reality systems, in which performance is intrinsically linked to user experience. Then, the metaverse was the next logical step.”
What do you study/research in web3? What is your niche?
“My primary focus is developing large-scale (campus, city, and beyond) XR systems, following a joint system-user-centric approach. With such large-scale systems, it is critical to consider the user’s perspective, assess the adequacy of the developed methods, and ensure a seamless experience when the system’s performance cannot be guaranteed.”
“For instance, many operations may rely on mobile or wireless networks that are highly unpredictable. Sometimes, the application will experience spurs in latency and capacity drops, affecting the entire XR pipeline.”
“We thus develop solutions for these unavoidable variations to remain invisible to the user. Similarly, how to interact with content at a large scale is both a human and technical problem. The content should be situated in the digital-physical world in a way that is both technically feasible and provides a good user experience.”
You are working on a white paper for a metaverse OS. Can you tell us more about it?
“The Metaverse will enable new ways of computing, distributed around a multitude of devices managed by different entities. New systems will be required to allow such shifts in paradigms from distributed computing to XR content display and user interaction.”
“Fundamentally, large-scale XR will lead to the appearance of new resources to be shared between applications and novel ways of displaying and interacting with content. These operations are at the core of every modern operating system.”
“It would thus make sense to create a new category of operating systems to share XR-specific resources (e.g., a map of the environment) among applications and develop standardized interaction methods for XR technologies. This OS would thus present the following advantages”:
- Better performance by enabling OS- and Kernel-level base XR operations
- Better integration between XR applications and the OS and between XR applications by sharing the XR physical-digital world as a single resource across applications
- Easier development of XR applications as the OS performs the primitive XR operations (e.g., localization and mapping) and exposes the results to the applications
- Increased privacy and security by applying systematic privacy protection mechanisms on-device sensor data
- More intuitive and coherent user interaction through OS-level UIs, similar to the screen/keyboard/mouse combo at the core of modern operating systems.
Can you tell us more about other metaverse papers you’ve worked on?
“One other significant paper on the metaverse my team has worked on focuses on how to develop a campus-scale AR metaverse on our university campus at HKUST”:
“This paper lays the foundations of what we believe will represent the major technical challenges for such large-scale AR metaverses. It also describes our plans to develop the HKUST campus metaverse over the upcoming years. Such an AR campus would enable a multitude of applications”:
- Education: One of the primary objectives of a university campus is education. Our AR metaverse campus will support education, allowing both to augment physical lectures and enable remote participants’ presence in class.
- Navigation: University campuses can often be confusing, sprawling across multiple interconnected buildings. An AR navigation application can precisely guide visitors to their destination by overlaying visual cues.
- Context-aware interaction with IoT data: It is common for Sciences and Engineering campuses to become living labs for novel applications. Pervasive sensing infrastructures provide a wealth of data on-campus. The AR metaverse camp provides a context-aware display of suchrmation. For instance, one application or the number of students in the library and the different canteens. Such data could be displayed at the main intersections at peak hours to guide students to less crowded areas.
- Digital augmentation of physical events: Campuses regularly host events, such as conferences and workshops. Through the AR metaverse, it is possible to augment these physical events with digital content, providing special guidance to visitors, and enabling remote participation.
- Arts, games, and entertainment: Besides technical and educational purposes, campuses also host cultural and student life. The AR metaverse can support such life through applications. For instance, we are currently building an AR treasure hunt for new students to discover the campus at their own pace during orientation.
“This paper was published at the Meta build Workshop, the first workshop for building the foundations of the metaverse, co-located with IEEE VR 2022. This first edition of the workshop was a significant success with high-quality publications and great panel discussions.”
“This paper develops the idea of classrooms at the intersection between physical and digital, where physical participants in a university classroom can interact and follow lectures face-to-face and with other classrooms, whether physical on other campuses or completely digital.”
“Mixed Reality and Virtual Reality are the two primary technologies allowing for digital avatars and physical participants to interact seamlessly. The idea behind such classrooms comes from the specific situation of our university, which will be spread over two campuses in 2022 while still facing issues with students having to attend classes remotely due to the pandemic.”
“With the metaverse classroom, we hope to bring together the two university campuses and increase the sense of presence for remote students.”
“Besides these works that explicitly target the metaverse, most of my prior results target some aspects of the metaverse, whether related to XR, user experience, or gaming.”
From an academic standpoint, what excites you most about the metaverse, and what worries you most about web3?
“From an academic standpoint, Web 3 and the metaverse represent significant technological challenges that will most likely take the next decade to address. Similar to the original web, the metaverse will portray a complete shift in how we consume, create, and exchange content on the Internet. For now, nothing is set in stone, and there is a huge field for experimentation.”
“On the other hand, I have two primary worries about Web 3”:
- Lack of openness: If we consider the original Web, everybody could deploy a server, which led every user to become an actor. The first website was running on Tim Brener-Lee’s machine at the CERN. I believe that without such openness leading to ownership, the users would not have appropriated the Web, and it would not have seen the exponential growth that happened during the 90s. These days, the metaverse effort is driven by large companies such as Microsoft or Meta and many startups, none of which propose bridges to other platforms, leading to a variety of metaverseS instead of a single metaverse.
- Commercialism before utility. Many projects related to metaverse and Web 3 focus on blockchain technology. Some propose a complete ecosystem centered around a specific virtual world (leading to my first concern). However, many others develop a financial/commercial system before defining a utility for the space (token/NFT-centric projects). I am worried that the metaverse will become a digital space where simply existing will require some financial transaction before the public adopts the space. If this were to be the case, the metaverse would lose its universality, and adoption by the public will likely be much slower (if any).
I’ve seen you were involved in the Metabuild workshops. What are these?
“The MetaBuild workshop (co-located with IEEE VR 2022) is one of the first academic workshops dedicated to the metaverse. In this workshop, we focused on building the metaverse as a whole, whether through studies of existing metaverse platforms, the development of new technologies for the metaverse, or proposals for the future of the metaverse.”
“This workshop got great success with more than 40 participants and concluded with a panel that led to a discussion with all participants. The discussions led to very interesting ideas and concepts that we hope to develop in the upcoming years further.”
How do you think the future of education/academia will change as we enter the metaverse?
“The metaverse is likely to be driven by the industry and academia alike. Many projects will target education, whether it is metaverse classrooms, metaverse campuses, or other applications aiming to reinforce learning through immersive technologies.”
“Overall, I believe that nothing can replace physical interaction with an instructor in a physical classroom. Similarly, labs can only be superficial in virtual environments, while skills and practice need to be developed in the physical world.”
“However, the metaverse can augment such practice, either by preparing the students for tasks in the physical world or providing them with additional information during lectures and labs. It is also an efficient tool to handle university closure and remote learning following unexpected events (e.g., the COVID-19 pandemic).”
Over the next year, what other academic projects are you working on? How can our readers get involved?
“Over the next year, I will primarily focus on developing large-scale AR systems to support the development of a metaverse that penetrates our physical world (AR-driven metaverse) and studying the application of the metaverse to the classroom environment.”
“The AR-driven Metaverse requires significant amounts of visual data, and we plan to release an app for people to help us with the collection. We aim to rebuild accurate urban environment maps and provide precise localization and situated content display to the users through crowdsourcing.”
“Another aspect I want to develop is opening the metaverse and providing technical solutions for interconnecting many virtual worlds hosted on different platforms and servers. There is a significant challenge to bring together persistence and distributed hosting of digital resources.”
“Currently, we only have solved the problem of ownership persistence through blockchain and NFT solutions. However, the objects themselves are still not persistent. This may become a problem when the server hosting the objects a user paid for disappears.”
“Similarly, how to enable any user to create and host digital objects with minimal cost and effort becomes challenging when it comes to sharing and eventually exchanging such content with other users.”
If you found this article to be informative, you can explore more current Digital Twin news here exclusives, interviews, and podcasts.
