The “metaverse” is the collective marketing term for a set of virtual reality media experiences. It is accessed using headsets such as the Oculus Quest, Valve Index, and HTC vive. It is often presented in marketing materials as newly enabling the digital replication of physical space, despite the fact that this has always been possible using normal (i.e., not head-mounted) displays, and the replication is often a worse experience. The metaverse is also being used, maliciously, to promote cryptocurrency scams to uninformed consumers.
I argue that we are missing the true promise of virtual reality. By doing one thing in the physical world while seeing another in the virtual world, we can trick our brains with sensory misdirection. We can exercise more, eat more healthily, and become more imaginative. In this article I share a few examples from the bounty of scientific evidence showing how virtual reality can hack our brains for our benefit.
What is the metaverse?
The metaverse can be thought of as a new Web, but for virtual reality and accessed through virtual reality devices such as Facebook’s (i.e., “Meta’s”) Oculus headset. It is a collective term for the set of connected experiences we might have through these devices. Note: the term is not well defined. It is not a scientific term and is not widely used in serious academic research on virtual and augmented reality. It is a colloquial, commercial, marketing term, promoted by a set of corporations.
The term comes from Neal Stephenson’s 1992 science fiction novel “Snow Crash”, in which the metaverse is a virtual space, using the metaphor of the physical world, where people interact with each other as virtual avatars. A single firm holds a monopoly over the virtual ‘real estate’, which is available to be bought and developed.
Stephenson’s vision of the metaverse, while extraordinary and fascinating, appears to have had two problematic influences on how we conceive of the value of VR: the focus on recreating the real world, and the opportunity for capitalistic exploitation.
Problem 1: the way the metaverse is presented places a misleading emphasis on the replication of the physical world
Many large companies have publicly affirmed their serious commitment to building the metaverse, including NVIDIA (who makes graphics processors for computers), Epic (who develop the Unreal game engine and the popular game Fortnite), and of course Facebook (now Meta). But what exactly is it that they envision we will do in the metaverse?
Well, mostly meet and hang out. Facebook has presented a VR meetings app for work, a social space, and a VR chatting app. Video envisionments of the metaverse depict social meetings, glitzy clubs, marketplaces, and large social gatherings such as concerts. NVIDIA imagines that “we will buy and own 3D things like we buy 2D songs and books today. We will buy, own, sell homes, furniture, cars, luxury goods and art in this world. Creators will make more things in virtual worlds than they do in the physical world.”
This sounds great, except it makes it seem like these visions are new and unique to the metaverse. They are neither new nor unique. All these applications are already possible, and indeed thriving, in non-VR settings. Second Life, which has been running since 2003, has been used for art exhibitions, live music and theatre, religious meeting houses, scientific collaboration, virtual workplaces, as an educational platform, and even hosts embassies for several countries including Sweden, Estonia, and the Philippines. It has a thriving virtual economy, estimated to be worth over half a billion US dollars, based on trading digital goods and services in its digital currency (the “Linden Dollar” — at the time of writing, a US dollar is worth 320 Linden dollars). Second Life is far from the only example. Jagex’s 2001 game RuneScape also developed social uses and its own economy. For contemporary examples one need look no further than the spectacular successes of Minecraft, Roblox, and Fortnite at creating versatile digital spaces for people to commune.
All these pre-existing platforms look and work exactly like the visions of VR worlds presented to us in metaverse promotional videos. It’s just that we look at them on a screen on our desks instead of on a screen strapped to our heads.
One can object that strapping the screen to your head is not a trivial difference, that VR creates a different level of immersion, and that is indeed true. However, do these applications demonstrate a fundamentally new and unique quality of virtual reality technology being put to good use? I would argue that they do not. First, VR being ‘more immersive’ is a difference in degree, and not in fundamental quality. It is envisioning a use case for VR as a ‘faster horse’, not an automobile.
Second, VR is fundamentally worse than screens in many situations. You can do things with screens that you can’t do in VR. You can use screens for long periods of time with much less fatigue. You can use multiple screens simultaneously for different purposes. You can retain peripheral awareness of things not in VR, such as keeping an eye on a dog or a child or a bubbling pot. You can engage in intellectual activity without coupling it to physical (in)activity, by using a treadmill desk, or going for a walk outside while taking a phone call. You can use screens while using other things: reading a physical book while taking digital notes, playing the guitar while watching a tutorial, or watching the TV while knitting.
Yes, some limitations may be mitigated with better VR and AR technology, but it may take years to solve certain fundamental technological challenges, and there is no guarantee we will. Moreover, some experiences are purely digital and screen-based in origin, and there is no physical interface to replicate, leading to an awkward recreation of screens within a virtual world. For example, email in VR is going to look exactly like email on your screen. Reading documents and webpages will be the same. Watching videos will be the same. Reinventing these experiences with physical metaphors would be contrived; a hammer looking for a nail. One can argue that there’s nothing stopping us from recreating screen-like interfaces in VR either, but what is the point of strapping a screen to your head in order to recreate (badly) the experience of a screen on your desk?
Problem 2: the metaverse places a problematic emphasis on crypto-speculation.
Perhaps unsurprisingly, the metaverse has turned into a gold rush, providing plenty of ammunition for scams and schemes. Conflation and confusion are the tools of the con man’s trade. And what better confusion than cryptocurrency, NFTs, DAOs, and Web3?
Consider Decentraland. It is a virtual world where people pay real money (like, nearly USD $1,000,000) for digital ‘parcels of land’ on which they can construct buildings and offer services. The Wikipedia article for Decentraland has this delightful description: ‘Users may buy virtual plots of land in the platform as NFTs via the MANA cryptocurrency, which uses the Ethereum blockchain.’
NFTs! Cryptocurrency! Blockchain! The density of these buzzwords alone is enough to make one wonder whether it makes sense to ‘invest’ in ‘land’ in the metaverse. Of course, it does not. You can build a pretty virtual building on it, and it therefore has recreational value, but in that sense you have bought a (very expensive and bad) game, not land. You can’t live on that ‘land’. You can’t grow crops on it, or raise livestock, or mine it. And it can disappear if anything happens to Decentraland or its servers. Besides not-land, you can spend real money on other not-things, including clothes, accessories, and even names. Not nice names either: the community failed to vote against allowing the character name ‘Hitler’, and at one point the name ‘Jew’ was on sale for USD $362,000.
So why do people spend this kind of money on literally not-things? Because they believe they may ultimately turn a profit by duping someone else into spending more. It is transparently a game of greater fools. While Decentraland is currently used primarily by a small number of hobbyists and speculators, it sets a problematic precedent. There is no real scarcity of such digital assets. Any company can start a new Decentraland, offering slices of not-stuff for whatever you’re willing to pay.
The game-like quality of these experiences makes them attractive to casual consumers, and the veneer of digital ‘goods’ makes them seem like legitimate marketplaces. Further legitimacy is conferred when familiar names like Sotheby’s wade in to ‘invest’ too. But make no mistake: this is gambling and speculation in disguise. And when you can enter at any price, gambling becomes a way of parting people of all socioeconomic strata from their money, but especially takes a toll on the poor and working classes.
Dear reader, a word of advice: if someone is trying to sell you something, and you hear the word ‘blockchain’, run.
Sensory misdirection is the fundamental value proposition of virtual reality
So far we have seen that VR cannot add fundamentally new value to our lives through the applications typically presented (socialising, trading, and speculation), and may even be a step backwards. What is VR good for then?
It turns out that hijacking the entire visual field allows us to play some pretty impressive tricks with the brain. Academic research has explored and demonstrated that virtual reality can be used to help you exercise more without becoming fatigued, eat less and still be satisfied, rehabilitate patients of degenerative illnesses, and even help children with autism develop their sense of imagination in play. How is this possible?
VR can help us exercise better
Researchers at the University of Sydney conducted an experiment where participants played a variety of different virtual reality exercise games (‘exergames’) while having their heart rate monitored. These included Fruit Ninja, where the objective is to slice through falling fruit using a virtual sword; Hot Squat, where you squat your way repeatedly through an obstacle course; Holopoint, where you are an archer shooting enemies coming at you from all directions and dodging their projectiles; and Portal, where you move from room to room solving puzzles.
If you have never played a VR game before, it might help to know that playing VR exergames is not like playing traditional computer games. You move in the physical world in a fashion similar to the activity you are doing in the virtual world. To slice a falling fruit, you swing your arm through the air. To dodge an enemy arrow, you step aside or lean back.
After participants played these games, they completed a questionnaire about their perceived exertion, i.e., how hard they thought they had exercised. The researchers then compared this subjective self-assessment of exertion to their heart rate readings, an objective measure of their true exertion. They found that for Hot Squat, which was relatively monotonous and engaged the large muscles of the legs, participants felt they were exercising harder than they actually were. But for Fruit Ninja, Holopoint and Portal, which engaged the body more holistically and which offered interesting and varied movements, participants underestimated how much they had exercised. Moreover, they reported enjoying the exercise more. Other studies have also found similar results: reduced perceived effort, increased enjoyment.
Virtual reality exercise requires physical and cognitive work simultaneously, and therefore promotes neuroplasticity. Several studies show that VR exergames are better at promoting cognitive gains than isolated physical activity. This result has been found across different age groups. It has been found both in clinical settings, where the exergame is being used as part of a physiotherapy or rehabilitation program, as well as non-clinical settings, i.e. everyday exercise.
VR can help us eat better
Our experience of eating is deeply bound up in what we see. The feeling of satiety, or fullness after a meal, is ambiguous. VR can take advantage of this.
Researchers at the University of Tokyo ran an experiment where participants ate cookies in the real world, and were shown themselves eating cookies in VR. However, the cookies in VR were either shrunk, so that they appeared two-thirds the true size, or enlarged, so that they appeared one and a half times the size. Participants were asked to eat until they felt satisfied. When participants were seeing themselves eat enlarged cookies, they ate significantly less than when they were seeing themselves eat shrunk cookies, and reported feeling less hungry afterwards.
The sensory misdirection potential of VR in food can go further than changing how full we feel; it can even make us believe we are eating something we are not. Food marketing has long known the importance of colour in consumer preferences. Did you know, for example, that the flesh of commercially farmed salmon is naturally grey, and the pink colour is added artificially to replicate the appearance of wild-caught salmon? Or that 19th century butter manufacturers lobbied for margarine to be dyed pink so that it would be less attractive to consumers?
These tricks can now be played in VR, altering the appearance much more dramatically and without needing to alter the food itself. In another experiment, researchers at ETH Zurich showed participants chocolate cake in VR, while in reality they were eating a lemon cake. Nearly a third of them identified the taste of chocolate in what they had eaten. Another experiment by researchers at Aarhus and York Universities found that participants who saw light coloured coffee in VR while drinking black coffee rated it as creamier in comparison to when they drank the same coffee, but saw black coffee in VR. Researchers at the University of Tokyo found that the illusion can be made even stronger with accompanying olfactory (smell) misdirection: participants eating a plain cookie, but seeing and smelling a different kind of cookie, experienced a change in the cookie’s taste in 80% of the trials.
Virtual reality can train the mind
Through diet and exercise, VR can improve our bodies. But, perhaps more profoundly, VR experiences can also improve our minds.
When young children play, they often pretend that objects are something else. A banana is a telephone, a sofa is a fortress, a stick is a sword. Besides object substitution, pretend play can also attribute pretend properties to an object (e.g., the cracks in the pavement are dangerous), or invent imaginary objects (e.g., feeding imaginary food to a doll). Rather than being frivolous or fanciful, pretend play lays important foundations for symbolic and abstract thinking.
Unfortunately, children on the autistic spectrum often find it hard to engage in imaginative play, and this can cause developmental delays or permanent inhibition. My colleague Zhen Bai, with whom I shared an office during my PhD, conducted a fascinating series of experiments showing how an augmented reality system could facilitate pretend play in children on the autistic spectrum.
Zhen built a system that operated like a “magic mirror”. Children sit in front of a screen that shows a video feed from a camera pointed at them, essentially like a mirror. The magic is when you bring one of a set of special wooden blocks into the field of view. The blocks are plain apart from stickers bearing a small marker code, like a bar code or QR code. The system can recognise these markers and overlay the image of something else in the video feed. So the child can hold up a block but see themselves holding up a car. A block can become a train, an airplane, a railway station, a school, a bridge, a traffic light, a ball of fire.
This is not a true VR system, it is more accurately described as AR (‘augmented reality’) and no headsets are worn.
In multiple experiments with different versions of this system, Zhen demonstrated that playing with such an AR system encourages children on the autistic spectrum aged 4-7 to produce symbolic play more frequently, and for longer, compared to equivalent play without computer assistance. The most positive effects were experienced by children with the most developmental delay in symbolic play. With typically developing children too, there were benefits: a variation of the system focused on augmenting virtual characters with emotional states (i.e., giving them faces that looked happy, sad, angry, etc.) was found to promote social symbolic play. In using this system, typically developing children were encouraged to express the emotional states of pretend roles in a social context, explain the cause and effect relationship of emotional states, and communicate with playmates to construct joint pretense.
The applications are not limited to children. Surveys and meta-analyses of dozens of experiments have shown that virtual reality is effective for neurorehabilitation. Virtual reality therapy, when combined with conventional therapy, is more effective in improving the balance of post-stroke patients than conventional therapy alone. It is better for improving the upper-limb function of stroke patients than conventional therapy. It is more effective and faster than conventional therapy for dementia patients. When used in conjunction with conventional therapy, VR therapy can improve the effectiveness of treatment for traumatic brain injury, Parkinsons, and multiple sclerosis.
In each of these latter applications, the immersive nature of virtual reality is important. Strapping on a headset and replicating a flat virtual screen inside of VR is not sufficient to gain these benefits. It is the fundamental value of immersive VR that it can directly exploit and affect the wiring of our brains.
Of course, this same power can also be misused. This article aims to showcase some of the more beneficial applications. It is not difficult to imagine opportunists wanting to harness the neurological impact of VR to manipulate its users in the same way that they use the psychology of persuasion and gambling addiction to retain users today.
Conclusion
Virtual meeting rooms, social spaces, nightclubs, land, cities, and economies are fun to imagine, and they may yet become interesting and useful. But ultimately these are games and recreation at best, and scams at worst. Recreation is wonderful, of course, but it is not something fundamentally new that we couldn’t do before the metaverse; it is just a different way of having fun. To me, focusing on these applications, and making them the core of how we present and talk about the metaverse, is completely missing the point of the amazing potential of this technology.
VR exergames can create a gap between actual and perceived exertion that can be harnessed to improve our fitness. We can exercise more while believing we have exercised less, having promoted our cognitive function, and having enjoyed ourselves more. I find that amazing.
VR misdirection can create a gap between actual and perceived consumption that can be harnessed to improve our diets. We can eat less while feeling fuller. We can consume foods with fewer calories (e.g., a black coffee) while tasting foods with more calories (e.g., a coffee with cream). I find that remarkable.
VR therapies can create a gap between actual and perceived environments that can be harnessed to improve our minds. We can use VR as a scaffolding to build our imagination, or to create rich sensorimotor stimulation to prevent and reverse the decline of the mind in old age. I find that miraculous.
These are the applications of the metaverse we should be getting excited about. This is what is truly new and fundamentally valuable about VR. This is a new frontier in our relationships with our brains and bodies.
References
Costa, Marcos Túlio Silva, et al. “Virtual reality-based exercise with exergames as medicine in different contexts: A short review.” Clinical practice and epidemiology in mental health: CP & EMH 15 (2019): 15. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6407662/
Yoo, Soojeong, et al. “Evaluating the actual and perceived exertion provided by virtual reality games.” Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 2017. https://dl.acm.org/doi/10.1145/3027063.3053203
Mestre, Daniel R., Marine Ewald, and Christophe Maiano. “Virtual reality and exercise: behavioral and psychological effects of visual feedback.” Annual Review of Cybertherapy and Telemedicine 2011 (2011): 122-127. https://ebooks.iospress.nl/doi/10.3233/978-1-60750-766-6-122
Zeng, Nan, Zachary Pope, and Zan Gao. “Acute effect of virtual reality exercise bike games on college students’ physiological and psychological outcomes.” Cyberpsychology, Behavior, and Social Networking 20.7 (2017): 453-457. https://www.liebertpub.com/doi/abs/10.1089/cyber.2017.0042
Ammann, Jeanine, Michelle Stucki, and Michael Siegrist. “True colours: Advantages and challenges of virtual reality in a sensory science experiment on the influence of colour on flavour identification.” Food Quality and Preference 86 (2020): 103998. https://www.sciencedirect.com/science/article/pii/S0950329320302676
Narumi, Takuji, et al. “Augmented perception of satiety: controlling food consumption by changing apparent size of food with augmented reality.” Proceedings of the SIGCHI conference on human factors in computing systems. 2012. https://dl.acm.org/doi/abs/10.1145/2207676.2207693
Wang, Qian Janice, et al. “A dash of virtual milk: altering product color in virtual reality influences flavor perception of cold-brew coffee.” Frontiers in Psychology 11 (2020): 3491. https://www.frontiersin.org/articles/10.3389/fpsyg.2020.595788/full
Narumi, Takuji, et al. “Meta cookie+: an illusion-based gustatory display.” International Conference on Virtual and Mixed Reality. Springer, Berlin, Heidelberg, 2011. https://link.springer.com/content/pdf/10.1007/978-3-642-22021-0_29.pdf
Bai, Zhen. Augmented Reality interfaces for symbolic play in early childhood. No. UCAM-CL-TR-874. University of Cambridge, Computer Laboratory, 2015. https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-874.html
Mohammadi, Roghayeh, et al. “Effects of virtual reality compared to conventional therapy on balance poststroke: a systematic review and meta-analysis.” Journal of Stroke and Cerebrovascular Diseases 28.7 (2019): 1787-1798. https://www.sciencedirect.com/science/article/pii/S1052305719301648
Henderson, Amy, Nicol Korner-Bitensky, and Mindy Levin. “Virtual reality in stroke rehabilitation: a systematic review of its effectiveness for upper limb motor recovery.” Topics in stroke rehabilitation 14.2 (2007): 52-61. https://www.tandfonline.com/doi/abs/10.1310/tsr1402-52
Maggio, Maria Grazia, et al. “The growing use of virtual reality in cognitive rehabilitation: fact, fake or vision? A scoping review.” Journal of the National Medical Association 111.4 (2019): 457-463. https://www.sciencedirect.com/science/article/abs/pii/S0027968418303468
Acknowledgements
I received valuable feedback on drafts of this post from friends and family.
Disclosure
I am affiliated with the University of Cambridge, and with Microsoft, which has interests in AR, VR, and the metaverse. All views in this post and website are mine alone and are written in an independent capacity. This post and website do not reflect the views of any individuals at Microsoft or at the University of Cambridge. This post and website do not reflect the views of Microsoft Corporation, Microsoft Research Ltd., or the University of Cambridge.
Advait Sarkar 