Summary of the context and overall objectives of the project (For the final period, include the conclusions of the action)

The past is behind us and can never be relived except to some extent through photographs and video, and of course imaginary reenactment in memory. Although, looking at a photograph may recall to mind some of the sensations of being and acting in those past events, in imagination, this is still not the same as being and acting there. This project explored the possibility that by using virtual reality (VR) it may be possible to give people the sensation of participating in reconstructions of past events. 

This goal then leads to a number of scientific problems. What does it mean to have the sensation of ‘being there’? This has been studied for 3 decades by researchers under the general name of ‘presence’ – the sense of being in the virtual world depicted by the VR displays, referred to specifically as ‘Place Illusion’ (PI). Under what conditions will people tend to act in a VR scenario as if it were real? This requires the operation of another illusion referred to as ‘Plausibility’ (Psi), which is the sensation that events that occur in the VR are really happening. For example, imagine in VR there is a crowd of virtual people in a street, they part as you move through them and  a virtual person passing by smiles at you.  You will tend to smile back even though you know for sure that nothing real is happening. You use your body to perceive in VR much as you do in reality (looking around by turning your head and body, listening by turning your ear towards a sound, reaching out to touch) which leads to PI.  The crowd parting, and the person smiling specifically at you leads to Psi. Typically PI and Psi together (‘presence’) lead to you act in the VR much as you would in similar circumstances in reality. 

Hence particular goals of this project were to deepen understanding of PI and Psi building on theory through the results of experimental studies. The approach used was to reconstruct musical performances from the past, immerse people in those and to understand their responses, and the extent to which PI and Psi were achieved. Specifically we reconstructed part of a 1983 Dire Straits concert at the Hammersmith Odean in London performing ‘Sultans of Swing’ (https://youtu.be/bOSWaKT88j4 )and also the performance of the Spanish singer Massiel  at the Eurovision Song Contest that Spain won in 1968 (https://youtu.be/dGcmRNYFtt8).

This required the fulfilment other objectives. In order to reconstruct the Dire Straits concert we had to have a way of creating virtual copies of the band members from photographs and video, and similarly for Massiel. This led to the development of a method for the construction of virtual bodies that have a close likeness to actual people just from photographs, by using a deep learning neural network. It also required a method to construct large virtual audiences that would behave appropriately in response to the music of the band. We also had to provide a method for people to move through virtual spaces larger than the physical space in which they were located, and without experiencing simulator sickness. But VR affords experiences that are not possible in reality – as well as placing people in the audience of a concert they could be on stage, and be part of the band. Being part of the band – how would they behave depending on their relationship to the virtual body of the band player in which they were embodied? If they had a strong feeling that the virtual body was their own body (referred to in the scientific literature as ‘body ownership’) would they behave differently than if they did not have this feeling? Would they have a sense of agency over the movements of this virtual band player body even if these were not their own movements? Finally, in reality people do not attend concerts alone, but go with friends or family. Therefore we had to have a way that multiple different people, separated geographically in space, could simulataneously be together in the same VR scenario, see and interact with one another, each represented by an avatar with their own likeness. Hence we developed a system called ‘VR United’, that also had many other applications – such as allowing immersive meetings between members of our lab during periods of lockdown, and applications in journalism where a journalist could interview a remote person with both together in an immersive setting.

This project has several applications potentially beneficial for society. During the course of the 6 years of the project VR has moved to become a consumer product. Hence the more that we know about it the better, especially regarding its ethical and data protection implications. To this end we spun off an additional project funded by the Spanish Government to allow us to investigate such ethical issues. Second, the VR United application can be of wide usage, and we are investigating taking this forward in as a commercial venture. One application that we are now currently exploring is to help people overcome loneliness – specifically, older people geographically separated from their relatives may nevertheless meet and interact with them immersively in the same shared virtual space. 

The most significant application is for older people to reexperience their past. For example, for a certain generation in Spain, winning the Eurovision Song Contest in 1968 is an iconic moment. We can place older people back in that time, and experience being in the audience in a virtual reconstruction of Massiel’s performance. The experiment to evaluate this was carried out in the last 3 months of the project, and the results are waiting to be analysed, though based on past data it is likely that such re-visiting the past in an immersive way will have a positive effect on various indicators of ageing. This led to a further spin-off project (an ERC Proof of Concept) where this will be investigated further, and taken out to society.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far (For the final period please include an overview of the results and their exploitation and dissemination)

Two versions of a Dire Straits concert were completed and experimental studies were carried out in order to understand how people responded to the novel experience of attending a virtual concert. We used a machine learning technique called ‘sentiment analysis’ that required participants to write short essays describing their experiences, as well as standard questionnaires. These short essays could then be analysed by sentiment analysis that assigned a numerical score based on the negative or positive sentiment expressed by the essays, or by each sentence.  The results were surprising, the performance of the band itself was less impactful than the audience surrounding the participants. Even though everyone knew that the situation was entirely virtual, some participants become disturbed by the imagined intentions of their surrounding audience. In particular, some women became concerned that the virtual men around them would approach to start a conversation. Some felt that they were being ‘stared’ at by members of the virtual audience, which made them uncomfortable. Using sentiment analysis was a breakthrough – we would never have found these deeper responses to the event using standard approaches such as questionnaires. The level of ‘place illusion’ and ‘plausibility’ were high, but independently of this, the sentiment was variable depending on the particular disposition of each participant, and how they interpreted the virtual events around them. 

We developed a new method for the quantification of presence (place illusion or plausibility, or any other subjective feeling associated with the evaluation of a virtual event). Suppose a number of factors might be involved in the generation of presence (for example, whether or not virtual characters look at the participant during the experience, or the level of photorealism of the depicted scenario). The participant would start the experience in a particular configuration of these factors (for example, the characters never looking at them, and a cartoon style of rendering, and others). Then every so often an artificial intelligence program, based on a method called ‘reinforcement learning’ (RL) would offer to the participant a change in one of the factors (e.g., a change to a more realistic style of rendering) (https://youtu.be/WS-Khiv4ecw). The participant could accept or reject this change depending on whether or not it enhanced their sensation of presence. Then later the RL would offer a further change, which again the participant could accept or reject. This continued throughout the scenario, and the RL would learn which changes the participant would be most likely to accept or reject. After the participant had rejected a sequence of changes, e.g., 10 times, the RL would stop and this was taken as the equilibrium or optimal configuration for the participant. In this way we would discover the optimal settings for participants without the use of questionnaires after the whole experience, rather the optimal setting was determined by the real-time choices made by them. We refer to this as ‘Adaptive Multimodal Matching’(A3M) which is an advance on our previous method ‘Multimodal Matching’ (3M), which is similar but without the use of the AI. 

This novel method (3M/A3M) has subsequently been applied in other applications, and by others – for example, 3M in collaboration with Facebook to determine the style of presentation of a virtual TED talk most preferred by people. This method also allowed us to show experimentally, towards the end of the project, that ‘place illusion’ is indeed dependent on the method by which people perceive and act in virtual reality (using their bodies the same way that they do in physical reality). We also were able to show that place illusion is related to how people move their eyes – the greater the presence, the lower the randomness in eye movements. This also shows that presence is a genuine psychophysiological phenomena, and not just a theoretical construction by researchers. 

The 3M/A3M methods are being exploited in various further projects. A project called PRESENCE (i2cat.net/presence/) which is investigating advanced methods and technologies to maximise presence, environMENTAL (www.environmental-project.org) which is considering the impact of climate change on mental health, and GuestXR (guestxr.eu) where it is being used for the evaluation of social presence. 

Using our technology that enabled the reconstruction of concert performances in VR we carried out an experiment with older people in order to examine if a form of ‘reminiscence therapy’ would be effective in VR. Reminiscence therapy is based on the idea that putting people back in times of their youth, for example, by showing photographs, videos, or revisiting salient places, improves their quality of life, for example, by cognitive improvement. We used the Massiel performance at the 1968 Eurovision Song Contest to explore this idea. We have not yet completed the analysis of these data, from the last 3 months of the project, but an earlier pilot experiment suggested that the results would be very successful. With respect to exploitation a further ERC Proof of Concept project has just started to explore this method and to take it out to society. 

In previous work, ourselves and others had found that body ownership over a virtual body of a different race would reduce implicit bias against people of that race. For example, embodying ‘white’ people in a ‘black’ virtual body, just for a few minutes, tended to reduce the implicit bias of those people. Implicit bias refers to non-conscious bias, and is not related to beliefs or explicit attitudes. However, previous work did not take into account the social setting of the scenario, which had been neutral or positive. We carried out an experiment to show that if the scenario is a negative one, where people may become stressed, then embodiment in a body of another race actually increases their implicit bias. This is very important, because it is assumed by many that VR is an ‘empathy machine’ – that simply putting people in VR in the situation of disadvantaged and discriminated-against groups would automatically increase their empathy towards those groups. This may or may not be the case, but our results suggest that irrespective of empathy, their bias may be increased. 

Further to this we developed a new paradigm to encourage pro-sociality, that is the behaviour to support others, in this case people who are being attacked or abused. This is called the ‘Golden Rule Embodiment Paradigm’ since it is based on the ‘Golden Rule’ of many religious and philosophical standpoints: “Do not do to others that which you would not want done to yourself”. It also involved embodiment and body ownership. The idea is that the participant would first be (in VR) involved in an action that causes harm to a (virtual) other – for example, acquiescing while a virtual human character insulted another virtual human character. Participants would then re-live a playback of the whole scenario and be able to experience, from the embodied perspective of the victim their own previous acquiescence to the negative situation. Compared to a control group we found experimentally that in a subsequent test scenario those who had re-experienced the scenario from the victim’s standpoint, were more likely to intervene to stop the abuse. We carried out this scenario in an inner city police department in the United States, in the context of police bias against African Americans, and indeed found that in the subsequent test scenario, participants who had experienced the scenario from the standpoint of the African American victim, were more likely to intervene to stop harassment by a (virtual) fellow officer, than those in a control group (https://youtu.be/tb9QAUkZWic).  This project was carried out with in collaboration with Google Jigsaw.

The Golden Rule Embodiment paradigm is the fundamental method used by our spin-off company kiin.tech, to reduce discrimination in the workplace, and has been exploited in multiple trials with various global companies. 

In another study we found that it is possible to have agency over a virtual body, one that looks like the participant, even though it detaches itself from the location of the participant in VR and carries out its own independent actions. By ‘agency’ is meant that people have the sensation that the actions of the body are their own actions. In this experiment the participants were people who experienced a non-clinical level of paranoid ideation, meaning, for example, that they imagined that other people were hostile to them. This has negative consequences for their social interactions with others. In the experimental study, one group saw their virtual body successfully interacting with other (virtual) people, and another group saw their virtual body randomly walking around without ever interacting with the virtual people present. We found that the first group improved with respect to their feelings of paranoia, even though the measurements were taken one week after their VR experience, compared to the second group. This showed that the agency over their virtual body could have important and beneficial psychological consequences (https://www.youtube.com/watch?v=V5oreDMnR8Q).

Participants in a VR need to move around. They are typically in a real space (a lab, or a home living room) that is much smaller than the virtual space in which they can move (for example, think about a concert hall). In VR people typically move by unnatural actions, such as beaming instantaneously to positions that they point at by using a hand-held controller and pointing and pressing a button. This has several  disadvantages, not least of which is its impact on plausibility and also disorientation. The best way to move around is to walk, since this excites normal bodily and brain responses. But if the real space available is too small and they walk, they will quickly intersect with barriers such as real walls or furniture. We devised a method where they are able to walk long distances by actually walking, but whenever they reached a boundary, they could swivel the virtual world around so that they would be still able to walk and keep within the boundary. 

We developed a new VR programming library called QuickVR which is available for anyone to use (via GitHub) called QuickVR. We developed a method for the rapid creation of virtual bodies based on photographs of people, as explained above, and deployed these in the VR United application to allow multiple people to simultaneous interact together in a VR scenario each person with a virtual body that looks like themselves. VR United has been exploited publicly in a number of ways. A journalist from the Financial Times in London used it to interview Professor David Chalmers of New York University for a series of articles called ‘Lunch with the FT’. The interview was about Prof. Chalmers’ book ‘Reality+’ (https://www.youtube.com/watch?v=1dACicAYdYg). The video journalist Raul Gallego Abellan (currently with TV3 Catalonia, Spain) conducted a remote interview from Jerusalem with Professor Hiroshi Ishiguro in Osaka Japan (https://youtu.be/njVlI8409fs). A conference panel was organised using VR United with Prof. Chalmers (Australia), Prof. Doron Friedman (Tel Aviv, Israel) and Prof. Slater (Barcelona, Spain) together with a guest appearance by Dr Albert Einstein, a virtual character controlled by ChatGPT (https://youtu.be/qkN1F3QAhp8). We were invited to a conference called ‘Global Impact Summit’ in Malaga, Spain (October, 2023) as part of a panel to discuss creativity. The remit of the discussion was to include the work of the artist Pablo Picasso, and the founder of modern neuroscience Ramon y Cajal. Using VR United and ChatGPT we created a Barcelona cafe scenario where these two discussed the nature of creativity, while current and past members of our lab were seated around the cafe (https://youtu.be/aEC1SogJfk8). 

Research over a long period (6 years in this case) is highly unlikely to stick rigidly to a plan. By its very nature research leads to new avenues not thought of at the time of submission of the original grant proposal. One example of this is based on the idea of ‘change blindness’. This occurs in reality where people can be oblivious to dramatic changes in their environment. We showed that this occurs also in VR, but surprisingly, even with respect to the appearance of virtual people, including the participant’s own virtual body. Participants were embodied in a virtual body that they saw directly when looking down towards themselves and also in a mirror. They interacted with a teacher of Qigong, who demonstrated various movements that they had to follow. Over time both the faces of the teacher and the self-body changed gradually but dramatically. Just over 70% of participants did not notice the change in their own virtual body, and 85% did not notice the change in the virtual teacher’s body (https://youtu.be/XPkUIjBKqUU). Participants looked towards both bodies an approximately equal amount of time. This shows that change blindness not only occurs in VR (which was already known) but that it operated even though participants were looking continuously at the items that changed. Moreover, it offers a further sign of ‘body ownership’ since although there was change blindness with respect to their own virtual body, it was significantly less than the body of the teacher. 

The change blindness paradigm was also exploited in a further application, concerned with helping people with fear of public speaking. In this case people with such anxiety spoke to a standing virtual counsellor, who asked them to carry out a series of exercises (such as reading a script on a white board, or describing a short movie) and gave them advice about how to overcome their fears. Over time the single counsellor morphed into an entire seated audience, by duplicating his virtual body which then moved off and imperceptibly transformed into another person. Although participants would, of course, notice each duplication, they were unlikely to observe the transformation to other persons, nor their adoption of a seated position, since their focus was on the counsellor always in front of them, and the changes occurred in peripheral vision. This occurred over approximately 20 minutes. The method was compared to traditional exposure therapy, where another group returned 5 times, on the first occasion speaking to the single counsellor, the next time to a small audience, the third time to a larger audience, and eventually to the whole audience. It was found that the single session method was at least as good, in helping people overcome their fear of public speaking, as the multiple exposure method, which is an excellent first result for exploring this paradigm further as a new approach to classical exposure therapy. 

A change blindness method was also applied to the problem of simulator sickness in VR. We referred above to methods of moving through a scenario. A very typical method is to point a hand-held controller in the direction of desired travel, and then pull a trigger on the controller to move forward in that direction, with speed controlled by how much the trigger is pulled. Most people experience simulator sickness with this paradigm – unless the movement forward is very slow. This is because their visual system is indicating forward movement but their sensorimotor and vestibular systems indicate that they are standing still. The conflict between these systems typically causes sickness. We discovered that people will move faster through a darker environment than a brighter one, without experiencing simulator sickness. We carried out two experiments to test this. The first was used to construct a statistical model of the relationship between velocity of movement and brightness of the environment, and the second was used to test this model. The model developed from the first experiment well-explained the results of the second. The conclusion is that starting off a VR scenario with a darkened environment and then gradually and imperceptibly increasing the brightness results in less chance of simulator sickness. This is important because very often the first VR experience of people leads to simulator sickness, and as a result they never want to try VR again.

Progress beyond the state of the art and expected results until the end of the project

The use of sentiment analysis for the analysis of the responses of people towards virtual environments has been an important breakthrough and is now regularly employed in our lab, and the two papers published on this, 2021 and 2023, have been cited by 68 and 12 times, respectively. 

The  3M and A3M methods have been followed up by others, we are aware of at least 7 papers by others that have used it, the most recent in 2023. We continue to use and develop it in our lab, where we have used it in 6 studies with corresponding papers, most recently in 2024.

The discovery that the entropy (disorder) of eye movements is related to presence, and the method of discovery itself is new, and throws new light on the meaning of presence. Prior to this we considered presence as composed of  ‘place illusion’ and ‘plausibility’, the theory of which was further developed and published during the course of the project. Now we add a third dimension that we refer to as ‘familiarity’ since the low level of disorder (entropy) of eye movements associated with presence is similar to the known result that in reality eye movements have low entropy in familiar environments. This goes deep into the reason why virtual reality ‘works’ in spite of low fidelity with respect to reality – people are perceiving the virtual world based on their own internal world models. The VR offers cues about what type of place is being represented, and the rest is ‘filled in’ from the participants’ internal world models. We are now persuing the notion of presence in totally unfamiliar environments. 

The many uses of VR United and its components (look-alike self-representations) provide an important step forward in the exploitation of VR in the developing idea of the metaverse. 

The discovery that change blindness occurs even though people are looking at the areas of the scene that dramatically change, and notwithstanding that the parts of the scene that changed were vital (the representation of self and another) is new, and has many applications that are being now explored. 

The operation of the Golden Rule Embodiment paradigm is new, and similarly is being exploited in other applications.

The link between environmental brightness and simulator sickness and its exploitation in a new method to reduce the chance of simulator sickness is an important step beyond the state of the art.