Mixed reality (MR), sometimes referred to as hybrid reality,^[1] is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. Mixed reality takes place not only in the physical world or the virtual world,^[1] but is a mix of reality and virtual reality, encompassing both augmented reality and augmented virtuality^[2] via immersive technology. The first immersive mixed reality system, providing enveloping sight, sound, and touch was the Virtual Fixtures platform developed at the U.S. Air Force's Armstrong Laboratories in the early 1990s. In a study published in 1992, the Virtual Fixtures project at the U.S. Air Force demonstrated for the first time that human performance could be significantly amplified by the introduction of spatially registered virtual objects overlaid on top of a person's direct view of a real physical environment.^[3]

Virtual reality (VR) versus augmented reality (AR) versus mixed reality (MR)

The definitions in the modern contemporary economy makes the distinction between VR, AR and MR very clear:

•   Virtual reality (VR) immerses users in a fully artificial digital environment.
•   Augmented reality (AR) overlays virtual objects on the real-world environment with spatial registration that enables geometric persistence with respect to placement and orientation within the real world. Prior technologies that overlaid data or images not spatially registered to real world geometries are referred to as heads-up display technologies.
•   Mixed reality (MR) not just overlays but anchors virtual objects to real world objects and allows the user to interact with combined virtual/real objects.

Definition

Virtuality continuum and mediality continuum

In 1994 Paul Milgram and Fumio Kishino defined a mixed reality as "...anywhere between the extrema of the virtuality continuum" (VC),^[2] where the virtuality continuum extends from the completely real through to the completely virtual environment with augmented reality and augmented virtuality ranging between. The first fully immersive mixed reality system was the Virtual Fixtures platform developed at US Air Force, Armstrong Labs in 1992 by Louis Rosenberg to enable human users to control robots in real-world environments that included real physical objects and 3D virtual overlays called "fixtures" that were added enhance human performance of manipulation tasks. Published studies showed that by introducing virtual objects into the real world, significant performance increases could be achieved by human operators.^[5][6]

The continuum of mixed reality is one of the two axes in Steve Mann's concept of mediated reality as implemented by various welding helmets and wearable computers and wearable photographic systems he created in the 1970s and early 1980s,^{[7][8][9][10][11]} the second axis being the mediality continuum, which includes, for example, Diminished Reality (as implemented in a welding helmet or eyeglasses that can block out advertising or replace real-world ads with useful information)^[12][13]

"The conventionally held view of a Virtual Reality (VR) environment is one in which the participant-observer is totally immersed in, and able to interact with, a completely synthetic world. Such a world may mimic the properties of some real-world environments, either existing or fictional; however, it can also exceed the bounds of physical reality by creating a world in which the physical laws ordinarily governing space, time, mechanics, material properties, etc. no longer hold. What may be overlooked in this view, however, is that the VR label is also frequently used in association with a variety of other environments, to which total immersion and complete synthesis do not necessarily pertain, but which fall somewhere along a virtuality continuum. In this paper we focus on a particular subclass of VR related technologies that involve the merging of real and virtual worlds, which we refer to generically as Mixed Reality (MR)."

Interreality physics

In a physics context, the term "interreality system"^[14] refers to a virtual reality system coupled to its real-world counterpart. A paper in the May 2007 issue of Physical Review E^[15] describes an interreality system comprising a real physical pendulum coupled to a pendulum that only exists in virtual reality. This system apparently has two stable states of motion: a "Dual Reality" state in which the motion of the two pendula are uncorrelated and a "Mixed Reality" state in which the pendula exhibit stable phase-locked motion which is highly correlated. The use of the terms "mixed reality" and "interreality" in the context of physics is clearly defined but may be slightly different in other fields.

Augmented virtuality

Augmented virtuality (AV), is a subcategory of mixed reality which refers to the merging of real world objects into virtual worlds.^[17]

As an intermediate case in the virtuality continuum, it refers to predominantly virtual spaces, where physical elements, e.g. physical objects or people, are dynamically integrated into, and can interact with, the virtual world in real time. This integration is achieved with the use of various techniques. Often streaming video from physical spaces (e.g., via webcam)^[18] or using 3-dimensional digitalisation of physical objects.^[19]

The use of real-world sensor information (e.g., gyroscopes) to control a virtual environment is an additional form of augmented virtuality, in which external inputs provide context for the virtual view.

Applications

A topic of much research, MR has found its way into a number of applications, evident in the arts and entertainment industries. However, MR is also branching out into the business and education worlds with systems such as these:

• IPCM – Interactive product content management

Moving from static product catalogs to interactive 3D smart digital replicas. Solution consists of application software products with scalable license model.

• SBL – Simulation based learning

Moving from e-learning to s-learning—state of the art in knowledge transfer for education. Simulation/VR based training, interactive experiential learning. Software and display solutions with scalable licensed curriculum development model.

• Military training

Combat reality is simulated and represented in complex layered data through HMD.

One of the possible applications mixed realities is for training military soldiers. Training solutions are often built on Commercial Off the Shelf (COTS) technologies. Examples of technologies used by the Army are Virtual Battlespace 3 and VirTra. As of 2018, the VirTra technology is being purchased by both the civilian and military law enforcement to train personnel in a variety of scenarios. These scenarios include active shooter; domestic violence; military traffic stops, etc.^[20][21]  Mixed reality technologies have been used by U.S. Army Research Laboratory scientists to study how this stress affects decision making. With mixed reality, researchers may safely study military service men and women in scenarios where soldiers would not likely survive.^[22]

As of 2017, the U.S. Army was developing the Synthetic Training Environment (STE). STE is a collection of technologies for training purposes that has been estimated to include mixed reality. As of 2018, STE was still in development without a projected completion date. Some recorded goals of the simulation were to increase simulation training capabilities, and the availability of the environment to other systems, and to enhance realism.^[23] It was claimed that training costs to be reduced with mixed reality environments like STE.^[24][25] For example, using mixed environments could reduce the amount of munition expended during training.^[26] It was reported in 2018 that STE would include representation of any part of the world's terrain for training purposes.^[27] STE would offer a variety of training opportunities for squad brigade and combat teams, including, but not limited to Stryker, armory, and infantry.^[28] It is estimated that STE will eventually replace the Army's Live, Virtual, Constructive – Integrated Architecture (LVC-IA).^[29]

• Remote working

Mixed reality allows a global workforce of remote teams to work together and tackle an organization's business challenges. No matter where they are physically located, an employee can strap on their headset and noise-canceling headphones and enter a collaborative, immersive virtual environment. Language barriers will become irrelevant as AR applications are able to accurately translate in real time. It also means a more flexible workforce. While many employers still use inflexible models of fixed working time and location, there is evidence that employees are more productive if they have greater autonomy over where, when and how they work. Some employees prefer loud work spaces, others need silence. Some work best in the morning, others at night. Employees also benefit from autonomy in how they work because everyone processes information differently. The classic VAK model for learning styles differentiates Visual, Auditory and Kinesthetic learners.^[30]

Machine maintenance is also a subject that can be executed with the help of mixed reality. Larger companies that have multiple manufacturing locations with a lot of machinery can use mixed reality to educate and instruct their employee. The machines need regular checkups and have to be adjusted every now and then. These adjustments are mostly done by humans, so these employees need to be informed about every small adjustment that needs to be done. By using mixed reality, employees from multiple locations can put on a headset, and get live instructions about the changes. Instructors can operate the representation that every employee sees and can glide through the production area, zooming in to technical details and explain every change of a machine. It has shown that a five-minute training session with such a mixed reality program has the same results as the employees reading a 50-page training manual.^[31]

• Functional mockup

Mixed reality is applied in the industrial field in order to build mockups that combine physical and digital elements.^[32] With the use of simultaneous localization and mapping (SLAM), mockups can interact with the physical world to utilize features such as object permanence.^{[citation needed]}

• Consciousness

It has been hypothesised that a hybrid of mixed and virtual reality could pave the way for human consciousness to be transferred into digital form entirely - a concept known as Virternity, which would leverage blockchain to create its main platform.^[33][34]

Display technologies

Here are some more commonly used MR display technologies:

• Cave Automatic Virtual Environment

• Head-up display

• Head-mounted display

• Tablet PC

• Computer display

• Personal Digital Assistant

• Mobile phones

• Handheld PC

Notable examples

• BAMZOOKi
• EyeToy
• FightBox
• Magic Leap
• Microsoft Hololens
• Nokia Point & Find
• zSpace

See also

• Mixed reality portal