Enhancing military pilot training: Overcoming challenges in integrating VR and MR technologies

As air forces worldwide grapple with pilot training challenges, some view the increased use of simulators and modern technologies as the way forward.

Virtual reality (VR) has been increasingly utilised in military pilot training to enhance the effectiveness of training programmes. VR offers a realistic and immersive environment that allows pilots to practice and refine their skills in a simulated setting.

It has been enjoying its popularity, particularly for pilot training, due to its cost-effectiveness, risk mitigation and scenario replication qualities. While it has become increasingly integrated into pilot training, it has faced several challenges over the years.

Related Articles

Royal Air Force pilot training troubles not going away

Vrgineers to showcase F-15E fighter simulator that could replace live training at IT2EC 2023

Updated: Elbit departs Project Selborne submarine training, Metaverse VR steps in

Problems that had to be addressed included mitigation of motion sickness for users, and limitations of the technology itself, such as visual fidelity, latency issues, integration with existing systems and training content development challenges.

‘Mixed reality (MR) is at a similar point in its life where VR was around three years ago,’ Marek Polčák, CEO and founder of virtual reality and pilot simulation training specialist Vrgineers. 

Challenges of MR pilot training

Military pilot training has faced certain challenges with the integration of MR technologies, particularly issues related to tracking accuracy, latency and processing power – reminiscent of the issues that VR once grappled with.

Polčák mentioned that the majority of training technology users are at least partially convinced that MR is the way to go, but the broader implementation of the technology is still about a year or so away.

‘With VR, you see different programmes or projects, especially in the US, that include the rolling out of hundreds of headsets for pilot training,’ Polčák noted. ‘MR is not quite there yet, but it’s starting to get there.’ He said this was due to multiple technological hurdles which make such platforms inconvenient to use.

‘What I found out during my presentations around the world, through direct feedback from pilots and trainees, is that [one of the main concerns] is visual quality, including [image] fidelity and readability,’ Polčák said. ‘Another problem is linked to the resolution of the cameras and the overall performance.’

He explained that VR devices work on a very high refresh rate (75Hz 4K per eye or 120Hz QHD per eye) and adequate MR training requires the same performance or higher.

‘When you are sending two high-quality streams into a computer, it becomes quite difficult to calculate them,’ Polčák explained. ‘The problem is that the [data] streams are not sufficient to fully utilise the displays. For instance, if you have an 8K camera, you can only utilise a 4K display, but you are unable to feed an 8K refresh rate into the computer to make it work.’

Vrgineers believes it can introduce an MR product free from currently present issue by the end of 2024-2025. (Photo: Vrgineers)

The second main issue that pilots encounter during using MR head mounted displays (HDM), Polčák said, is the discrepancy between the VR and MR environments.

The difference occurs ‘because the cameras are positioned in front of the users’ eyes, but the virtual camera in the scene is positioned exactly where the eyes are,’ he said. ‘This causes misalignments which they can see in every headset when they are moving around.’

Polčák added that due to the physical nature of this issue, overcoming it will be quite a challenge for the industry. There are, however, two approaches that already exist seeking to address the problem. ‘You have to optically or virtually move the chip itself into the centre of the eyes,’ he said, adding that Vrgineers has been working on some solutions.

‘There are some advanced algorithms which we are looking into,’ he said. ‘These basically re-project the physical world into the 3D model and then shift it to the eyes’ point.’

Another solution is to develop a periscope-like solution which can extend the length of the light beam so that it can travel to the chip. To make users trust MR technologies, Polčák said, companies who work on MR solutions for military applications need to invest more in software and algorithm development, as well as on increasing performance.

Projected timeline for MR technologies for pilot training

Vrgineers anticipates that it can introduce an MR pilot training product free from the currently present issues by the end of 2024 at the earliest, with the most pessimistic projection extending to the end of 2025.

Polčák thinks that a few minor inconveniences are also holding back the popularity of MR devices in pilot training. ‘One of these is interconnecting VR and MR platforms,’ he said. ‘When you use a VR device, you need to load in a 3D model of the cockpit, or you have to be in a greenroom screen so you can map it out.’

According to Polčák, introducing extra adjustments and requirements can disrupt the smoothness of training and add inconvenience. He believes that MR pilot training solutions will gain widespread adoption when they can be simply plugged in without the need for additional technicians.

Shephard's Dubai Airshow 2023 coverage is sponsored by: