Extended Reality Inspection

The Problem

At the Composites Department at Konsberg Aerospace, a number of steps are followed to produce high-quality composite panels for aerospace applications. The final phase of the process is to inspect whether the panels meet the standards. In this final inspection of the parts, panel thickness is measured at specified points (illustrated on a paper sheet) around the edge of the panels to ensure that the thickness is inside of the tolerances required. The current inspection process is conducted by human inspectors. He/she prints out a schematic of the part and performs manual measurements with a micrometre guided by only the schematic's illustration. The results are written down on paper and punched into a computer system later. The XRI team was given a task of improving workflow, efficiency and reliability of this inspection process using HoloLens.
Image of micrometre

The Solution

The solution was developed for the Microsoft HoloLens headset. The first commercial device to merge virtual and augmented reality features in a hybrid technology known as Mixed Reality. With high-tech front cameras, the see-through Head Mounted Displays of the HoloLens project can not only project digital content into physical world but also register user's hand gestures allowing one to interact with the holograms. Our team used this technology to improve workflow, efficiency and reliability of an inspection process in an industrial production setting described above.
Images of HoloLens

The XRI application was designed and developed with user-friendly menus and voice command to make the inspectors work routine as seamless and intuitive as possible, enabling hands-free access to data while taking measurements.
Images of application menus

Using microcontroller and Bluetooth, the team enabled wireless communication between the micrometre (the measurement tool used by the inspector) and the XRI application running on the HoloLens, making the inspection process even smoother. This way the measurement value from the micrometre is displayed in the HoloLens and registered directly in the application.
Image of Bluetooth-module prototype

Another feature demonstrated by our team is barcode recognition. This concept can optimize the process of selecting parts by simply looking at a binary pattern that uniquely identifies a given part through the HoloLens instead of scrolling through lists of data.

The XRI application not only discards manual notation of the measurements but it replaces paper schematics all together by projecting digital replicas of the composite parts and allowing the user to interact with the measurement points.
Image of XRI-application in operation

The reliability and documentation of the quality assurance process is improved by an automatic image capturing the moment the measurement takes place. The images are stored along with measurement data, ready for an inspection report.

To advance the holographic guidance and indicate where to measure on a physical part, we explored the complex feature of overlaying AR content onto physical objects and established guidelines for camera calibrations for precise image recognition and tracking. To assess the computer vision feature and to ensure the accurate alignment of AR content onto the parts, we created a dynamic multi-axis fixture capable of locking panels of any sizes in specific positions.
Image of fixture

The important question is, what now? Well, anything really. If one can use XR technology in an industrial setting, then imagine a doctor receiving patient information in front of their eyes and retail customer service checking the inventory on the spot for the customer — all others are begging to be the next.