Voice interactions to reveal human expertise in virtual twins

Voice interactions to reveal human expertise in Virtual Twins

Digital & Virtual twins are ready today for operation. Despite their original purpose, their use is most of the time limited to the engineering and method offices of industrial companies. We will investigate the expected benefits of virtual twins for manufacturing industries and focus on their use on the shop floor. We will realize the benefit industrials may find spreading the use of virtual twins among field workers.

In parallel, the statement is clear: virtual twin data is still too complex to be operated as is on the shop floor by field workers. Alternative interfaces are needed to ease the interactions with the virtual twin’s data on the shop floor. Easy and safe interactions are also required to feed the virtual twins with more data from the shop floor.

Spix industry explains how voice interactions may provide the shop floor worker a safe and easy way to work with virtual twin data and feed his shop floor experience back.

Introduction to digital twins

The concept of digital twin comes from NASA to improve physical-model simulation of spaceships in 2010. Digital twins are the result of continual improvement in the creation of product design and engineering activities. The digital twin often exists before there is a physical entity of the modelled object.

A digital twin is a mathematical representation of a real-world object or system. It can be used for different purposes: industry, education, healthcare, … For the manufacturing industry, it establishes a digital representation of a real-world component, a product and its life cycle, or a shop-floor process.

The digital twins are data-driven models taking their inputs from the real-world, which allows a process to be forecasted, optimized, or surveyed. It serves as the effectively indistinguishable digital counterpart of it for practical purposes, such as simulating, integrating, testing, monitoring, and maintaining. The constant feedback from the field or from deployed products creates a constant cycle of improvement.

The digital twin integrates artificial intelligence, deep learning algorithms and big-data analysis. With such data a simulated model is created, able to change and update as function of the evolution of its real-life counterpart. A digital twin “learns” from the actual data coming from the real-life of the digital model. Evolutions of the model can be made in real-time or quasi real-time to be as close as possible of the physical process or product life cycle.

For few years now, the industry leaders recognize the power of the digital twin to push forward the digitalization effort of the production and unleash the introduction of innovations.

“From this introduction, we may easily conclude the digital twin concept is very much oriented toward engineering, studies, and optimization. Nothing in this first level of definition targets the shop floor worker. Something more is needed…”

Virtual twin to rise the interest of the shop floor

While digital twins bring tremendous value to the manufacturing sector, the virtual twin may close the gap between the engineering and the shop floor.

A virtual twin is not limited to a mathematical model of an object or a system (as the digital twin definition) but considers the systems in which the object is involved. The virtual twin includes the environment where the physical object exists. This makes it possible to simulate in the virtual worlds the consequences of each modification or optimization of an individual product globally and within the entire system model.

Virtual twin solutions (like the one from Dassault Systèmes) combine the science-based representation of a product, factory or company through modeling and simulation with the power of prediction brought about by incorporating real-world data. This convergence of the virtual and real worlds and the continuous cycle of information between the two creates a closed-loop capability that enables optimization in virtually any scenario.

“With this first approach of virtual twin as an extension of a digital twin, not sure it is clear how it may close the gap with the shop floor… Let’s see where the shop floor workers play a key role.

Virtual twins learn from different source of data, generated by the engineering at first and gradually collected from the real-world. Correspondingly, the results of the computation and simulation from virtual twins can be made available through the mean of 3D data to the shop floor. Here we are, virtual twins need shop floor data to be efficient, and in return may help shop floor workers with relevant data.”

The source of data used to feed virtual twin can be described as follow, with a gradual move from engineering data to field and real-world data.

  • At first engineering data are needed.
    Engineers and experts in modeling, with a deep knowledge of industrial processes, modeling strategy, and data processing, create the first version of a virtual twin. This version most of the time exists even before the real object does. A 3D model of the object is used to communicate to potential customers early in the development process, and mainly to the operations in charge of the production, maintenance, and recycling.

  • Life-cycle data collected from the field.
    Field data from IoT sensors are collected to reflect different status of the equipment or process under surveillance and feed the virtual twin models. As the virtual twin also considers the environment of the object or product under investigation, related data are needed. Such data may come from many different sources, including from human expertise.

To summarize, the virtual twin rises the interest of the shop floor operations, as it may provide efficient information in the form of 3D models or 3D data representation. Equally, it needs the participation of the shop floor workers to collect the necessary real-world data for it to improve and become more accurate and relevant.

“Now comes the time where a field worker seats on the shop floor and asks: how are we expected to use 3D data and give our feedback to a virtual twin with our gloves and helmets, without falling while looking at a tablet, and without being experts in digital solutions? Actually, he is not wrong…”

Voice interactions to close the gap with humans…

To really give back the expected return on investment, the virtual twin needs to find a way to equally deliver its benefits to the engineering and to the operations on the shop floor. Industrials agree that up to 25% of the time of workers is lost looking at the relevant information to perform his tasks or reporting his actions in a digital system. Engineers may spend time on digital systems. Their influence may be minored one day by technologies like ChatGPT (oupsy!), but field operators can’t. They have real-world actions to perform, sometime in harsh or hazardous conditions: digital solutions need to serve them and not vice-versa.

“Let me introduce the use of voice technologies and voice interactions to transform the digital experience of field workers into a success story, a value-added action, and an effective assistance for their jobs.”

The benefits of the virtual twins are commonly classified in seven different categories for the manufacturing industry. For all criteria, the role of voice interactions to the increase the expected benefits of virtual twins will be described.

  • Unlimited simulation.
    Simulation is a good practice to study the “what if” situations. Virtual twins are perfectly suited to perform this task, and 3D is a very good mean to communicate the results of simulation. From the shop floor perspective, the access to 3D content is much more complex. On a tablet with gloves, the manipulation of 3D information is not easy.
    Voice interactions (turn right please, zoom on the place XYZ, select the new items, show me the deviation between run 1 and 2) ease the access to simulation results at the shop floor level.

  • Continuous improvement.
    The improvement of the virtual twin comes from the quality of the models on the one hand, and from the real-world feedback on the other hand. The more real-time accurate shop floor data will be used, the more efficient the virtual twin models will be. Thus, the workers need to be assisted and motivated to feed the virtual twin with real-world data.
    Voice is an easy mean to help workers report the status of their equipment (the machine XYZ was down for 3 minutes due to cause 4, the default on the machine 1 is of category critical) and provide the virtual twin precious data.

  • Improved human factors.
    3D representation of the production plant is a perfect way to find the best compromise between productivity objectives and workers safety and comfort. Nevertheless, the human behavior is very hard to model from the engineering department. The experience of the workers is the only asset that can be used to properly model a production plant that fits the workers expectations and constraints.
    Voice is a good way to help them express their feedback to one design or the other, based on 3D information, or based on a real plant. Other ways of giving feedback are hardly used, because workers may have pain to write, may fear to misspell words, or to make mistakes. Voice eradicates these barriers and unleashes the rise of comments and feedbacks.

  • Added agility.
    Agile design and agile adaptation of production is possible if the information circulates back and forth between the parties involved in the process. When virtual twin is properly used, it enables this easy circulation of the information. Unfortunately, at the shop floor level, this is not as easy. The workers have little time to spend on digital solutions to find information nor to feed data.
    Voice interactions is an innovative way to help and motivate the field workers to access to complex information. The use of voice at the shop floor can be a way to smooth the circulation of the virtual twin information, including the operation level as all the others.

  • Global collaboration.
    The virtual twin helps to involve and motivate the field workers in the design and improvement of the production processes. The use of 3D data on the shop floor, event prior to the creation of a new production plant helps the workers to project themselves in their future work.
    The simplification of the manipulation of 3D data by voice on the shop floor (show me the heat map, select the high-pressure pipes only, cut at the 4th stair), helps the cooperation between the worker’s level and the engineering’s one.

  • IIoT integration.
    IIoT data are of mandatory importance for the surveillance of industrial assets. To be relevant for the virtual twin analysis, some IIoT data need to be completed with the context of their collection. The context is given by the expertise of the men and women working on the shop floor, close to the machines and familiar with their assets.
    If they can generate the context easily by voice (oil was replaced yesterday, humidity is not as usual, night shift reported the same issue), the IIoT data will be contextualized, and the virtual twin model will be more accurate.

  • Knowledge retention.
    This”knowledge retention” is the wrong wording to presenting knowledge capture and transmission… The motivation of digital tools regarding knowledge shall not be to retain it, but rather to share and disseminate the knowhow. Virtual twin when properly presented and used, it a good shell to share the experience accumulated on the shop floor.
    Again, voice interface provides the workers an easy way to generate expertise feedback and retex on operational situations. The access to such knowledge data by voice from the shop floor is also a way to help the up skilling of the field workers.

Based on the evaluation criteria of the benefits of the virtual twin, a key role of the voice interactions has been enlightened.

“At this point, one word is needed about augmented reality devices and smart glasses. Such devices have found their place at the engineering level to enhance team cooperation, and for training purposes. Training of skilled workers on complex tasks is mandatory. Virtual twin and smart glasses are perfectly adapted to this task. The issue on the shop floor is the dramatic increase of full fall incidents, observed while introducing digital solutions based on visual interfaces. Workers, like consumers on the street on their smartphone, focus on the screen or on the smart vision, and create injuries due to full fall accidents. The use of voice avoids such situation, helping the workers to focus on their duty, keeping their hands and eyes free. They can put their tablet or smartphone away or in their pocket, and focus on the environment, colleagues, and assets around them.”


The potential applications of the virtual twin, as an enhancement of the digital twin, for the benefit of the industry are very promising. The limits of its use and accuracy when applied to the shop floor, considering the key role of people at work is now clearly identified.

Voice interactions are suggested to overcome the difficulties of the field workers to access to virtual twin information. Similarly, the use of voice is explained to help the workers generate expertise feedback to enhance the virtual twin models with real-world data.

At the end of the day, the spread of virtual twin technologies on the shop floor will be associated to its evaluation by the workers: is the virtual twin useful, usable, and acceptable. At Spix industry we believe that voice interactions will help the workers increase their evaluation of virtual twin at the shop floor, helping them to make it efficient for their work.


(1) – Dassault Systèmes: a first look to Virtual Twin

(2) – Delmia 3D experience: use of 3D data on the shop floor

Press contacts
André JOLY – Managing Director
Phone. : +33 (0)6 25 17 27 94
Email: andre.joly@spix-industry.com

Legal entity
Website : spix-industry.com
Linkedin : linkedin.com/company/spix-industry
Simsoft3D SAS – 1244 rue l’Occitane – 31670 Labège (France)
“SPIX” and “SPIX industry” are registered trademarks of Simsoft3D SAS.