While robotics and artificial intelligence get plenty of attention in supply chains and customer service, a more fundamental change is happening in manufacturing. The Internet of Things, virtual reality, and augmented reality — or IoT, VR, and AR — are being adopted in response to the need for speed and flexibility in production.

Last week, events across the U.S. associated with Manufacturing Day provided an opportunity for enterprises to assess new technologies, as well as for students and workers to learn about new opportunities.

PTC partnered with Rockwell Automation and the National Association of Manufacturers to create an AR-infused comic book for Manufacturing Day. It illustrates modern manufacturing and how science, technology, engineering, and mathematics (STEM) can lead to fulfilling careers.

Robotics Business Review recently spoke with two executives at Needham, Mass.-based PTC Inc. about how AR and VR are already helping manufacturers.

Mark Schuetz, vice president of AR product management at PTC, has worked at Siemens in Germany. He then worked with the Creo 3D CAD software and digital twins.

“I realized that defining IoT and AR is part of our story, and we acquired Vuforia,” he noted. “Five big forces are transforming manufacturing: IoT and big data analysis, AR and VR, additive manufacturing, robotics, and the concept of the digital platform tying it all together.”

Jean-Philippe Provencher, vice president of manufacturing, has been at PTC for more than 15 years, after the company acquired ThingWorx. He helps shop floors digitize operations and achieve higher performance with IoT, VR, and AR.

“I’m focused on factory IoT strategies and packaging these technologies for tough use cases,” he said.

Q: What are some of the challenges in applying augmented reality and virtual reality to manufacturing processes?

Schuetz: As manufacturing processes become more complex and there are more product configurations, there’s another trend: Automation is coming to factories more and more.

Manufacturers are trying to do more with less people, but they still need people to assemble, operate, or maintain machines. They need a product skill set, people who know how to fix things, so there’s more to deal with.

We need to give workers those skills and information they may not have on time, using VR.

One challenge is hardware. Digital eyewear is evolving. It’s at a stage like the iPhone 1. It took years to get it to where we need it — starting with tablets, touchscreens, then head-mounted devices. You need a camera to identify objects and a way to provide the AR experience. You can have 2-D information on a tablet.

The advantage of those devices is that they are certified for factories and have a long battery life.

From a technical perspective, a HoloLens or Magic Leap is better, but they’re not certified for factories. They’re good for only a couple of hours, so it’s not yet a complete shift.

Provencher: Hardware is definitely a challenge. PTC has been serving manufacturers for more than 30 years. We know our customers. We help by being hardware-agnostic.

HoloLens is great, but customers can start with a smartphone or a tablet and a fixed camera. We can help them adopt AR today, and in some use cases, they might need something more advanced.

Q: How do PTC customers identify where AR and VR can be most useful?

Schuetz: We try to help customers use the right hardware for their use cases with what’s available now. If they want hands-free, it’s possible to have a touchscreen on the wall and a camera over the workbench. That’s one way of dealing with hardware limitations.

Provencher: Companies first need to see where AR and VR can add value. Then we can identify use cases where different devices could be useful.

How do we help manufacturers figure out where to start? There are different methodologies and documents that we make available.

The first step in adopting AR and IoT isn’t generating more data, but [figuring out] how we can better leverage what we have in the factory. We want to help operators make sense of the information being thrown at them.

Business objectives haven’t changed in factories — more efficiency, controlling labor costs — so there is some consistency in the use cases that customers have in mind for AR. They tell us that they’re interested in the following things:

1. AR-guided work instruction, for assembly, end-of-line inspection, and training.
2. Setup and changeover instructions, in mixed-product assembly lines, such as food and consumer goods versus automotive. Changeovers are highly sensitive and can affect lead time and product quality. AR can definitely help.
3. Remote expert guidance, for helping someone in real time. Someone else could look at a camera feed remotely and annotate in AR — the annotations stick with real objects. That’s often one of the first use cases.

We’re bringing AR to the industrial world, combining it with 3D modeling and digital twins. Creating 3D animation of step-by-step instructions is very rapid with AR, which includes recognizing physical objects and annotation with the right scale and orientation.

A user can send instructions, “print” to AR, and then have the right scale of AR on an object.

With PTC, you have expertise combined in one company. In one AR experience, you not only see the device in front of you; you also see 3D instructions that rapid to create, plus real-time feedback. You know the impact of manipulating a machine.

Q: How important are audio and 3D to the AR/VR experience? Is there any thought to haptic feedback, particularly for training?

Schuetz: For AR experiences, the audio has changed a lot. People want to use voice recognition more and more. [Microsoft’s] Cortana is evolving, but factories are noisy environments.

Interacting with voice or hearing some commands is great. In the VR space, it’s an additional layer of reality if you have it next to the visual.

Haptics doesn’t play a big role in AR. Overlaying reality doesn’t have to haptics. We’ve spent a lot of time in the past few years in computer vision. Earlier this year, PTC released model-tracking technology that builds a 3D CAD model.

When looking at an object, the camera tries to recognize 3D geometry. This is more common in the manufacturing space, where you move around machines.

Another challenge that companies have struggled with is the content itself. Creating content took a long time because 3D CAD models were built for high-end devices that were too big to run on the floor.

Bringing data and AI together is a big selling point for our Vuforia Studio for AR design. There are still environments where you don’t have the 3D content. You may have a connected machine with IoT, but it didn’t come with CAD data. We’re working on in-situ recording of repair and changeover procedures.

The technology can record how you do a job — including voice, location, what you’re doing — and hand this over from an expert to someone who’s doing it for the first time.

Ultimately, we don’t want to create work. We just want to record experts doing their jobs, which is helpful for others without spending extra time. If we can give tech that records how to do certain processes and operations, that can deliver big value and expertise.

Q: Where does robotics fit in? Can you give some examples?

Provencher: When it comes to maintenance, we see customers combining data from a lot of sources — physical devices combined with AR, instructions, and feedback from the controller.

For example, take a medical device that includes AR and confirmation from the equipment itself that you performed an operation.

Also, it’s not just maintaining technology, but also operators on the assembly line. They have to deal with a lot of equipment and an HMI [human-machine interface] screen. They use AR as a way to unify and show the data to the operator on one screen.

AR summarizes and simplifies data from different sources. PTC Vuforia can combine data from ThingWorx, ERP [enterprise resource planning] systems, and machines and create role-based apps.

Vuforia can transform some of the role-based information for AR experiences. We do it at the ThingWorx/IoT level.

One research project we’re working on includes a plant manager and maintenance technicians at a refinery. We can set up a persona based on the specific equipment they’re looking at and identify the right assets, based on their role.

Q: Can you describe how digital twins are useful to manufacturing and other industries?

Provencher: We’re working on creating a digital twin of an entire factory so you don’t have to model it in 3D. You just scan it and create a representation in space. Then we know what you’re looking at based on where you are in reality and in terms of the assets around you.

We can serve the right AR experiences on demand by leveraging digital twin of entire factory and individual location information.

Users could see faults or problems where red flags raised by maintenance. For instance, in a refinery, people could look at equipment and assess whether it’s properly running.

Modeling and scanning also has uses in factories with small-batch production. Maintenance crew and operators need detailed work instructions.

In a refinery, you can have a virtual control room and continuous processes. By adding access to KPIs [key performance indicators] and to the systems controlling devices, AR gives workers the opportunity to have them in one clear interface. For example, it can show the temperature of a specific pump or valve.

Schuetz: We see a couple of examples right now with identifying products by VINs [vehicle identification numbers]. We use a camera feed to identify components and deliver that data to an IoT system.

Q: What can end users expect in the AR/VR space in the near future? As one of PTC’s five areas of developing digital-physical convergence, how much growth do you expect?

Schuetz: From a technology perspective, we’re moving from products to spaces, taking us into an entirely new area of AR. In the past couple of years, we’ve been tracking a single product, which is very product-centric.

We’re moving to smart connected products and operations. It’s not just how to fix an individual project but also to how to run an entire factory.

Nonetheless, we’re also making improvements to projects with six to eight different steps. What you and the camera see changes every time; these are multimodal targets. Tracking has gotten better while the product is changing.

We’re also trying to provide more out-of-the-box solutions. We see customers building AR experiences. We want to make it more difficult to build bad experiences by adding components including the user experience in AR.

People have had good experiences with 2D on screens for years. With 3D, we need to bake it into our products as use-case-specific solutions.

As for industries beyond manufacturing, we have a free trial where you can sign up and test our stuff for a month. Then we ask people, “What did you do? What was your timeframe?”

So far, 12,000 people have participated. They gave us good data about actual use cases. They’re fairly well-distributed across the value chain. It starts with a digital design review, then goes to manufacturing, then product visualization, demo in marketing, then service.

One of the biggest areas for AR and IoT information is that there are 30 pages of boilerplate in service manuals now. If it’s a connected machine, you could know that it’s hot. If you have tablet, it can show where it’s hot.

There are lots of areas to improve in services, parts identification. People often order the wrong part and then need to ship it back.

Training is another big area for AR. With more product configurations and complexity, companies ship more iterations of products.

You either need to take people out of the regular business to train them, or you typically have to give configuration-specific training in operations and logistics. We don’t take people out of business operations.

Provencher: What PTC’s customers are really looking for is speed and scale for business value. There are new ways of trying software before you buy it — the entire Vuforia platform is available.

We also analyze and look at what customers are doing with AR, and we share with them what others are doing.

Our most recent whitepaper on the state of industrial innovation provides a snapshot of how customers are generating value with AR.

We’re bringing digitization to manufacturing, including 3D, IoT, and AR. We’re probably one of the few vendors that focuses on AR, not just from a technology point of view, but also from a business one.

At the end of the day, it’s the ability to create content and deliver it at speed, at scale.