NASA uses AR to train astronauts

Augmented reality shows users a view of the real world supplemented with sound, music, graphics, or data. Now the same technology that brings Pokémon GO to life is helping scientists prepare for space exploration. Here to explain is Matthew Clausen, creative director in the operations lab at NASA’s jet propulsion laboratory.


Augmented reality shows users a view of the real world supplemented with sound, video, graphics or data.

Now the same technology that brings Pokémon Go to life is helping scientists prepare for space exploration.

Here to explain is Matthew Clausen, creative director in the operations lab at NASA's Jet Propulsion Laboratory.

Matthew, thanks for joining us.

Thank you.

So, first of all, we know that this is the technology that puts Pokémon onto our desks and backyards.

But can you really explain -- what is augmented reality and how does HoloLens work?

With Pokémon, it's using the geographic data for GPS to give it a place a two-dimensional object in space or at least create the illusion.

But with HoloLens, it actually maps the environment in a three-dimensional model so that you can actually place three-dimensional objects onto the space as if they were really there.

So HoloLens looks like those virtual reality goggles that we have seen, right?

But it's spewing out a hologram in front of us?

Yeah, so HoloLens, it has a sort of like a visor that goes around your head, whereas like with the Oculus or with the Vive, the HoloLens actually has a translucent screen so you can see through into the world.

So if you and I were wearing it at the same time, we could still see each other but also have conversations about three-dimensional objects.

So that's another thing that actually separates it from two-dimensional augmented reality such as Pokémon, where you and I could both be on different sides of a three-dimensional object and have a perception of it being there as if it was actually real.

A real solid volumetric object.

So how is NASA Jet Propulsion Laboratory using HoloLens for Mars exploration and planning?

So for Mars exploration, we've reconstructed the Martian landscape using all the photogrammetry data, the photographs that the Curiosity rover takes while it's on the surface.

And we've combined that with orbital data.

We have two different orbiters that are rotating and orbiting around the planet.

And we create a mesh that is the best reconstruction that we can create.

We put that into a system which allows scientists using a HoloLens to walk around the surface virtually as if Mars was in their office.

You can still see your office, but you can also see Mars at the same time.

And so scientists that would be, say, in France or scientists here in JPL, they can have a conversation together as if they naturally would, say, if they were walking around in the desert and doing their geology work that they are used to doing.

And you've found that when scientists are using the HoloLens in order to get a feel for the Mars landscape and kind of measure things, that their spatial accuracy is better?

One of the fortunate things is that here in the ops lab we develop the older tools that the scientists have been using, and the way this works is we usually have sort of an unwrapped panorama of the environment, so if you just imagine -- think of it as sort of turn around, taking pictures in 360 degrees and then unwrap that into one sheet -- that's how the standard way of doing things was.

And so they are very used to that, and so what we did was we just devised one test where we had them estimate the distance between the objects using that tool and then create a prototype that used a Oculus Rift actually and have them estimate the distances and angles between objects in the exact same environment.

And what we determined was that scientists were two times better at being able to accurately detect distances using immersive technology and they are more than three times as good at estimating angles.

And another one of the applications is Project Sidekick, where you are helping scientists, astronauts who are already at the International Space Station.

Can you explain how that works?

Yeah, so Project Sidekick has basically one goal, which is to help astronauts do their jobs better and help them to have more information when they need it.

When an astronaut is preparing to go to the space station, they spend months and up to years actually learning all these complex procedures.

As you can imagine, it's a ton of work, and it's a ton of information to have to hold in one person's head.

So, we have all these long documented procedures that are basically text documents, and they go through these books, and they read them and they're yet to kind of remember how to do them.

And we thought, what if we could actually just superimpose the three-dimensional model of the procedure on the environmental space station to enable them to just be able to follow a step-by-step thing as if they could basically have their imagination projected into the real space?

And I can see potential consumer uses for this, too.

Is there any chance that any of these applications could make it to the consumer market down the line?

So the stuff we are doing, I think, has helped in the design of some aspects of what is going to be coming commercially.

JPL doesn't necessarily take our products and push them out to the public, but we like to open up the resources that we've created and the designs that we've made to other partners to help them to create those commercial products.

You can expect there will be a number of commercial products that have similar applications that will be coming up.

Really? And one more application with the HoloLens is ProtoSpace -- is that right?


Can you explain how that works?

One of the really important tasks that we do here at JPL is we design spacecraft and rovers specifically.

So our focus is usually on robotic exploration, rather than on human exploration.

So what that means is that we have a lot of engineers who are working to create these very, very complicated pieces of equipment and machinery.

And oftentimes we have teams that are in France, we have teams that are in other parts of the country, that are designing very specific subsystems that all have to come together in order for the spacecraft to work properly.

It's difficult sometimes to imagine how all that stuff's coming together using traditional Autocad -- or computer automated drafting programs.

We can pull the 3D models into the space, and spacecraft designers can actually walk around, put their arm in between things, and actually just recognize things as simple as, oh, there's not enough space to put a screwdriver in here.

And what might seem trivial to the construction worker, for example, where materials are cheap, not being able to put your hand between something that is actually the combination of very, very expensive pieces of equipment, you know, on the scale of millions of dollars can save the taxpayer a lot of money.

And so we have been able to put that to use on multiple missions so far, and it's been a huge success.

So it's sounding like this is really... The biggest thing is it's strengthening the accessibility and the communication.

And I've seen you say that this could even, down the line, make Mars exploration virtually accessible to people all over.

What do you envision?

And when might we expect that?

The wonderful thing that's happening right now is that these technologies are becoming ubiquitous -- or they should be soon.

You can go and you can buy your own Google Cardboard right now for fairly cheap.

The HoloLens is now available to consumers.

The Oculus and the Vive are all available.

So it's really just a matter of taking the stuff that we have done and sort of being able to repackage it and give it to the public.

And so we are hoping that in the future, as we go and explore other planets, as we do our Europa fly-by in March 2020 mission, which is another -- curiosity that we are going to be sending to Mars in 2020.

We are hoping that right there with us using the same sorts of tools and exploring the surface of Mars together so that everyone can experience it.

Great. Matthew Clausen from NASA's Jet Propulsion Lab, thanks very much for joining us.

Thank you. It's a pleasure.