Getting Miners Used to Gigantic Tires in VR, with Sheridan College’s Nick Ullrich

XR for Business

Surface miners, like the ones safety instructor Nick Ullrich teaches, get to play with some pretty big toys, like loading trucks with tires three times taller than the average person. But tires that big come with some pretty big blind spots, and Nick is using VR to get them used to those blind spots before putting them behind those gigantic wheels.

Alan: Welcome to the XR for Business Podcast, with your host, Alan Smithson. We all know safety comes first, and today, we're speaking with Nick Ullrich, a safety instructor from Gillette College, who's focused on using VR to train mining employees to become safer and better operators. Coming up next on the XR for Business podcast.

Nick, welcome to the show.

Nick: Yes, Alan, thank you very much for having me. I appreciate it.

Alan: Oh, it's my absolute pleasure. I read an article about how you're using VR to train mining employees. How did you get into this? Tell us a little bit about your background.

Nick: Ok. Yeah, so the first project that we've done is a blind spot recognition, using virtual reality 360 photography as well. And it's to help miners -- when they're on the mine side -- understand the blind spots of heavy equipment. So we started with that because MSHA -- the Mine Safety and Health Administration -- has an initiative out there about powered haulage. And that's kind of where the idea came from. We see fatalities every year in the mining world. So I wanted to give everybody an understanding of what the blind spots for the heavy equipment that they're working around are on, and give them an idea of that. And virtual reality gave us an opportunity to do that without actually having to have all the equipment here at the college. So it works out great to give them a vision of what they would see if they were in that equipment.

Alan: That's pretty cool. So you're talking about those big, huge dump trucks with the giant wheels and loaders and all these type of things?

Nick: Yeah, absolutely. We have-- in the program that we have now, we have 10 pieces of equipment, it includes the 400 ton haul trucks, which is the largest haul trucks in the world. Those tires are approximately 15 feet high.

Alan: Whoa.

Nick: Yeah, next to them, you will go about to the middle of the tire.

Alan: That's incredible. So you've got these trucks. Now, did you create them as 3D models and then climb inside of it? Or is this taken from like a 360 video type of thing?

Nick: So we do a couple different things with it. We do have 3D models of all the pieces of equipment. So like I said, we have about ten pieces of equipment right now, and we just have 3D models of those, where people can walk around them virtually and see how big they are, put them to actual size or as close as I could get to actual size, by my recollection of them. They can get into them -- for the most part -- and just kind of see it in a virtual spot. And then we did go out to all the different mine sites and take 360 photography of each of those pieces of equipment. And we did it a really cool way. We set up a scene, so we had a whole lot of different things around the piece of equipment. So let's just say a haul trip, we have several different people and smaller vehicles, like light duty vehicles, such as just your normal pickup or van. We had those all set up in a special way, where you couldn't see them from the cab, so they were *in* the blind spots of that equipment. We took that 360 photo from the cab of that piece of equipment, so we can show everybody what the cab looks like and what they could see outside of it, knowing that they couldn't see any of the things in the blind spots. They were able to look around the cab, they were able to see out the windows, they could even use the mirrors. Then what we did is moved outside and we took several different 360 photos from the piece of equipment. And every 360 photo that we took, there's more of the things that were around you at the time. We would jump them from one spot to the next. And with every spot that they jumped, then more of the full scene became visible. We're talking about 10 to 15 pieces of either other equipment or people that were all around them the entire time. So they were able to see that 360 photo eyes and then get back in the cab and then really realize that, "Man, we really could not see any of that stuff that was around us at the time."

Alan: That's pretty awesome.

Nick: The virtual reality part was great, because it gives them an opportunity to look around. We're mainly focused on the new miners that are just going through the initial class. MSHA requires that people do a 24 hour training before they go out to a mine site. That's what we teach here at the college. We focus on those people that have never been around that equipment. So having the virtual model, and then being able to see how big it is and actually stand next to the tire -- that I was telling you about -- that got their attention. And then the 360 photo allowed us to show them what the actual piece of equipment looked like, that they'll be working around. It worked out really good to do both things.

Alan: Almost like a hands-on approach. You get to check it out, be there, experience it before you even get on. I can imagine this would also be good for recruiting people as well, because it's pretty exciting to stand next to a 15 foot tire.

Nick: We should use it as a recruiting model. That would be great. [laughs] But yeah, it's mainly focused on giving them an opportunity to see that stuff before they get out there, because people don't-- if you've never seen it and you've never been around it, you don't really get the full capacity of it, if I'm telling you about it. But if I can put you next to that tire -- even in a virtual space -- you get the understanding. Everybody asks, "Is this really how big it is?" And you get to say, "Yeah, this is-- you're standing next to the actual truck. It's just not in front of you at this point in time." So it really does get their attention. So that part, just to capture the sheer size of it and what they're getting into.

Alan: That's pretty awesome. I actually want to try this now. I want to stand next to one of these. I want to drive one of these monster truck things.

Nick: I'll get into that a little bit later, probably. But our next project is going to be a mine tour, where we'll have them fully immersed in a whole mine situation, so lots of stuff to come with that.

Alan: How are you combining the kind of 3D models with the photos?

Nick: The 3D models, when you first start the virtual reality part of it, you're in a warehouse. And then we have all of the 10 pieces of equipment at full size, so you can imagine how big the warehouse looks. And they can choose any of them that they want to go to first or whatever. The full program, they have to visit all 10 of them and go through the whole thing. But they start off in that warehouse. Video plays from MSHA on a big screen that drops down from the ceiling. It's pretty awesome, it gets their attention right away. And it just talks about blind spots. It was put out-- it's a video that's just put out by MSHA, to show that. But then they simply teleport to that piece of equipment and then they can walk around it. They can teleport, we have spots where they can teleport to specifically, like the cab, and some different areas around it, next to the tyre. And they get to take that in. And then we have a spot where they go to the-- we'll take them into the 360 photos. And then they look at a certain spot every time. And then they can move to the next photo, and move all the way around. There's only a specific direction that they can go. They have to see it a certain way. And then there's probably for each one between six and ten different photos of the piece of equipment as they're moving around it. So then they get to move around that. Then when they go back to the cab, once they exit out, they go back to that warehouse where all the equipment is at, and then they can choose their next one and go through that. Once they get done with that blind spot, then they virtually go to a classroom where we have a lot of review information, a lot of photos, a lot of industry specific safety posters hanging on the wall that they can watch, there's another video. And then they can play with the models, we have it so they can pick them up and hold them and see them in a smaller size, so they can actually see what that looks like. And that always gets them, because they're picking up the piece of equipment, moving it around in a virtual world. So that's pretty awesome. Then finally -- especially MSHA-related stuff -- you got to have a test. You start off in a-- it's in another room, and you have a haul truck in front of you, a full sized haul truck. And you have a lot of pickups parked around you in the blind spots of that haul truck. And here's the great thing about using virtual reality is, I told the programmer that I wanted to become a giant, and be able to move those pickups out of the blind spots of the haul truck. And they said, "Yeah, I think we can do that." So you go above the whole scene and you're looking down through a glass ceiling at the whole scene below you. And you have a model of it sitting on a podium next to you and you pick up and move those little pick up models, and it moves the big pick up models down below you. So you move those outside of the blind spots. If you have them in the blind spots, we have it where it will register, where you don't pass the test. So you have to put them outside the blind spots and then you can go back down and see your scene, get in the haul truck and make sure you can see all the pickups before you submit. Make sure they understand -- at least in the haul truck -- where those blind spots are added, and where they can place those pickups to get out of them.

Alan: So people, when they're operating even smaller equipment, they know not to park or even drive in those blind spots.

Nick: Alan, that's exactly what we use it for. We train a lot of contractors, so a lot of the people that we train are not necessarily going to be driving those haul trucks, but they're going to be interacting with them. And it's important that they know what they can't see. They might not ever get in one, but at least now when they go out there, they understand if they were in one, what they can and can't see.

Alan: I would assume this is going to save lives as well.

Nick: That's my intention, yes. And I have no doubt that it will. You can talk about KPI and what you can actually gain from it. That's a tough one to know for sure, if what you did saved them. But I have no doubt in my mind that it is going to make the mine site a safer place and prevent injuries, yes.

Alan: It's saving lives, which in turn makes everybody work a little bit smarter. And I think the God view of being able to stand back and look at it and move things around as if you were in that huge giant view, I think that's a really smart idea of doing it. And the fact you've combined CG graphics with 360 photos of the real site is really an impressive part of that. How long did it take to put this together?

Nick: So the project itself, I'll tell you, I first saw virtual reality about a year and a half ago, it was October of 2018. And that was the first time I ever tried goggles on. NIOSH -- the National Institute of [Occupational] Safety and Health -- they had a setup at one of their conventions that I went to, and they were using it in an underground mining situation. We're surface mining out here, so once I saw that, I said, "Man, we could use this for surface mining." The initiative from MSHA about powered haulage, and we see fatalities from blindspots every year, people getting into them and getting ran over. So that was the start of the project, coming up with how we were going to do that. Did some research then came up with an idea for it. We got a programmer fresh out of school, he just graduated. So we got him for one summer. So we literally had June, July, and August. And me and him worked together every day, pretty much, for that entire time, coming up with it, and putting it all together, and taking the photos. And we crammed a lot into that three months. [chuckles] It took us about three months to get it done. And we can always add to it. At this point time, we have the 10 main things that we use around this area. So it's a done project. Yes, you could always build on it more, but we're moving on now a little bit, getting into the next project. From beginning to end, it took me about eight months.

Alan: Yeah, and a lot of it was just trying to organize site visits, and making sure that you had all the equipment ready.

Nick: Yes, that was a huge part. We were lucky that we had mines that would work with us really well.

Alan: The paperwork usually takes longer than the actual production of the thing. So did you guys build this in Unreal or Unity or...?

Nick: Yes, we used Unity.

Alan: And what type of headsets? Are you using the Vives, or are you using...?

Nick: We're using the Vive headsets in our department. We're currently working on getting a full classroom setup, so we can utilize it for everyone. We have classes that get up to 24 people. So we're working right now to get that setup, where we can have 24 of them going at one time and be able to utilize it in our class that way. Right now it's a little clunky getting them through it all and only having a couple different headsets. We do hook it up to the projector as well, so the whole class can see what one person is looking at, but it's not the same. You need to experience it in the goggles. So we're working hard to get that. Hopefully that's going to be here by the summer, and we'll have that full classroom immersion and everybody will be able to put them on at one time and go through them.

Alan: You built this for your college. Is this something that you have a mandate to also provide to other mining companies, or other schools, or anything like that?

Nick: So that's a great question. I'm glad you brought it up. I do work at Gillette College. We are kind of a little bit different in how we do it. We have a grant -- an MSHA-provided grant -- that we operate under and it's just through the college that we do it. It's a weird partnership and I don't know exactly all the rules of it. But with that being said, yes, one hundred percent, we made this with grant money. So if anyone wants this product, I will give it to you. You can use it in your training. I will send you a jump drive. It's only about three megabytes once you put it on the jump drive. So I can send you the entire program and people can use it, view it, whatever they want to do with it. It's provided for free.

Alan: Did you say three megabytes or three gigabytes?

Nick: It fits on a small jump drive, so...

Alan: Yeah, it's got to be gigabytes. Well, I want to try it. I want to go and sit in one of these haulers. You've got to send me one.

Nick: Absolutely.

Alan: [laughs]

Nick: That's truly our goal, is that we save lives.

Alan: I'll tell you what, if you send me a drive, I will actually go and share it with some friends of ours at a mining company here in Toronto. Because Toronto's a kind of a hub for the mining head offices.

Nick: Absolutely. I'll send you one.

Alan: They'll be great. Well, Nick, this is really incredible. I love how you guys pulled this really holistic system, and I can't wait to give it a try. And is there anything else you want to share with everybody before we wrap it up?

Nick: Yeah, just virtual reality. And I know you guys -- and most of your audience -- is going to know, but I see it as truly the future. Now, I'm focused on safety and safety training, but this is the future of safety training. When you can put people in a certain environment and show them that, without putting them in harm's way, I know it's revolutionary and I know that-- there isn't a word that I could use to explain how I feel about it. But I've only been doing it for about a year and a half. And I know that this next project -- which is a mine tour -- I'm going to put people in situations that they could come across in mining, and I might be able to have a high wall falling on them, and some kind of incident or accident that could happen at a mine site or that we see all the time. You can always talk about those, you can tell them in their class, you can show them pictures of the times when it has happened. But I'm a big believer in the virtual reality side of it, and I can make it happen to you. When you go out to the mine site and that picture brings up a feeling, that's a training that people will not forget. If they have a high wall fall on them in a virtual world, yes, they get to walk away from it. They have no lasting injuries. You can have them fall, not wearing their fall protection. You can do all kinds of things for this safety side of it. And when they get out to the actual job site and they see it, and that triggers the training that you gave them, they'll look at it differently. So I know your audiences, it knows all about virtual reality and they know it can be done to do amazing things. But the safety side of it is where I'm focused. And I can see so much benefit from giving them an idea of what they are seeing before they get out there. That's the type of training that I want to give, and that's what virtual reality has allowed.

Alan: So basically, what you're saying is you can study it all you want, but virtual reality gives you the feeling of being there.

Nick: Absolutely. Mine safety, it's a huge deal. And mining is going to be here, it has been here forever, it's going to be here forever. And people don't understand what they're getting into when they go out there. If I can show them what they are getting into and and what they're doing -- without putting them there -- before they even get there, it will literally save lives. And I don't think there's anything more important than that.

Alan: Thank you so much for sharing this. And if anybody wants to get in touch with you, how can they they reach you?

Nick: Email is [e-mail]. I wish I had a website or something fancy that I could give out, but I don't.

Alan: Well, I've got the sheridan.edu website, so they can follow up there, and take a look at the courses that are offered through Sheridan and Gillette College. So Nick, thank you again so much for taking the time to share this, and also the generosity of sharing this experience with our listeners and everybody to keep the world safer.

Nick: Absolutely. Thank you. I appreciate it.