"I Consider Myself Crucial to the Success of that Company": Jack McCauley Recalls the Early Days of Oculus and Looks to the Future

This article first appeared on USgamer, a partner publication of VG247. Some content, such as this article, has been migrated to VG247 for posterity after USgamer's closure - but it has not been edited or further vetted by the VG247 team.

Sony's Advantages

USG: What's your take on the PlayStation VR?

JM: Ok, here's my take on Sony. Let's just talk about GoPro for a second. I mouth off about GoPro when I talk, because that's a shitty company. And the reason they're a shitty company is, I'm an investor in GoPro. Pretty early on I lost my shirt, and everything that I know about the business I ignored when I invested in GoPro. I got caught up in GoPro's hype, and I did a start up company based on GoPro selling accessories for the camera. I had a very cool product for the camera - it was a remote control that you could wear on your key ring or you could put it on your wrist or wherever you want and start and stop the camera. It was a radio. We started looking at, and when you do something like this with a hardware company, any company, the first thing you do is you lay out a spreadsheet with your cash flow on it. How are you going to manage the business and the cash flow? Who are you going to find as your partner to build it who will work with you with terms, which means net ninety, which means ninety days after they ship it they get paid, if they can trust you? You've got to work that whole thing out with the cash flow. We did that with my little start-up company with the GoPro remote and couldn't see the positives on it. So we canned the business.

Apparently no one did that at GoPro. They should have. Because, they produce a camera and like HTC it's just a piece of hardware they make margin on. Sure, they sell accessories, they sell little mounts and things, which cost pennies to make, incidentally, and they sell them for five bucks. They make pretty good profit on that. But to grow that kind of a business you're going to need re-occurring revenue on software. It's just the way the world is now. So, on GoPro, I could see it starting to slide. What I said to myself is, the first thing, when I got their camera, the Hero 2... I said the first thing they're going to do is they're going to put a cellular radio in this thing so I can livestream full HD video from my sport experience onto a network, so people can look at it live. That's what I said to myself, that's a natural. Then they can sell the cameras and get money on reoccurring use of the cellular network. But they didn't do that. I was going, why didn't they do that? Well, we're going to use Wi-Fi. Well, you have to be inside of a building, or you have to be near a Wi-Fi. And so, there's Wi-Fi on ski slopes and stuff like that, but, a lot of it didn't make sense to me. And I was wondering why they were doing that, and I've just watched the company. Their stock was at $89 or something six months ago, it's at $12 today. I just watched them, and I'm watching them, it's like watching an explosion of how to manage a business. Furthermore, when you buy the camera and you shoot with it, the image quality of the camera is not as good as GoPro's videos. I go online and GoPro's videos look like they're professionally shot to me, and I said, I want to reproduce that, I want to be able to take it out there. Well, it's misleading to consumers to sell something that based on professionally edited videos that they think they can get out of a product, and they don't get it.

So let's go back to Sony. Sony was also in the sports camera business, but I watched Sony very carefully. I said, “They're not really really doing it.” They've kind of got their foot in the water, they've kind of made a product, they're kind of seeing how it goes. I go, “Why not?” Well, because Sony's smart. They're in the camera business, they know that business really well. They know sales cycles and marketing and all those things, and, they got into it very very gently. With VR, that's not been the case. They're diving in. And the reason they are, my feeling is, is that they've got 36 million PlayStation 4s. They have the infrastructure to download content and play it on the PlayStation 4, and they have those pieces in place already, all done. Now they have this VR headset. Granted, it's probably not as good as the Valve one. It's not a walkaround experience, it's kind of a seated experience. They've got their Move controllers, which... they're OK. But they have all those pieces in place. So, who's the dark horse here? Well, Sony. They've got a very large install base. If they get just 10 percent attach rate, which means just 10 percent of the PlayStation 4 consoles buy Sony VR, that's 3.6 million units to write software around that will run on a PlayStation 4 without a doubt. They will run. That's phenomenal. That's a really big win.

Of course, people say, "Well, it's not going to be as good as the high end PC or the gaming rig," but it's going to be a VR experience, and Sony will eventually figure that out. So, that's my call. Sony had a very good driving game that came out on the PS2 that blew people away. The graphics were incredible for a very low-powered console, and people were wondering how they did it. Well, they re-wrote all the middleware, and they had a really hot programmer who was working on the game for them. So, in answer to your question, it's a matter of time before Sony takes a big chunk of business away, and Sony has, I don't know what launch titles they have planned. They've been a little bit quiet about that. Sony will have a launch title for it, and Sony will have these other pieces in place, and there are only 17 million gaming rigs that can run Oculus at full performance in the world. So, 36 million versus 17 million. The 17 million are split between two companies. Half of the company is Valve, and half the company is Oculus.

The natural thing to say is, why doesn't Oculus go over to Microsoft and say, “Hey, let's sell the Oculus headset for Xbox One?” I would do that, wouldn't you? Makes sense. Well, you know who, Microsoft was one of Facebook's early investors, Microsoft put 270 million dollars into Facebook. And, that paid off pretty well. That's a possibility. I don't pretend to have inside knowledge of that, but that seems like a natural place for Oculus to be. But they've only sold 16 million Xbox Ones. Not very big. And ten percent attach rate, which is probably pretty close to what you'll get, is 1.6 million units. But maybe split between the PCs and the Xbox One they'd have enough to have something there.

What VR Needs to Reach Its Full Potential

USgamer: So how's work on the new VR tracking system going?

Jack McCauley: It's going really well, actually. I'm a very enthusiastic person generally, but I'm very easily distracted. I have a tendency to work on something and then switch to another thing when I start getting frustrated or bored with the thing I was previously working on, and then with all intents of finishing the new thing, switch back to the old thing, and I go back and forth. But, I eventually finish all of them, it's just that I don't do them in a style that – if I start rushing on something just to get it done I won't be happy with the results, so, I'll stop and put it on a shelf and then come back to it later, when I get the energy for it. So, to answer your question, it's actually going pretty well. I can give you some background on that so you understand where it comes from, if that helps.

USG: My understanding is that you started developing it in response to issues with motion sickness.

JM: Exactly. The problem was, with the developer kits. We attempted to fix the vestibular issues, which is the inner ear issues. The greatest source of information on motion sickness and vestibular issues comes from the military research, and all the indications are that it has to do with the subtleties of the inner ear and micro-adjustments to posture and head position to keep your head pointed in the same direction as your ears are, if you can understand that. The ocular system and the vestibular system are located right next to each other in the brain for a reason. The neural pathways are very short, and the reaction time is very short. If you miscue that system by providing false cues to either one, it can cause some pretty nasty side effects. There's a reflex called the vestibular ocular reflex that can be miscued and mistriggered, which causes vertigo, which is really bad. Vertigo can be dangerous, from falling. So, my work there was to try to eliminate that miscues and so I started doing research on that. I got a lot of stuff from the military, read all kinds of papers, and it turns out that the greatest source of those kinds of things are, the miscues are real subtle things.

I don't know if you've ever experienced this: you get onto a train or mass transit, and you're inside of a subway station, and the train starts to move, and just briefly, sometimes you get a little pinch of nauseas or pinch of dizziness. And that's because your eyes, looking out a window, or looking out the other side of the subway, don't see the train moving the same way your ears feel it. Your ears are much more sensitive to those kinds of changes, and your body, the balance system is cued into that to keep you balanced and standing upright. So, I originally started working on that back in 2014. Before that, we were using a camera at Oculus to do this sort of positional tracking, and the main purpose of that was just to be able to move around in the game. We were trying to address some of this motion sickness stuff, but it was really just to get you to be able to move around in VR. And I started looking into that based on my own experiences, one of which was being on the Tokyo Tower, standing on the observation deck, and feeling nauseous, and going, "What's going on here, because, it's not moving, I don't feel it moving, but I feel like getting motion sickness." Well, what was happening was blowing in the wind, and it was swaying, but at a very very low frequency that was undetectable to your eyes, but your ears still felt it. And then this is where I started thinking, "Ah, this is what these researchers are talking about, the subtlety."

"My goal with this thing is to not make money, because I don't need money. My goal is to improve VR, and I think if I can take this one small piece of VR and fix that, it's a profound problem. It's a very difficult problem to fix."

So, I started looking at that, and I started looking at cameras and how you would fix that with a camera. Turns out, I don't believe you can fix that with a camera based tracking system, because there's not enough pixels to catch those subtle movements. Camera based systems, which are basically like the mo-cap systems, you see the professional mo-cap systems, and these systems like Oculus for positional tracking have a limited resolution, which is just the image sensor. The closer you are to the image sensor, the more subtleties you catch. The further away, the fewer subtleties you catch. And, I went through and tried to figure out what the subtleties were, what a number is. There's really no number for it. These are physiological things that are not well understood even today. Even experts in vertigo, researchers who get DARPA contracts and so forth, really haven't put their finger on what it is. It appears, though, that you can be trained away from vestibular issues with a certain amount of easing into the experience. And so, with some training and sort of leading people in and easing people in to the VR experience in particular, you can alleviate some of those things, and accurate positional training certainly helps, like the work that Oculus has done, and Valve. But I started to attack the real subtle motions, and I determined that a camera can't do it. Besides, I'm not a camera guy. I'll take that back. You can attack that with a camera, it just requires mechanical complexity, it's just some complexity. And trying to dial that in and get it working, I just don't think I could do that in five years. It would take me a long time.

USG: I mean, realistically speaking, it's still pretty early days for the tech in general. They're charging upwards of $800 dollars for technology that's pretty immature. Would you say that?

JM: I would say you're right on the money. You're exactly right, and, the thing about VR, you've tried it, I know. It's profound, it's going to change things for sure. How long that will take is the question. It could take five years to get a good depth camera working that has really sensitive positional tracking and get these fine microadjustments in your inner ear, could take five to ten years to get something. Now, Intel has a consumer product, RealSense, which has a depth camera, but the technology's not there yet. It uses basically like an image sensor; and incidentally, a lot of the work is done at Berkeley, at BSAC and their sensors group, but Liwei Lin is the guy that did a lot of work [Editor's note: McCauley is a Berkeley graduate]. But basically, what it does is steers a packet of photons at an image sensor and tries to map the depth of this thing by the return time of the photons. So, you fire the photons out with a laser, they bounce off the object, they bounce back and you do trip time. You measure it with an image sensor, and it's just not there yet. RealSense, I have it, it doesn't work very well, it's not that great. It's a very preliminary SDK. It might be there in five years, to the extent that it's inside of cameras. I mean, excuse me, inside of cell phones, they can just snap a 3D picture and look around, but it's just not around yet. And how good it will be as far as the motion sickness thing, I'm not sure it will be that good. I think it would have the same sort of limitations that a camera would have.

So, back to my tracking. Let me just tell you where I'm at with that. I think I'm onto something profound. And my sense here is that, and I don't have really good insight. I have a major entertainment company that you know about coming into my facility on Monday to see it. It's at a state now where I can pretty much pick up microadjustments and movements from about forty feet with this device. The videos that I show online are not the one I'm using. Those are the ones I can show. The one I have is much more precise and much more accurate than that, and this thing could be like the lighthouse for Vive. You stick it up on the wall like a speaker, it's kind of invisible, it doesn't look like much of anything, you don't notice it, but when you walk in a room, it can find you and track you very precisely and get those microadjustments, and eliminate a lot of vestibular artifacts. Plus, it's very quick. It's got very very high performance, and it's not expensive, it's a couple bucks. So, the goal of the design is... I know I can track a person in the room using military hardware, which is really expensive hardware. But I could never sell something like that, and it would never be a product. So, how can you do this as cheaply as possible, and as accurately as possible?

Basically, what I did was took what I did before. A long time ago I built something, and it worked very similarly to this, and I modified it so that it works in VR. Back then, I had that thing working very very precisely. But all the work I did was predicated on work done ten, fifteen years ago, that was actually a product that I sold. Now I took the core guts of that out and put it in this device, and got it working. So, I'm going to try to take this thing and start... what I'm going to do is show it to people to get feedback on it, to see what it is they're working on and how well this thing fits. If this thing that I've made doesn't fit what they're trying to do, then I'm going to do something else to try to make it work. My goal with this thing is to not make money, because I don't need money. My goal is to improve VR, and I think if I can take this one small piece of VR and fix that, it's a profound problem. It's a very difficult problem to fix. You hear people talk about it. I don't know if you have.

USG: It seems like the toughest nut to crack for video games, just because, when you're playing a first person shooter you can't move in-game, because if you move you're going to induce nausea. Well, that's a huge limitation, a game-breaking limitation, and unfortunately, just being able to teleport around, it's a novel answer, but it feels gimmicky still.

JM: Yeah, [the MEMS Tracking System] I'm building is cheap enough to put ten of these, twenty of these in a volume, like an arcade, and run around inside the arcade with other players and play against them, as an avatar. That's what I want to do. It's not practical to have... no one has a home that big, right. But you can have one or two of these to have a smaller experience. I think the limitation with Vive is the small play volume. What are you going to do with eight feet by eight feet?

USG: Are you much of a gamer?

JM: Not so much anymore, it takes too much time. I used to be. But, I'm more of a technical person, and executor, so people ask me to do stuff and I execute on it.

USG: The reason that I'm asking is one of my hypotheses for VR is that it's amazing for hobbyists who want to play cockpit games, like, driving games.

JM: Mech games.

USG: Flying games, mech games, anything that puts you in a cockpit of a thing is just great, but, I'm actually kind of doubtful that it has application before that, and if anything, gaming is actually going in the opposite direction, where the most popular games are really simple. The top selling game on Steam right now is called Stardew Valley. It's a 16-bit looking game that could have been made on the Super Nintendo. So, I was wondering if you had a perspective on that.

JM: Well, I don't think a first person shooter in the traditional form of a first person shooter where you move around with a game controller and you see the position is a good VR experience. I don't think a cockpit game is a good VR experience, because with a cockpit game, you don't feel, like a driving game going around a corner, you feel no Gs. You're sitting there, and you can look and you can see the games next to you, you can go around the corner and you can see your feet on the brake pedals, big deal. That's just kind of gimmicky. But the walkaround experience where you can go into the holodeck and explore a new environment is intriguing and is useful entertainment. I want to go to a different planet with different creatures and walk around and look at them and interact with them. That's what I want to do. And it's not going to be a tethered device, it's going to be a mobile device with the computer inside of that, inside the headmount. It's going to be a standalone device. That's five, six years out, who knows. But, that's what I want out of it. I don't care about cockpit games. They're boring. First person shooters, they're all kind of the same stuff over and over again. So, is there application in entertainment? That is the application. Is it a game like a first person shooter? Probably not. It's going to be probably an exploratory experience or a novel experience, where you can actually – What are you going to do, you're going to sit down and interact with people on Facebook in 3D? What's the use of that? I could do that on my phone.

USG: My other perspective has been that, this has become kind of a huge hype engine in that you have these really big companies like Facebook and Sony jumping in and just all kinds of money flying around, and in that alone it's become hugely newsworthy. Is this just the race – It doesn't even really matter how viable VR really is, it's the case of large companies saying, “We don't want to be left out in the cold if VR truly is the next big thing. We've got to throw everything we've got at this.”

JM: Do I think it's wise to do that? Well, OK, I'll just quote someone that I know, and this doesn't come from me, it comes from someone else who owns a giant company. His dad was the founder. He's now the chief of the company and its board of directors, said to me, “I'm on board with VR. I think it's profound.” When I hear people like him say that, it's not coming from me, and I kind of feel that way, then it is. What form will it take? If Oculus or Vive or Sony has a bad rollout on the consumer version, it's not going to stop it. They're just going to have a bad rollout, and they're going to have to sit down and reformulate and try again until they run out of money. And I don't think that Facebook is going to run out of money, and I don't think that Valve is going to run out of money. So if the initial experience is not good, the other experiences that come out of it will be better and better. Remember how bad cellular phones were when they first came out? I mean, flip phones, they sucked. Actually, they didn't suck, they were good phones, but they didn't have much you could do with them besides make phone calls. And look at where they are now - you're getting rid of your desktop because you have a phone.

USG: Aside from motion sickness, I think that in terms of gaming applications I find that the lack of weight is a huge problem for VR. Like, you were mentioning, you don't feel the Gs when you're driving a car. Well, similar to a first person shooter, you don't feel the weight of the gun. So, it's like a toy. It didn't work for me, unfortunately.

JM: Yeah.

USG: Is that solvable?

JM: It would have to have gravity on it, it would have to be a weight. I guess you could make an accessory for a video game you could see in your hands and shoot.

USG: Yeah, exactly, or force feedback. The way games now do it is they use a combination of force feedback, sound effects, visual cues, and that kind of thing to kind of channel a sense of weight, but it seems like a tough nut to crack with VR itself. But maybe an accessory is the way that you have to kind of do it.

JM: Yeah, it'd have to be an accessory. I don't see that as much of a problem as getting the content part, having the head and the body feel right. If you're running, I want to feel like I'm running. It may be that there are these huge open spaces or huge indoor spaces that are all fixture for playing VR games in, walk around VR games with other players. I can see that.

USG: That's entirely possible. Well, it sounds interestingly enough that we're on the same page when it comes to VR, because I'm actually pretty doubtful about its broader gaming implications, but I do think it has some pretty amazing commercial implications.

JM: Yes, I think people are aware of the same things you're aware of. I think we discussed those things early on, how well certain paradigms in programming or styles fit, it doesn't fit a lot of things. But it does fit the holodeck, that's what it's going to be. You're going to be transported to another world. It may be an imaginary world or part of the real world, who knows, where you can walk around in it. I think that would be great.

USG: Yeah, I think that would be really great. Star Trek for real.

JM: Yeah, there you go.