Inside Sony’s Magic Lab, PlayStation R&D Gets Weird (Videos)

To get to Sony’s Magic Lab, you take an elevator to the fifth floor of a building at its San Mateo, Calif., campus. You step out of the elevators onto an ordinary office floor — and that’s a problem, because Magic Lab director Richard Marks wants you to feel like you’re about to step on lava.

Once he figures out what kind of projector to use, Marks said the floor would become a 3-D optical illusion that looks like something out of a PlayStation game.

“A lava floor was an easy-to-understand example,” he said later. “We really want to use content from our games studios, so it feels like you are walking into one of those worlds as you step out of the elevator.”

Here on the fifth floor, that’s a totally normal endeavor.

The four-man team at the Magic Lab has helped develop some products already found in homes, including the PlayStation 4’s camera and the PlayStation Move motion-tracking controllers. Marks and his skunkworks team’s main task, though, is figuring out what consumers don’t yet know they want.

In the past year, Sony has sold 13.5 million units of its latest gaming console, the PlayStation 4, ahead of both Microsoft’s Xbox One and Nintendo’s Wii U. Even though it is currently winning the console war, the company knows that gamers are easily swayed — it won big with the PlayStation 2, but then fell behind Microsoft and Nintendo when it fumbled the PlayStation 3.

To understand the Magic Lab’s projects, then, is to understand various potential futures for gaming. It’s an attempt to create disruptive innovation from within a behemoth corporation, with a tolerance for the weird in the name of entertainment.

Easily the most high-profile experiment in the Lab right now is Project Morpheus, a virtual reality headset that connects to the PlayStation 4 console and is poised to compete with Facebook’s Oculus Rift, though neither has yet announced a release date. Marks demoed a VR experience developed with NASA, using the Morpheus and two PlayStation Move controllers to manipulate virtual models of two Athlete rovers:

Another Magic Lab project that has attracted the attention of Sony’s game development circles is Eric Larsen’s work on eye tracking. Using an eye-tracking camera made by SensoMotoric Instruments, Larsen showed off multiple ways in which a game might be affected by a players’ gaze, ranging from using one’s eyes to aim in a shooter game to making a virtual character uncomfortable by looking at them too intently.

“If you’re in a combat situation, your eyes sort of give away what your next move is,” Larsen said. “And that’s the kind of thing that can let characters predict your attention, your next move. The [artificial intelligence] can see you’re looking at a weapon and know you’re thinking about going for it.”

The problem with eye tracking at the moment, other than the fact that Sony is currently dependent on outside hardware, is distance — users have to be sitting within a couple feet of the tracker, which is a better fit for gaming at a desk than on the couch.

Other projects in earlier stages of research include work on how to pair haptics — that is, technology that simulates physical feedback from familiar real-world objects — with the PlayStation Move controllers. This could be especially important for Project Morpheus, as virtual reality doesn’t yet have a clear “winning” method of input.

Osman said he’s also intrigued by the integration of PlayStation tech with robotics. He piloted a four-rotor toy helicopter around the office with a PlayStation 4 controller.

“Let me see if I can get it to roll on the wall without hitting the sprinkler and making a bad day for everybody,” he said.

Marks likened Magic Lab’s research into consumer robotics to out an already-in-the-wild videogame called Johann Sebastian Joust. Developer Douglas Wilson wanted to make people interact with each other rather than the TV screen, and built a party game around the sensors inside PlayStation Move and Dualshock controllers; players dance around one another and try and force each other to jostle their controllers. They move too much, and they’re out.

Robots like the one Osman is tinkering with might be able to fly around the room, Marks said, as players use controllers to bat them back and forth. To avoid property damage, a robot could report its location back to the PlayStation 4, where all the game processing is happening.

“You could create invisible walls so it wouldn’t slam into your actual walls, because it knows where it is,” Marks said.

At the end of the day, though, Magic Lab’s goal is to find something that excites the people outside of the fifth floor, an idea that Sony and its affiliated content creators can throw their full weight behind as it is starting to do with Project Morpheus. And that’s much easier said than done, Marks said, because technological progress has a trajectory, while the public’s willingness to accept and use new technology is more abrupt — like flicking a light switch.

“Hewlett-Packard’s favorite thing was, ‘Until it fits in your pocket, it doesn’t matter,’” Marks recalls. “A calculator, in their case. Not a phone. That’s kind of why we exist. Technology goes (moving his arm smoothly) like this, but we want to see when the consumer experience changes.”

This article originally appeared on Recode.net.