Leap Motion focuses on SDK and embedded, 2nd-gen controller will not be sold separately

In December, Leap Motion has announced a new hardware, its 2nd generation version, with a companion mobile faceplate, that suited perfectly onto GearVR headsets. We were all super-excited about it, thanks to the amazing specifications: 180°x180° tracking, improved tracking distance (beyond arm length), improved tracking and so on. We were all waiting for the moment when this Leap v2 (notice: in this article I’ll use v2 to refer to the 2nd version of the Leap Motion hardware, not the Leap Motion v2 software released in 2014)  would come out to stick it onto our GearVR headset and live amazing mobile VR experiences with natural interactions. Well, guys, I can tell you that for sure this moment will never come. I mean, Leap Motion 2nd-gen is coming, but we won’t be able to use it that way. Follow me and I’ll explain everything…

How this post has born (in case you’re curious to know, otherwise skip this part)

This post comes thanks to my passion for technology: I’m a Leap Motion fan and since December I was waiting for info on the new device: improved tracking angle and distance means hands tracking in VR even when you’re looking at something else, means true hands presence. But since December, all big VR blogs never came out with news on Leap Motion. I wanted a Leap Motion second generation hardware release date. So, I got brave and said: “If they’ll not make a post about Leap Motion v2, I’ll made my own post about it with great news and internet memes…”

I remember the great quote of Steve Jobs about how he contacted HP, about people that don’t take the phone…

… so I just got brave and emailed Leap Motion asking for an interview. They were super-kind and agreed to talk with me.

So, at 7.30pm, after a long day of work, I was talking on Skype with an awesome guy called Alexander Colgan. He works at Leap Motion and is not only a very skilled techie, but also someone with which talking is very enjoyable. I was really excited, scared, tired… but he managed to make me feel comfortable. For me it has been an emotional experience like having a talk with Scarlett Johansson (apart from the fact that I don’t want to marry Alex) and my brain got crazy… I was so excited that instead of configuring OBS to record all the screen stream, I set it to record my webcam, so now I have 1 hour video stream of my bearded face. Damn.

But, apart from looking like an idiot, the interview has been reaaaaally interesting. Scarl…ehm Alex has been super-kind and informative and managed to give me great insights on Leap Motion present and future. I’ll split what he has said to me in two posts. Today I’ll tell you the big news, the ones that we were all craving, while in the next one I’ll post a series of interesting but less WOW information (like some techie stuff).

Have a nice reading, hope you’ll like it…

Leap Motion second generation hardware

Leap Motion second generation hardware is already out there. It’s finished. Leap v2 is like Leap Motion v1 in the sense that the technology is the same: 2 IR (infra-red) cameras with a little horizontal offset. Now lots of companies are working with RGB cameras (like Microsoft with HoloLens), so I asked Alex if they also started employing RGB cameras and the answer has been a firm “no”.

Of course, since Leap Motion is full of smart guys and girls, they have experimented with them, but didn’t find them useful. Processing data from RGB cameras requires more computational power (you have to analyze 3x the amount of data of a grayscale camera), so this leads to more battery consumption in mobile/standalone headsets and this is surely a problem, since standalone VR will be an important share of the VR market. Furthermore, since I’m a computer vision guy, I can assure you that RGB cameras are also more prone to problems depending on change of lighting of the environment: for example, if I’m tracking hands on sunlight, the skin will be of a pink-yellowish color, while if the day is cloudy, my skin will appear of a dark pink. Color is not helping a lot (I know, CV purists may consider that there is the “skin locus”, but this is an advanced topic…) and adds only complications, so they discarded RGB.

There are two big differences in the hardware with respect to older version:

• IR cameras are no more low-resolution ones, but are high-definition ones;
• The baseline is bigger, meaning that cameras are more distant each other, to allow for better tracking precision and depth estimation;

The result is a device that is slimmer and wider, ideal to be integrated into standalone headsets. Look the comparison between the Leap Motion v1 and v2 in this photo that Alex has made me shoot:

As Alex has stated, the Leap Motion v1 has born as a PC gadget, while the Leap Motion v2 has been conceived as something that could be embedded into VR standalone headsets, that’s why the design is completely different.

This hardware is the one that will make possible the 180°x180° tracking with improved distance, tracking and precision. For each tracked hand we will have the astonishing detection of 26 data-points. But as Mc Hammer would say “You can’t touch it” and we’ll see later why you will never have it into your hands.

Leap Motion focus is now on SDK

Having finished the hardware, it’s now turn for Leap geniuses to invest in their SDK. They really want developers to create with Leap Motion amazing virtual reality experiences, where people can use their hands in VR in a super-natural way. So now they’re all-in in working on the SDK.

Their focus is in giving the developers the tools to create amazing VR experiences using full hands interactions in an easy way. As a VR developer, I really know the hurdles of creating a realistic hand interface with the surrounding virtual world. I mean, when we take an object in our hand in VR in Unity, the thing that we usually do is take that object and put it as a child of the transform of the hand. This results in an un-natural object pose, with the object seeming like fixed with a screw onto our palm… and making something better is hard and requires lots of coding. Even frameworks like VRTK don’t help a lot in this.

Leap Motion wants to avoid all of this and that’s why it has created its own interactions engine, that this week has reached version 1.0 .

With the interactions engine, that basically redefines the physics rules for objects interacting with hands, Leap Motion takes care of handling complex stuff like interaction of hands with objects, interaction of objects with multiple hands and objects with objects. As you can see from the above GIF, you can take objects with your fingertips and then passing them from one hand to the other, in a very natural and realistic way. The greatest thing is that we won’t have to code anything, it will be Leap Motion SDK managing all this complex stuff for us. Cool, isn’t it?

Along with the Interaction Engine, they’re also releasing the Graphic Renderer, a tool to render complex curve user interfaces in a super-optimized way.

They’re focused on ultra-optimizing everything in the SDK for both mobile and PC platforms, so that Leap Motion SDK could be used everywhere with little computational consumption. So their SDK will suit completely every kind of VR device.

Leap Motion Interaction Engine for every VR platform

So, they’re investing a lot on their SDK… but I know what you’re thinking: “Cool, but why should I use this interaction engine? Very few people have Leap Motion out there”. I agree with you and so did a lot of developers that gave feedbacks to Leap Motion.

So Leap Motion had a brillant idea and made the Interaction Engine and the Graphic Renderer available for all platforms, not only for Leap Motion ones! I mean, you’ll be able to use this natural interaction engine even when developing an application for Oculus + Touch or Vive + VR controllers!! They structured these tools so they could be used with any controller, so that anyone of us can use them!

This is an amazing news in my opinion, since will give us developers the possibility to use a stable and greatly functioning, super-optimized and continuously updated SDK to handle interactions between hands and objects in the virtual world without writing a single line of code! I’m so excited about it… can’t wait to test it with my Touch…

Special features on the work…

Given all these updates on hardware and SDK, I’ve asked Alex if Leap is going to introduce some amazing innovative features like multiplayer (more people using the same Leap Motion), environment reconstruction (like with Structure), object tracking (currently Leap v1 is able to track only a pencil) and so on.

Alex has answered that behind the curtain a lot of things are happening and they’re experimenting, toying around and trying a lot of innovative things. But he can’t make any announcement, because nothing of this is already in a stable version, they can’t guarantee that any of these will surely happen in the future. So we have to let them continue their experiments and then we’ll see.

He finished with a “Keep an ear out…” that made me understand that they’re woking on some amazing stuff. So I started dreaming about what they’re going to announce in the future… stay tuned!

Leap Motion App Store is closing, apps will be free

Leap Motion has announced the “retirement of their app-store”. It was something that was in their thoughts since a while, but in the end they officially took the decision. The reason is that having a separate app store for a peripheral has little sense, when the headset producers are really pushing hard their app stores. Think about how much money Oculus is pouring into the VR ecosystem to push its Oculus platform and the Oculus Store (Robo Recall has costed them around $4M). Lead an app store to success requires lots of investments and has to be kept only if it is really beneficial for the company… which is not the case. It has surely more sense that some experiences on external stores will support Leap Motion as an input device (so for example on SteamVR we could see that a game requires Leap Motion).

To help the transition between the old Leap Motion model and the new one and to consider also experiences not requiring virtual reality (remember that Leap has born as a PC device!), Leap is talking with developers of the most popular Store apps. In preview, I can announce you that Leap will make all these apps available for free on their platform. So, you’ll be still able to download amazing experiences like Geco Midi and GameWAVE and they will be completely free! This is huge!

Leap Motion goes all-in with embedded, goodbye separate controller

In December we have been all delighted by the demos of new Leap Motion 2nd-gen mounted onto a GearVR. The faceplate had an unofficial name: the Rigel mount. We all wanted it, but no one has ever had it, only some selected journalists have been able to try this awesomeness.

Here comes the sad news: it will never see the light on market. Rigel has only been crafted to showcase to people the potentialities of Leap Motion 2nd-gen embedded into a standalone device. The only device that at that time could somewhat emulate a working standalone VR headset was Gear VR, so Leap engineers created an awesome faceplate that would give people the look and feel of a Leap Motion embedded into a standalone VR headset. This let people try embedded Leap Motion in exhibitions and also let Leap engineers work with a good-enough emulation of a mobile headset for some months. But it has not been made with the idea of selling it. Sorry, guys. It hurts, I know.

Because this is the path that Leap Motion has chosen for the future: be embedded into standalone headsets. Leap Motion is part of the Qualcomm 835 VR reference design and has already been showcased inside this headset in some exhibitions. Some journalists have been able to try it, along with a modified version of the Blocks demo and feedbacks are pretty postive (read this Road To VR review, for instance).

They’re working with different OEMs to integrate Leap Motion inside various standalone headsets and this is the only road they’ll take. Consider that Qualcomm headset isn’t truly a headset, but more a headset reference design, so a lot of Qualcomm-based-headsets made by different brands should come in the future. And all of them will integrate Leap Motion out of the box. This will allow to create super-headsets having features like:

• Wireless (they’re standalone);
• No need for external cameras/emitters (Qualcomm inside-out tracking doesn’t require external devices like Lighthouse);
• Natural hands tracking (thanks to Leap Motion hardware and SDK)

If you think about it for a moment, we’re really talking about super-devices, that have the power to disrupt the market. I mean, Lighthouse tracking has been super-improved, but still requires external devices (laser emitters), which are a nuisance for non-techie people. Microsoft cheap headsets are great and require no external devices, but their controllers are tracked only if they’re in the field of view of the users and this is bad for user experience. These Qualcomm headsets combine the best of both, since they require no external hardware and thanks to new Leap hardware, they’ll able to track hands wherever they are, even if they’re not in the field of view of the user. You’ll have wireless VR with your full hands in it, wherever you’ll want it. Powerful, isn’t it?

So, no Leap Motion 2nd-gen controller? Why?

Alex has been clear on this: “There’s no plan for a separate peripheral“. Even about faceplates, add-ons, whatever… the answer has been “no” to everything. And he also clearly added there’s no way to make Leap send it to us, so don’t ask him. I’ve tried also to bribe him, but with no luck.

This means that, for example, I won’t be ever be able to have a Leap Motion v2 sticked onto my Oculus CV1 headset. Leap Motion 2nd-gen will be always embedded. No way out.

Why all of this? Well, as Alex says, releasing it as a new controller would have been easier and the release data would have been surely nearer. But, if you release it that way, “you’re forever relegated to accessory”.

Leap Motion people has made lots of meetings to decide their vision of the future, thinking about which kind of technology they would like to become, which kind of user experience they would like to provide. And so they realized that “embedded was the way to go“. Reason is, if you are embedded, for the user everything is easier: nothing to assemble, no cables, nothing. For the developer everything is easier too, since if he/she wants to develop an experience for a certain headset, he’s sure that 100% of users will have Leap Motion. “Input fragmentation is bad for everyone” and as a developer I can say that this is right, since when developing for a headset, thinking about the various kind of controllers the user can have is a nuisance, while if I’m sure that that headset contains Leap Motion, I’m sure about how to design my application. Even as a user I agree with them, since very few time I’ve the willing to take Leap Motion and mount it onto my headset, if it isn’t already there. Having it out-of-the-box integrated into headset reduces the friction and makes sure the user will use natural interactions all the time.

“Our roadmap is embedded in a wide range of headsets. Peripheral is a slow death.” he added. I think that one of the reasons why they took this road is because they know that in the future headsets will integrate hand tracking out of the box: we all remember that Oculus some years ago have bought Nimble VR (a Leap Motion competitor), so it is possible that a Oculus v2 will include hands tracking out of the box and no one would buy Leap as an accessory for a headset that already does the same thing.

So Leap Motion is going from a B2C business model to a B2B2C one. They say it’s been a hard decision, but they firmly believe it has been the right one.

But I’m still sad about it, since this will mean that we won’t be able to experiment with this awesome device for whatever purpose we’ll want to use it. There will be less freedom. This can be an issue for research centers and for makers.

Qualcomm headset release date

It’s a secret when Qualcomm headset will be officially released. Alex knows it, but can’t tell me.

The fact that he knows it, that the device has already been showcased in various events and that the competitor Intel Alloy would be out at the end of this year, makes me think that the release date will be Q4 2017, but it’s just a speculation of mine.

Leap Motion will not be part of Vive standalone headset

Some weeks ago, Vive has announced a standalone headset for Daydream platform, made starting from Qualcomm VR reference design. In my article talking about this announcement, I was wondering if this would mean the standalone Vive headset would include Leap Motion, too. Alex has clearly stated that this is not the case.

And that’s it for today’s article! Hope you enjoyed all these news! If this is the case, share the article to let all VR innovators to know. And to not miss the part two, subscribe to my newsletter using the widget on the right sidebar!

(Header Image by Leap Motion)