All the AR and VR news from iPhoneX Apple Event, and much more

Yesterday for the first time I watched live a keynote by Apple. The reason was simple: some times ago I watched a video by Robert Scoble where he announced that Apple could completely disrupt the AR and VR market with the iPhone 8, because it could be the first device completely capable of AR and VR. I wrote an article reporting his thoughts and I was excited and scared by this possible revolution of the XR ecosystem.

The event has been pretty interesting, but I came out pretty deluded. I’ll tell you why in the remainder of this article.

Since I’ve assisted to the whole show waiting for the disruption moment, I’ll give you a summary of all its parts, highlighting the most important news and giving my point of view on them.

Intro

The event streaming began with the song “All you need is love” and a general overview of the Steve Jobs Theater. Then Tim Cook started speaking and he presented Apple’s new house, the Apple Park, that is super modern and super innovative and obviously with a supreme design. There is the Steve Jobs Theater, where events like this will be showcased. There is also a special Apple Store there, where people can try, among other things, a special augmented reality experience about the company. At the words “Augmented Reality” pronounced by Tim Cook in the first minutes of the event, I become super excited… AR seemed one of the main themes of Apple keynote.

After that, they commemorated Steve Jobs. This commemoration and all the event itself seemed to me something related to a religious sect more than to a company. It was moving but also weird.

Retail

Apple announced improvements of current Apple Stores and the opening of new big Apple Stores in major cities of the world. For instance, they’re opening a wonderful shop near the river in Chicago. Some shops are going to become plazas where you go there, assist to events, talk with people, try Apple devices, work with other people, exchange ideas, etc…

A new shopping experience, for sure. Like a church of their sect… cough cough

Apple Watch

Apple Watch is now the most sold watch in the world. This is impressive, for a product that is considered to be a failure. A product that Tim Cook has called “A tiny computer on your wrist” (a very nice definition).

On September, 19th they’ll release a new version of the Apple Watch iOS that will include an improvement of various apps (fitness, activities, etc…) and most importantly the Apple Heart Study. This is a project thanks to which your Apple Watch monitors your heart rate to checks for possible problems and notifies you if you can have some heart issues. This is really important since most of the people that are starting having heart problems don’t notice that. I really liked this initiative since it can save lives, so congrats to Apple for that. The project will start only in the US for now.

Then they presented the new Apple Watch: the series 3. It is an evolution of the previous Watch in everything (even in colors and bands), but it has as disruptive feature the fact that it doesn’t need a companion phone anymore. With a special version of Apple Watch 3 (that works in some countries… and of course Italy is not among them), you can make phone calls, listening to music through iTunes and much more stuff directly from your watch. Your watch has the same phone number of your phone. Someone pointed me that the Samsung Gear S3 already had the same cellular functionality, but the fact that also Apple added it is interesting: it does mean that these smartwatches have now the potential of really becoming a tiny computer on your wrist. You can go to the seaside only with your smartwatch and be able to answer important calls or read the news without having to carry your smartphone with you.

Furthermore, about AR, I see smartwatches becoming interesting as well. I think that at least at the beginning, AR glasses will need a companion input device, to switch among options and click easily… because raising your arms to select everything or to type on a floating keyboard is tiresome as hell. I always thought this device should be the smartphone, but now I’m in doubt if it can be the smartwatch. Furthermore, the technology to put a tiny computer on your wrist can be the same to put a tiny computer on your head. And the smartwatch is a contextual device, giving you contextual infos, exactly as an AR glass should do during all day.

So I think that smartwatches are also something that is in the path for the AR future. And we all know that Apple sooner or later will release an AR glass. So all these improvements on Apple Watch are very interesting to me.

Apple TV

The new Apple TV 4K has been revealed. It uses HDR and Dolby technologies, so images and audio are awesome. The new TV is an improvement of previous one and the software now lets you see also live sports, live news, tv shows from your local TVs.

They called on the stage ThatGameCompany to showcase their game Sky, all focused on graphics and lights. I almost slept during all this session, so don’t ask me further details.

iPhone 8 and iPhone 8 plus

They presented two new models of iPhone: 8 and 8plus.

Long story short: they’re just an evolution of 7 and 7plus, with better colors, more processing power, more battery, etc… They should have called them 7.1 and 7.1plus. They presented the core processor of these phones. The previous one was called A10, so they called this one A11. Wow, so much fantasy! But, to make it sound cooler, they’ve added the world BIONIC after it, so its complete name is A11 BIONIC. Marketing fluff at its best. From tomorrow I’ll call myself Tony BIONIC, I think this would double my followers… Bionic makes everything look so cool. Anyway, this processor has lots of features and it is optimized for all technologies like AR, machine learning, gaming, etc…

Thanks to the two rear cameras, these devices are able to do “portrait lighting” in photos and videos. This means that the machine learning computer vision algos recognize where your face and body are and can do fancy stuff like removing completely the background and/or adding particular light effects to your face. This should make your photos more professional as if they were made in a studio. Nice feature.

Augmented Reality

“iPhone8 has hardware and software specialized for augmented reality” and there I was waiting for the big innovations.

I saw the big words “Augmented reality” on the screen and I was waiting for amazing news. Instead, they just talked about ARKit and how the A11 BIONIC will make it faster and better. The man on the stage talked about 3 upcoming AR apps, but to me most of them didn’t even seem ARKit related: one of them showed the possibility to see stars and constellations written in the sky (cool, but surely ARKit can’t track the sky since it’s too far away… so the app is made using geolocation and gyros data); another one let people in a stadium see where the various baseball players are in the field (this is really interesting, but again is made using another kind of data).

Then they called on stage Directive Games, to show its upcoming ARKit game. Their upcoming game The Machine was just a tabletop game where you can move around the table and shoot enemies. This was my reaction

I mean, I could make a similar game on Vuforia. There’s no need of ARKit. Furthermore, a game where you have to turn around a table to shoot is cool for the first time, then you find that rotating around the table with your phone in your hands is tiresome. And there’s no need to fuse the real and the virtual to do this shooting game and so they broke one of the most important rules when designing an AR app. Why play this game in AR? The table adds nothing to the experience. It would be better in VR.

I think they missed completely the point on augmented reality. An experience like MeasureKit is less impressive than a game, but shows the true spirit and utility of augmented reality on a phone. An app like this one has no sense in VR and has no sense without the real world images. And it wouldn’t be possible without ARKit. This would have been the ideal demo.

The fact that Apple is pushing on AR makes me happy because it is beneficial for the whole ecosystem, but I expected the announcement of some more evolutions of ARKit or the showcase of astonishing AR demos. This has not been the case. And I continue having the same doubts about ARKit revolution.

iOS 11 (that includes ARKit) will be distributed on September, 19th. So this is the date when we’ll start seeing people downloading AR apps on their iPhones.

iPhone 8 and 8plus (again)

All upcoming iPhones will have wireless charging. Finally.

The iPhone 8 and 8plus will be available from September 22 for $699 and $799.

At this point we were on social media quite angry, saying “They can’t have made just these phones”. I mean, announced features were interesting, but nothing worth an Apple presentation. These 8 and 8plus are just a nice evolution of 7 and 7 plus. Spoilers on the web teased that there would have been an iPhone X model for the 10th anniversary of the first iPhone and we were all waiting for this one.

Until now, no big news, no disruptions. They should have kept the surprise cake for the end of the party.

One more thing…

Tim Cook returned on stage and said that he has to say “one more thing…”. Everyone went nuts. The disruptions were coming. The iPhone X, the new model that will change the future was about to be presented.

A video started and introduced the cool design of iPhone X. Applauses everywhere. Tim Cook said that the first iPhone has set the road for the past decades of smartphones and so this iPhone will set the road for the next decade. I was excited by the possible features.

iPhone X

iPhone X has no borders and no buttons. You just swipe from bottom side to simulate the pressure of the Home button and press a side button to evocate Siri. The fact that has no borders makes the device elegant and clear. It has glass on the front and the rear of the device (someone says that this way you can break the glass both on the front and the rear :D).

The screen is finally OLED: has 2436*1125 resolution and 458ppi. Screen technology is still Retina, but to make it cooler, they called it Super Retina. I still wonder why they didn’t call it Super Retina BIONIC… maybe it was too long. 😀

iPhone X FaceID and face tracking

True innovation inside this iPhone X has been the addition of a depth sensor on the frontal camera. This allows for face detection and tracking in all lighting conditions (it is based on IR depth sensors, so the day or the night is the same for it).

Apple has so announced FaceID: after you train your phone by moving your face in various directions for some seconds, your phone can detect your face and use it to authenticate you (to unlock the phone or to make payments using Apple Pay). While this kind of technology is not new (there’s Windows Hello on Windows 10 and some Nokia phones already did something similar), Apple has improved it a lot.

They worked so that photos and 3D masks can’t trick the system and only your true face can authenticate you: from their tests, it can authenticate someone else only in 1 case out of 1 million (with fingerprints, was 1 out of 50000). Furthermore, the system updates the model of your face every day, so it can recognize you even if you grow a beard, change hair, etc… and it is also insensitive to clothes and glasses. Seems a great piece of tech to me. If you’re interested in technical details, they said that the system uses a point cloud of 30000 points that are fed to a neural network that can recognize your face. A11 BIONIC is optimized for such kind of algorithms and the whole system has a dedicated neural engine: all the recognition happens on the device, so there is no sensible data sent over the network.

About unlocking your phone, it just works if you look at it (so if your face is frontal). So the procedure to use your iPhone X is: you take your phone, look at it and then swipe from bottom to open the home screen. The recognition can work even with 3rd party apps.

But this technology doesn’t allow only for face recognition, but also for face tracking. iPhone X can track your face and so lets you develop augmented reality applications tailored to the human face. Again, this is impressive, since using a depth sensor is possible to perform a much precise tracking than with only an RGB camera, so it is possible to create AR applications that are very effective. And what have they showcased to astonish us with this technology? Animojis.

These are funny things that you can use in messages: the standard emojis are now 3D and they follow your facial expressions. So, you can send a vocal message with a cute panda talking with your voice and following your facial expressions. In the demo they showcased various animojis and in the moment that they showed a talking poop, I think that Steve Jobs had the instinct to return to Earth and slap some people’s faces.

Again, they have an amazing tech and they use it to showcase a talking poop. Funny, but they just attracted irony for something that instead is important for business. Some times ago, I was contacted for a possible AR project and so I made some research: to perform a valid face tracking using phone frontal camera it is necessary to download some expensive 3rd party tool, like Visage. Price of such licenses is around $3000 per year per application and the results, since they’re based on a single RGB camera, are very good, but still not perfect. In this video, you can have a taste of my experiments.

With this new Apple framework, it is possible to have better tracking results, without having to pay any additional license, since this AR technology will be included in ARKit. This can enable a lot of new application, like for instance in e-commerce, enabling customers to try products onto their faces (e.g. glasses or makeup); or in plastic surgery to make people preview how they’ll be after a surgery. Since it can detect your face, it can also detect your emotions and this, combined with the heart rate monitor of the Apple Watch, enables a lot of psychological applications. And also marketing ones: they can show you a product and analyze your reactions to it.

If it works as it seems, this is an amazing technology for augmented reality, it will unleash a lot of opportunities. Instead, they showed us a talking poop and an old guy pretending to be a chicken.

iPhone X (again…)

Backward cameras now are aligned vertically and they’re super stabilized and super-optimized. Machine learning magic recognizes the various elements of the photos you’re going to take and optimizes everything to make your photos appear wonderful.

Battery lasts 2 hours more than iPhone 7.

There’s a device called AirPower that lets you charge wireless all your apple devices (phone, watch and earphones) at the same time.

You can preorder this device from October, 27th and have it from November, 3d for $999. It is the most expensive phone in history.

The end

Tim Cook says again how he’s proud of his new phone and then quotes Steve Jobs again.

In the end, he says goodbye to everyone. The event is over.

What has missed: VR and other delusions

So, this has been what the event has shown. But… what about things that weren’t shown?

First of all, there have been no surprises: almost everything had been leaked in the previous days. We all knew about all the announced features, so watching the event has been quite useless. I was expecting Apple having some special ace in the hole, instead, they had not.

Then I honestly don’t get why releasing the iPhone 8 and 8 plus, when the true star of the event is the iPhone X. It is like saying: we have a new revolutionary phone, but if you want, you can spend a bit less and have a “meh” device. iPhone 8 and plus are too similar to iPhone 7 to be relevant. IMHO only the X was worth announcing.

The iPhone X had been announced as a revolutionary device, but it doesn’t carry any revolution. It is full of incremental innovations, but I see no disruption. For instance, I’ve shown you how AR with the face was already possible with current devices, they’ll just make it better. And some of their “innovations” like wireless charging or OLED screens are just features that some Android devices have since years.
First iPhone was true disruptive because allowed you to have a computer in your pocket. This one is just “a better iPhone”. Furthermore, I hope that in the next decade we won’t have smartphones anymore and we’ll all wear AR glasses, so I hope that we don’t need a smartphone to inspire the next decade.

About AR and VR I’m quite deluded by the shown apps, but especially by the lack of rear depth cameras. Rear depth cameras (or another revolution in rear cameras, like additional IR cameras distant one from the other) would allow for a better tracking of the environment. This would not only allow ARKit to be more precise and stable but also would allow the iPhone to have a perfect positional tracking that could be employed in VR, making the iPhone the first VR 6DOF device. Scoble predicted the possible use of rear cameras to track hands and so have the iPhone VR headset capable of doing what the Snapdragon 835 VR promises: standalone 6DOF device controlled by hands and voice. A dream for us all. Instead, no one of these dreams will become true.

Yesterday Apple has never talked about VR: Tim Cook has always said that Apple is not interested in VR (even if it is now supported on Mac) and yesterday’s keynotes confirm this. Expectations on iPhone X being the best VR capable phone out there have been disenchanted. There will be no revolution in the VR market.

I miss Jobs and his vision of the future.

The irony

Of course, yesterday announcements generated a lot of ironies… so I’m leaving you the best that I’ve seen until now on 9GAG to make you smile a bit.

Also Reverend Kyle (a VR enthusiast) did its best to write funny comments:

'Innovation.'

I do not think it means what you think it means.

— Kyle Riesenbeck (@reverendkjr) September 12, 2017

Apple Watch is the #1 watch in the world, with a total of 1,300 owners!

— Kyle Riesenbeck (@reverendkjr) September 12, 2017

'This isn't some generic sky. This is the sky around you.'

Huh? What?

— Kyle Riesenbeck (@reverendkjr) September 12, 2017

Wow. I've never seen a commercial for an electronic device that only showed a screensaver. #iPhoneX

— Kyle Riesenbeck (@reverendkjr) September 12, 2017

Hope you liked this post and if this is the case, share it and register to my newsletter to support my hard work 🙂

(All screenshots have been made by me, but represent an Apple event, so I guess I have to credit them)