All the AR and VR news from iPhoneX Apple Event, and much more

Yesterday for the first time I watched live a keynote by Apple. The reason was simple: some times ago I watched a video by Robert Scoble where he announced that Apple could completely disrupt the AR and VR market with the iPhone 8, because it could be the first device completely capable of AR and VR. I wrote an article reporting his thoughts and I was excited and scared by this possible revolution of the XR ecosystem.

iPhone X presentation Apple event AR VR
This apple appeared when the event was about to begin

The event has been pretty interesting, but I came out pretty deluded. I’ll tell you why in the remainder of this article.

Since I’ve assisted to the whole show waiting for the disruption moment, I’ll give you a summary of all its parts, highlighting the most important news and giving my point of view on them.

Intro

The event streaming began with the song “All you need is love” and a general overview of the Steve Jobs Theater. Then Tim Cook started speaking and he presented Apple’s new house, the Apple Park, that is super modern and super innovative and obviously with a supreme design. There is the Steve Jobs Theater, where events like this will be showcased. There is also a special Apple Store there, where people can try, among other things, a special augmented reality experience about the company. At the words “Augmented Reality” pronounced by Tim Cook in the first minutes of the event, I become super excited… AR seemed one of the main themes of Apple keynote.

iPhone X presentation Apple event AR VR
A photo of the stairs of the Steve Jobs Theater. It seems wonderful

After that, they commemorated Steve Jobs. This commemoration and all the event itself seemed to me something related to a religious sect more than to a company. It was moving but also weird.

iPhone X presentation Apple event AR VR
Tim Cook is always a bit moved when he talks about Steve.
Retail
iPhone X presentation Apple event AR VR
A moment of the video where they showcased their innovations in their stores. They say that Store is one of their products and as with other products, they want to make the experience better and better

Apple announced improvements of current Apple Stores and the opening of new big Apple Stores in major cities of the world. For instance, they’re opening a wonderful shop near the river in Chicago. Some shops are going to become plazas where you go there, assist to events, talk with people, try Apple devices, work with other people, exchange ideas, etc…

A new shopping experience, for sure. Like a church of their sect… cough cough

Apple Watch
iPhone X presentation Apple event AR VR
The new Apple Watch

Apple Watch is now the most sold watch in the world. This is impressive, for a product that is considered to be a failure. A product that Tim Cook has called “A tiny computer on your wrist” (a very nice definition).

iPhone X presentation Apple event AR VR
I’m not joking, they’re at number one place!

On September, 19th they’ll release a new version of the Apple Watch iOS that will include an improvement of various apps (fitness, activities, etc…) and most importantly the Apple Heart Study. This is a project thanks to which your Apple Watch monitors your heart rate to checks for possible problems and notifies you if you can have some heart issues. This is really important since most of the people that are starting having heart problems don’t notice that. I really liked this initiative since it can save lives, so congrats to Apple for that. The project will start only in the US for now.

iPhone X presentation Apple event AR VR
Apple Watch should help in detecting atrial fibrillations. That’s really amazing

Then they presented the new Apple Watch: the series 3. It is an evolution of the previous Watch in everything (even in colors and bands), but it has as disruptive feature the fact that it doesn’t need a companion phone anymore. With a special version of Apple Watch 3 (that works in some countries… and of course Italy is not among them), you can make phone calls, listening to music through iTunes and much more stuff directly from your watch. Your watch has the same phone number of your phone. Someone pointed me that the Samsung Gear S3 already had the same cellular functionality, but the fact that also Apple added it is interesting: it does mean that these smartwatches have now the potential of really becoming a tiny computer on your wrist. You can go to the seaside only with your smartwatch and be able to answer important calls or read the news without having to carry your smartphone with you.

iPhone X presentation Apple event AR VR
In this moment, the speaker started a call with a girl that was rowing in the middle of a lake, to showcase how vocal quality of this device is huge

Furthermore, about AR, I see smartwatches becoming interesting as well. I think that at least at the beginning, AR glasses will need a companion input device, to switch among options and click easily… because raising your arms to select everything or to type on a floating keyboard is tiresome as hell. I always thought this device should be the smartphone, but now I’m in doubt if it can be the smartwatch. Furthermore, the technology to put a tiny computer on your wrist can be the same to put a tiny computer on your head. And the smartwatch is a contextual device, giving you contextual infos, exactly as an AR glass should do during all day.

iPhone X presentation Apple event AR VR
Apple Watch 3 release prices: it ain’t cheap

So I think that smartwatches are also something that is in the path for the AR future. And we all know that Apple sooner or later will release an AR glass. So all these improvements on Apple Watch are very interesting to me.

iPhone X presentation Apple event AR VR
New fancy colors of Apple Watch and its bands. From the design side, Apple always rules.
Apple TV
iPhone X presentation Apple event AR VR
Apple TV has HDR technology that makes colors brighter

The new Apple TV 4K has been revealed. It uses HDR and Dolby technologies, so images and audio are awesome. The new TV is an improvement of previous one and the software now lets you see also live sports, live news, tv shows from your local TVs.

iPhone X presentation Apple event AR VR
Apple TV release prices

They called on the stage ThatGameCompany to showcase their game Sky, all focused on graphics and lights. I almost slept during all this session, so don’t ask me further details.

iPhone 8 and iPhone 8 plus
iPhone X presentation Apple event AR VR
Apple iPhone 8 and iPhone 8 plus, that are waterproof as previous models

They presented two new models of iPhone: 8 and 8plus.

Long story short: they’re just an evolution of 7 and 7plus, with better colors, more processing power, more battery, etc… They should have called them 7.1 and 7.1plus. They presented the core processor of these phones. The previous one was called A10, so they called this one A11. Wow, so much fantasy! But, to make it sound cooler, they’ve added the world BIONIC after it, so its complete name is A11 BIONIC. Marketing fluff at its best. From tomorrow I’ll call myself Tony BIONIC, I think this would double my followers… Bionic makes everything look so cool. Anyway, this processor has lots of features and it is optimized for all technologies like AR, machine learning, gaming, etc…

iPhone X presentation Apple event AR VR
Presentation of the features of the mesmerizing A11 BIONIC

Thanks to the two rear cameras, these devices are able to do “portrait lighting” in photos and videos. This means that the machine learning computer vision algos recognize where your face and body are and can do fancy stuff like removing completely the background and/or adding particular light effects to your face. This should make your photos more professional as if they were made in a studio. Nice feature.

iPhone X presentation Apple event AR VR
Portrait picture mode: with a menu, you can choose the lighting for your face to have a fantastic photo
Augmented Reality

iPhone8 has hardware and software specialized for augmented reality” and there I was waiting for the big innovations.

iPhone X presentation Apple event AR VR
Features that make the iPhone 8 optimized for AR: better cameras and better sensors

I saw the big words “Augmented reality” on the screen and I was waiting for amazing news. Instead, they just talked about ARKit and how the A11 BIONIC will make it faster and better. The man on the stage talked about 3 upcoming AR apps, but to me most of them didn’t even seem ARKit related: one of them showed the possibility to see stars and constellations written in the sky (cool, but surely ARKit can’t track the sky since it’s too far away… so the app is made using geolocation and gyros data); another one let people in a stadium see where the various baseball players are in the field (this is really interesting, but again is made using another kind of data).

iPhone X presentation Apple event AR VR
The baseball AR application: surely interesting, but seems more geolocation than environment reconstruction to me

Then they called on stage Directive Games, to show its upcoming ARKit game. Their upcoming game The Machine was just a tabletop game where you can move around the table and shoot enemies. This was my reaction

(Image from Know Your Meme)

I mean, I could make a similar game on Vuforia. There’s no need of ARKit. Furthermore, a game where you have to turn around a table to shoot is cool for the first time, then you find that rotating around the table with your phone in your hands is tiresome. And there’s no need to fuse the real and the virtual to do this shooting game and so they broke one of the most important rules when designing an AR app. Why play this game in AR? The table adds nothing to the experience. It would be better in VR.

iPhone X presentation Apple event AR VR
Showcase moment of the AR game: just a tabletop game (see the guy on the right pointing the phone towards the table). Buuuuuuuuuuh

I think they missed completely the point on augmented reality. An experience like MeasureKit is less impressive than a game, but shows the true spirit and utility of augmented reality on a phone. An app like this one has no sense in VR and has no sense without the real world images. And it wouldn’t be possible without ARKit. This would have been the ideal demo.

The fact that Apple is pushing on AR makes me happy because it is beneficial for the whole ecosystem, but I expected the announcement of some more evolutions of ARKit or the showcase of astonishing AR demos. This has not been the case. And I continue having the same doubts about ARKit revolution.

iPhone X presentation Apple event AR VR
iOS 11 will be released in a week

iOS 11 (that includes ARKit) will be distributed on September, 19th. So this is the date when we’ll start seeing people downloading AR apps on their iPhones.

iPhone 8 and 8plus (again)
iPhone X presentation Apple event AR VR
Wireless charging your smartphone is now possible. Apple has changed the case of the iPhone to make it possible

All upcoming iPhones will have wireless charging. Finally.

The iPhone 8 and 8plus will be available from September 22 for $699 and $799.

iPhone X presentation Apple event AR VR
iPhone 8 Plus price. It costs as a high-end Samsung phone

At this point we were on social media quite angry, saying “They can’t have made just these phones”. I mean, announced features were interesting, but nothing worth an Apple presentation. These 8 and 8plus are just a nice evolution of 7 and 7 plus. Spoilers on the web teased that there would have been an iPhone X model for the 10th anniversary of the first iPhone and we were all waiting for this one.

iPhone X presentation Apple event AR VR
Innovations added by the iPhone 8 and 8 plus. Nice things, but nothing marvelous

Until now, no big news, no disruptions. They should have kept the surprise cake for the end of the party.

One more thing…
iPhone X presentation Apple event AR VR
The magical sentence

Tim Cook returned on stage and said that he has to say “one more thing…”. Everyone went nuts. The disruptions were coming. The iPhone X, the new model that will change the future was about to be presented.

A video started and introduced the cool design of iPhone X. Applauses everywhere. Tim Cook said that the first iPhone has set the road for the past decades of smartphones and so this iPhone will set the road for the next decade. I was excited by the possible features.

iPhone X
iPhone X presentation Apple event AR VR
The magical X, at the end of the presentation video

iPhone X has no borders and no buttons. You just swipe from bottom side to simulate the pressure of the Home button and press a side button to evocate Siri. The fact that has no borders makes the device elegant and clear. It has glass on the front and the rear of the device (someone says that this way you can break the glass both on the front and the rear :D).

iPhone X presentation Apple event AR VR
iPhone X. We were all waiting for this moment

The screen is finally OLED: has 2436*1125 resolution and 458ppi. Screen technology is still Retina, but to make it cooler, they called it Super Retina. I still wonder why they didn’t call it Super Retina BIONIC… maybe it was too long. 😀

iPhone X presentation Apple event AR VR
iPhone X home screen. The big display makes the design very sleek
iPhone X FaceID and face tracking
iPhone X presentation Apple event AR VR
Facial recognition technology working on frontal camera. This image conveys a lot the idea of what happens

True innovation inside this iPhone X has been the addition of a depth sensor on the frontal camera. This allows for face detection and tracking in all lighting conditions (it is based on IR depth sensors, so the day or the night is the same for it).

iPhone X presentation Apple event AR VR
These are all the tiny sensors embedded in the top part of the iPhone. There’s really a lot of electronics!

Apple has so announced FaceID: after you train your phone by moving your face in various directions for some seconds, your phone can detect your face and use it to authenticate you (to unlock the phone or to make payments using Apple Pay). While this kind of technology is not new (there’s Windows Hello on Windows 10 and some Nokia phones already did something similar), Apple has improved it a lot.

iPhone X presentation Apple event AR VR
iPhone X lock screen, that can be unlocked using your face

They worked so that photos and 3D masks can’t trick the system and only your true face can authenticate you: from their tests, it can authenticate someone else only in 1 case out of 1 million (with fingerprints, was 1 out of 50000). Furthermore, the system updates the model of your face every day, so it can recognize you even if you grow a beard, change hair, etc… and it is also insensitive to clothes and glasses. Seems a great piece of tech to me. If you’re interested in technical details, they said that the system uses a point cloud of 30000 points that are fed to a neural network that can recognize your face. A11 BIONIC is optimized for such kind of algorithms and the whole system has a dedicated neural engine: all the recognition happens on the device, so there is no sensible data sent over the network.

iPhone X presentation Apple event AR VR
Reconstruction of the point cloud that the devices make starting from your real face

About unlocking your phone, it just works if you look at it (so if your face is frontal). So the procedure to use your iPhone X is: you take your phone, look at it and then swipe from bottom to open the home screen. The recognition can work even with 3rd party apps.

But this technology doesn’t allow only for face recognition, but also for face tracking. iPhone X can track your face and so lets you develop augmented reality applications tailored to the human face. Again, this is impressive, since using a depth sensor is possible to perform a much precise tracking than with only an RGB camera, so it is possible to create AR applications that are very effective. And what have they showcased to astonish us with this technology? Animojis.

iPhone X presentation Apple event AR VR
Chicken animoji: the 3d model of the chicken follows completely the face and the expression of the user

These are funny things that you can use in messages: the standard emojis are now 3D and they follow your facial expressions. So, you can send a vocal message with a cute panda talking with your voice and following your facial expressions. In the demo they showcased various animojis and in the moment that they showed a talking poop, I think that Steve Jobs had the instinct to return to Earth and slap some people’s faces.

Again, they have an amazing tech and they use it to showcase a talking poop. Funny, but they just attracted irony for something that instead is important for business. Some times ago, I was contacted for a possible AR project and so I made some research: to perform a valid face tracking using phone frontal camera it is necessary to download some expensive 3rd party tool, like Visage. Price of such licenses is around $3000 per year per application and the results, since they’re based on a single RGB camera, are very good, but still not perfect. In this video, you can have a taste of my experiments.

With this new Apple framework, it is possible to have better tracking results, without having to pay any additional license, since this AR technology will be included in ARKit. This can enable a lot of new application, like for instance in e-commerce, enabling customers to try products onto their faces (e.g. glasses or makeup); or in plastic surgery to make people preview how they’ll be after a surgery. Since it can detect your face, it can also detect your emotions and this, combined with the heart rate monitor of the Apple Watch, enables a lot of psychological applications. And also marketing ones: they can show you a product and analyze your reactions to it.

iPhone X presentation Apple event AR VR
A virtual mask super-imposed to the face of the user. The tracking was damn stable

If it works as it seems, this is an amazing technology for augmented reality, it will unleash a lot of opportunities. Instead, they showed us a talking poop and an old guy pretending to be a chicken.

iPhone X (again…)
iPhone X presentation Apple event AR VR
All iPhone X innovations, in a simple recap

Backward cameras now are aligned vertically and they’re super stabilized and super-optimized. Machine learning magic recognizes the various elements of the photos you’re going to take and optimizes everything to make your photos appear wonderful.

iPhone X presentation Apple event AR VR
A photo taken with the iPhone X. Quality is astonishing

Battery lasts 2 hours more than iPhone 7.

There’s a device called AirPower that lets you charge wireless all your apple devices (phone, watch and earphones) at the same time.

iPhone X presentation Apple event AR VR
You can now recharge all your Apple devices on a single station

You can preorder this device from October, 27th and have it from November, 3d for $999. It is the most expensive phone in history.

iPhone X presentation Apple event AR VR
The most expensive smartphone in history: $999 at launch! Surely this phone has already set a new record…
The end
iPhone X presentation Apple event AR VR
Steve Jobs’s quote from the end of the event

Tim Cook says again how he’s proud of his new phone and then quotes Steve Jobs again.

In the end, he says goodbye to everyone. The event is over.

iPhone X presentation Apple event AR VR
Goodbye, Tim. Thanks for your time.
What has missed: VR and other delusions
iPhone X presentation Apple event AR VR
iPhone X multitasking screen

So, this has been what the event has shown. But… what about things that weren’t shown?

First of all, there have been no surprises: almost everything had been leaked in the previous days. We all knew about all the announced features, so watching the event has been quite useless. I was expecting Apple having some special ace in the hole, instead, they had not.

Then I honestly don’t get why releasing the iPhone 8 and 8 plus, when the true star of the event is the iPhone X. It is like saying: we have a new revolutionary phone, but if you want, you can spend a bit less and have a “meh” device. iPhone 8 and plus are too similar to iPhone 7 to be relevant. IMHO only the X was worth announcing.

iPhone X presentation Apple event AR VR
The iPhone X has been presented as the future of the smartphone. A part of Apple future is actually the past for Android devices

The iPhone X had been announced as a revolutionary device, but it doesn’t carry any revolution. It is full of incremental innovations, but I see no disruption. For instance, I’ve shown you how AR with the face was already possible with current devices, they’ll just make it better. And some of their “innovations” like wireless charging or OLED screens are just features that some Android devices have since years.
First iPhone was true disruptive because allowed you to have a computer in your pocket. This one is just “a better iPhone”. Furthermore, I hope that in the next decade we won’t have smartphones anymore and we’ll all wear AR glasses, so I hope that we don’t need a smartphone to inspire the next decade.

iPhone X presentation Apple event AR VR
Features of Apple frontal camera

About AR and VR I’m quite deluded by the shown apps, but especially by the lack of rear depth cameras. Rear depth cameras (or another revolution in rear cameras, like additional IR cameras distant one from the other) would allow for a better tracking of the environment. This would not only allow ARKit to be more precise and stable but also would allow the iPhone to have a perfect positional tracking that could be employed in VR, making the iPhone the first VR 6DOF device. Scoble predicted the possible use of rear cameras to track hands and so have the iPhone VR headset capable of doing what the Snapdragon 835 VR promises: standalone 6DOF device controlled by hands and voice. A dream for us all. Instead, no one of these dreams will become true.

iPhone X presentation Apple event AR VR
Rear cameras have been improved, but nothing more

Yesterday Apple has never talked about VR: Tim Cook has always said that Apple is not interested in VR (even if it is now supported on Mac) and yesterday’s keynotes confirm this. Expectations on iPhone X being the best VR capable phone out there have been disenchanted. There will be no revolution in the VR market.

I miss Jobs and his vision of the future.

The irony
iPhone X presentation Apple event AR VR
Apple, Apple…

Of course, yesterday announcements generated a lot of ironies… so I’m leaving you the best that I’ve seen until now on 9GAG to make you smile a bit.

 

 

https://i2.wp.com/img-9gag-fun.9cache.com/photo/am21wb2_700b.jpg?resize=643%2C1206&ssl=1

Also Reverend Kyle (a VR enthusiast) did its best to write funny comments:

 


Hope you liked this post and if this is the case, share it and register to my newsletter to support my hard work 🙂

(All screenshots have been made by me, but represent an Apple event, so I guess I have to credit them)

Skarredghost

AR/VR developer, startupper, zombie killer. Sometimes I pretend I can blog, but actually I've no idea what I'm doing. I tried to change the world with my startup Immotionar, offering super-awesome full body virtual reality, but now the dream is over. But I've still not waken up...

4 thoughts on “All the AR and VR news from iPhoneX Apple Event, and much more

    • September 14, 2017 at 4:50 pm
      Permalink

      LOL.

      Furthermore Apple has not pushed on AR the right way… I’m angry in seeing all people making irony on the poop-tracking (not a nice name :D) and not understanding how face AR can be important

      Reply
  • September 14, 2017 at 2:30 pm
    Permalink

    I didn’t realize whether the pair of rear cameras can be actually considered a stereo camera or not – if so (somewhere you stated the cameras are “calibrated” or such… it could be an external calibration leading to a metrically-precise stereoscopic camera system), it’s a depth camera! so, we (V&ARers’) could rely on depth data to support 6-dof tracking, maybe hand/finger tracking (a-la Leap Motion and Fingo), and AR 3D pose estimation. Maybe hardware is yet there, and new software releases could add VR-AR services in a second time.

    Reply
    • September 14, 2017 at 4:49 pm
      Permalink

      You’re completely right: two cameras can be used to detect depth, but there are some issues:
      – The two cameras have to be a bit distant otherwise calculation is too prone to error;
      – RGB cameras are sensible on change of lighting (in the dark the tracking becomes bad)
      – Depth estimation on monochrome surfaces (i.e. non textured one) with only RGB data is impossible.

      The addition of depth cameras solves these issues and adds more precision. This smart guy writing this interesting post about AR and a world model shared by people too highlighted how depth cameras are important https://medium.com/super-ventures-blog/arkit-and-arcore-will-not-usher-massive-adoption-of-mobile-ar-da3d87f7e5ad.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *