AI & XR are made for each other! But is this also the end of our privacy?

Soon, AR glasses with AI and advanced machine vision capabilities will be able to analyze not only our environments and hand gestures, but maybe even people’s faces, expressions, and emotions. In this article I’ll talk about why I’m so excited about XR (VR+AR) and just as I mentioned in my previous article, why I believe that XR & AI are made for each other, which is also why I’m a little bit terrified by combining the two. But, it’s up to us (the creators) to take action and create ethical use-cases for these technologies.

A decade ago, when AR sparked my curiosity (because of an Israeli TV show, Mesudarim about Startups and Innovation), I made a simple marker-based AR app. Which was my 1st time to wonder about the future of AR & AI…
And around the same period, I learned a little bit about body language and micro-expressions (as a teenager who was interested in business and tech). And since the topic was so fascinating to me, I figured that one day robots and AI might be able to do it even better than us, humans. But I didn’t imagine that AI revolution would happen in our lifetime, but maybe in a distant future, like in Hollywood’s dystopian movies such as I, Robot And Terminator.

Even now, you can know what public figures and Leaders feel, with Emotion Labs

The future is now!

And now (for better or worse), these technologies are already here! As for AI analyzing body language and expressions, well, it’s sort of working but without AR (for now). But our role as Tech-enthusiasts is to connect the dots, right? So imagine how powerful you would feel if you had an AR glasses with AI and Machine Vision capabilities to analyze expressions and understand the feelings of the people you interact with. For example, during business meetings, while negotiating with investors, or during a job interview. It could feel like these kind of technologies actually giving you Super-Powers, right? But because technology is a double-edged sword, others can use it against you as well! So I wonder — where do we draw the line ?!

So people’s concern about AR originally started with Google Glass back in 2013. Knowing that those who walk around with AR glasses (some called them, GlassHoles! Is it relevant for Apple Glass too? ????) are violating your privacy, is a creepy experience. Because you can’t tell when they’re recording they see, and what do they do with it… So now you should also worry about people with AR glasses having an advantage over you! What if they are analyzing your behavior, body language, and micro-expressions during a job interview, or even during a date? Thy can see your, insecurities and your flaws while you can’t do anything to prevent it! Creepy? Yes!

But just before this kind of technology is available commercially, and long before it will be available in AR glasses (probably I’m not the only one thinking about it just because reading this research papers). Current use cases are: Airports, Courts, Security gateways, Job interviews (video calls), Public Locations (like shops and supermarkets), etc.

The good: AI & XR can change our experiences and the way we see the world!

The Good vs. The Bad

And as always, we can use technology for positive use-cases, for example this VR Tech to be used prevent domestic violence. And we can do the same with AR Emotion Analyzer, to create empathy between married people (Husband understanding the wife, and vice versa). And even creating empathy between police and civilians, etc.

On the other hand, these kind of technologies are even more scary in different regimes. China for example has their famous “Big Brother State” 24/7 Surveillance Cameras, with Biometric Databases, and a Social Credit system, so it’s only natural to be worried about privacy and human rights in the age of AR, especially if you live in a Democracy…
And now with this CoronaVirus pandemic — police forces are already equipped with some sort of AR glasses / headset to measure body temperature. You can only assume that it is only a matter of time until they’ll be able to integrate Biometric capabilities to identify people, their medical status, and maybe even their emotions.

The bad: Do you want to share what you see and say with Tech-Giants and their AI?

What about Privacy?

One can only wonder about what are the Tech-Giants gonna do with such an information about us (the users)… Most VR & AR (XR) glasses now have between 2–10 cameras and sensors (inside and outside) that are capable to see and analyze your surroundings (Home / Office). They are also designed for Eye-tracking and Hand-Tracking, to use our hands and natural gestures in order to enable us to interact with holographic user-interfaces. Also for 3D Meshing (mapping our environment) and to place virtual objects on it and even behind real objects and furniture (Occlusion). Eye tracking cameras are good not only to provide you with better experience based on your gaze, but it can also identify each user (via Iris recognition) and it’s great for automatic log-ins (Spatial Computers are just like PCs with multiple-users).

So what happens to all this data? Is it stored on our devices only? Is it goes to the cloud (Telemetry / Analytics)? Remember, we are talking about our personal life, personal data (way more personal than PCs and Phones), and our privacy and personal information (Everything we see and say, from our POV) imagine all of it(!) uploaded to the cloud, analyzed, and kept forever …
Well, it’s creepy, but take it as a food for thought! Although it’s not Westworld’s Delos Corporation, yet (unless there are some Neuralink sensors in Magic Leap ML1, Microsoft HoloLens, Google Glass, and Facebook’s Oculus headsets that we didn’t knew about ???? ) but it’s still creepy…

The State of XR: I have mapped the market

There’s hope!

If you are following the leaks and rumors about Apple Glass, it looks like Apple want’s to maintain their “We care about privacy” even when it comes to AR glasses. According to the reports, Apple won’t have a camera on their AR glasses (at least not in their 1st-gen device), they’ll use a Lidar technology (just like in the new iPad Pro) to analyze your surroundings. But we’ll have to wait and see how and if it’s gonna work. Although Apple is a big Privacy advocate, and they are proud of their “On-Device AI”, we should remember that Siri is still a cloud-based AI (Just like Google Assistant and MS Cortana). So the question is always how much do you really trust them, after all, just like Microsoft, Google, and Facebook —Apple’s software is closed-source and walled-garden. So can we trust them? not really…

Windows10 is the only Cross-Platform OS, for Desktops, Tablets, Phones, IoT and even VR/AR Headsets.

But the only true and transparent XR initiatives we’ll be able to trust are from FOSS organizations, such as: OpenXR, Ultraleap, LeapMption (Project NorthStart) etc. And in Spatial-Computing (unlike Desktop Computing) you can’t replace your Closed-source OS with and Open Source one (it’s not easy as installing Linux instead of Windows and macOS). Even on Mobile phones, Installing AOSP instead of OEM’s Android is not easy for most people (and on the iPhone it’s nearly impossible), and the same goes for VR/AR as for today.
So, for now, we’ll have to take action and watch over these Tech-Giants if we want them to create ethical XR devices, because they are the ones who drive innovation, competition, and commercial adoption (I hope AppleGlass will revolutionize the industry just like the 1st iPhone did).

So, once again — Technology is a double edge sword, so where do we draw the line? You decide…

Research & Papers:

Paper 1: https://dl.acm.org/doi/pdf/10.1145/2993148.2993165

Paper 2: https://link.springer.com/article/10.1007/s00607-019-00722-7

Paper 3: https://arxiv.org/abs/1902.03284

Paper 4: https://ieeexplore.ieee.org/abstract/document/8673885

Paper 5: https://www.sciencedirect.com/science/article/abs/pii/S0304394017304986