Blog

You are here: Home » Blog
14th June 2023
14 Jun 2023

Discussions about Apple Vision Pro

I’ve been asked several times about the Apple Vision Pro announcement, so here is the summary of some of the main points that have come up in discussion.

 

Certainly Apple does design extremely well. Some of the parts of the announcements and demonstrations were very slick, but if you look closely some of the parts were obviously not finished and I expect some features will change.

Hardware (Most of It)

Apple designs nice hardware. There are echoes of several former systems (some of them quite old).  There is lots of other commentary on the basic features, such as resolution and field of view. If you want to experience something approaching the resolution touted for the Vision Pro (50-70 pixels per degree), you’ll need to try something such as the Varjo VR3 (70ppd). The Vision Pro panels appear to have a larger number of pixels in total, whereas the Varjo systems have a foveal region of high clarity and resolution (which is lovely to look at!). The field of view has been compared to the Valve Index (a HMD approaching its 4th birthday). The audio is rumoured to be excellent and the tracking is probably state of the art, but it isn’t clear if these are notably different than other current systems such as the Meta Quest Pro or VIVE XR Elite.

Video passthrough is an important feature. Ever since Apple bought VRVana it has been the most likely way that they would achieve augmented reality views. We had ordered the Totem display after I had tried it at the IEEE VR conference. It was excellent – they had custom hardware to transfer the video direct to the display so the latency was impressively low (especially compared to the homebrew hardware we had built). There is now a whole range of impressive video passthrough modes on mobile systems (e.g. Quest Pro again) and desktop systems. At a recent count we had nine different systems doing this to some extent. The best, by far, is the Varjo XR3, and it is unlikely that the Vision Pro can beat that on quality, though it might beat it on latency (12ms latency was mentioned, so approximately 1 frame). We know that there are still effects of latency at that range, even down below 4ms but this becomes a real challenge for frame-based rendering architectures (e.g. see Sebastian’s work), so no doubt there will be more research on this and faster systems (e.g. Microsoft was working on the Prism system when I was on sabbatical there. Some details in this paper about latency of video AR systems).

Eyesight (the External Panel)

The external panel is by far the most interesting aspect of the system. It was mentioned that the HMD can scan your face, then track its movements while you are wearing the HMD and then show your face on the front. The first part is not that unusual, the computer vision and reconstruction techniques have been explored for a while (e.g. see the FaceVR paper and its citations). No doubt Apple will do a state-of-the-art job here, but their goals are somewhat different than, say, Meta’s Codec avatars (see Upload VR’s summary).

The external panel is really interesting. The need for awareness of the immersed user and their expressions has long been explored in HMD design as it is a potential barrier to social use. Sony tackled this one way with the PSVR (with the default being that there is a screen for spectators to watch so that they can see what you are doing), but while mobile systems can support second screens this is probably uncommon in use. It is not always well supported on wired systems as it is up to the client software to generate a view for spectators (or use a network solution).

So the basic idea of putting a screen on the outside has been pursued by academic researchers. You can just attach another panel (or smartphone) on the outside of the HMD (as long as it doesn’t obscure tracking sensors). Below is an image from the paper TransparentHMD: Revealing the HMD
User’s Face to Bystanders by Mai et al.

TransparentHMD from Mai et al.

This display has some limitations. You can already see that the image is flat, so it is obvious the face on the display is not facing the same way as the wearer. The most advanced display of this type, is Meta’s research work on “Reverse Pass-Through VR”, where they reconstruct the face and put the eyes of the user on an external light field display.

Why did Meta go to the bother of a light field display? There are a couple of reasons: image depth and apparent direction. A light field display can give the impression that an image is behind the display surface. So to viewers, the eyes appear to be coincident with the face of the wearer. This alleviates the effect that the user’s eyes appear to be in the wrong place. The second is more subtle and is related to a couple of effects of how images appear on flat displays. One is easy to understand: the TV screen turn effect. That is, if you draw a face on a flat panel, as a viewer moves around, they have a bias of seeing the eyes of the user on screen as if they were pointing orthogonal to the plane of the display. This means that two people looking at the screen will not agree on the direction the image of the user on the screen is looking. The biases are quite systematic and would be a real hindrance to communication if the HMD wearer were not directly in front of the person they were speaking to. This has been well-studied, including by us. Below are three systems built by Ye Pan and others, that explore how users in different directions perceive gaze and eye contact with a representation of another person. From left to right, a spherical display, a curved lenticular display and a multi-view display (based on a random-dot style display).

Spherical display, lenticular display and multi-view display. Ye Pan et al., UCL.

So Apple’s display is most like the middle of these: a lenticular display. This is a very practical choice since there are lots of lenticular displays available, so a lot is known about their capabilities. They only support one viewing direction though, and a limited number of views. They can produce apparent depth (as you can tell from many lenticular stickers or cards). However, they don’t produce variation vertically, and if the viewer or wearer tips their head over (so the viewer’s eyes aren’t along the main axis of the display) the effect will disappear.

Apple have taken on a real challenge to make this look good. It is the most interesting part of the HMD by far. I suspect that the reason Tim Cook and others weren’t filmed wearing the display is that the reconstruction and presentation is not looking as good as they want yet, and if you were to film a very recognizable person, the limitations would be much more obvious. Whether limitations matter in practice for the purpose they intend (local collaboration) is a different issue, but it is a feature that will not be easy to represent on images or videos of it.

Finally, on this aspect: the dial. I like dials. Dials are under-appreciated in HCI in general. The Vision Pro has a dial to blend between AR and VR modes (and between Eyesight modes). There is one other display in our collection with a dial, and it is very old: the Sony Glasstron PLM-S700E (circa 1998). I am still surprised that no company has taken this approach to a mixed AR/VR display. They solve the viewing awareness problem, by having the display out of the way of the wearer’s eyes (thus viewed through a half-silvered mirror) and then the external surface is an LCD panel that can go partly transparent so it becomes an optical AR display. It has a dial on the battery pack (which you would wear on a belt); I did say that there were echoes of past displays. The image below shows a fully mobile VR display we made with the Glasstron.

Sony Glasstron plugged into an iPhone making our first portable VR system (demoed at Mobile World Congress 2012!). The dial is very hard to see, but is on the top-right of the near side of the battery pack. The shiny outside of the HMD (which is very small and light, but narrow field of view), is a screen that can be switched from almost opaque to partly transparent.

 

Software

Apple has the advantage that it has lots of users that are familiar with the basics of its existing interfaces. They showed a panel with icons that looks like a transparent tablet with circular icons. It might be just me, but this looks quite dated these days, and no doubt widgets will come. Most other platforms have moved to a more media centre-like or console model with large panels with animated previews. Circular icons, clearly separated might make for good eye-tracking demos, but the edges of more packed displays will be challenging. This interface has already been simulated by others on existing displays (for example).

There was lots of polish to the software, but again nothing particularly new. They have excellent spatial scanning so support local awareness. It wasn’t clear what they were doing for a guardian system. I may write more about these separately, but otherwise what they showed is somewhat reminiscent of Hololens’s approach to pinnable apps and volumes, versus fully immersive.

Possibly the most interesting aspect of their software will be how apps will be published on their store. Their app model for iPhone/iPad is very different than the prevailing curated store model.

 

Overall, it is a very exciting package of XR functionality. No doubt we will get one and start to port our demonstrations and tools to it.

 

 

in Uncategorised /by /#permalink
23rd September 2020
23 Sep 2020

Thoughts on Facebook/Oculus Accounts

I turned this into a post because it covers many discussions I’ve had in different forums.

 

So Facebook Oculus have a new policy about use of Facebook accounts to enable new devices or for new users after October. Its new Quest 2 devices will need a Facebook account. We have lots of Quests in the lab and have used them extensively for teaching over the past year. They are really great devices with impressive capabilities packed into a very usable package.

 

Facebook urgently needs a better solution for researchers and educators going forward. 

 

I can’t see that we can make significant use of Quest 2 in our labs in the future. Yes, we will buy devices. Yes we will probably do lots of demos of them. But we will not be able to use them as the default device in teaching. It also seems that we can no longer run some of our experiments on them because we can’t guarantee to satisfy GDPR requirements.

Further, their timing of changes is absolutely awful as teaching and research programmes are set months in advance. Christmas may be coming, the Quest 2 may have to go out with Facebook accounts, but now I have to rethink the coursework for my VR module that I was going to start in 3 weeks. I expect a lot of my colleagues are also having to rethink what they were going to do.

The issues in increasing order of difficulty:

 

1) Content and history

 

We have lots of Oculus devices from DK1 onwards. We have bought lots of content. This is spread around various “lab” oculus accounts, because, you know, they weren’t always owned by Facebook. These oculus accounts now need to be given to “people” so the content can be migrated. It’s a minor issue but the idea that there is one person for one device, is really not a good match to how a lab or teaching facility works. Who should get the content? (Are there tax implications?)

Further questions:

  • Can I log into multiple devices with the same account and run content on both? I can do this now, but it surely doesn’t fit with the personal profile model.
  • If I log into multiple devices, what does the logging do?
  • Will I be able to transfer content between accounts, because what happens if someone leaves or we lose access to their Facebook account?

Perhaps we should have bought corporate devices, but they haven’t always been around, the programme has requirements that I don’t think the university can match, it is really expensive for a university and there is no store, so we can’t buy Rez Infinite. Whoever does their sales in the UK hasn’t been keen to talk to me in the past.

Apparently our old devices will keep working for a couple more years, but we can’t demo content we already own on new devices without facing this issue. This is a shame because one of the main things our lab has done over the years (20+ years) is act as a place people in the university and in the broader community can come to experience current VR technologies. We have demonstrated to 1000s of people.

 

2) Facebook identities

Why don’t I just link my oculus account to my Facebook account? (Can I link more than one?) I do have a Facebook account, but like many people I don’t use it for work at all, just family and friends from outside work (sorry to those colleagues I have been ignoring for >10 years). I personally don’t want to connect my Facebook and Oculus identities. 

That is not the big issue though: I probably can’t require students to have Facebook identities in order to work in the labs.

Perhaps legally I might be able to do this. I am not sure, and that is the main problem: I don’t have the time to ask and find out in 3 weeks. My university  has bigger problems to deal with, such as how to make sure that our online teaching functions. Earlier in the year we were advised to not use teaching tools that use external sign-ups. For example, there were a few discussions about whether we could use Zoom before we got a university-wide account.

The problems are:

  • Students might have Facebook accounts, but rightly don’t want to share their details with anyone at the university. I can probably do coursework without having to know their identities, but we are going to do our VR coursework “networked-first”, so we need some thought there. When it gets into supporting the students, TAs might have to find their identities, etc.
  • Students might not want or be able to create Facebook accounts. There are various reasons here, but the obvious one is that they are Chinese students and are currently based in China because of Covid-19 travel restrictions.
  • Students might just be concerned about data gathering.
  • Students might have accounts in a name that they can’t validate if challenged. E.G. they use a nickname or alternate identity. We don’t want to trigger anything that might cause a validation or query (see below the recent change about needing telephone number or credit card for developer access)
  • Facebook maintains the accounts. What if Facebook decides to suspend an account just before a deadline? We have no power here to intervene.

 

3) Security

I have a whole bunch of questions about security on shared devices. Here is just one to get started:

  • What if a student or staff member forgets to log out of a device?

There isn’t much security on Quests right now. I use an Oculus not a Facebook account, but presumably there are interactions that I can push through apps into the wider Facebook system if I had logged in with my Facebook account.

Perhaps someone’s account is going to get polluted with lots of irrelevant and non-characteristic information from multiple people that we show demos to. There is a minor risk to that, obfuscated by the fact that we don’t keep records of who does demos which leads to the elephant in the room, GDPR.

4)  General Data Protection Regulation (GDPR)

Once you start touching upon the issues in GDPR you enter a world of complexity that even UCL (which is sophisticated in these matters) hasn’t quite resolved:

  • Can I have a participant do an experiment in the lab on an Oculus Quest 2 logged into someone else’s Facebook account? The person with the Facebook account signed up to some data privacy agreement. The participant signed up with us and we only collect data that is allowed under a data protection agreement agreed with the college prior to getting ethics permission. What data is Facebook actually going to collect? 
  • How can we run certain types of distributed experiments out of the lab?

In our lab, this isn’t really an issue as we commonly don’t use Quests for experiments. However, we are trying to run more experiments out of the lab both because of current Covid-19 conditions, but also to gather more representative samples and increase the number of participants in our studies (join the Distributed-VR3DUI Slack to discuss)

In the lab, we should note that the risks are extremely small. It is standard practice for us not to have devices online during experiments just in case they decide to background download or even install something. But if we were online, e.g. we wanted to do something multi-user, Facebook might also gather data and that might not be allowed under GDPR. To some extent we have ignored some of the implications of this; indeed much of the HCI field has, as the same concerns would apply over, say, experiments run on smartphones. Again, the risks are tiny as we usually don’t keep any personally identifiable information for our experiments and the data would be properly anonymous, but we’ve got to that through years of experience in running experiments and refining our data collection process to minimize risks. 

Hence, Facebook really needs to support researchers by having devices, or policies that ensure we can use the devices without risks or ambiguities. Especially in an EU context. 

For example, a really simple mitigation would be confirming that developer accounts can be created that aren’t associated with specific people. Perhaps there would need to be a process to apply for these? There is a separate issue about the move to having all developers confirm a telephone or credit card, but I haven’t thought that through other than I can see that it may make someone more reluctant as it ties more information to their Facebook account.

Maybe I should just get a corporate account, but I expect that even if I had the money, it could take weeks or months to get the legalities sorted as I would need central contracts support from the university – I am not an authorised purchaser of software and licenses for the university. 

 

Overall, for this term I am not too worried. We have enough Quests that I can give each student group a Quest logged in with one of our lab oculus accounts. My experience is that some students will buy their own Quest 2s anyway having considered the Facebook account issue. We also have a lot of Vives and other systems and laptops to lend out. However, for future iterations an alternative standalone VR platform or a better solution from Facebook would be highly desirable.

 

in Uncategorised /by /#permalink
14th August 2020
14 Aug 2020

VR’s “Cat Problem”

 

This came up in an online discussion again: VR has a “cat problem”. I have been using this phrase myself for a few years, and was challenged to come up with a referencable source about it, but couldn’t find one. Please contact me by email or on Twitter (@anthony_steed) if you have another source that I’ve forgotten. So here is a definition of the cat problem.

Head-mounted VR mostly obscures your view of the real world. While you can switch to a view of the real world (e.g. on the Vive Pro or Oculus Quest), you generally don’t see your surroundings. Today, chaperones and guardian systems are ubiquitous in platform software, but we forget that this feature hasn’t been with us since these devices launched. E.G. Oculus added a guardian system in late 2016. The Quest launched with a guardian, but then it pretty much had to because being untethered there was little guide that you would walk into a wall.

Guardian system on the Oculus Quest. A wire cage appears around the user when you get to the border of your safe region, with the hand controllers poking holes through this cage.

Guardian system on the Oculus Quest. A wire cage appears around the user when you get to the border of your safe region, with the hand controllers poking holes through this cage.

The guardian usually keeps us from walking into furniture and walls, but like many users I expect, I occasionally still punch something solid, especially when playing disc golf in Rec Room.

The cat problem is neatly explained by this Tweet:

The consumer HMDs can’t track moving objects coming into the safe region, be they cats or humans. Thus the user may be unaware of danger. In principle it seems that with the prevalence of cameras on HMDs for tracking that this feature might become available in the future. It would mesh nicely with academic work that has looked at recruiting real objects as obstacles or utensils for the virtual world.

 

On the term itself. I first remembering talking about this when I met with Henrique Olifiers at  Bossa Studios to discuss some VR projects (late 2015 I think). They had early prototypes of the Vive HMDs including one with external cameras. Perhaps the term came from Henrique or one of the VR developers whose names I’m afraid I have now forgotten.

 

Anthony

 

 

 

in Uncategorised /by /#permalink
8th February 2019
08 Feb 2019

Sabbatical Part 2

I am now in Christchurch, New Zealand visiting the Human Interface Technology Lab. I am supported by the University of Canterbury’s Erskine Programme for visiting fellows.

in Uncategorised /by /#permalink
17th September 2018
17 Sep 2018

Sabbatical

I am on sabbatical during the academic year 2018-2019. I am currently at Microsoft Research in Redmond in the EPIC Group. In late January I will be at the Human Interface Technology Lab in New Zealand.

in Uncategorised /by /#permalink
11th July 2017
11 Jul 2017

MRes Virtual Reality

 

We have launched our MRes Virtual Reality for 2017-2018 entry. This MRes is designed for students who want to go into research in virtual reality, augmented reality or spatial interfaces. It is good training for anyone who wants to study a doctorate or go into industry in the area. We used to offer an earlier version of this MRes as part of our four-year EngD programme in the VEIV Centre, but this new MRes is designed to be standalone.

 

 

in Uncategorised /by /#permalink
5th May 2016
05 May 2016

IEEE VGTC Technical Achievement Award

The  2016  Virtual  Reality  Technical  Achievement  Award  has been awarded to  Anthony  Steed, Head of the Virtual Environments and Computer Graphics group at UCL Computer Science, in  recognition  of  developments  that  have  significantly advanced research, engineering and applications of virtual reality systems.

in Uncategorised /by /#permalink
4th November 2015
04 Nov 2015

ICAT-EGVE 2015

I presented a version of my Yther system at ICAT-EGVE 2015.

Pre-print of this paper.

in Uncategorised /by /#permalink
4th November 2015
04 Nov 2015

VR Club

For students and staff at UCL we run a “VR Club” on the 3rd Wednesday of each month.

Unfortunately because of numbers, this is not an open event. You will not be able to access the rooms we use without a UCL security pass! We can however host a few external visitors each month. Please contact me or David Swapp for details and an invite.

 

 

 

in Uncategorised /by /#permalink