Multi-user VR Displays Or… What is my PhD about in simple words.

TL;DR: Different ways have been attempted to generate multiple images from the same display for different users across the years. In this post I wrote about six categories that try to multiplex imagery from the same display and if the reader is interested a paper on the state of the art on different ways for multiplexing images up to 2018 is shared at the end (or also grab it here).

 

This topic is part of my PhD thesis and I have been wanting to write about this for the past few weeks so people in general can understand what I am working on without too much technicalities.

The core idea of my thesis is to be able to show different images to different users in the same display. As simple as it sounds. One could naively argue, “Well Juan, that already exists, TVs nowadays offer to split the screen so I can watch a channel while my friends watch other things  at the same time.” Which I can reply to you. Well, yes… But the idea is to display images from the same display without other users being able to see the other user’s views. Like person A being able to watch discovery channel on the TV while person B at the same time is watching something else on  Netflix without each other being able to see what the other person is watching.

Another question that one can ask could be: Why would this be important?, why would people care?. Turns out that a really bright person in a paper (Pollock[1]) found that collaboration times get significantly increased when users are at different locations vs center of perception. This means when people are trying to work in groups looking at screen in 3D when they cannot perceive 3D images crafted with correct perspective depending on where they are relative to the screen; collaborations get significantly longer and perspective distortions prevent precise collaboration scenarios.

Ok Juan, but  Where would this be useful? This is also another good question; Pretty much everything that involves collaboration and screen projected VR (Yes VR is not only Head mounted displays) :). Some examples could be surgery simulation training, Mechanical design, general teaching, training and pretty much everything that involves collaborative object manipulation / discussion.

Pretty much what the PhD thesis aims to do is when different people point certain parts of a 3D object, they point to different places in the physical space. In other words;  what I want is several people coincide in the physical space when pointing to things.

With this, collaborative scenarios would result in correct perspective views for each of the participants like this: (And yes, you could eventually watch Netflix and different other things at the same time  with this ;)).

So  the design challenge lays on “How can we create a system that displays multiple images in a common area occluding all but the appropriate image for each user?”.

Sounds like magic, right? Well… Turns out this has been researched for long time and there have been several attempts that try to achieve this. Another bright researcher in another paper (Bolas[2]) proposes four categories for generating multiple images from the same screen for different users. These categories are based on the approach used for generating images and are: Spatial barriers, Optical Routing, Optical Filtering and Time Multiplexing. additionally to these categories I think is worth adding volumetric displays and light field displays.

 

Spatial Barriers.

Pretty much they use the display’s physical configuration and user placement in advantage to generate specific views. Depending on the approach used, brightness is reduced as a result of the physical barriers, resolution also can be affected due to generating images in the same screen.

Some examples of this approach are the use of parallax barriers, the Protein interactive Theater[3] or the illusion Hole[4,5] among other projects:

 

Optical Routing:

This approach uses angle sensitive optical characteristics of certain materials to direct or occlude images based on the viewer’s position, color Banding, lose of resolution, moiré patterns, are some of the challenges of this approach.

Pretty much everything that routes light follows this approach; some examples include lenticular lenses (like the cards that used to come on cereal boxes that you flip and display different images), holographic screen materials that reflect light depending on where users are, etc.

Optical Filtering:

This category involves systems that filter viewpoints using light’s electromagnetic properties. The most common used devices for this category are polarization filters (like the 3D glasses for viewing 3D movies), anaglyph filters, etc. Pretty much anything that filters light or manipulates light’s wavelength. Brightness reduction, color washing/loss are some of the problems associated with this approach.

Time Multiplex:

This category encompasses solutions that use time sequenced light and shutters to determine which users sees an image at a given point in time. Brightness reduction, flickering, crosstalk, hight refresh rates requirements are some of the challenges of this approach. Normally this approach involves highly custom circuitry for syncing shutters and high refresh systems to allow image multiplexing.

One of the most prominent projects are Bernd Froehlich’s[11,12] multi-viewer stereo displays; here he uses a projector per eye for each user and with shutter glasses and shuttered projectors he sequences between users to generate 3D views and separate users.

 

 

 

Volumetric Displays:

In volumetric displays. the image is produced within a volume of space. This content is always confined within the physical device enclosure. normally imagery is displayed  with spinning mirrors, screens, translucent discs, stacked LEDs on a grid etc. Resolution is one of the biggest drawbacks and the volumetric content normally is not possible to interact directly but with virtual wands.

On the bright side, the content can be seen from any point of view and there is no need for special glasses to be used to perceive the rendered images.

Light field Displays:

Finally, in light field displays; the light emitted from a point on screen varies with the direction hence the generation of different views. Without going into too much detail; this is an active area of research, unfortunately there are no solutions out of the box and mostly everything is made from scratch; there is a small field of view (~10deg) from where the users can see and processing times to generate frames are prohibitive for realtime rendering.

Which approach is better?

No approach is better than other. Depending on the needs of your project, one approach would work better than other; one thing to not is that hybrid approaches try to exploit the best from different techniques and are commonly used to either generate stereoscopy or increase the number of users.

Parallax methods face challenges regarding brightness (with some exceptions) Optical routing approaches still pose some drawbacks that can be solved with today’s technology with increased resolution on displays among others. Optical filtering approaches on the other hand are very limited by the nature of their filters for separating users and also brightness is another factor that gets affected. Time multiplexing is an interesting technique that with fast enough refresh rates can increase number of views, still added synchronization, custom circuitry and shuttering nature of the approach poses new challenges to be solved like brightness and flickering among others.

Volumetric displays have challenges of resolution and occlusion. Unfortunately with the nature of the displays, users cannot pinpoint inside the rendered volume. Finally, Light field displays is an active area of research. One of the biggest challenges this approach faces is that there are not out of the box solutions to work on. Long processing times for generating frames, reduced viewing angles also offer a lot of room for research.

If you managed to read up until here and you are still interested on the state of the art up to 2018 on multi user displays, feel free to read this paper that I published in Electronic Imaging 2019 in the Stereoscopic Displays & Applications conference (SD&A http://www.stereoscopic.org/)

References

  • [1]. Pollock,B., Burton, M., Kelly, J. W., Gilbert, S., & Winer, E. (2012). The right view from the wrong location: Depth perception in stereoscopic multi-user virtual environments. IEEE transactions on visualization and computer graphics, 18(4), 581-588
  • [2]. Bolas, M.,McDowall, I., & Corr, D. (2004). New research and explorations into multiuser immersive display systems. IEEE computer graphics and applications, 24(1), 18-21.
  • [3] W. (1998, May). Designing and building the pit: a head-tracked stereo workspace for two users. In 2nd International Immersive Pro- jection Technology Workshop (pp. 11-12).
  • [4] Kitamura, Yoshifumi, et al. ”Interactive stereoscopic display for three or more users.” Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 2001.
  • [5] Kitamura, Yoshifumi, et al. ”The illusionhole with polarization filters.” Proceedings of the ACM symposium on Virtual reality software and technology. ACM, 2006.
  • [6] Nguyen, D., & Canny, J. (2005, April). MultiView: spatially faithful group video conferencing. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 799-808). ACM.
  • [7]Nguyen, D. T., & Canny, J. (2007, April). Multiview: improving trust in group video conferencing through spatial faithfulness. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 1465-1474). ACM.
  • [8] Jorke, H., & Fritz, M. (2006, February). Stereo projection using interference filters. In Proc. SPIE (Vol. 6055, p. 60550G).
  • [9] Jorke, H., Simon, A., & Fritz, M. (2008, May). Advanced stereo projection using interference filters. In 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video, 2008 (pp. 177-180). IEEE.
  • [10] Jorke, H., Simon, A., & Fritz, M. (2009). Advanced stereo projection using interference filters. Journal of the Society for Information Display, 17(5), 407-410.
  • [11] Froehlich, B., Hoffmann, J., Klueger, K., & Hochstrate, J. (2004, May). Implementing multi-viewer time-sequential stereo displays based on shuttered lcd projectors. In 4th Immersive Projection Tech- nology Workshop, Ames, Iowa (May 2004).
  • [12] Froehlich, Bernd, et al. Implementing multi-viewer stereo displays. (2005).
  • [13] Favalora, Gregg E., et al. ”Volumetric three-dimensional display system with rasterization hardware.” Photonics West 2001-Electronic Imaging. International Society for Optics and Photonics, 2001.
  • [14] Jones, A., McDowall, I., Yamada, H., Bolas, M., & Debevec, P. (2007). Rendering for an interactive 360 light field display. ACM Transactions on Graphics (TOG)26(3), 40.
  • [15] Lanman, D., Wetzstein, G., Hirsch, M., Heidrich, W., & Raskar, R. (2011). Polarization fields: dynamic light field display using multi- layer LCDs. ACM Transactions on Graphics (TOG), 30(6), 186.