Read how I managed to render elementary OS Desktop windows in Virtual Reality using xrdesktop.
elementary OS is rendering its windows in Virtual Reality
If you like my contributions, ❤️ Sponsor Me. It would mean the world to me!
What already works
Thanks to the plugin I wrote to support xrdesktop in elementary's window manager "Gala", we are now able to mirror desktop windows to Virtual or Augmented Reality. Although the support is still in a very early stage, its already possible to interact with the windows using the Virtual Reality Controllers and doing some re-positioning stuff. On the other hand, mouse or keyboard interaction is not possible as of now and the current state has been tested in SteamVR using a HTC Vive only.
How is it done?
Mirroring the elementary OS desktop to Virtual Reality needs three pieces to work together:
Thankfully, xrdesktop uses GObject. That means, we were able to build most of the needed VAPI's automatically using GObject Introspection. Although there were some fixes needed upstream to get this running correclty (in gxr, gulkan as well as in xrdesktop itself).
Mouse and keyboard interaction is still missing. So this would be a natural next step to implement. Unfortunately, Gala uses libmutter-6 to do so - wherelse xrdesktop only supports libmutter-5 in libinputsynth. Another potential solution for this would be to add Wayland support. However, this may lead to a bunch of other problems on elementary.
User facing xrdesktop settings
There is no way a regular user can adjust xrdesktop's settings in elementary OS yet. I think the best way to do so, would be to develop a Switchboard Plug with the UI needed to tinker with the settings.
Add support for Monado
Monado is the reference runtime for the OpenXR specification. At the time I started development, SteamVR provided the better end user experience and was much more stable. Therefore I chose SteamVR over Monado to jump start initial work. However, in the long rung we really want to support open software - therefore getting things working on Monado is definitely something we need to work on.