Dipping elementary OS' toes into Virtual Reality
Read how I managed to render elementary OS Desktop windows in Virtual Reality using xrdesktop.
April 2021
If you like my contributions, ❤️ Sponsor Me. It would mean the world to me!
What already works
Thanks to the plugin I wrote to support xrdesktop in elementary's window manager "Gala", we are now able to mirror desktop windows to Virtual or Augmented Reality. Although the support is still in a very early stage, its already possible to interact with the windows using the Virtual Reality Controllers and doing some re-positioning stuff. On the other hand, mouse or keyboard interaction is not possible as of now and the current state has been tested in SteamVR using a HTC Vive only.
How is it done?
Mirroring the elementary OS desktop to Virtual Reality needs three pieces to work together:
The possibility to adjust some settings on how the mirroring should work
As of now, I worked on 1. and 2. So we have a neat Wingpanel indicator which allows us to enable/disable desktop mirroring to Virtual Reality:
In addition, the backend which does all the mirroring work is developed as a Gala plugin. Fortunately, there is a reference implementation for the GNOME desktop environment available. So the main work was to translate the existing C code into Vala. To do so several VAPI's for the xrdesktop libraries were needed as well as doing some OpenGL stuff in Vala.
Thankfully, xrdesktop uses GObject. That means, we were able to build most of the needed VAPI's automatically using GObject Introspection. Although there were some fixes needed upstream to get this running correclty (in gxr, gulkan as well as in xrdesktop itself).
Big thanks also to Maia Everett, who provided the awesome ValaGL project on GitHub - a jump start for OpenGL VAPI in Vala.
What's next
Mouse and keyboard support
Mouse and keyboard interaction is still missing. So this would be a natural next step to implement. Unfortunately, Gala uses libmutter-6 to do so - wherelse xrdesktop only supports libmutter-5 in libinputsynth. Another potential solution for this would be to add Wayland support. However, this may lead to a bunch of other problems on elementary.
User facing xrdesktop settings
There is no way a regular user can adjust xrdesktop's settings in elementary OS yet. I think the best way to do so, would be to develop a Switchboard Plug with the UI needed to tinker with the settings.
Add support for Monado
Monado is the reference runtime for the OpenXR specification. At the time I started development, SteamVR provided the better end user experience and was much more stable. Therefore I chose SteamVR over Monado to jump start initial work. However, in the long rung we really want to support open software - therefore getting things working on Monado is definitely something we need to work on.
Last updated
