FragmentVR review and how to: an useful utility to see your real world from inside SteamVR

Have you already found yourself inside a virtual reality experience with the willing of having a window open to the real world so to be able to be aware of your surroundings, so to be able to find your keyboard or to keep an eye on your baby? Well, then FragmentVR is the right program for you.

FragmentVR is a little utility for SteamVR that lets you do exactly what I specified above: open, inside your virtual world, some windows that let you see the real world around you. These windows will exploit the frontal camera of your headset to show you the stream of the real world. Of course, to work properly, it needs that you own a headset featuring a frontal camera and at this time, the only available one is the HTC Vive.

But how does it work? Well, it is quite simple:

1. You get FragmentVR on Steam;
2. You open it inside your Steam app (you won’t find it under the category “Games”, but under “Software”;
3. At this point, you should see a big window saying that FragmentVR has been added to your dashboard;
4. Put your headset on. If you see your camera stream in front of your eyes, go directly to step 10, otherwise, go on following these instructions;
5. Open the dashboard using the button on your controller;
6. You will find that there is a new button, the one for “Fragment VR”. Select it;
7. You will be prompted the FragmentVR dialog. Here you can set some options, whose names are self-explanatory. For instance, you can:
   • Change the transparency of the FragmentVR windows;
   • Show these windows only when you open the dashboard and not in game;
   • Hide the Fragment points in the real world;
8. Click on “Recreate fragment”;
9. Exit the dashboard;
10. You see your camera stream rectified in front of your eyes (if you don’t see anything, close everything and re-start SteamVR: it has happened to me the first time). Use your controller to select 4 vertices of a quadrilateral that will be your window to the real world. You just press the controller every time that you want to add a vertex, there is no need to draw completely the figure in the air;
11. Now you have a 3D window on your real world. Enjoy!
12. In case you want to change the window, go back to step 5.

As you can see, it is pretty easy. The tool is beneficial for various situations and in my opinion, the one of opening a window to constantly see your keyboard and mouse is one of them.

It is also great for eating while you are in VR :). I think that it can have various applications: it is not a complete passthrough mode that requires you to pause the app (as the one that I use on the Vive Focus), but just an unobtrusive window that lets you take a glimpse of the reality around you. If you put it in the right position, it can make you consider both the real and the virtual world at the same time.

I find interesting that the window that you are going to open is a 3D one in the virtual world. So, it is not attached to your headset, but it is fixed in 3D space. Thanks to some projection matrix, the program tries to show you the exact part of the 2D stream of the camera that matches the quadrilateral that you’ve drawn, from whatever perspective you’re looking at it. This 3D positioning has some advantages (the images are not shown on an annoying quad attached to your eyes), but also some disadvantages (if you move in a position where you don’t see the real window anymore, the program becomes useless). I also appreciated the precision, in the sense that the real and virtual world are well matched: for instance, the controllers in the camera feed and in the virtual world are almost in the same position.

https://gfycat.com/LazyConstantAlaskanmalamute

The tool is in early access and will probably remain in this state for six months, according to the author. For this reasons, it is currently very rough: there is no tutorial (I’m writing this post for people like me that had no idea on how to use it); the UX of the options dialog is very basic; sometimes the camera stream disappears; the initial camera stream has a low FPS; the points are not set by the controller in the position where you would expect them; etc… I’ve spoken with the author and he has planned various updates to the experience to add support for other headsets like the Vive Pro, to improve the UX and fix the bugs and also to make possible the opening of more than one window. So, with time, it will improve a lot.

It is a quite interesting tool, even if it still needs some refinements. If you are interested in using it, you can find it on Steam for 4.99€/5.99$. I think that it should cost a bit less, especially at this stage. Anyway, if you need something like that, 5€ can be worth the expense.

P.S. Some months ago I’ve met the people that later on would have developed this application and they offered me a lot of sushi for lunch… should this be considered a sponsored post, then? 😀 😀 😀

(Header image by FragmentVR)