Today I host another amazing article by the VR ergonomics expert Rob Cole, which has already written on this blog amazing posts, like the series about the ergonomics of the Valve Index or the viral article about the amazing Caliper VR controllers he worked on by himself.
A few weeks ago, he wrote here a detailed article on the Vive Trackers, and now he is completing it with this new article, which is a long and detailed story of his experimentations with the Trackers and their use cases, and especially their big potential for full-body VR. It’s going to be a very interesting post, so read it until the end!
Disclaimer: HTC provided three of their Vive Tracker 3.0 units for my use in these articles.
What is an HTC Vive Tracker?
As described in my first HTC Vive Tracker article earlier this year:
“Vive Tracker is a wireless, battery-powered SteamVR tracked accessory that provides highly accurate, low latency 6 Degrees of Freedom (6DoF) motion tracking within a roomscale environment.”
Or in HTC’s own words:
“Track movement and bring objects from the real world into the virtual universe”.
Whilst the first part of this article looked at the history of the HTC Vive Tracker and its three different versions, the second part will focus on some practical experiments with the latest HTC Vive Tracker 3.0
First though, a quick dive into the more technical side of the HTC Vive Tracker 3.0 for those who are interested in how these things work.
Vive Tracker tracking
The HTC Vive Tracker 3.0 is a SteamVR-compatible device, using Valve’s second-generation (2.0) base station tracking technology. To understand the overall concept, the first generation of 1.0 base station tracking was described in this way by Valve
“How It Works: The SteamVR Tracking Basestations sweep the room with multiple sync pulses and laser lines, reaching out to about 5 meters. By keeping careful track of the timings between pulses and sweeps, the SteamVR Tracking system uses simple trigonometry to find the location of each sensor to within a fraction of a millimeter. By combining multiple sensors, 2 basestations, as well as adding a high-speed IMU (inertial measurement unit), SteamVR also calculates the tracked object’s orientation, velocity, and angular velocity, all at an update rate of 1000Hz.”
In April 2018 HTC introduced the Vive Pro headset, Pro controllers, and Vive Tracker 2.0 each fitted with next-generation optical sensors and updated electronics. Second-generation base station (2.0) tracking had evolved by eliminating one rotor and encoding the beam. This technical difference is well described by Redditor /u/Crozone:
“The laser will encode the base station ID as well as the current laser sweep angle. There is no longer a ‘timing pulse’, the base station does the timing internally and sends the angle directly to the Vive on the laser. This means that whenever the Vive detects a laser on any sensor, it already knows the base station that is sending that laser, as well as the exact angle of that laser from the base station. This also means that the base stations don’t need to sync at all, and the Vive doesn’t even need to time anything.”
Introduced in March 2021, Vive Tracker 3.0 is a smaller, more power-efficient evolution of the 2.0 Vive tracker, providing longer battery life and less mass / inertia when quickly moving around.
HTC Vive Tracker 3.0 has 18 optical sensors, carefully positioned across a specially designed shape that evolved from the original Vive controller, and has since been shrunk in size. The shape of the Vive Tracker 3.0 maximizes the number of non-co-planar sensors that the base station rays can illuminate, to provide highly accurate (sub mm) roomscale 6DoF tracking that is highly resistant to occlusion.
It’s not uncommon to obscure several sensors during movement in roomscale sessions: having 18 sensors at different angles aims to minimize this problem, and works very well in the real world.
Each sensor is hidden behind an optical window made from a special plastic manufactured to reduce stray visible light and minimize the effect of reflections whilst providing a clear optical path for infrared rays emanating from the base stations.
With HTC Vive the sensor window has been part of their design language since the first Vive SteamVR kit in 2016, this was continued with Vive Pro, Vive Trackers, and Vive Pro 2.
While holding the Vive Tracker it’s very easy to see the sensor locations placed across the entire device. Valve and Pimax SteamVR products instead have subtle sensor windows with optical material co-molded into a structural casing: sometimes a feint circle is all that can be seen.
Each SteamVR tracked device has “known physical geometry” which includes the exact position and orientation of each optical sensor relative to the object’s physical center, creating a “sensor mesh” within the physical bounds of the tracked object.
The image above, clearly an homage to the legendary Unity Cube, demonstrates the position and orientation of optical sensors across different surfaces of the object… this would ensure that sensors are always available as the object is turned or moved.
Controllers, headset, and trackers are individually calibrated to a high degree of accuracy after manufacturing, as small variations exist when manufacturing plastic casings and when placing individual sensors into the casing. For example, the calibration data of a single sensor is this one:
model Points [ -0.00322245085, -0.0423639342, 0.110276416 ]
model Normals [ -0.0380270146, -0.700318217, -0.712817252 ]
“Points” are 3-dimensional co-ordinates for the sensor, whilst “Normals” are the relative orientation of the optical sensor’s centroid (light-facing surface). A device that is not properly calibrated can suffer from poor tracking performance or failure to boot. I’ve read that initial sensor placement needs to be within 1mm of the plotted Point location for SteamVR’s tool to work with during the calibration phase (this is done post-assembly).
This high degree of geometric accuracy allows the host computer to easily determine the exact orientation and relative position of the tracked object because it can compare incoming angular data against the geometry of the tracked object to determine its position in the roomscale 3-dimensional space, and its orientation within that space.
The image above shows optical sensor hits being detected on the Vive Tracker from a 2.0 base station on the wall above the playspace. If the Tracker is turned, sensors become occluded, causing the data to change in real time. The locations of the optical windows for each sensor can clearly be seen across the casing of the Vive Tracker.
Optical sensor hits are combined with direction, acceleration, and gravity data from an onboard IMU (inertial measurement unit). The IMU itself is a 9DoF unit, operating at a much higher polling rate (1000hz) than the optical sensor system (100hz). The IMU used on the Vive Tracker is common to 6DoF tracked objects, using a 9-axis motion tracking device. An example is described by market leader TDK of their InvenSense IMU:
“InvenSense’s 9-Axis devices combine a 3-axis gyroscope, 3-axis accelerometer, and 3-axis compass in the same chip together with an onboard Digital Motion Processor™ (DMP™) capable of processing the complex MotionFusion algorithms.”
However, IMU devices suffer from small data errors which start to compound over time: without correction (world reference) IMU based devices tend to “drift away” from their starting point, or become disorientated, something experienced with older 3DoF VR systems which required regular recalibration.
The lighthouse optical system of SteamVR therefore provides an angular “world reference” at 100hz for 1000hz IMU based systems, providing error correction for IMU drift with an accurate provision of position and pose across a large dimensional tracked volume.
The image below shows a Vive Tracker outputting live gyro and accelorometer data using a function in “Lighthouse Console” (a command line interface). If you move physically the device around you would see the data immediately changing to reflect your real-world movements.
The data is sent wirelessly back to the host PC through a USB-mounted Vive “Dongle”. The Dongle communicates with a wireless chip inside the Vive tracker through a proprietary version of Nordic Enhanced Shockburst on the 2.4 GHz frequency. As described by the manufacturer:
“Enhanced ShockBurst (ESB) is a basic protocol supporting two-way data packet communication including packet buffering, packet acknowledgment, and automatic retransmission of lost packets. ESB provides radio communication with low power consumption, and the implementation is small in code size and easy to use.”
SteamVR tracking operation is overall very efficient: it minimizes calculation, power, and communications load on the host PC. Another big advantage of lighthouse base station tracking is no direct connection is required between the base stations and the host PC, since the base stations are self-contained “dumb” emitters.
This means that your PC can be in another room, if your cabling is long enough, and you can put multiple tracked devices into that room, each using the same base station to provide accurate tracking. Base Station 2.0 also supports up to four base stations in one room, providing up to 10 m x 10 m coverage.
The JSON
When each SteamVR device is built and calibrated, all the data is written to a file Valve calls “the JSON”: effectively the on-board driver, the JSON is unique to each different tracked device.
Each tracked device has a unique serial number generated by an onboard CPU, whilst the file contains highly accurate geometry of all device sensors, USB device identity, IMU setup, and system information for SteamVR to understand what the device is and how to interact with it.
Finally, it’s common to see some “firmware” references in the JSON. On the Vive tracker this seems to be related to power management whilst on other hardware like Index controllers it’s mainly related to finger tracking and grip sensors. I’ve broken different areas into colors to make it easier to understand:
- RED: Device identity for USB interface
- ORANGE: SteamVR light house information
- GREEN: Firmware (“embedded” programs in hardware)
- YELLOW: Head orientation (orientation)
- PURPLE: IMU (gyro) setup fully calibrated
- AZURE: Channel Map (directory of physical connections to sensors)
- LIGHT GREEN: ModelNormals (orientation of each sensor) fully calibrated
- LIGHT PINK: ModelPoints (three-dimensional position of each sensor) fully calibrated
One particular item of interest for my experiments is the render model reference (in the orange section), because it tells SteamVR where to look for a file stored on the host PC that provides the render model that is seen as the virtual representation of your device in VR. If we change the render model we can tell the Vive Tracker to display something else that doesn’t look like a Vive Tracker.
Caution: You can download the JSON from any SteamVR device using Lighthouse console, but make sure to store its unique JSON somewhere safe and don’t get it mixed up with other JSON files.
Uploading an improperly configured or corrupted JSON file, or trying to fix an issue by copying the JSON from another device (i.e. another Vive tracker) will tend to stop the device from working, as this person on Reddit recently found out:
“Hello all, In a desperate attempt to fix a bugged tracker, I flashed a 3.0’s config onto a 2.0 and now SteamVR won’t find the 2.0 tracker at all. It shows up as a device, but is stuck in searching?”
Attaching the Vive Tracker
As mentioned during my first article, the Vive Tracker 3.0 is attached using 1/4” screw thread camera style “docking mechanism” with stabilizing pin (ISO standard ISO 1222:2010).
Docking Mechanism
-1/4” Screw nut to fasten the accessory.
-Stabilizing pin recess for constraining the tracker from rotating.
-Pogo pin port (spring contact-type) for optional electrical connection to the accessory.
-Friction pad to provide a surface with friction between the accessory and VIVE Tracker (3.0)
Electrical pogo pins
The bottom of the Tracker has six “pogo pin” contacts allowing electrical signals to pass between the Tracker and any compatible peripheral equipment. This allows hardware developers to place suitable buttons on tracked objects and use the Tracker to send this data back to the PC.
HTC’s Marios Bikos: “The HTC VIVE Tracker allows you to not only track objects in VR (getting tracker pose in real-time) but also use the POGO pins to simulate input buttons (Grip/Trigger/Trackpad/Menu) as if you were using a Vive controller.”
The image below shows a layout of the pogo pins, with Pin.3 as the Grip button, Pin.4 as the Trigger button, Pin.5 as the Trackpad button, Pin. 6 as the Menu button.
Over the years since the release of the Vive Tracker, a number of enthusiasts and hobbyists have made pogo pin connectors for the Tracker, and there are many examples found online showing the different approaches taken. To use the Vive Tracker with active pogo pin inputs you need to follow the advice, again from Marios Bikos:
“The only way to get input from the Vive Tracker’s POGO pins in this scenario is to change the tracker role to “Held in hand” and select Left/Right depending on how we plan on using the Vive Tracker.”
There is a wealth of community information found online about development with the Vive Tracker, but the best starting point is perhaps to look at HTC’s Vive Tracker Developer guidelines which can be found here: https://developer.vive.com/documents/850/HTC_Vive_Tracker_3.0_Developer_Guidelines_v1.1_06022021.pdf
If not using the pogo pins, for example using a Tracker for full body tracking where it’s not attached to an object with electrical inputs (but a soft strap for body attachment), then it’s attached using a suitable 1/4” screw and rubber pad, ideally with stabilizing pin to stop the tracker from rotating around the mount.
If no pin is available then rotating the tracker into the rubber pad provides enough friction it probably won’t come loose even if dancing frantically in VR Chat.
Dongles and cabling
Each Vive Tracker requires its own USB Dongle, as each unique Tracker and Dongle are paired together during the controller pairing procedure in SteamVR.
It’s now possible to have up to 9 trackers running though this will probably cause USB havoc on most PC systems, so some proper research would be required to get the right USB cables and a good PCI-E card.
Each Vive Tracker is supplied with a USB cable, Vive Dongle and Dongle base; the USB cable is 1.2 meters length, to provide some distance between your PC and the Dongle, in an effort to minimize electrical interference from the PC.
However, this may not be enough to get the Dongle into your playspace, depending on your VR setup. For my domestic setup, I have a 3m x 3m playspace but my PC sits outside the room on a trolley with the headset tether passing under the door.
The standard 1.2 meters cable (right side, image above) is much too short to get the Dongle from my PC, under the door and into my playspace. This was quickly solved using a five meters long active USB 2.0 cable extender left over from my ownership of an Oculus Rift CV1 with a 3-camera tracking setup.
After purchasing another active USB 2.0 cable extender of the same style from Amazon for £10, it was easy to use the USB extension cables to place two Vive Tracker dongles right into my playspace, by passing under the door alongside the headset tether. The extension cables are shown at the top and on the left side in the image above.
The use of an active extension cabling ensures a stable USB connection over a longer cable run, as you don’t want USB instability causing your Trackers to keep disconnecting. Your mileage with USB stability will vary depending on the quality of your motherboard and its ability to support multiple connections; you may need a PCI-E card or similar solution if having USB connectivity issues.
The third Vive Tracker dongle was easily connected by plugging it into the “Frunk” of my Index headset. For non-Index owners, a third USB extension cable would be required. If your PC is inside your playspace you may not need cable extensions, but they could allow you to optimize the placement of your Dongles without compromising the position of your PC, so well worth looking at for a small cost.
With all three Vive Tracker dongles finally connected, my PC recognised them as USB devices. Now it was time for the setup in SteamVR.
Pairing in SteamVR
Vive Trackers are paired just like Vive or Index controllers: under “devices” and “pair controller”, you will find the Vive Tracker option, so just follow the easy-to-understand on-screen instructions.
Several seconds later you should find your Tracker paired with a Dongle and glowing steady green (as seen in the images below). If it does not pair, make sure your Dongle is within reach of the Vive Tracker, to enable them to easily find each other. For pairing problems, there is a lot of technical information online both from HTC and the SteamVR community.
Once each Tracker was paired (a total of three in my case) I could see all operational SteamVR devices in the menu shown below. I’ve labeled these in different colors to highlight what type of device each one is, whilst small icons make it easy to identify at a glance.
Going further into the menu reveals “Manage Trackers” which provides a list of the fourteen different tracker locations and options. These mirror the body locations (i.e. left and right foot) as well as providing single locations for the chest and the waist.
The different wearable (body) options are demonstrated by our wooden friend below, showing the high degree of precision that is possible given enough Vive trackers and USB bandwidth:
In addition to the body-worn locations, you can also choose:
- Disabled (ignore)
- Camera
- Keyboard
My three active Vive Trackers are shown in the image below, with locations assigned for full body tracking in VR Chat and similar applications. I’ve scrubbed off the serial numbers of the trackers but usually, this is something like “LHR-A8368513”. You can see the roles assigned to each tracker: “Left foot”, “Waist”, and “Right foot”.
First time – What do you see?
As soon as you put your headset on and try pausing SteamVR home, you will immediately see any active Vive Trackers shown as high-resolution render models with glowing indicators. If you are using a Vive Tracker assigned as “held in hand”, you can hold it, rotate it, move it around which is very cool and immediately cements the idea of a “tracked object” in the user’s brain.
The images below were shot through the lens of my Index headset… I apologize for the distortions, but it’s hard to photograph the dual compound lens with a smartphone! The left image shows a single Vive Tracker with Index controllers whilst the right image shows a waist-mounted Vive tracker as well as left-and-right-feet mounted Vive Trackers.
Unpausing SteamVR will cause the Vive Trackers to disappear from the view inside your headset, and you won’t see them again unless pausing or using an application designed to work with them, usually swapping their render model for an in-game object (for example a gun or tennis bat).
The practical experiments
I tried a bunch of different stuff including:
- Attaching to bought objects
- Finding some games to play
- Attaching to a custom object
- Full Body Tracking setup used in VR Chat.
Attaching to bought objects
Well, this plan came unstuck quickly as it wasn’t possible to buy a Vive-tracker-compatible gun in the UK anymore. Both Amazon and Ebay redirected me to sellers in the USA, but by the time International shipping and sales taxes were added it was getting close to £100, which was getting a bit expensive for playing a few rounds of Arizona Sunshine.
Even Computer Exchange didn’t have anything second hand, though their stock constantly changes so I set up several stock alerts on their website. Other brands that offered assault rifles and pistols for Vive Trackers had discontinued their products, with no current alternatives available.
There seemed to have been a flurry of peripheral activity in this space during 2016-2017, following the launch of the HTC Vive, before it went cold probably due to limited sales.
Even if every SteamVR customer had bought a Tracker and peripheral in the first 2 years, that would still have been a very small user base in the grand scheme of hardware sales. Tooling up injection mold tooling and assembly isn’t inexpensive even for a simple gun design.
These guys are still offering their gun for $899, but with the following somewhat confusing caveat (quoting exact words):
- The offer is Vive Gun VR + tracker 1.0 version= 100% perfect GamePlay.
- Due to latest version vive tracker 2.0 released in 2018, it’s not compatible with Vive gun perfectly, that missing function below: 1- Unusable with trackpad function on vive gunVR. 2- Shorter battery life on tracker 2.0 and can’t charge by vive gunVR directly in game playing.
Looking further into this, some reviews online complained about compatibility problems with later updates of SteamVR breaking something that was never fixed. I’m assuming this relates to the pogo pin inputs used by the different brands and models of gun, but it was unclear exactly what had gone wrong.
“1.0 out of 5 stars. Reviewed in the United States on 27 December 2019. Waste of $160 for the gun and tracker. Doesn’t work with any games.”
Some users mentioned having to use an earlier version of SteamVR to get their guns to work, though this then didn’t seem to work for Valve Index headset users who required a later build of SteamVR.
With the relatively high pricing and considerable hassle of returning an imported gun if it didn’t work properly, I didn’t waste any further time looking into this. I had fond memories of using a Vive-tracked gun in co-operative Arizona Sunshine at a VR Arcade some years ago, and left it at that!
Anyone for Tennis?
The other option for an easily bought object was “HTC’s Racket Sports Set” which they’ve stopped selling on Amazon and only seem to sell directly as a special item. I did look at buying the Tennis and Ping Pong game “Virtual Sports” (which comes free with the set) but it seemed exclusive to Viveport as it wasn’t available on Steam.
The Racket Sports set contains 2 different-sized plastic props with realistic grips, there is no “pogo pin” connection as it doesn’t require button input (you would use a Vive or Index controller to launch and control the application).
The paddle and racket provide a much closer feeling of weight and inertia compared to waving a controller about and removing vulnerable, expensive controllers from the playspace in favor of using a tougher / cheaper plastic racket and Vive Tracker.
PADDLE
- Weight without Tracker: 4.9 oz.
- Size: 6″ W x 10″ H x 1″ D
- Connection: 1/4-inch UNC threaded mount (standard tripod mount)
- Safety: Lanyard strap
RACKET
- Weight without Tracker: 10.8 oz.
- Size: 4.75″ W x 15.75″ H x 1.5″ D
- Connection: 1/4-inch UNC threaded mount (standard tripod mount)
- Safety: Lanyard strap
Looking at the list of Racket Sports Set compatible games, you can see the majority are from the early days of the HTC Vive consumer launch, releasing in 2016 and then 2017. Some have not been updated since launch, whilst others have received a couple of updates.
Thankfully I found all these titles available on Steam for people who don’t have Viveport access. This list shows the release date and current pricing.
- Holoball, 2016, £10.99, free demo
- Racket NX, 2018, £15.49
- Selfie Tennis, 2016, £7.19
- Racket Fury, 2017, £15.49
- Blobby Tennis, 2016, free of charge
- VRSquash, 2017, £14.49
- First person Tennis, 2019, £19.49
- Paddle Up, 2016, £6.99
To try using a racket game with one of my Vive Trackers, I downloaded demos of Racket NX and Blobby Tennis, then I built myself a racket for the Tracker using a spare ping-pong bat. This is what happened.
I couldn’t get the Racket NX demo to work properly if just using a Vive Tracker. With no trigger, I couldn’t actually start the game. It also seemed to require a trigger action to “pull” the ball toward the bat.
It was easy to play Racket NX using an Index controller and it was fun, but it wouldn’t allow me to have an active Index controller in one hand and the Vive Tracked bat in the other, it just ignored the bat completely. I tried adjusting this problem in the controller settings, but no controller settings were actually available. However the full game is sure to feature settings and much more so it could be worth revisiting.
Mr. Blobby Tennis
I used a Valve Index controller in my left hand to throw the ball, and the Vive Tracker fitted in the bat held in my right hand to control the tennis racket. This looked very basic but felt really fun with a “blob” opponent on the other side of the net occasionally returning my terrible serves.
Despite me holding a wooden handled ping pong bat with a tracker attached, the in-game tennis racket didn’t feel too dissimilar due to the feel of “holding a handle” instead of gripping a plastic motion controller body at a strange angle.
I then decided to lengthen the handle of my “bat” to make it feel more like a tennis racquet which immediately helped things feel more balanced.
As soon as I got used to using my longer paddle I started to become more proficient, sometimes making a dozen returns before hitting one straight into the net or splicing it off at a random angle.
Despite the solid ping-pong bat somewhat obscuring the Vive Tracker from the second base station, I had no issues with tracking, which was surprising. The first base station had clear line of sight of the Vive Tracker on the bat, whilst rapid hand movements provided a rich stream of IMU data, especially acceleration.
Rays from the second base station would have hit some sensors on the sides of the Tracker as the bat angle changed during my movement. A “hollow” (cut out) bat paddle would have exposed more sensors on the Tracker to the second base station, which is what HTC has done with their plastic racket set.
Quickly though, I realized my 3 x 3 meters playspace was a little too small and I stopped after hitting my front wall with my bat, and then my overhead light fitting (2.5 meters up). I didn’t want to risk damaging the Vive Tracker, so I called an end to my Tennis game and put my bat away.
For players wanting to get into ping pong or tennis, there seems to be a decent selection of older titles with Tracker support for use of practical bats and rackets. Overall I thoroughly enjoyed the sensation of using a tracked bat and could see myself buying the HTC Racket Set for a more durable setup with two size options.
Attaching it to a custom object – AIM Controller
After the disappointment of not being able to buy a gun for playing Arizona Sunshine, I found a PSVR AIM Controller on sale for just £15 in my local gaming store. It seemed like an ideal time to make my own custom object. The AIM controller was a very cool peripheral built for Sony’s PSVR system. Using an AIM in “Farpoint” was an amazing experience, providing a tactile feel and much improved controls with many different input options.
I downloaded a Vive Tracker render model from HTC’s developer site and bought a digital model of the AIM Controller from a digital model storefront for $12. These assets were imported into CAD software where they were aligned to create a hybrid of the two.
In the physical world, I completely stripped down the AIM controller which turned out to be a simple shell predominantly empty but stuffed with input components at key locations. After removing two haptic motors, a rechargeable battery, and numerous flexes, it was ready to accommodate the Vive Tracker.
I’d already decided to mount the tracker on top of the gun, to present the most visible aspects to the base stations for good tracking. After quickly manufacturing a low-profile mounting plate from thermoplastic, I attached the tracker and took an accurate measurement of its position for use later on.
After getting things aligned and adjusting the render model file using the PC, it was time to carefully download “the JSON” using the lighthouse console. This command downloadconfig triggers the download of the JSON to your host PC.
The updated JSON was uploaded to the Vive Tracker using uploadconfig. Now when starting SteamVR Home, if pausing SteamVR, the AIM+Vive Tracker controller is seen instead of the original Vive Tracker render model.
The next step would be to connect a trigger and buttons through the pogo pins, to see if it’s possible to get it fully working with the current version of SteamVR. It should also be possible to fit one of the haptic motors (ERM) when triggering haptic events to provide basic rumble feedback for a game like Arizona Sunshine. However, I ran out of time before this article was due, so I thought about doing this properly as a fun project in the New Year.
Full body tracking setup used in VR Chat
Full Body Tracking (FBT) has become essential for many users of the popular social media application “VR Chat”. Whilst other applications are available, VR Chat is the best known and best populated with many users able to engage using desktop PC as well as VR headsets.
VR Chat has a thriving scene with people using Avatars compatible with full body tracking, and there is a growing community of creators making and selling elaborate Avatars to the VR Chat community. There are also thousands of free Avatars available so it’s quite easy to find something you like the look of and take an Avatar for a test drive.
VR Chat is perfectly usable with a regular VR headset and controllers, but this only provides “three points” of tracking (head, left hand, right hand) to drive an Avatar.
Taking these three points and applying a software model allows a calculated position for the avatar arms, pelvis, and lower limbs known as IK (inverse kinematics).
From developer Runevision: “Inverse kinematics (IK) is a way to automatically calculate the joint angles of e.g. a leg or arm based on where the end effector (typically the foot or hand) should be. So you could specify that the hand of a person should be at some specific position, and IK can then be used to calculate the joint angles of the shoulder and elbow in order to make the hand be there.
Inverse kinematics need special algorithms to “solve” the problem of finding the right joint angles. This is not trivial, especially if a limb has more than two segments. (A human leg would have two segments while a horse hind leg would have three.)”
More information about Inverse Kinematics can be found here https://docs.unity3d.com/Manual/InverseKinematics.html
However, IK tends to go wonky when users move about especially the hips, leading to strange uncomfortable sensations when the Avatar legs are going in a different direction to the user’s own legs. Elbows are another well-known problem area, leading many game developers to use a “hand” or “glove” rather than presenting a full arm to the user.
For effective Full Body Tracking in SteamVR, we can add a tracked hip/waist and tracked left and right feet using three SteamVR trackers. The image above from Terrel Strong at the University of Washington was from a detailed study into IK and body tracking.
Adding the Vive trackers provides three additional points of tracking, providing a total of six (including HMD and two controllers) points that can drive the avatar skeleton with control on each lower limb and the waist. It is possible to add extra trackers in SteamVR, up to nine total to further tighten up skeletal control for the fullest body tracking experience available.
Equipment used
- SteamVR-compatible headset (Valve Index)
- SteamVR controllers (Valve Index)
- VR capable PC
- SteamVR account (free)
- VR Chat (free) on Steam
- Three SteamVR Trackers (HTC Vive Tracker 3.0 with USB dongles)
- USB cable extenders (if required)
- Tracker straps (as required)
Strapping up: Waist
The Vive Trackers come with 1/4” threaded screw mounts in their bases, but require mounting on straps or a belt so they can be attached to the body.
Of course, many different companies make Vive-Tracker-compatible straps, belts, and equipment, but being short on time (couldn’t wait for delivery) I decided to make my own.
The waist strap was made from a spare belt and some thermoplastic I molded into a suitable mount, after embedding a 1/4” bolt to let me screw the Tracker into place.
A long 3mm machine screw was used as a stabilizing pin to prevent the Tracker from trying to rotate under movement. The rear of the plastic mount was faced with industrial-grade velcro, I then added a thick foam pad to make it very comfortable against the stomach.
The waist mount worked out great and didn’t move around, feel uncomfortable or cause any tracking issues. The smaller and lighter Vive Tracker 3.0 helped in this respect with less inertia when quickly moving about.
The feeling of having a tracked pelvis really opens up body movement in VR… this was a similar revelation to using the free Decagear application for Smartphones and Windows PC which added tracked hip direction in games like Boneworks with great success.
The Vive Tracker takes this to the next level, tracking the pelvis in 6DoF adding a big dimension to body interaction and feelings of immersion.
Strapping up: Feet
For my feet, I decided to make simple shoe mounts with velcro straps and padding. The tracker was attached using a 1/4” threaded fitting taken from some home cinema speaker wall mounts, these fittings had flat plastic bottoms so they fitted snugly against the top of the shoe.
Wearing shoes was best in terms of stability, they let me strap the Trackers without being excessively tight, the mounts were stable and easy to get into in seconds. I used a long velcro strap on each foot to wrap over the built-in strap, adding another layer of security against unwanted movement.
Despite doing some terrible dancing and lots of random movement, the feet mounts never caused any trouble with the Vive Trackers behaving very well during all my VR Chat sessions. I also tried strapping them straight to my feet without shoes, and found socks were comfortable but the Trackers tended to move around a little unless strapped really tight, so preferred using shoes (these have very thin soles so you can feel your tether as necessary).
Getting Started: Avatar time
On launching VR Chat, you choose an Avatar. There are lots of different free models available with full body tracking support. I tried using lots of different avatars including a toy rabbit piloting a mech suit, a red-eyed robot, a suave fox, and a small furry in cyber gear (which has a bugs bunny style face under the visor).
Each Avatar has a “feel” to it when using FBT, and I found I really liked using the toy rabbit in the mech suit because it felt right in terms of scale and movement, whilst the toy rabbit driving the suit made small body and mouth movements; giving a strange sense of a puppet being puppeted.
The small furry gave the worlds a different scale, making normal-sized avatars tower over my rabbit. I soon noticed a lot of small and very small Avatars in different worlds during my journey through VR Chat.
The suave fox was welcome in worlds inhabited by furries (seen in the picture above carrying my drinking companion, a duck, don’t ask!) and the red-eyed robot was really precise for seeing the full body tracking at work with its stripped-down skeleton ideal in that respect.
Calibration – VRChat
It’s very easy to calibrate full body tracking in VRChat: you just open the user menu in game, select “FBT Calibration” and then adopt the “T” shape (arms out at 90 degrees).
You should now find your Avatar is fully calibrated to your tracking setup. If you need to recalibrate, just use the menu. Once calibrated, you will suddenly notice that fine movement is tracked very accurately, including something unprecedented… LEGS!!
“Hey Mark, we’ve got some legs over here…”
Despite all the recent hype about the Metawhatsit, and how we don’t have tracked legs yet in VR… well… I moved my real legs and my Avatar legs immediately reacted, my brain nearly fell out of my head with the sheer surprise of it all 🙂
After taking off my headset to recover, I checked online and it seems that “tracked legs” have been possible since 2017 using SteamVR, so does perhaps Meta need to double-check their data? If they want a demonstration, they can let me know: I have a “SteamVR” room where they can try out this awesome new technology 😉
So the first impression of having legs was quite striking, though not fully appreciated until standing in front of a “mirror” where suddenly it all made an immediate sense and explained why people spend all that money on full-body tracking setups.
A current limitation with single-focal length headsets is VAC (vergence accommodation conflict) where your eyes find it difficult to resolve very close objects in sharp focus.
This means that looking directly down at your avatar’s upper body is somewhat obscured by the issue of VAC making it harder to focus clearly on your torso or upper arms, but when standing in front of the mirror the focal distance of the headset is as intended making things much sharper.
Making minute movements that are precisely tracked in real-time driving your avatar gives a very strong connection in the brain, feeling akin to a mild electric shock, just like when I first used the HTC Vive Pre in 2016. This was experienced every time I used Full Body tracking.
As many VR enthusiasts will sadly attest to, that electric excitement of being in VR during those early weeks or months soon fades as you become accustomed to using VR, making that desirable feeling of strong presence less regularly attainted. Some users report it becomes too obvious they are staring at two screens strapped to their heads, despite stereoscopic view and 6DoF tracking.
I thought a lot about this whilst using Fully Body Tracking, and went back to Skarredghost’s recent interview with the legendary Jeremy Bailenson:
“There’s a joke I tell at cocktail parties, and it’s not really a joke because it’s not funny at all, but if you’re going to ask me, what are the five most important features that cause presence in VR? My answer for you is going to be tracking, tracking, tracking, tracking, tracking. I’ve nothing against resolution… it’s always nice to see a high res image, and, of course, field of view is important, but if you’re going to inspire presence, the thing that’s most important is low latency, high frame rate, high accuracy of tracking (which is really important)”
I’ve read many comments from people still happily using the original HTC Vive system for VR Chat, in their experience having very accurate Fully Body Tracking is much more important than having a higher resolution headset. It’s also easy to pick up used Vives at low prices, providing an inexpensive route to FBT (Full Body Tracking) for cash-strapped enthusiasts. Just be aware that base station 1.0 and 2.0 technology doesn’t always mix.
To demonstrate FBT from my own perspective, I took my “toy rabbit in the mech suit” avatar for a spin and made some different poses in front of an HQ mirror in VR Chat. At the same time, photographs were taken of what I was doing at exactly the same time in the real world (these have been flipped horizontally to match the mirror image from VR Chat).
Subtle body language seems to translate straight through the avatar which is thrilling stuff. The use of full body tracking with sub-millimeter precision really helps with immersion and experiencing strong feelings of presence again.
Being able to lie down, stand up, roll over, karate kick, stand on one leg, and dance about felt really liberating despite the tether. Dancing in a mirror alongside other FBT users was really vivid with all the subtle body movements doing a very convincing job on my subconscious lizard brain.
As soon as you start using FBT regularly you start to notice other user Avatars and who has FBT or who is using a standard VR system with three points of tracking. The difference in the body language is night and day, with the FBT users very lifelike whilst standard users look somewhat stiff and unnatural. No judgment or elitism here, just an observation. I tried using VR Chat a couple of times without the Vive Trackers and something was definitely missing…
Pricing
HTC Vive Tracker 3.0 are available for a recommended retail price of £129 in the UK from different electronics retailers or from HTC directly at https://www.vive.com with free shipping.
For some reason, they are found to be a little less expensive through HTC’s Amazon store at £127.50 including free shipping and free returns.
Alternative SteamVR tracker-style devices are also available from Tundra Labs and Manus (aimed at the professional market).
Conclusions
The second part of this article was a lot of fun, and a learning experience about the power of full body tracking to dramatically increase immersion leading to stronger feelings of presence on a repeatable basis, which for a long-term PCVR user was a revelation. I’m late to the FBT party but better late than never, and the Vive Tracker 3.0 is a useful improvement on the previous generation, so very easy to recommend. The software on the SteamVR side seems to have matured with good Tracker support making it easy to set up and manage. I had no issues during weeks of use no matter how I moved or what I did.
I went into this expecting to have lots of fun building tracked objects and exploring different games, but truthfully the revelation of FBT meant more time was spent in VR Chat where it was easy to put FBT to good use. Other applications are available, so look for FBT support when checking out new software.
There are sports games that will work with the Vive Racket Sports set, and if you are into tennis, ping-pong, or similar sports then it should prove well worth the purchase of a Vive Tracker and a suitable bat. Whether guns currently work I cannot confirm nor deny, but I’ll try to build out my tracked AIM gun in the New Year, so I guess I’ll find out then.
As I mentioned in my first article, the use of FBT for applications like VR Chat seems to be driving healthy sales of Vive Tracker (I met a lot of FBT users) whilst peripherals are thin on the ground and not supported in many (if any) contemporary games.
What about the costs?
It’s not a secret that it’s relatively expensive to set up full body tracking, if doing it properly with SteamVR and tracker-style devices.
You have to start with a SteamVR system like Valve Index (£920) or Vive Pro (currently £719 on sale), then add three Vive Trackers (£387 with shipping) and some Tracker straps (typically £40-70 for set), and of course a VR capable PC.
Are there alternatives?
For people using VR Systems that are not SteamVR native (like Vive or Index) it’s possible to add base stations and SteamVR controllers and then combine the Quest/Pico/WMR and SteamVR playspaces using an application like Open VR Space Calibrator, which you can find at this link: https://github.com/pushrax/OpenVR-SpaceCalibrator. It is described by the developers as:
“This helps you use tracked VR devices from one company with any other. It aligns multiple tracking systems with a quick calibration step. It may not work for your setup, but there are many cases that work to a degree, and some work very well.”
However, results may vary, with some users reporting it’s almost unusable whilst others mention it works but regular recalibration is required as the two tracking systems tend to drift apart. For this reason, some users will swap to a SteamVR headset (even an old Vive) to provide a reliable and hassle-free Fully Body Tracking solution.
There are other full-body tracking systems on the market, using multiple IMU, external cameras, magnetic fields, or other tracking technology. They don’t tend to be very accurate, or suffer from drift, but do provide some access to body tracking at a more affordable price than going the base station route. Truly a case of “you get what you pay for”…
However, as esteemed boffin Jeremy Bailenson said, “what are the five most important features that cause presence in VR? My answer for you is going to be tracking, tracking, tracking, tracking, tracking.”.
From my experience with the HTC Vive Tracker 3.0 during these months, they will certainly provide what you need for FBT and sports gaming, providing that all-important feeling of presence. Highly recommended!
Thanks! Big thanks to HTC for providing the Vive Trackers and answering all my questions, thanks to Tony Skarredghost for hosting my article, and most of all thanks to you the reader.
(Header image by Rob Cole)