I took a second look at the video and I think the tracking may be drifting because it seems as though most of the sensors are being covered with your hand and controller. Because of the size of the pucks, it's pretty easy to obscure the sensors, and unfortunately, the more sensors you obscure, the less tracking quality you're going to get. How are you planing to mount these trackers for your experience?
Regarding the offset, can you shoot a top down video of all of the devices inline with the virtual view? Also make sure the firmware is updated on all of your devices, including base stations.
FOO