When we open sourced the Kinect Fusion algorithm a few weeks ago, we were not expecting so many people to join our community and start helping co-develop and improve the system. As we said in our initial announcement, the new Kinect Fusion implementation was still experimental, and it still has a lot of rough edges. However, that didn't scare off Weipeng Xu, one of our fearless new users. He saw the potential of this code and quickly integrated it with the XNA framework, creating a foundation to support the development of future augmented reality games and applications.
Augmented Reality is a challenging application domain, and there are a number of difficult tasks that must be performed in any AR system. Notably, for many applications, it's necessary to be able to construct a good 3D model of the environment and accurately track the movements of the camera in real time. Weifeng was able to use a Kinect 3D camera and PCL's experimental implementation of the KinectFusion algorithm to solve both of these problems. The result, as shown in the videos below, is a markerless AR system with the ability to handle occlusion between the real scene and virtual objects.
If you're interested in finding out more technical details about KinectFusion, check out Richard Newcombe's ISMAR 2011 publication.