PCL Developers blog

Justin Rosen

PCL Weblog and Scratchpad | My Roadmap - WIP

email:jmylesrosen@gmail.com
web:justinrosen.com
project:Out-of-core multi resolution visualization of massive point clouds.
mentor:Steve Smith, Trimble

About me

I’m a Technical Director at DreamWorks Animation working on “How to Train Your Dragon 2”. I’m currently in the Crowds department supporting the needs of production artists, pipeline and crowds technology.

Prior to DreamWorks I worked at CIS a visual effects studio in Vancouver. While there I worked with many talented veterans as a Pipeline/Show TD helping integrate and develop a preexisting pipeline at a new studio for feature film production.

Before moving to Vancouver I worked at Nanotoon Entertainment as a Lead/Supervising TD on the film “Molecules to the Max!”, an IMAX feature animated film. I helped setup and develop for the first time a naive visual effects pipeline. In the process I touched all aspects of production including the development of molecular visualization software for use in Maya.

I graduated from Rensselaer Polytechnic Institute (RPI) in 2005, with a dual major in Computer Science and Electronic Media Arts and Communications. My senior year I had an internship with Nanotoon working on “The Molecularium” with some of the most inspiring people which threw me into the world of computer graphics and visual effects.

Recent status updates

Pressing On
Wednesday, April 25, 2012
../../_images/trcs4.png

Back from vacation and getting into some of the more exciting stuff that is required to get our out-of-core viewer up and fully functional. There has been quite a bit of talk on implementation and this week I plan on adding three important features:

  • Frustum Culling - Providing a way to query the octree for nodes on disk given the camera frustum. This will provide a good start in determining what data needs to be streamed in.
  • Threading - All data processing needs to happen in a separate thread as to not block the main UI.
  • Caching - Implement some type of Least Recently Used(LRU)/Least Frequently Used(LFU) cache to store streamed data and discard the less relevant.

I’ve spent most of my time recently learning and understanding VTK and getting the new mapper working with Radu to get this integrated into the pcd_viewer. The current viewer doesn’t handle large datasets and this integration will allow for much heavier data sets in realtime.

The new VTK classes are far from finished and will require quite a bit of work to handle all the data VTK can throw at it. I’ve sent Marcus, a core VTK developer, my work in progress in hopes to get some help on proper and stable integration with VTK. Here are some of the todo items still to be addressed.

vtkVertexBufferObject - Vertex Buffer Object Wrapper

  • Needs to support both VTKs float and double data types
  • Needs to support indices via GetVerts and *vtkIdType
  • First implementation supports vertices, indices and colors. Need to add all of VTKs attributes, i.e. normals, textures, ...

vtkVertexBufferObjectMapper - Vertex Buffer Object Rendering Wrapper

  • Need the ability to set vertex, fragment and geometry shader (Currently uses simple shaders)
  • The current mapper has vertex, indices and colors VBOs. Need to support all of VTKs attributes, i.e. normals, textures, ...
  • Determining whether to pull point/cell data and if colors are present on the data passed in.
  • Handle all VTK data types to make sure calls to glVertexAttribPointer are correct and consistent.
vtkVBOPolyDataMapper Commit - WIP
Tuesday, April 03, 2012
../../_images/trcs4.png

I’ve committed a working version of my vtkVBOPolyDataMapper that the outofcore_viewer is now using. This is the working version from my previous post.

I started breaking out the VBO functionality into the separate class vtkVertexBufferObject that’ll be used by the mapper, vtkVBOPolyDataMapper. I’m running into a few issues I need to resolve before committing and getting feedback from Marcus at VTK and the PCL group. The interface for the new vtkVertexBufferObject should be quite simple and handle the basic VTK objects. Will post another update once I’ve got this all working.

New Commits and More on Visualization
Thursday, March 22, 2012
../../_images/trcs4.png

I’ve updated the pcl_outofcore_process runtime to dump numerous pcd files into an octree. There are currently a few issues with this tool that need to be resolved:

  • numpts isn’t written to the json file
  • Given that numpts isn’t written it’s hard to tell if all points specified within the pcd files are getting written to disk
  • Should we store bounding box info within the header of a pcd file so an entire cloud doesn’t require parsing?
  • Should we write a pcd reader that can iterate through the points so the entire cloud doesn’t have to be loaded into memory? This’ll be useful when loading up clouds in the millions+

I’ve committed the octree_viewer from my last post with very basic VTK functionality. This required some additional methods that already existed within the pcl_octree. Although, I’ve noticed that the voxels displayed aren’t square, which I think should be. I’ll have to look into this further.

On to the more exciting things!

I’ve been in talks with Radu and Marcus Hanwell, one of the developers on VTK on how we should move forward with our visualizations tools. The current version of VTK is based on OpenGL display lists, which is a very old technology deprecated in 3.0, removed in 3.1 with support via the ARB_compatibility extension and a compatibility profile introduced in 3.2. That being said, there are classes within VTK that use the newer technology, just not the pieces we’re interested in.

Why is this important? Well, because writing an out-of-core viewer doesn’t make much sense with display lists and requires a more dynamic implementation.

As VTK 6 evolves various newer OpenGL features will be integrated, vertex buffer objects (VBOs) among them. Until then, I plan on helping Marcus get these new features in faster by helping prototype and possibly develop the classes required. The good news is the VTK guys now have immediate testers with a simple test case, billions of points!

I’ve started to hack together a new VTK mapper vtkVBOPolyDataMapper to replace the vtkPolyDataMapper. I have a very barebones version working. It’s got a long way to go, makes quite a few assumptions and’ll need some love to work generically in the VTK framework. I’ll post more on this when I update the octree_viewer with the newer functionality.

Visualization with VTK
Wednesday, March 07, 2012
../../_images/trcs4.png

I’ve started to wrap my mind around VTK and the PCL Visualizer. I wrote an application similar to the octree_viewer using straight VTK. The following is a processed outofcore cloud with 4 LODs.

../../_images/OutofcoreViewer01.png
Urban Robotics Octree Framework
Tuesday, February 28, 2012
../../_images/trcs4.png

The following diagrams the general framework PCL received from Urban Robotics for use in the out-of-core project. The diagram is broken up into three parts:

  • Creation

    • Points of type PointT are added to the octree_base data structure. This data structure is in charge of managing the underlying child nodes and subsequently divided data.
    • As the points are subdivided, octree_base_nodes are created containing a random subsample or LOD of the points that are contained within each node (branch). These nodes are in charge of managing bounding box and meta data on disk and hold payload data read from and written to disk, but doesn’t handle the lower level read/writes.
    • Once a max depth or leaf node is reached a container type is created to manage disk or ram access. These are currently the only types of containers available within the framework.
    • The disk containers handle the low disk I/O
  • Directory Structure

    • At the top level of the directory structure lives a .octree file containing the octree depth or LOD, number of points at each LOD and various other bits of meta data. This maps to the octree_base.
    • Each directory from the top level root directory maps to an octree_base_node. Each node directory contains a .oct_idx file providing a nodes bounding box and LOD data. Leaf nodes have no children (child directories) and are found at the max depth of the tree providing access to the original payload data (Not a subsample).
  • Query

    • When reading or querying the tree the octree_base provides an interface to the underlying data structure.
    • Querying the tree is accomplished by providing a bounding box that intersects with the underlying octree_base_nodes. These octree_base_nodes provide access to the point data via containers or filepaths containing the binary point data.
../../_images/UrbanRoboticsOctree.png

Stephen and I have been documenting and refactoring the underlying the code and are at a point where we can start investigating some of the more interesting features to be implemented.

In addition I’ve started to commit tools that’ll be useful in the processing of pcd files for use in the framework.