PCL Developers blog

Jeremie Papon

email:jpapon@gmail.com
project:Modular Interactive Application for Point Cloud Data
mentor:David Doria, Marcus Hanwell, Alexandru Ichim

About me

I am currently pursuing my PhD in Computer Science at Georg-August University of Goettingen in Germany, working as a research assistant in the Bernstein Center for Computational Neuroscience under Prof. Florentin Woergoetter.

Project Description

This project aims to develop a full featured GUI for visualization and application of the Point Cloud Library’s algorithms. Interfaces will be provided for filtering, registration, surface reconstruction, model fitting, and segmentation of point cloud data. The project will leverage the modular nature of the library to create an application which is easy to maintain and extend as the library develops.

Recent status updates

Cloud Manipulations!
Monday, August 20, 2012
../../_images/gsoc126.png

Just a quick update to let you know that you can now manipulate clouds using the mouse. This allows you to do some pretty interesting things when combined with other tools, like Euclidean Clustering.

For instance, in the image below, there are three frames. The first (left-most) shows the original point cloud. I use the Euclidean clustering tool on it, which allows me to pull out the hand as a separate cloud, and select it (now shown in red in the middle). I can then use the mouse manipulator to move the hand as I please, and, for instance, have it pick up the red peg (right most frame).

../../_images/ManipulateHand.png

Everything is coming together; the core of the app is now in place... basically I just need to implement the rest of the PCL tutorials as plugins, and you can do basically everything PCL can do... with the exception of working with movies.

Handling templated clouds, Double Dispatch in cloud_composer
Monday, August 13, 2012
../../_images/gsoc126.png

So I finally decided to bite the bullet and add support for the templated classes to the GUI, rather than just using the sensor_msgs::PointCloud2. After some discussion on the dev boards and in irc (thanks to everyone for their input, especially Radu), I decided to put together a system which allows use of the templated classes even though point type is unknown at compile time. This is inherently kind of difficult, since templates and run-time polymorphism are two opposing concepts.

Before I get into the technical stuff, let me just say that the consequence of everything that follows is that for every CloudItem, we maintain both a PointCloud2 and a templated PointCloud<> object. Because of the way the model was designed to enable undo/redo, these are always synchronized automatically, since they can only be accessed through the CloudItem interface.

Everything is centered on the CloudItem class, which inherits from QStandardItem. For those who haven’t been following, the application is built around these types of items, which are stored in a ProjectModel object. There GUI is essentially a bunch of different views for displaying/editing this model. Anyways, clouds are always loaded from file as binary blobs (PointCloud2 objects). The header of the PointCloud2 object is then parsed to determine what the underlying data looks like. By doing this, we can figure out which template PointType we need, and using a nasty switch statement, can instantiate the appropriate PointCloud<T> object.

We then take the pointer to the PointCloud<T> and store it in a QVariant in the CloudItem (we also store the PointType in an enum). When we fetch this QVariant, we can cast the pointer back to the appropriate template type using the enum and a little macro. This means one can write a tool class which deduces the template type and calls the appropriate templated worker function.

One interesting thing popped up when I was doing this. The tools use run-time polymorphism (virtual functions) to determine what work function to call. That is, I manipulate base-class Tool pointers, and let the v-table worry about what type of Tool the object actually is. A problem arises with templated types though, since virtual function templates are a no-no.

To get around this, I worked out a double dispatch system - the visitor design pattern. This allows me to determine what code to execute based on the run-time types of two different objects (rather than just one, which could be handled by a vtable). The core idea here is that we first do a run time look up in the vtable to determine what tool is being executed, passing it a reference to the CloudItem. We then take the QVariant, deduce the PointType of the cloud it references, then execute the tool using the appropriate template type.

I’m sorry if none of that made sense... I’ll draw up some diagrams of how the whole application functions in a week or so, which should help alot. I’m now finishing up the code which allows manipulation of vtk actors with the mouse to be applied back into the model (with undo/redo as well).

tl;dr: Templates now work even though you don’t know the PointType at compile-time. Mouse manipulation of actors in the PCLVisualizer render window now works, and will shortly be properly propagated back to the model.

Cloud Commands: Now with 99.9% Less Memory Leakage!
Wednesday, August 08, 2012
../../_images/gsoc126.png

Today I reworked the Cloud Command classes and their undo/redo functionality so that everything gets properly deleted when the time comes. There are two different scenarios where a command needs to get deleted:

  • When the undo stack reaches its limit (set to 10 commands atm), we need to delete the command at the bottom of the stack (first in)
  • If we eliminate the existence of a command by undoing it, then pushing another command on the stack

These two cases have different deletes though. In the first case we need to delete the original items, since we’ve replaced them with the output of the command. In the latter case, we need to delete the new items the command generated, and leave the original ones alone.

Additionally, different commands need to delete data which is structured in different ways. For instance, a split command (such as Euclidean Clustering) needs to delete one original item or many created items, while a merge command needs to delete many original items or a single created item... and so forth...

Oh, on a side note, I’m not sure what I should have the app do when one tries to merge two Cloud items which have different fields. For now, I just say “Nope”... but there may be a way to combine them that makes sense. Unfortunately this currently doesn’t exist for PointCloud2 objects in PCL afaik. Anyone feel like adding some cases to pcl::concatenatePointCloud(sensor_msgs::PointCloud2&, sensor_msgs::PointCloud2&, sensor_msgs::PointCloud2&) that checks for different but compatible field types? For instance, concatenating rgb and rgba clouds could just give a default alpha value for all the rgb cloud points. This would be pretty easy to code if one did it inefficiently by converting to templated types, concatenating, then converting back...

Merge Cloud Command, Create New Cloud from Selection
Tuesday, August 07, 2012
../../_images/gsoc126.png

Just a quick update on what I’ve added.

The frustum selection now connects back to the model through an event it invokes. It can search for points if needed, but right now I’m sanitizing all clouds (no NaNs) so there’s a 1-1 correspondence from vtk to PCL. Of course I have to figure out what cloud the vtk point belongs to, but that can be done using the CloudActorMap from PCLVisualizer.

I’ve added the last undo/redo command, merging. I also added a tool which creates a new cloud from the current selection. There are now two “selectors”; a green one for points selected in the VTK window, and a red selector for points selected in the cloud browser dock. Right now all commands applied to selections work on both, but that may change in the future.

Axes Widgets, Signal Multiplexer, InteractorStyleSwitch, Rectangular Frustum Selection
Friday, August 03, 2012
../../_images/gsoc126.png

Crunch time is here, so I’ll just give a short update on the things I’ve added:

There’s an axes widget now that shows the orientation of the camera, as in ParaView.

I added a signal multiplexer class which greatly simplifies the architecture. This class allows you to specify connections without having to specify the receiving object when you make the connections. You then can switch receiving objects, and all the connections are adjusted automatically. For me, this means that I have all my GUI actions connected to the multiplexer, and then I just tell the multiplexer which project is currently visible. The GUI actions are then routed to that project, and all the other projects just idle. It also allows me to update the GUI state automatically to project state when projects are switched. Why a class like this isn’t included in Qt itself is kind of a mystery to me.

I added an InteractorStyleSwitch class which allows you to switch what interactor style PCLVisualizer is using. This means that with one function call we can switch from the normal PCLVisualizer camera nteractor style to one which lets us select sub-clouds with the mouse.

Which brings us to the last addition, the ability to select subclouds using a rectangular frustum selector. As seen in the images below, this works, and it works on multiple clouds in the scene. What it doesn’t do yet is take the selected vtkPolyData points and figure out what indices (and what clouds) they correspond to in the original PCL clouds contained in the Project Model.

That’s for tomorrow... plus the ability to split off the selected points into a new CloudItem (with undo/redo of course).

../../_images/frustum_select.png

Stay classy, PCL blog readership.

Tool Input Checking
Monday, July 30, 2012
../../_images/gsoc126.png

So I implemented a way of showing what inputs are required for a tool and if the currently selected items are a valid input for the tool. It’s a pretty simple system really; if the selected item(s) is valid input for a tool, it becomes enabled, and you can use it. If the selected item(s) don’t match what is needed by a tool, the tool is greyed out, and you get a tooltip which says why you can’t use the tool. In the case of the image below, that’s because calculating FPFH features requires normals.

../../_images/Tool_Input.png

Now I’m working on mouse selections in the QVTKWidget/PCLVisualizer view. Things like rectangular-drag and point by point.

Then I’ll add registration, the merge command, and the ability to manually shift point clouds (using the keyboard, and perhaps the mouse).

Then I’ll add as many additional tools as I can manage before GSoC ends. It’s crunch-time; it’s going to be tough to implement all the features I would like to by the middle of August. This has turned out to be a pretty large project for one person... Fortunately the end of GSoC won’t be the end of this App; I’ll be contributing one day a week this Autumn/Winter to continued development of it.

New Plugins, Deleting, Properties, a Screenie
Friday, July 27, 2012
../../_images/gsoc126.png

Another quick pre-sleep update... Voxel Grid Downsample and Statistical Outlier Removal have been added, which means the cloud modification undo/redo functionality works now. Delete undo/redo command has also been added. The only remaining command type to be added is the merge command, as would be needed for registration. This is up next.

Then on to selection tools... which may need to be implemented directly, rather then as plugins, since plugins have no way to interact with the VTK render windows (and they shouldn’t, since their processing is done in non-GUI threads). Adding additional tools now is really quite fast... though I think at some point in the coming days I’m going to polish the plugin interface a little bit, as well as clean up the items, mainly by giving them a templated way to get/set data that doesn’t rely on Qt’s awkward UserRole enum system.

I’ll leave you with a little screenshot of what it looks like now.

../../_images/CloudComposer_SS_27JUL12.png

Oh, one other thing, if you’re fooling around in the GUI, and something goes horribly wrong, please submit a bug report. The whole app is starting to become a little more complex these days, so ugly bugs that I haven’t found are bound to start cropping up.

Feel free to submit feature requests too... but I make no guarantees as to how long it might be before they can be implemented!

Clustering, Splitting clouds, Selection highlighting
Tuesday, July 24, 2012
../../_images/gsoc126.png

Just a quick update before I go to sleep... Euclidean clustering now works; this means that the cloud splitting command & undo/redo functionality is implemented now too. If you load a cloud, run clustering, it will split it into clusters and whatever remains from the original. Basic (ugly) highlighting now works too; the cloud selected in the browser on the left gets “highlighted” in the view by turning red... which is terrible. I want to do a more subtle highlighting effect, maybe some sort of glow, or tinting, but my attempts are failures so far. I played around with adding a colored ambient/diffuse light to the selected cloud actor (in pclVisualizer) but none of the lighting changes I made had any effect. I suspect I’m making some sort of noobish vtk mistake. I think I need to read through a quick vtk tutorial tomorrow morning.

To sanitize or not to sanitize?
Saturday, July 21, 2012
../../_images/gsoc126.png

So I’ve stumbled upon a design issue that I’m not sure how to handle. The problem is that some functions in PCL require sanitized input (such as FPFH), namely, they can’t handle NANs. This means I have to use either a passthrough filter or the removeNAN filter at some point. It might seem natural to do this within the plugin that needs sanitized inputs, but then the result won’t really correspond to its input cloud, ie, it will have less points. To get around this, I’m just sanitizing all input clouds when they are loaded, but this isn’t a satisfactory solution either, since it means you break up organized clouds (and modify the cloud automatically, which the user might not want to do).

So, what should I do here? Should plugins specify that they require sanitized input, and have their tool icon greyed out unless the selected input cloud is valid? What would be the best way to notify the user about why a tool is greyed out? Tooltips which give the reason it is disabled when you hover over the tool icon?

This seems like a sensible solution to me, but please, if anyone actually reads these blogs, let me know if you have a better idea, or if my solution seems foolish to you.

Oh, and more tools are coming, I promise, I’m just trying to make the plugin specification as complete as possible before I start implementing more tools... and I keep discovering things it is missing.

Changes to displaying, interaction with models
Wednesday, July 18, 2012
../../_images/gsoc126.png

I haven’t been posting much on the blog, I’ll try to keep up with this better. I’ve done some rewriting of how things are displayed. Now the item classes define their behavior with respect to the various view classes. This means that a CloudItem defines its own paint function, which the View window (which uses PCLVisualizer) just calls when it needs to paint the CloudItem (ie, when it changes, when its added/removed). Also, now when changes are made in the inspector to properties they propagate through the models and update all the other views. For now, I’m only making properties editable which don’t require another call to the actual tool functionality, just ones that change how things look. For instance, you can change the scale and level of Normals, but you can’t change the radius, since that would require recalculating them, which is an expensive operation. This may change in the future, but for now, if you want to change the radius, you just have to calculate a new set by running the tool again (with a different radius).