PCL Developers blog

All blog posts

Wrap-up posting for 3D edge detection
Tuesday, August 28, 2012
../_images/gsoc121.png
../_images/screenshot-1341964387.png

My primary goal of GSOC‘12 was to design and implement a 3D edge detection algorithm from an organized point cloud. Various edges are detected from geometric shapes (boundary, occluding, occluded, and high curvature edges) or photometric texture (rgb edges). These edges can be applicable to registration, tracking, etc. Following code shows how to use the organized edge detection:

pcl::OrganizedEdgeFromRGBNormals<pcl::PointXYZRGBA, pcl::Normal, pcl::Label> oed;
oed.setInputNormals (normal);
oed.setInputCloud (cloud);
oed.setDepthDisconThreshold (0.02); // 2cm
oed.setMaxSearchNeighbors (50);
pcl::PointCloud<pcl::Label> labels;
std::vector<pcl::PointIndices> label_indices;
oed.compute (labels, label_indices);

pcl::PointCloud<pcl::PointXYZRGBA>::Ptr occluding_edges (new pcl::PointCloud<pcl::PointXYZRGBA>),
        occluded_edges (new pcl::PointCloud<pcl::PointXYZRGBA>),
        boundary_edges (new pcl::PointCloud<pcl::PointXYZRGBA>),
        high_curvature_edges (new pcl::PointCloud<pcl::PointXYZRGBA>),
        rgb_edges (new pcl::PointCloud<pcl::PointXYZRGBA>);

pcl::copyPointCloud (*cloud, label_indices[0].indices, *boundary_edges);
pcl::copyPointCloud (*cloud, label_indices[1].indices, *occluding_edges);
pcl::copyPointCloud (*cloud, label_indices[2].indices, *occluded_edges);
pcl::copyPointCloud (*cloud, label_indices[3].indices, *high_curvature_edges);
pcl::copyPointCloud (*cloud, label_indices[4].indices, *rgb_edges);

For more information, please refer to following codes in PCL trunk:

It was a great pleasure to be one of the GSOC participants. I hope that my small contribution will be useful to PCL users. Thank Google and PCL for the nice opportunity and kind support. Lastly, thank Alex Trevor for mentoring me.

Benchmarking PNG Image dumping for PCL
Thursday, August 23, 2012
../_images/gsoc1211.png

Here is the result we got from PNG dumping benchmarking: 640x480 (16 bits) depth map + 640 x 480 (24 bits) color image.

08-23 20:57:43.830: I/PCL Benchmark:(10552): Number of Points: 307200, Runtime: 0.203085 (s)
08-23 20:57:54.690: I/PCL Benchmark:(10552): Number of Points: 307200, Runtime: 0.215253 (s)

If we are dumping the result to the /mnt/sdcard/, we are getting:

08-23 21:02:23.890: I/PCL Benchmark:(14839): Number of Points: 307200, Runtime: 0.332639 (s)
08-23 21:02:40.410: I/PCL Benchmark:(14839): Number of Points: 307200, Runtime: 0.328380 (s)

There is a significant overhead (about 0.1 second!) with the SD Card I/O.

We shall verify these with a faster SD card. The /mnt/sdcard seems to be mounted onto an internal SD Card on the Tegra 3 dev board that I have no access to? I tried to open the back and so already.

Also, I have tried different compression levels and it seems that level 3 is giving the best compression ratio vs the speed. More plots will come next to justify my observations.

Code Testing
Tuesday, August 21, 2012
../_images/gsoc1211.png

Have gone through the whole setup again and replicated on Ubuntu 12.04 + Mac OSX 10.8 environments. The README file is now updated to reflect what is needed to have the environment setup.

At the end, it was only 4 scripts, and maybe we can automate these completely. We have also added the scripts for dumping the images store in the SD Card. check out all .sh files in the directory and that may save hours of your time.

Cloud Manipulations!
Monday, August 20, 2012
../_images/gsoc126.png

Just a quick update to let you know that you can now manipulate clouds using the mouse. This allows you to do some pretty interesting things when combined with other tools, like Euclidean Clustering.

For instance, in the image below, there are three frames. The first (left-most) shows the original point cloud. I use the Euclidean clustering tool on it, which allows me to pull out the hand as a separate cloud, and select it (now shown in red in the middle). I can then use the mouse manipulator to move the hand as I please, and, for instance, have it pick up the red peg (right most frame).

../_images/ManipulateHand.png

Everything is coming together; the core of the app is now in place... basically I just need to implement the rest of the PCL tutorials as plugins, and you can do basically everything PCL can do... with the exception of working with movies.

Drag and Drop(dnd) Support and More Filters
Sunday, August 19, 2012
../_images/gsoc1212.png

dnd support inside the scene tree is added, the users can drag and drop point cloud items to different render window, which will help them observe the point cloud. It will be nice to add more dnd support, for example, drag point cloud directly from the file system to the modeler app, there can be many such nice features, and I will add them after GSOC. And new filters can be added very easily to the current framework. But there seems an unknown exception thrown from file save module, which crashes the app, I will anaylse what’s the problem and finish file/project load/save things.

A new point type to handle monochrome images is now available on the trunk!
Saturday, August 18, 2012
../_images/gsoc124.png

I’ve developed a new point type to handle monochrome images in the most effective way. It contains only one field, named intensity, of type uint8_t. Consequently, I’ve updated the pcl::ImageViewer so that it is able to manage the point clouds related to the new point type, named pcl::Intensity. Finally, I’ve changed the PNG2PCD converter by adding more functionalities: now the user can choose if the written cloud should be based on pcl::RGB or on pcl::Intensity. For more information, please see the documentation related to each single class or file.

Lossless Image Dumping with libpng + Android and Tegra 3
Friday, August 17, 2012
../_images/gsoc1211.png

Today, I’ve added the support of libpng + libzlib for the Tegra 3 project and so we can dump the raw images from the Kinect (or any OpenNI supported devices) onto the SDCard for post-processing or debugging. After hours of fiddling with the parameters and hacking away on the code, now we can capture and compress 4-6 images per second (2-3x 24-bit RGB image + 2-3x 16-bit depth image) on a Tegra 3. I believe these libraries are already NEON optimized and thus we shall be getting the best performance from them. Here is the little magic that gives me the best performance so far.

// Write header (16 bit colour depth, greyscale)
png_set_IHDR(png_ptr, info_ptr, width, height, 16, PNG_COLOR_TYPE_GRAY, PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_DEFAULT, PNG_COMPRESSION_TYPE_DEFAULT);
//fine tuned parameter for speed!
png_set_filter(png_ptr, PNG_FILTER_TYPE_BASE, PNG_FILTER_SUB);
png_set_compression_level(png_ptr, 1); //1 is Z_BEST_SPEED in zlib.h!
png_set_compression_strategy(png_ptr, 3); //3 is Z_RLE

Next step, if time permitted I will use the PCL library compression code instead. Using the libpng, however, has taught me where the critical paths are and how we shall handle the data. Right now, I am sure that I wasn’t introducing any overheads from the data copying or manipulations. I was handling the raw data pointers the whole time.

For the longest time, I have had trouble getting any performance out from the Tegra 3, mainly because of the floating point operations! Again, avoid these operations at all cost unless we have a more powerful processor!

Here is a screenshot of some of the images that were dumped from my Kinect in real-time!

../_images/png_dump_screenshot.jpg
Tutorials, bug with pcd_viewer, PCLVisualizer and important additions
Thursday, August 16, 2012
../_images/gsoc127.png

Wrote tutorials for the 2D classes. Got another weird bug found by Radu. Plotter not working in pcd_viewer on point picking. Still struggling in it.

Tried to make PCLVisualizer cleaner and readable. Removed some unnecessary function calls. Didn’t commit yet.

Added some important functionalities in Plotter and a seperate vtkCommand event handler.

Placement for ICP Registration
Wednesday, August 15, 2012
../_images/gsoc1212.png

The initial position and orientation can be tunned in a dialog triggered by double clicking the point cloud item ICP registration, as shown in the following snapshot. And ICP paremeters are exposed to give more control to the users. I will add more workers, add project support and document the functions next.

Handling templated clouds, Double Dispatch in cloud_composer
Monday, August 13, 2012
../_images/gsoc126.png

So I finally decided to bite the bullet and add support for the templated classes to the GUI, rather than just using the sensor_msgs::PointCloud2. After some discussion on the dev boards and in irc (thanks to everyone for their input, especially Radu), I decided to put together a system which allows use of the templated classes even though point type is unknown at compile time. This is inherently kind of difficult, since templates and run-time polymorphism are two opposing concepts.

Before I get into the technical stuff, let me just say that the consequence of everything that follows is that for every CloudItem, we maintain both a PointCloud2 and a templated PointCloud<> object. Because of the way the model was designed to enable undo/redo, these are always synchronized automatically, since they can only be accessed through the CloudItem interface.

Everything is centered on the CloudItem class, which inherits from QStandardItem. For those who haven’t been following, the application is built around these types of items, which are stored in a ProjectModel object. There GUI is essentially a bunch of different views for displaying/editing this model. Anyways, clouds are always loaded from file as binary blobs (PointCloud2 objects). The header of the PointCloud2 object is then parsed to determine what the underlying data looks like. By doing this, we can figure out which template PointType we need, and using a nasty switch statement, can instantiate the appropriate PointCloud<T> object.

We then take the pointer to the PointCloud<T> and store it in a QVariant in the CloudItem (we also store the PointType in an enum). When we fetch this QVariant, we can cast the pointer back to the appropriate template type using the enum and a little macro. This means one can write a tool class which deduces the template type and calls the appropriate templated worker function.

One interesting thing popped up when I was doing this. The tools use run-time polymorphism (virtual functions) to determine what work function to call. That is, I manipulate base-class Tool pointers, and let the v-table worry about what type of Tool the object actually is. A problem arises with templated types though, since virtual function templates are a no-no.

To get around this, I worked out a double dispatch system - the visitor design pattern. This allows me to determine what code to execute based on the run-time types of two different objects (rather than just one, which could be handled by a vtable). The core idea here is that we first do a run time look up in the vtable to determine what tool is being executed, passing it a reference to the CloudItem. We then take the QVariant, deduce the PointType of the cloud it references, then execute the tool using the appropriate template type.

I’m sorry if none of that made sense... I’ll draw up some diagrams of how the whole application functions in a week or so, which should help alot. I’m now finishing up the code which allows manipulation of vtk actors with the mouse to be applied back into the model (with undo/redo as well).

tl;dr: Templates now work even though you don’t know the PointType at compile-time. Mouse manipulation of actors in the PCLVisualizer render window now works, and will shortly be properly propagated back to the model.

Openni interface for 3D edge detection
Friday, August 10, 2012
../_images/gsoc121.png

An openni interface for 3D edge detection is added in PCL trunk. Once it starts, you can show and hide each edge by pressing the corresponding number:

  • 1: boundary edges (blue)
  • 2: occluding edges (green)
  • 3: occluded edges (red)
  • 4: high curvature edges (yellow)
  • 5: rgb edges (cyan)

The high curvature and rgb edges are not enabled for fast frame rate, but you can easily enable these two edges if you want to test.

A new tool for PNG to PCD conversions is now available on the trunk!
Friday, August 10, 2012
../_images/gsoc124.png

I’ve developed a simple utility that enables the user to convert a PNG input file into a PCD output file. The converter takes as input both the name of the input PNG file and the name of the PCD output file. It finally performs the conversion of the PNG file into the PCD file by creating a:

pcl::PointCloud<pcl::RGB>>

point cloud. Now, the PNG2PCD converter is available on the trunk version of PCL under the tools directory.

Supervised Segmentation
Friday, August 10, 2012
../_images/gsoc122.png

The supervised segmentation is a two step process. Consisting of training phase and segmentation. In the training phase we extract the objects from the scene. We use the FPFH features as classifiers and as a prior assignment of the unary potentials of the CRF. We compute the FPFH histogram features for all points in one object. To reduce computation and feature comparisons in the recognition step we use a k-means cluster algorithm and cluster the feature into 10 classes. The training objects can be seen in the following image.

../_images/training.png

In the segmentation and recognition step we use the learned features to assign prior probabilities to a new scene. The prior assignment of the most likely label can be seen in the following image. As you can see in the image, many of the points of the objects we want to segment and recognize are not labeled correctly. This is because the distance of two FPFH features are two far apart. However, as a first initial estimate, FPFH features are well suited. The advantage of using these features is the fact that it only captures the geometry of the features and not color information. Whit this the training data set can be much smaller.

../_images/prior.png

As a second and to refine the assignment we use the fully connected CRF. The following image shows the segmentation and labeling after 10 iterations.

../_images/seg.png
Cloud Commands: Now with 99.9% Less Memory Leakage!
Wednesday, August 08, 2012
../_images/gsoc126.png

Today I reworked the Cloud Command classes and their undo/redo functionality so that everything gets properly deleted when the time comes. There are two different scenarios where a command needs to get deleted:

  • When the undo stack reaches its limit (set to 10 commands atm), we need to delete the command at the bottom of the stack (first in)
  • If we eliminate the existence of a command by undoing it, then pushing another command on the stack

These two cases have different deletes though. In the first case we need to delete the original items, since we’ve replaced them with the output of the command. In the latter case, we need to delete the new items the command generated, and leave the original ones alone.

Additionally, different commands need to delete data which is structured in different ways. For instance, a split command (such as Euclidean Clustering) needs to delete one original item or many created items, while a merge command needs to delete many original items or a single created item... and so forth...

Oh, on a side note, I’m not sure what I should have the app do when one tries to merge two Cloud items which have different fields. For now, I just say “Nope”... but there may be a way to combine them that makes sense. Unfortunately this currently doesn’t exist for PointCloud2 objects in PCL afaik. Anyone feel like adding some cases to pcl::concatenatePointCloud(sensor_msgs::PointCloud2&, sensor_msgs::PointCloud2&, sensor_msgs::PointCloud2&) that checks for different but compatible field types? For instance, concatenating rgb and rgba clouds could just give a default alpha value for all the rgb cloud points. This would be pretty easy to code if one did it inefficiently by converting to templated types, concatenating, then converting back...

Bugs and additional features
Tuesday, August 07, 2012
../_images/gsoc127.png

Fixing bugs takes time. PCLPlotter was behaving weird in pcd_viewer. A window was appearing just after the creation of an object of 2D classes (Plotter and Painter2D) without even the call of plot/spin/display functions. Thus, had to move vtkRenderwindowInteractor::Initialize() and therefore vtkRenderwindowInteractor::AddObserver() to the display triggering calls (plot/spin/display).

Added other small functionalities like setTitle* in Plotter.

Merge Cloud Command, Create New Cloud from Selection
Tuesday, August 07, 2012
../_images/gsoc126.png

Just a quick update on what I’ve added.

The frustum selection now connects back to the model through an event it invokes. It can search for points if needed, but right now I’m sanitizing all clouds (no NaNs) so there’s a 1-1 correspondence from vtk to PCL. Of course I have to figure out what cloud the vtk point belongs to, but that can be done using the CloudActorMap from PCLVisualizer.

I’ve added the last undo/redo command, merging. I also added a tool which creates a new cloud from the current selection. There are now two “selectors”; a green one for points selected in the VTK window, and a red selector for points selected in the cloud browser dock. Right now all commands applied to selections work on both, but that may change in the future.

Fair tests on the Stanford dataset 2
Sunday, August 05, 2012
../_images/gsoc124.png

I’ve finally completed the section Detectors evaluation: repeatability and time performances and it is now available for consulting.

reading code on MS
Saturday, August 04, 2012
../_images/gsoc123.png

Tomorrow I am going on vacation for two weeks, I just hope I will be able to take the time to finish the API for MeanShift and get it up and running until the deadline. In my spare time I am checking out some other existing implementations of MeanShift, namely the Edison library and the one existing in OpenCV as well as some existing Matlab implementations. I’ve also wrote a first script doing segmentation based on color, but results were not exactly what I was hoping for.

Fair tests on the Stanford dataset 2
Saturday, August 04, 2012
../_images/gsoc124.png

In what follows I’ll show the main results of the tests on the Stanford dataset. The complete results will be given in Detectors evaluation: repeatability and time performances. The fairness of the comparison is ensured by fixing properly some common parameters among all the detectors. ISS has clearly the best relative repeatability among all the PCL detectors that have been under testing. With regards to the legend, the same considerations made in the previous blog post apply this time.

../_images/absolute_repeatability_all3.png
../_images/relative_repeatability_all3.png
../_images/time_performances_all_6cr1.png
../_images/iss_time_performances_6cr1.png
Axes Widgets, Signal Multiplexer, InteractorStyleSwitch, Rectangular Frustum Selection
Friday, August 03, 2012
../_images/gsoc126.png

Crunch time is here, so I’ll just give a short update on the things I’ve added:

There’s an axes widget now that shows the orientation of the camera, as in ParaView.

I added a signal multiplexer class which greatly simplifies the architecture. This class allows you to specify connections without having to specify the receiving object when you make the connections. You then can switch receiving objects, and all the connections are adjusted automatically. For me, this means that I have all my GUI actions connected to the multiplexer, and then I just tell the multiplexer which project is currently visible. The GUI actions are then routed to that project, and all the other projects just idle. It also allows me to update the GUI state automatically to project state when projects are switched. Why a class like this isn’t included in Qt itself is kind of a mystery to me.

I added an InteractorStyleSwitch class which allows you to switch what interactor style PCLVisualizer is using. This means that with one function call we can switch from the normal PCLVisualizer camera nteractor style to one which lets us select sub-clouds with the mouse.

Which brings us to the last addition, the ability to select subclouds using a rectangular frustum selector. As seen in the images below, this works, and it works on multiple clouds in the scene. What it doesn’t do yet is take the selected vtkPolyData points and figure out what indices (and what clouds) they correspond to in the original PCL clouds contained in the Project Model.

That’s for tomorrow... plus the ability to split off the selected points into a new CloudItem (with undo/redo of course).

../_images/frustum_select.png

Stay classy, PCL blog readership.

Fair tests on the Kinect dataset
Thursday, August 02, 2012
../_images/gsoc124.png

In what follows I’ll show the main results of the tests on the kinect dataset. The complete results are given in Detectors evaluation: repeatability and time performances. The fairness of the comparison is ensured by fixing properly some common parameters among all the detectors. NARF and ISS are clearly the two best PCL detectors among the ones tested.

I think it is required a brief explanation of the graphs legend. First the harris 3D detector is tested with all its possible response methods, so the abbreviations HA, NO, LO, TO, and CU respectively refer to the response methods: HARRIS, NOBLE, LOWE, TOMASI, CURVATURE. The abbreviation ISS - WBE refers to the execution of the ISS 3D detector without the border extraction, while ISS - BE refers to the execution of the detector with the extraction of boundary points. Finally, the abbreviations 1 th, 2 th, 3 th and 4 th stand for 1, 2, 3 and 4 threads and they are related to the OpenMP optimization of the code.

../_images/absolute_repeatability_all2.png
../_images/relative_repeatability_all2.png
../_images/time_performances_all_6cr.png
../_images/iss_time_performances_6cr.png
PCL Android (milestones)
Wednesday, August 01, 2012
../_images/gsoc1211.png

The lack of hardware accelerated libraries for Android is the key bottleneck I’ve been facing. After spending many hours on NEON assembly and other tools, I finally come across this...

Ne10: A New Open Source Library to Accelerate your Applications with NEON http://blogs.arm.com/software-enablement/703-ne10-a-new-open-source-library-to-accelerate-your-applications-with-neon/

http://projectne10.github.com/Ne10/

Next I’ve verify the actual speedup we can get with such library, and see how we can accelerate some of the PCL calls with these. With NEON + multithreading, I am looking for a 10x speedup on Tegra 3.

More to come next... Update: I’ve added Ne10 to the project tree, and have it compiled. Shall be ready to verify the theortical speedup we can obtain with the new hardware accelerated libraries. =)

starting work on mean shift segmentation
Wednesday, August 01, 2012
../_images/gsoc123.png

I have started working on implementing mean shift. I am going to try to keep the API as modular as possible, thinking of the fact that mean shift can be used for a variety of things among which segmentation is only one. First I am just writing a script like program to check if the algorithm works, and then I’ll implement it in the API. (segmentation will be based on color space and on normals at a first try)

News on tests
Wednesday, August 01, 2012
../_images/gsoc124.png

While developing the evaluator for the ISS 3D detector, I realized that I set the harris 3D and 6D normal estimation radius to:

multiplier * cloud_resolution

where multiplier is set to 6, 9, 12 and 15 cloud_resolution each time. Instead of this setting, I should have set the normal estimation radius to:

4 * cloud_resolution

in order to obtain fair tests. The previous tests are valid but the reader should take into account this consideration. While testing the ISS 3D detector I re-run the tests to obtain the desidered fairness. I have also decided to collect this final tests in a specific blog page, so as the user can immediately reach the results without looking for it in all my posts. I have just completed the evaluation on the Kinect dataset and I will post it soon both on the blog and on Detectors evaluation: repeatability and time performances.

Additional features and spin*() functions
Tuesday, July 31, 2012
../_images/gsoc127.png

As I posted before, my work is now to improve and add additional features to my two classes PCLPlotter and PCLPainter2D. I always had in my mind that I will add spin*() functions (spinOnce(time) and spin()), which are a part of all the existing visualization classes (like PCLVisualizer, PCLHistogramVisualizer, etc), to my classes. Frankly, I did not understand these functions much based on the documentation and the code, perhaps because of my no knowledge of vtk’s event handling (vtkCommand and all). All I knew that this functions someway start the interactor.

So, I finally understood those function after getting familiar to vtkCommand and going through their implementation. I kind of find the names confusing. spinOnce(t) runs the interactor event-loop for time t. spinOnce sounds like spinning (looping) one time which is confusing. spin() runs the interactor event-loop for indefinite time using spinOnce(t) thereby providing the ability to update the scene with time. But following is the description for spin() provided in the documentation in verbatim: “Calls the interactor and runs an internal loop.”. Either I am missing out something or the documentation is misleading!

Apart from the above, I was stuck for the most of the time figuring out the usage of RepeatingTimer. The repeating timer event is caught and the timer is destroyed- right in the first time! A SingleShotTimer very well suited this purpose. I did not understand the use of RepeatingTimer. I used SingleShotTimer in my spin* methods and it works as it should.

Other than spin*() functions, I added other features about which I will post in the next blog. I would also like to comment here on the general design of the “visualizer”s in pcl. Unfortunately, blogging takes time and I will post about them in the next few days, one by one.

Better interface for various point cloud types
Tuesday, July 31, 2012
../_images/gsoc121.png

I have modified the previous code so that it finds possible edges from various point cloud types:

  • OrganizedEdgeBase accepts PCL_XYZ_POINT_TYPES and returns EDGELABEL_NAN_BOUNDARY, EDGELABEL_OCCLUDING, and EDGELABEL_OCCLUDED.
  • OrganizedEdgeFromRGB accepts PCL_RGB_POINT_TYPES and returns EDGELABEL_NAN_BOUNDARY, EDGELABEL_OCCLUDING, EDGELABEL_OCCLUDED, and EDGELABEL_RGB_CANNY.
  • OrganizedEdgeFromNormals accepts PCL_XYZ_POINT_TYPES with PCL_NORMAL_POINT_TYPES and returns EDGELABEL_NAN_BOUNDARY, EDGELABEL_OCCLUDING, EDGELABEL_OCCLUDED, and EDGELABEL_HIGH_CURVATURE.
  • OrganizedEdgeFromRGBNormals accepts PCL_RGB_POINT_TYPES with PCL_NORMAL_POINT_TYPES and returns EDGELABEL_NAN_BOUNDARY, EDGELABEL_OCCLUDING, EDGELABEL_OCCLUDED, EDGELABEL_HIGH_CURVATURE, and EDGELABEL_RGB_CANNY.

OrganizedEdgeFromRGB and OrganizedEdgeFromNormals are derived from OrganizedEdgeBase. OrganizedEdgeFromRGBNormals is then derived from both OrganizedEdgeFromRGB and OrganizedEdgeFromNormals.

ICP Registration
Tuesday, July 31, 2012
../_images/gsoc1212.png

ICP registration is added. The input point clouds and the result can be shown in same or different render windows for better inspection, as indicated by the following snapshot.

../_images/snapshot-2012-07-31.png

Initial positions of the input point clouds are important for the registration to converge. I will implement two types of tools for the initial placement of the input point clouds: setting corresponding points and directly tuning the orientation/position of the point clouds. The tuning will be accomplished by interactively adjusting the orientation/position parameters or by draggers(if it’s not very complicated to implement draggers with vtk...).

Tool Input Checking
Monday, July 30, 2012
../_images/gsoc126.png

So I implemented a way of showing what inputs are required for a tool and if the currently selected items are a valid input for the tool. It’s a pretty simple system really; if the selected item(s) is valid input for a tool, it becomes enabled, and you can use it. If the selected item(s) don’t match what is needed by a tool, the tool is greyed out, and you get a tooltip which says why you can’t use the tool. In the case of the image below, that’s because calculating FPFH features requires normals.

../_images/Tool_Input.png

Now I’m working on mouse selections in the QVTKWidget/PCLVisualizer view. Things like rectangular-drag and point by point.

Then I’ll add registration, the merge command, and the ability to manually shift point clouds (using the keyboard, and perhaps the mouse).

Then I’ll add as many additional tools as I can manage before GSoC ends. It’s crunch-time; it’s going to be tough to implement all the features I would like to by the middle of August. This has turned out to be a pretty large project for one person... Fortunately the end of GSoC won’t be the end of this App; I’ll be contributing one day a week this Autumn/Winter to continued development of it.

ARM Optimization
Sunday, July 29, 2012
../_images/gsoc1211.png

Often time we have ignored the importance of writing efficient source code. With the mobile platform, every bit of computation matters. Imagine you have a video player that can only achieve 10 fps, while the competitors are running at 60fps. These differences may define a successful or failure application.

To get started, these weeks I’ve gathered some NEON material and wrote some small functions that’s optimized with NEON instruction set. In fact it is surprisingly difficult due to the lack of documentations and example sometimes (maybe I’ve not tried hard enough?)

It wasn’t very difficult to have NEON Intrinsics code compiled and run on Tegra 3 after all.

All we need is adding the #include <arm_neon.h> and compile with -mfloat-abi=softfp -mfpu=neon options.

With the native c code, we can perform a simple array sum (i.e., adding all elements in an array) in about 0.034 second.

07-30 02:46:03.170: I/PCL Benchmark:(1426): Number of Points: 65536, 65536, Runtime: 0.034658 (s)

With the NEON, we get about 2x the perform.

07-30 02:48:04.070: I/PCL Benchmark:(2392): Number of Points: 65536, 65536, Runtime: 0.015879 (s)
int16_t sum=0;
for (; size != 0; size -= 1)
{
sum+=array[size-1];
}
return sum;


.. line-block::

int16x4_t acc = vdup_n_s16(0);
int32x2_t acc1;
int64x1_t acc2;
assert((size % 4) == 0);
for (; size != 0; size -= 4)
{
int16x4_t vec;
vec = vld1_s16(array);
array += 4;
acc = vadd_s16(acc, vec);
}
acc1 = vpaddl_s16(acc);
acc2 = vpaddl_s32(acc1);
return (int)vget_lane_s64(acc2, 0);

Code Example Source:

Reference Links:

The next step would be optimizing for the floating point operations, and it seems to be a rather difficult task. It seems to be promising now as if I can multithread the work to 3 cores together with NEON, we can get 4-6x speed up, and thus a 5fps application will now run smoothly at 30fps. That’s a big improvement for sure.
First pictures from the ISS detector
Saturday, July 28, 2012
../_images/gsoc124.png

In what follows I’ll show some snapshots related to the behaviour of the ISS detector. They have been obtained by not setting the border radius.

Shapshots from the Kinect dataset.

  • Model:

    ../_images/model_screenshot041.png
  • Scene:

    ../_images/scene_screenshot095.png
  • Results:

    • Absolute repeatability: 8
    • Relative repeatability: 0.195122

Shapshots from the Stanford 1 dataset.

  • Model:

    ../_images/model_screenshot058.png
  • Scene:

    ../_images/scene_screenshot070.png
  • Results:

    • Absolute repeatability: 413
    • Relative repeatability: 0.769088
ISS is available on trunk now!
Saturday, July 28, 2012
../_images/gsoc124.png

Now, ISS is available on the trunk and it is properly documented. In the section how to use the ISS 3D keypoint detector of this blog I will post some code snippets useful to the user who wants to exploit the ISS detector. Currently, the ISS detector is under testing. It will be tested for different configurations:

  1. using 1 thread and disabling the boundary estimation.
  2. using 1 thread and enabling the boundary estimation.
  3. using 1, 2, 3 and 4 threads (4 is the maximum number of threads allowed in my system) and chosing the best configuration among that described previously in 1. and 2. .

The tests will show both the repeatability results and the time performances of the detector. The results related to the configurations 1. and 2. can be compared with the results already obtained for the other detectors tested at the beginning of my GSoC work.

Coming soon: the test results for the ISS detector.

New Plugins, Deleting, Properties, a Screenie
Friday, July 27, 2012
../_images/gsoc126.png

Another quick pre-sleep update... Voxel Grid Downsample and Statistical Outlier Removal have been added, which means the cloud modification undo/redo functionality works now. Delete undo/redo command has also been added. The only remaining command type to be added is the merge command, as would be needed for registration. This is up next.

Then on to selection tools... which may need to be implemented directly, rather then as plugins, since plugins have no way to interact with the VTK render windows (and they shouldn’t, since their processing is done in non-GUI threads). Adding additional tools now is really quite fast... though I think at some point in the coming days I’m going to polish the plugin interface a little bit, as well as clean up the items, mainly by giving them a templated way to get/set data that doesn’t rely on Qt’s awkward UserRole enum system.

I’ll leave you with a little screenshot of what it looks like now.

../_images/CloudComposer_SS_27JUL12.png

Oh, one other thing, if you’re fooling around in the GUI, and something goes horribly wrong, please submit a bug report. The whole app is starting to become a little more complex these days, so ugly bugs that I haven’t found are bound to start cropping up.

Feel free to submit feature requests too... but I make no guarantees as to how long it might be before they can be implemented!

fixed problem
Wednesday, July 25, 2012
../_images/gsoc123.png

I managed to fix the region growing, so now, if the seed point is on a plane paralell to the table top the method does not fail. Below you can see some screen shots of the

../_images/screenshot-1344673807.png ../_images/screenshot-1344674049.png ../_images/screenshot-1344674128.png
Clustering, Splitting clouds, Selection highlighting
Tuesday, July 24, 2012
../_images/gsoc126.png

Just a quick update before I go to sleep... Euclidean clustering now works; this means that the cloud splitting command & undo/redo functionality is implemented now too. If you load a cloud, run clustering, it will split it into clusters and whatever remains from the original. Basic (ugly) highlighting now works too; the cloud selected in the browser on the left gets “highlighted” in the view by turning red... which is terrible. I want to do a more subtle highlighting effect, maybe some sort of glow, or tinting, but my attempts are failures so far. I played around with adding a colored ambient/diffuse light to the selected cloud actor (in pclVisualizer) but none of the lighting changes I made had any effect. I suspect I’m making some sort of noobish vtk mistake. I think I need to read through a quick vtk tutorial tomorrow morning.

Simplify Scene Tree and Point Cloud Storage
Tuesday, July 24, 2012
../_images/gsoc1212.png

After some efforts on making the scene tree with model/view, I gave up and just build it with QTreeWidget, which is simpler and works for desired UI interactions. And the work for making a PCL 2.x PointCloud is more than I expected, so I also gave up on this point cloud and found that pcl::PointCloud<pcl::PointSurfel> meets the requirements and I just store the point cloud in it for now. The storage will be upgraded to PCL 2.x point cloud type when it’s ready. After I removed the over kill code, it becomes much easier to progress, and I will move to the registration part next.

PCLPainter2D
Sunday, July 22, 2012
../_images/gsoc127.png

I added all the transform functionality in the Painter2D class now. So, one can perform transform operations like:

painter.addCircle(0,0,5);

painter.translatePen(5,0);

painter.addCircle(0,0,5);

painter.setIdentityTransform();

. . .

Since applying transform is a real time operation, it is required to keep track of when it is called. I solved this issue in similar way I tackled the underlaying vtkPen and vtkBrush. I stored a transformation matrix for each of the figure and updated it with the current transform stored in the painter class as state. Like I said before, implementation of this 2D class was not straightforward as I thought before. In fact it has been full of tricks.

Adding this functionality more or less completes the Painter2D class and my proposed work for the gsoc. For the rest of the period I will try to improve these two classes (PCLPlotter and PCLPainter2D) and add more features on request. So, if you think something should be added just email me; I will add if I think it is feasible ;-) I will also probably be assigned more work by Alex.

To sanitize or not to sanitize?
Saturday, July 21, 2012
../_images/gsoc126.png

So I’ve stumbled upon a design issue that I’m not sure how to handle. The problem is that some functions in PCL require sanitized input (such as FPFH), namely, they can’t handle NANs. This means I have to use either a passthrough filter or the removeNAN filter at some point. It might seem natural to do this within the plugin that needs sanitized inputs, but then the result won’t really correspond to its input cloud, ie, it will have less points. To get around this, I’m just sanitizing all input clouds when they are loaded, but this isn’t a satisfactory solution either, since it means you break up organized clouds (and modify the cloud automatically, which the user might not want to do).

So, what should I do here? Should plugins specify that they require sanitized input, and have their tool icon greyed out unless the selected input cloud is valid? What would be the best way to notify the user about why a tool is greyed out? Tooltips which give the reason it is disabled when you hover over the tool icon?

This seems like a sensible solution to me, but please, if anyone actually reads these blogs, let me know if you have a better idea, or if my solution seems foolish to you.

Oh, and more tools are coming, I promise, I’m just trying to make the plugin specification as complete as possible before I start implementing more tools... and I keep discovering things it is missing.

Working hard on ISS refinement
Saturday, July 21, 2012
../_images/gsoc124.png

Some time has passed since my last post. My recent and current activities regard mainly the ISS detector. First of all, I completed the detector so that it has a basic functionality (e.g. it works ! ). In order to be sure that it really works, I developed a simple evaluation framework at the same time. This framework looks like the basic frameworks I developed at the beginning of the GSoC code sprint and it allows the user both to compute the repeatability of the detector based on a pair of files (model + scene) and to visualize the resulting keypoints. My current activities regards the refinement of the ISS detector, and now I’m particularly focusing on time performances. After a brief analysis about the time performances of the detector, I refined it by using low-cost data structures. Now, I’m working on the introduction of the openMP directives in order to further speed-up the computation when 2 or more cores are avalilable. The detector will be available in the trunk when it will be fully completed and optimized.

Stay tuned with my roadmap: it always shows my current activities even if I’m not posting so much.

Changes to displaying, interaction with models
Wednesday, July 18, 2012
../_images/gsoc126.png

I haven’t been posting much on the blog, I’ll try to keep up with this better. I’ve done some rewriting of how things are displayed. Now the item classes define their behavior with respect to the various view classes. This means that a CloudItem defines its own paint function, which the View window (which uses PCLVisualizer) just calls when it needs to paint the CloudItem (ie, when it changes, when its added/removed). Also, now when changes are made in the inspector to properties they propagate through the models and update all the other views. For now, I’m only making properties editable which don’t require another call to the actual tool functionality, just ones that change how things look. For instance, you can change the scale and level of Normals, but you can’t change the radius, since that would require recalculating them, which is an expensive operation. This may change in the future, but for now, if you want to change the radius, you just have to calculate a new set by running the tool again (with a different radius).

PCLPainter2D
Tuesday, July 17, 2012
../_images/gsoc127.png

OK, so PCLPainter2D class is now available in the trunk. Currently it allows user to draw all the 2D primitives. The usage is same as discussed in design:

PCLPainter2D painter;

painter.addCircle(0,0,5);

. . .

painter.display();

The implementation is also exactly same as discussed in the previous blog. I’m storing drawing information in a data structure which is a vector of a class Figure2D. Later I’m using this information to re-implement paint() of the contextItem class (PCLPainter2D).

fixing things
Tuesday, July 17, 2012
../_images/gsoc123.png

This past week I have been fine tuning and rewriting the region growing part of the segmentation method in order to fix the eroneous segmentation shown in my last post.

PCL Performance Benchmark on Tegra 3 (Android 4.0)
Thursday, July 12, 2012
../_images/gsoc1211.png

I’ve rebuilt the PCL libraries using the build script (see pcl_binary/ in the svn respository) on Ubuntu 12.04 and the compilation works with a few hipcup. First we need to turn off the ENABLE_EXAMPLE flag due to the dependency problem. Second, we have to compile it with make -j 1 flag. Otherwise, everything ran smoothly.

I notice there isn’t any performance benchmark of PCL on Tegra 3. Here I’ve done a few simple testing. I believe it is important to see how such floating point operations we can do per second with the Tegra 3 architecture. float vs double? float vs int? Again, compiling the library as arm vs thumb mode may make different. Here I will provide a quick but throughout summary of what we can achieve with the Tegra 3 under different settings.

For simplicity, I’ve first benchmarked the passthrough filter by averaging the runtime of the filter over ten trials. I know this filter shall have a linear behaviour to the number of points, so far the benchmark results seem to be consistent.

// Create the filtering object
pcl::PassThrough < pcl::PointXYZ > pass;
pass.setInputCloud(cloud);
pass.setFilterFieldName(“z”);
pass.setFilterLimits(0.0, 1.0);
pass.filter(*cloud_filtered);
07-12 21:37:50.070: I/PCL Benchmark:(2785): Number of Points: 10000, Runtime: 0.002583 (s)
07-12 21:37:50.190: I/PCL Benchmark:(2785): Number of Points: 10000, Runtime: 0.002652 (s)
...
07-12 21:41:14.330: I/PCL Benchmark:(3614): Number of Points: 100000, Runtime: 0.036954 (s)
07-12 21:41:14.880: I/PCL Benchmark:(3614): Number of Points: 100000, Runtime: 0.038295 (s)
...
07-12 21:39:49.130: I/PCL Benchmark:(3344): Number of Points: 1000000, Runtime: 0.397860 (s)
07-12 21:39:53.720: I/PCL Benchmark:(3344): Number of Points: 1000000, Runtime: 0.392162 (s)
...

With these information, we can start optimizing our work by reducing the bottlenecks in each of these filters. But let’s recompile it in ARM mode and see if it will make a world of difference.

After I know what the Tegra 3 is capable of. It is time to design a simple 3D application. Argumented reality? and a few segmentation algorithms will do the tricks. What can we achieve with the current hardware?

Working in parallel on new detectors: GSS and ISS
Wednesday, July 11, 2012
../_images/gsoc124.png

Last week I started working on GSS but I stopped this week since I need some feedback from Alex and he may be on vacation now. In order not to lose much time, I have decided to switch to the implementation of ISS. After reading the related paper and after taking some knowledge on the code already implemented, I have defined the input and output parameters of the detector and the operations that have to be done in order to compute the keypoints. Then, I have prepared a skeleton class and now I’m currently filling and extending it with the implementation of new methods.

What were my other activities during the past and the current week? Bug solving and user support.

Segmentation using color and normals features
Tuesday, July 10, 2012
../_images/gsoc122.png

The following picture two pictures show the input data set. On the left side you see the captured point cloud captured using an Asus camera. On the right side you see the han labeled data set. I will show different segmentation results using different features and different levels of noise when setting the labels as unary potentials. One challenging part of the image (red circle) is the boundary between the box and the table. The box in the lower right corner has (almost) the same color as the table.

../_images/pc_anno.png

In the following image sequence you’ll thesegmentation using onlycolor information. The input labels arewith 50% noise assigned, meaning each unary potential is with 50% probability a random label assigned. From left to right the different results after x number of iterations can be seen. Whereas X is [0, 1, 3, 5, 10, 15]. Notice that when using only color information the table label grows into the box (red circle).

../_images/noisy50_it-0_it-1_it-3_noNormal.png ../_images/noisy50_it-5_it-10_it-15_noNormal.png

In the next image sequence we use only the normals as features. One can see that normals by itself are very powerful. However, we will also see that using only normal information has it’s limitations as well. The number of iterations per image is kept the same as well as the noise level.

../_images/noisy50_it-0_it-1_it-3_Normal.png ../_images/noisy50_it-5_it-10_it-15_Normal.png

Lastly Color + Normal features are used for segmentation. Notice that using color and normal features has extremely fast convergence. After only 5 iterations we have a very acceptable result.

../_images/noisy50_it-0_it-1_it-3_Normal+Color.png ../_images/noisy50_it-5_it-10_it-15_Normal+Color.png
RGB Edges
Tuesday, July 10, 2012
../_images/gsoc121.png

I added RGB edge detection, which is 2D canny edge detection from the RGB channels, in pcl::OrganizedEdgeDetection. Right now, the class only takes point types having RGB channels, but it will be changed so that possible edges can be detected from a given point cloud type. For example, ‘occluding’, ‘occluded’, and ‘boundary’ edges can be detected from any XYZ point types. And ‘high curvature’ and ‘rgb’ edges can be obtained from Normal and RGB point types, respectively.

Following images show the detected occluding (green), occluded (red), boundary (blue), high curvature (yellow), and rgb (cyan) edges:

../_images/screenshot-1341964342.png ../_images/screenshot-1341964387.png
Segmentation using color and normals features
Tuesday, July 10, 2012
../_images/gsoc122.png

In the second segmentation experiment I wanted to push the algorithm to the limit. For this I made the unary potentials extremely noisy. The potentials get with 80% a random label assigned. The first image sequence shows the segmentation result from left to right with different number of iterations [0, 1, 3, 5, 10, 15]. For the first test we use again only color features. We can see that by using only color features the algorithm performs poorly, which is not surprising. Changing the weights might help a little bit, however to make it a fair comparision I kept the weights and the standard deviations for the Gaussian kernels constant.

../_images/noisy80_it-0_it-1_it-3_noNormal.png ../_images/noisy80_it-5_it-10_it-15_noNormal.png

Next we use only the normals as features. Using the normals results in surprisingly good results. The background is labeled almost perfectly as well as the objects on the table. The table itselfhowever, remains unlabeled. To this point I have no good explanations why this is the case. Further investigation might be interesting.

../_images/noisy80_it-0_it-1_it-3_Normal.png ../_images/noisy80_it-5_it-10_it-15_Normal.png

Lastly we use Color + Normals features. To my surprise, I actually did not expect such a good result. The only part that seems to be mislabeled are table legs.

../_images/noisy80_it-0_it-1_it-3_Normal+Color.png ../_images/noisy80_it-5_it-10_it-15_Normal+Color.png
Design of the 2D painter class
Monday, July 09, 2012
../_images/gsoc127.png

In this post I will discuss about the design which I thought for the 2D painter class. The aim is to have a very simple interface (just like PCLPlotter) which allows user to add figures by simple add*() methods and, in the end, a display() method to show the canvas. Something like the following is desirable:

PCLPainter2D painter;

painter.addCircle(0,0,5);

painter.addLine(0,0, 5,0);

. . .

painter.display();

The underlaying implementation of PCLPainter2D in the above design will not be as straight forward as PCLPlotter where we have an instance of vtkChartXY and vtkContextView inside the class. The only job was to convert the plot data (correspondences) to a format (which is vtkPlot) appreciated by vtkChartXY. That is, we had a direct mapping in term of functionality from vtkChartXY to PCLPlotter (with difference in the type of data they process and an additional “view” object in PCLPlotter). The problem in the above design is the fact that we don’t have any vtkContextItem class which share similar properties of Painter2D class. Instead, 2D drawing works in the following way in VTK. The VTK user needs to first:

  1. Make a subclass of vtkContextItem
  2. Re-implement (override) Paint() of vtkContextItem. (shown in the figure)
../_images/contextItem.png

It would be really nice to have a vtkContextItem class which cuts off the overhead of subclassing and allows user to draw directly from the function calls. Unfortunately, we don’t have any (out of vtkChart, vtkPlot, vtkAxis,..., etc.) vtkContextItem class with that kind of behavior. So, before directly writing a Painter class for PCL it may be wise to write something like vtkPainter2D class for vtk and extend it to PCL. In this way it can be used to avoid subclassing in both VTK and PCL and its rendering could be further optimized in the future.

Now, the steps for creating “vtkPainter2D” (or PCLPainter2D) which would be a subclass of vtkContextItem are roughly the following:

  1. Store information of 2D primitives in some data structures in every call of add*() calls.
  2. Implement Paint() using those data structures.

These things have already been discussed with Marcus. It would be nice to hear your comments and suggestions.

More plugins, more fun
Monday, July 09, 2012
../_images/gsoc126.png

So I’ve finally gotten my satellite internet connection up and running here in France, so now I can start committing again every day. I’m finished with the FPFH plugin, with display of histograms in the inspector view. Now I need to work on a splitting plugin, segmentation, so I’ll have to spend a little more time in getting the undo/redo stuff working for that. I think I need to do some more thinking about the structure of how the items are working as well. Right now I’m still using a QStandardItem subclass, where it probably makes more sense to subclass from the QAbstractItem directly and implement some things myself. I’ll probably spend the next day working on that, along with the segmentation plugin. The plan is to have the following plugins working by the end of the week: Normals, FPFH, Euclidean Segmentation, Plane Segmentation, ICP for registering two clouds.

Snapshots
Sunday, July 08, 2012
../_images/gsoc127.png

I am adding snapshots of pcl_plotter in action showing examples of the functionalities which I discussed in my previous blogs. Till now, I didn’t get a good internet connection in my new apartment, but I am uploading them from my limited cellphone connection anyway.

Most of them are plots from a given function like polynomial/rational or a user defined custom function. Last two snapshots are provided for the comparison between PCLPlotter and PCLHistogramVisualizer.

  • Plot from polynomial
../_images/x2.png ../_images/x2x3.png
  • Plot from Rational function: y = 1/x
../_images/1byx.png
  • Plot from an arbitrary Rational function: y = (3x^2 + x + 2)/(6x^5 + 5x^4 + 4x^3 + 3x^2 + 2x + 1)
../_images/comprational.png
  • Plot from user-defined callback function (eg taken: step, abs, and identity)
../_images/callback_steps.png
  • Comparison between PCLHistogramVisualizer and PCLPlotter
../_images/histcomp.png
  • Multiple Histogram view in PCLPlotter
../_images/histmany.png
adding normals and more
Sunday, July 08, 2012
../_images/gsoc123.png

I’ve finaly managed to add the normals to the region growing algorithm and results are promising. I’ve conditioned adding of points to the region on the angle between the seed points’ normal and the current point. Results doing this are shown in the screen shot below.

../_images/scene_better.png

Adding this condition helped and not at the same time. Although now growing stops when we reach the table top, parts of the object that are parallel to it don’t get added as well. This was expected of course. Still...better then the first try.

Thanks to a friend of mine I found out about this theory:

The interesting part for me is that according to this theory every object can be broken down into piece primitives and there are only 32 kinds of these primitive shapes and all of them are convex, meaning that complex objects can be separated into these parts at their concavenesses.

The following is an extract from the book From Fragments to Objects - Segmentation and Grouping in Vision T.F. Shipley and P.J. Kellman (Editors) 2001 Elsevier Science B.V. All rights reserved.

“The simplest shapes are convex shapes—whose outlines have positive curvature throughout (see, e.g., Rosin, 2000, for the role of convexity in parsing). If the outline of a shape has regions of negative curvature, especially if these regions contain salient negative minima of curvature, this usually indicates that the shape can be further parsed to give simpler subshapes. [...] Three main geometrical factors determine the perceptual salience of a part (Hoffman & Singh, 1997): (1) its protrusion, (2) its relative area, and (3) the strength of its boundaries. Salience of a part increases as its protrusion, relative area, or boundary strength increases. In this section we briefly consider protrusion and relative area, and then discuss in more detail the strength of part boundaries. We restrict attention to 2D shapes; the theory for 3D shapes is more complex and discussed elsewhere (Hoffman & Singh, 1997).”

“Hypothesis of normalized curvature: The salience of a part boundary increases as the magnitude of normalized curvature at the boundary increases.”

“Hypothesis of Turning Angle: The salience of a negative-minimum boundary increases as the magnitude of the turning angle around the boundary increases.”

Based on these theories I introduces another constraint to my region growing: when adding a point the angle between the line connecting that point to the fixation point and the points normal is checked. If this angle is concave it means that the point currently being verified belongs to the object. Doing so resulted in the following results (blue points are the ones segmented out and the green patch on the objects is the neighborhood of the fixation point):

../_images/scene_1.png ../_images/scene_2.png ../_images/scene_3.png ../_images/scene_4.png ../_images/scene_bad_2.png

As it can be observed introducing that extra condition improved results a lot.The last scene is not a good result. My next step will involve investigating the cause of that erroneous segmentation. As a final observation, it was clear for me since the beginning, that the whole of this method is dependent on having a good fixation point. If this is not the case, the algorithm would not work at the moment. One of my next steps will be to investigate the method recently proposed by the authors of the original paper for automatic fixation point estimation.

With this occasion I would like to thank Zoltan-Cs. Marton for coming up with the idea of using convex vs. concave angles, it helped me a lot:). THX m8:)

Plot from rational functions and user defined callback function
Saturday, July 07, 2012
../_images/gsoc127.png

Added two functionalities:

  1. plot from rational functions which are the ratio of polynomials. Plot of 1/x looks nice.
  2. plot from a user defined callback depicting the relation between Y and X axis. The function should be continuous.

Snapshots coming!

Plot from polynomial
Friday, July 06, 2012
../_images/gsoc127.png

Added the functionality to plot from polynomial. This occurred to me as a useful functionality a plotter class should have. User needs to provide a vector which stores the coefficients of the polynomial and range. PCLPlotter will plot and display them on the screen.

I still don’t have a good internet connection. I will post the snapshots later.

I am very happy that the training period of my job will get over by the end of next week. It won’t be hectic after that as the office hours will go to normal (Currently its like 12-14 hours :()

Tegra 3 + Android + OpenNI + PCL’s VoxelGrid filter Sample
Friday, July 06, 2012
../_images/gsoc1211.png

I’ve collect some statistics and screenshots of our new sample app that demostrates the voxel grid filtering using the PCL library. The performance isn’t something I would be very proud of, i.e., only ~2fps with all of the processing with about ~0.3 million points (307200) as input. However, it is quite usable if we are using this for capturing something steady, perhaps to be used for 3D reconstruction in real-time.

Here are some screenshots of the sample apps, and it shows the RGB images, depth image, and also the 3D PointCloud data all using the OpenGL ES2. The downsampling does provide us at least 50% reduction on the number of points.

../_images/voxel_pcl_sample_july_6.jpg ../_images/voxel_pcl_sample_july_6_2.jpg

Here is a little video demo of the application (running at 2fps).

We also collects some simple statistics on the performance of the algorithm, both the runtime statistics and the compression ratio we can achieve.
...
07-05 21:46:59.150: I/Render Loop:(10204): Display loop 0.543398 (s)
07-05 21:46:59.730: I/PCL FILTER TESTING:(10204): Original: 307200, Filtered: 75208, Ratio: 0.244818, Sum 564766.304984
...

In some cases, the voxel grid filter can reduce the number of points to only a small fraction. I have seen cases where the ratio is below 10% for flat surfaces. We have only touched the surface of the PCL library, but I can see number of applications can be built using these. Possibilities are just limitless. ;)

TODO: need multithreading to utilizing the quadcore on the Tegra 3. It seems to be an easy task. CPU # 1: OpenNI engine; CPU # 2: main thread with GUI; CPU#3 & 4: PCL and other processing engines. :). That way we will be fully untilizing all for the cores on Tegra 3. Also, I wonder if we can use float instead of double for all operations, and also turn off the THUMB mode! Obviously, we need to squeeze more performance out from everything. NEON optimization? Anyone?

PCLHistogramVisualizer rewrite
Thursday, July 05, 2012
../_images/gsoc127.png

All important functionalities of PCLHistogramVisualizer are now incorporated in the PCLPlotter class. They are rewritten here so that this single class can take responsibility of all ploting related functionalities. The signatures of the PCLHistogramVisualizer functions are retained as of now so that one can directly use the PCLPlotter class with previous signatures to get similar result. I will post snapshots in the evening when I get a good internet connection.

I made some changes in pcd_viewer so that it can use this Plotter class instead of HistogramViewer. As signatures of the functions are same, the changes are minor; but the fact that it is now using this unified Plotter class. I will discuss with the community before committing the new pcd_viewer.

I really want to get some feedback about this Plotter class. Please update the trunk and use the class to plot something nonsense and tell me if you would like to have something in addition.

Reading up on mean shift
Thursday, July 05, 2012
../_images/gsoc123.png

Since I was a little swamped with work this week I did not have time to try out how adding the normals will change my region growing yet, so I did some reading in my spare time about mean shift segmentation and existing implementations. Since the original papers on mean shift are not the easiest to understand and to clarify to myself about steps involved in segmenting i started searching for other articles about this topic. I’ve found this article to be the most helpful:

Moving to the future PointCloud type
Wednesday, July 04, 2012
../_images/gsoc1212.png

I didn’t choose PointT as the core data structure, since different algorithms will interact with PCLModeler requesting different point cloud types, so I decided to use PointCloud2 because it is a container of many fields. But now I found it’s not easy to support some functions, for example, updating a specified field, or adding some fields, the current code is messy because of the accommodation for PointCloud2. According to this thread, PointCloud2 is going to be deprecated, and the proposed point cloud type in PCL 2.x is much more friendly for PCLModeler. So I will drop PointCloud2 and re-design the core based on the proposed data structure.

Final performances results for the Stanford dataset 2
Wednesday, July 04, 2012
../_images/gsoc124.png

In what follows, the time performances related to all the considered detectors have been collected with regards to the Stanford dataset 2 and graphically visualized to foster a better understanding. In our framework, time performances refer to the keypoint extraction time of each detector and different scales have been taken into account: 6, 9, 12, 15 * scene_resolution are considered. Results are given below.

../_images/time_performances_6mr_all1.png
../_images/time_performances_9mr_all1.png
../_images/time_performances_12mr_all1.png
../_images/time_performances_15mr_all1.png
Final repeatability results for the Stanford dataset 2
Wednesday, July 04, 2012
../_images/gsoc124.png

I just got the results for the execution of tests on the Stanford dataset. All the tests (both on the Kinect and the Stanford dataset) have been executed on a 2nd generation Intel® Core™ i5 processor with a speed equal to 2.4 GHz (3 GHz if TurboBoost technology is enabled). The results regarding the repeatability measure confirm the superiority of the NARF detector on the others. The repeatability results are graphically shown below:

../_images/absolute_repeatability_all1.png
../_images/relative_repeatability_all1.png
Changes to API
Tuesday, July 03, 2012
../_images/gsoc129.png

Based on some discussions on how people would want to use the 2D module, there have been some changes in the API for this module. Images are now being represented in the point cloud format itself. 2D filters will extend the pcl::Filter inteface and 2D keypoints will extend the pcl::Keypoint interface. This will lead to a code structure more consistent with the rest of PCL.

This module does operations only on the RGB/Intensity channels of Organized Point Clouds. It ignores all x,y,z information for now. There are lots of features in PCL which deal with the x,y,z data. Now that the 2D module works with the same data-types, the user could use these existing features for processing the x,y,z information and the 2D module to process the RGB/Intensity information.

I’ve been focusing on designing and implementing this new API in the past few days. I’m also converting the code I wrote earlier to comply with this new API.

Hard working days
Tuesday, July 03, 2012
../_images/gsoc124.png

While being in Toulouse I had to sacrifice some of the time devoted to PCL, so this week I’m going to work hard in order to recover that time. Currently, I am testing the detectors on a synthetic dataset (the well-know Stanford dataset). Unfortunately, since this dataset does not contain the RGB information the only detectors under testing are: Harris 3D, NARF and uniform sampling. Indeed, those three detectors are characterized by having a shape-based saliency measure. As for the tests executed on the Kinect-based dataset, Harris 3D is evaluated with regard to all the possible different response methods. While the tests are executing, today I’ve decided to run up with my roadmap and I started to take some knowledge about the 3DGSS detector. In particular, the reference paper I’ve read is:

Coming soon: the final evaluation results on the synthetic dataset.

Still looking for a PhD position
Tuesday, July 03, 2012
../_images/gsoc124.png

Finally, I decided to decline the offer by the LAAS-CNRS and I’m still looking for a PhD position. Any suggestion about it?

Region growing based only on boundaries.
Monday, July 02, 2012
../_images/gsoc123.png

For the approximation of the boundaries I used the class implemented by our fellow GSOC student Changhyun Choi.

Because the active segmentation method is one that is most likely to be used in a robotic application, as described in the paper scenes involved are table top scenes containing several objects. I chose the scene below as the one for testing because there are multiple objects on the table some occluding others, but as this is the scene that i will be running the algorithm on the first time I did not want it to be too complex (e.g. extremely cluttered scenes)

../_images/scene_rgb.png

After implementing and testing the first version of the region growing I got the results shown in the screen shot below....just to clarify....everything that is blue gets segmented out. The green points on the large box are the points near the “fixation point”. This version of region growing is based only on the borders and since there are no borders at the touching point of the box with the table growing does not stop. To get around this problem I will take into consideration the normals of the points as well when growing.

../_images/scene_bad.png
Android + Openni Autoscript
Sunday, July 01, 2012
../_images/gsoc1211.png

Thanks to Radu and others, we have finalized the autoscript for compiling the OpenNI for Android. These are all integrated to the Android sample project. Also, we have compiled the PCL for android, and in the coming weeks we shall have a complete sample project. These can be used as our standard template for development of PCL + OpenNI on Android.

Happy Canada Day.

Final performances results for the dataset based on kinect data
Saturday, June 30, 2012
../_images/gsoc124.png

In what follows, the time performances related to all the considered detectors have been collected and graphically visualized to foster a better understanding. In our framework, time performances refer to the keypoint extraction time of each detector and different scales have been taken into account: 6, 9, 12, 15 * scene_resolution are considered. Results are given below.

../_images/time_performances_6mr_all.png
../_images/time_performances_9mr_all.png
../_images/time_performances_12mr_all.png
../_images/time_performances_15mr_all.png
Final repeatability results for the dataset based on kinect data
Friday, June 29, 2012
../_images/gsoc124.png

I’m currently visiting the LAAS-CNRS in Toulouse, in order to see if it could be a good place to me to accomplish PhD studies. While doing so, I’ve finished to test the detectors on the kinect-based dataset. Again, I want to say thank you to Bastian Steder for helping me on dealing with the NARF detector. Results are graphically shown below:

../_images/absolute_repeatability_all.png
../_images/relative_repeatability_all.png
Segmenting around a fixation point
Friday, June 29, 2012
../_images/gsoc123.png

As promised in a previous post I took the time and created a basic flow chart of the involved steps of segmenting around a fixation point. Upper part of the chart (getting a boundary map, setting input cloud etc. ) are steps that need to be implemented by whoever want to use the ActiveSegmentation class(example of this will be shortly available pcl/examples), steps in the lower part are implemented in the class. It is left to the users discretion to choose an appropriate boundary map. I chose to do it this to have a bigger flexibility, and because many others are working on edge/boundary detection at the moment. My test will be based on the already implemented Boundary detection (which works for ordered and unordered point clouds as well) and trying it out with mapping 2d edge detections to the 3d cloud.

  • Flow Chart Active Segmentation
../_images/flow_chart_as.png

In my next post I will share some preliminary results as well.

First results
Tuesday, June 26, 2012
../_images/gsoc124.png

Here, I show the first results coming from testing all the detectors but NARF. NARF has been under investigation a little bit more, but thanks to Bastian Steder the issues I have encountered should be solved.

The absolute repeatability graph:

../_images/absolute_repeatability_all_but_narf.png

The relative repeatability graph:

../_images/relative_repeatability_all_but_narf.png

The time performances graph (related to a 6mr scale):

../_images/time_performances_6mr_all_but_narf.png
Current UI and Functions
Tuesday, June 26, 2012
../_images/gsoc1212.png

The current UI is shown as the following image. There are 3 groups of functions to be implemented, filters, registration and surface reconstruction. I’ve implemented one filter. More filters can be added quickly, but I will do it later after the whole framework is more stable. I will implement one function for each group, then add more. I am working on poisson reconstruction now, and then registration.

../_images/snapshot-2012-06-26.png

Features:

  • The point clouds can be rendered either in the main window or any other dockable windows, which is quite useful for registration, for example, the user can put the frames in the main window, and put each frame in a sperated dockable window, make interaction in the dockable windows and see how well the frames align in the main window.
  • The render windows and clouds are organized in the scene explorer, where contextual menus are supported, so the user can easily access the avaiable functions for the elements in the scene.
  • The user can turn on/off some channels when rendering the clouds.
Short notification
Sunday, June 24, 2012
../_images/gsoc123.png

After thinking and discussing about it I have decided to go on and implement a region growing method based on a boundary map and the fixation point. Calculating and setting the boundary map will be left up to the user the method segmenting the region of the fixation point that is enclosed by w boundary. I will also shortly add an example of how this is done in pcl/examples.

Tests launched today!
Friday, June 22, 2012
../_images/gsoc124.png

This week I’ve lost some time to debug my repeatability calculator. Since I had some misunderstanding about how the keypoint repeatability shall be computed, I’ve decided to devote a little section of my blog to this aim. You can find how to compute the absolute and relative repeatability here. After having solved such issues, I spent some time in defining more suitable parameters for the detectors under consideration. To this aim, I used the frameworks for simple keypoint detection evaluation since they allow to visualize results and so to immediately detect errors. I have also changed something in the visualization.

Now, the display of the model shows:

  • all the keypoints extracted from the model by applying a specific detector (green color)
  • all the model keypoints that are not occluded in the scene (red color)
  • all the model keypoints that are repeatable (blue color).

While, the display of the scene shows:

  • all the keypoints extracted from the scene by applying a specific detector (fuchsia color)
  • all the scene keypoints that are not occluded in the model (green color).

Tests have been executed on synthetic data. Here, I post two screenshots related to the Harris3D detectors:

  • model view
    ../_images/model_screenshot027.png
  • scene view
    ../_images/scene_screenshot046.png

Finally, today I’ve launched the first evaluation tests, so in the following days I will post the available results.

First functional plugin!
Friday, June 22, 2012
../_images/gsoc126.png

The normal estimation plugin works as you might expect it would- it calculates normals. It uses the undo/redo framework, and the work_queue system, so you can undo/redo adding normals to your project as much as you want, and calculations are done in a separate thread, so the GUI doesn’t lock up while it’s thinking.

I’ll add progress bars soon, but that begs the question, how can I estimate progress for pcl functions? I can emit progress updates BETWEEN individual PCL calls in a plugin (such as between KD tree calculation and normal estimation in the normal estimation plugin) but getting real timing info would require putting some sort of macro in the functions themselves.

Another consideration is how tools should be activated. Right now I have a button which you click, which runs the selected tool. This of course is only temporary, but I’m not sure what the best replacement is. For selector tools, it’s pretty easy, but for things like adding normals or segmentation, what’s the most intuitive way of activating the tools?

High Curvature Edges
Friday, June 22, 2012
../_images/gsoc121.png

During this week, I have implemented ‘high curvature edge’ estimation. Here, the high curvature edges are defined as ridge or valley edges that are not on objects’ boundaries. These sharp edges could be useful in registration, recognition, and tracking applications. I first tried the curvature values obtained from normal estimation, but it turned out that a simple thresholding scheme using curvature values does not work very well. Especially, it was hard to get clean and thin edges on these high curvature regions. I noticed that non-maximum suppression and hysteresis thresholding are required. So I employed a canny edge implementation (pcl::pcl_2d::edge::canny()). Instead of using the RGB values available in the given organized point cloud, I used ‘normal_x’ and ‘normal_y’ images, since high gradient responses on these normal images correspond to high curvature regions.

Following images show the detected occluding (green), occluded (red), boundary (blue), and high curvature (yellow) edges:

../_images/screenshot-1340407069.png ../_images/screenshot-1340407079.png ../_images/screenshot-1340407086.png
Qt Model/View and Thread
Thursday, June 21, 2012
../_images/gsoc1212.png

Now the point clouds and render windows are managed by the scene tree, and the thread support is added for the workers. QThread document seems to be a mess, and there’s a lot of discussion on it, finally I took the one that seems to be the best practice of using QThread(http://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/).

A Quick Update on Plugins
Thursday, June 21, 2012
../_images/gsoc126.png
Work is progressing on the plugin tool, undo/redo command, work queue. After spending quite a bit of time thinking about how the architecture would work, I had a basic idea of how the components would connect together. So, after making the interface and basic headers, I began writing the first plugin tool (normal_estimation). As I went along, I realized certain parts of the design weren’t practical, or inefficient (such as having commands handle all memory and preventing plugins from reading from the model items directly, also, having the work queue dispatch multiple worker threads at once). Overall though, the design is what I showed in my last post, and things are coming together well, even if somewhat slower then I would have hoped. Once the first plugin is finished, things should progress quickly in getting the other tools working, since, for the most part, I’ll just be implementing the tutorials as plugins.
Quick update
Thursday, June 21, 2012
../_images/gsoc123.png

I’m trying to push harder this week to try and get an implemented working version of the active segmentation approach. I started inserting my already existing code parts in the PCL API, created the base classes and added the basic functionalities to it. I’m planing on creating a flowchart to illustrate how things work, but life keeps getting in the way:). Anyway, I am having a little bit of trouble on deciding which way to go after I have the fixation point and the boundary map. I have to decide on doing this the way the guys did it in the original publication and implement a mapping between the 2d image and point cloud, or I could try implementing it using region growing having the fixation point as the seed. I will have to decide on this, or maybe try out both and see where that leads me.

Results from implementations
Wednesday, June 20, 2012
../_images/gsoc129.png

There was a PCL tutorial at CVPR recently, where some of the upcoming features in PCL were also presented. This included some of the work that I have been doing. Following are some of the results that I obtained :

Edge Detection :

../_images/edge.jpg

Morphological Operations :

../_images/morphology.jpg

Harris Corners :

../_images/harris.jpg
Modified Voxel Grid Filter for PointXYZRGBL
Wednesday, June 20, 2012
../_images/gsoc122.png

To be able to construct a Conditional Random Field from a point cloud I am using a voxel grid for sensor input discretization. This allows to work in grid cells rather than euclidean distances. The problem what I was facing was the following, which made it necessary to extend the current voxel grid implementation with a new subclass inherited from pcl::VoxelGrid<PoinT>. The current implementation of the voxel grid filter does a linear interpolation over the position and PointCloud field lying in the same grid cell. This is problematic for the PointCloud type PointXYZRGBL, which assigns a label to any point in the cloud. By interpolating these labels can become wrong since they are just simple unsigned integer values. In my implementation I modified the voting scheme for the field ‘label’. If many point lie in the same grid cell, the label number with the highest occurrence wins.

../_images/voxel_grid_label.png

On the left side you can see the behaviour of the modified voxel grid filter. You can see on the right side, which is the standart voxel grid filter, that the labels are wrong due to the interpolation.

First segmentation results
Wednesday, June 20, 2012
../_images/gsoc122.png

I implemented and modified the following two paper to handle n-dimensional data inputs given as a point cloud.

  • Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials, Philipp Krähenbühl, Vladlen Koltun
  • Fast high-dimensional filtering using the permutohedral lattice, A. Adams, J. Baek, and M. A. Davis.

The input for the segmentation algorithm is a point cloud with XYZRGB (will be extended further) ordered and unordered. To convert the input point cloud into a Conditional Random Field I am using a modified voxel grid for scene discretization. Each cell with a measurement becomes a node in the graphical model. For now the edge potential features incorporate position as well as color information.

As a first step I am using the algorithm for supervised segmentation, hand label an input scene. In the following picture the input cloud (left) and the labeled cloud (right) can be seen.

../_images/input_label.png

With the labels are used to initialize the unary potentials of the CRF. The potentials are initialized as follows. A point with associated label gets a 0.3 probability that the label is correct. Further more I am assigning 10% of the points a randomly chosen wrong label.

In the next image you can see on the left the the noisy point cloud initialized with the unary energies. On the right you can see the result after segmentation.

../_images/unary_seg.png
pcl::visualization::PCLPlotter
Monday, June 18, 2012
../_images/gsoc127.png

I completed the first draft of pcl::visualization::PCLPlotter which is available in the trunk. So, you can update your trunk to use it and give feedback and suggestions.

Using this class one can plot graphs, given the point correspondences. Several types of plots can be done which includes line, bar and points.

It also includes API for plotting histogram given data. This is a similar and does the same type of functionalities as done by the histogram plotting function (hist) of matlab.

The type of plot which can be created using this class is almost same as that of using its building block VTK classes. There is very less restriction without having any knowledge of VTK and its pipeline, which makes it powerful. I am adding some snapshots of the output of the plotter class to show its extent.

../_images/sinecos.png ../_images/sinecos1.png ../_images/histuniform.png ../_images/group.png ../_images/smoothred.png
Parameter and Worker
Monday, June 18, 2012
../_images/gsoc1212.png

It seems vtk doesn’t have ready to use draggers...so I leave it for now. Instead, I implemented a rough support for invoking workers and setting up the parameters for the workers. I will polish it and add support for more workers.

Android + PCL?
Monday, June 18, 2012
../_images/gsoc1211.png

Having a hard time replicating the result here.

http://dev.pointclouds.org/projects/pcl/wiki/How_to_compile_PCL_for_Android

My next task is to update the instruction on the mobile_apps directory and compile a binary that I can use my Android development. The mobile_apps directory will be updated with our new code for Android next. Most likely the first thing we will see is a port of OpenNI + PCL sample on Android. Then, a simple tracking will be added to it next.

I really wonder what the performance is like on Tegra 3. That will be my first report with some sort of side by side comparison. A few optimization would be added if I can get the code compile properly first.

Hectic Times
Sunday, June 17, 2012
../_images/gsoc127.png

Several things happened in the past weeks. It has been very busy and hectic.

Firstly, I got relocated to Chennai (about 1500 KMs away from my home) for my job after undergraduate. I never expected the joining date to be this early. Usually, joining starts around mid July for most of the companies recruiting from our college. But, it seems that I got little (un)?lucky! Whatever the case maybe, I will make sure that this work on PCL goes smooth and gets completed.

Before the relocation, I pushed myself hard to figure out the path, approach, and things I will use to get my API classes done. Browsing over vtk classes and getting help from Marcus, I made a concrete logical design (the way VTK objects should interact in the PCL API) before moving to this new place.

Finally, after relocation, coding in the night, I completed the PCLPlotter class which is now available in the trunk under visualization. I will ask the PCL community about what else they want from the class. I will post about the features of this ‘plotter’ class in my coming blog.

A framework for keypoint detection evaluation
Saturday, June 16, 2012
../_images/gsoc124.png

I spent the last two weeks working on a framework for keypoint detection evaluation in PCL. First, I’ve developed simple evaluators, one for each detector under consideration. The evaluators take as input a pair of files (model + scene) and they first set the right groundtruth rotation and translation. For each detector, keypoints are extracted from both the model and the scene cloud based on a set of typical parameters. Some parameters are kept fixed for each detector in order to insure the fairness of the tests. In particular the detectors are tested on various scales and the scale is considered as a multiple of the model resolution. Finally, the results are displayed by means of the PCLVisualizer: both keypoints, repeatable keypoints and keypoints that are not occluded in the scene are shown. I show here the visual results achieved by the NARF keypoint detector on data acquired by means of the Microsoft Kinect sensor:

  • model
    ../_images/model_screenshot0119.png
  • scene
    ../_images/scene_screenshot0115.png
  • Absolute repeatability: 7

  • Relative repeatability: 0.146

The objective of this week was to extend the simple framework evaluators to run on entire datasets. The values computed refers to the average absolute and relative repeatability. Recently, a new idea has come: the repeatability measures will be accompanied with an analysis about the detectors’ time performances. Time performances will be computed based on the extraction time of the keypoints. In the weekend, I plan to define some final parameters of the detectors and to add the time performance analysis to the frameworks. Finally, my roadmap is constantly updated with my recent advances.

Coming soon: tables and graphs about this work.

Progress on Active Segmentation
Saturday, June 16, 2012
../_images/gsoc123.png

As I mentioned in my first entry, I have decided to implement/adapt the Active Segmentation with Fixation approach first, developed by A.Mishra. As a first step I took the time and thoroughly studied the works of the aforementioned author, and found that there are several improvements to the first approach. Details on this can be found on the authors home page.

For those who are not familiar how active segmentation based on fixation works and don’t want or don’t have the time to read the initial paper here is a short description. The reasoning behind the choice of segmenting based on a fixation point originates from the way humans perceive objects in their surroundings. This method proposes an approach where we do not segment the whole image in several regions and then reason on what these regions might be, but segment only the region which contains our fixation point. So in short segmenting an object from the scene implies finding a “fixation point” and finding the contour of the object that encloses our point of interest. The main steps of the segmentation algorithm are(these steps are for rgb images using monocular cues):

  1. Obtain the boundary map
  2. Improve boundary map using using monocular cues.
  3. Set fixation points (points chosen must be part of the object we want to segment out).
  4. Convert to polar coordinates around the fixation point
  5. Search for shortest path/ apply graph-cut algorithm in order to find the contour of the object.
  6. Convert back to Cartesian space, points found on the “left” of the shortest path in polar space belong to our object of interest.

Later works improve on this method among other things by adding an automated fixation strategy for finding interest points and by using depth information. Although, as stated in the work of A. Mishra we can find object boundaries by checking for depth discontinuities, but often these boundary points do not correspond to the true boundaries of the objects. In order to find the true boundaries color and texture cues are recommended. While the purpose of my project is the implementation of the segmentation algorithm, if I find the time for it, I will pursue the implementation of a fixation strategy.

Implementation wise I have created the basic API of the segmentation based on the the other segmentation algorithms already implemented in PCL. I played around with existing implementations of boundary detection and I realized that I will probably be able to use the code of my fellow GSOC developer Changhyun Choi. I am looking forward to seeing how his different implementations of edge detection will influence the performance of segmentation.

I will be back with more information soon (mostly on implementation issues).

Multiple render window, scene tree and context menu
Friday, June 15, 2012
../_images/gsoc1212.png

PCL Modeler is able to render point cloud in specified render window, the scene tree is provided to keep a track of the objects in the scene, and context menu is provided to make functions more accessible to specified objects. The basic “view” related things are ready, and I will implement draggers to set the initinal positions for registration.

A Closer Look at the Framework
Friday, June 15, 2012
../_images/gsoc126.png

Well, I’ve spent the last week or so putting the model/view stuff together into a useable GUI, which is available in the trunk under apps/cloud_composer... but before you run off and try to use it, let me warn you... it doesn’t do anything (other then show you clouds). This is because the next major step in the project, the tools plugins, are just getting started. More about that later though, first lets go over the current state of the Model/View structure.

Lets go over the components:

  • Project Model: This is primary model “container” of the app. It can contain an unlimited number of clouds, which are stored in a tree structure. The top level of the tree (children items of the root item) are clouds. The children of the clouds are things like normals, point feature histograms, labels, calculated models, etc... All of these are stored in their own items, which are derived from abstract base class ComposerItem. They can represent any sort of data, but shouldn’t actually contain the data, but simply store pointers to the data. Currently the pointers are stored in QVariant objects, allowing me to use the standard Qt get/set data functions... but I think this may change in the future, as I think I’ll probably need finer control over what the set/get data functions are doing. In any case, along with pointer(s) to the data, the ComposerItems have another important member:a QStandardItemModel, which contains its properties. These are things like height/width for a cloud, or radius for normals.

These properties can be editable or not (though if they are, the plugin which defined the item will need to specify slots which can handle what happens when a property changes).

  • CloudBrowser - This is the main tree viewer for the project model, showing what clouds are in the project, and what items those clouds contain. Selecting an item or cloud here will bring it up in the CloudInspector.
  • CloudInspector - This shows the properties of an item (which is a model itself, contained within the item) in a tree view. It allows editing of fields if possible. Since the properties are a model themselves, one can easily specify that widgets should be shown here for a property, such as a QSlider for a parameter which has a range.
  • CloudViewer - This is a tabbed view which shows the projects currently open as different tabs. Each tab contains its own QVTKWidget and PCLVisualizer. When a new tab is selected, the current model for the cloud_composer application is switched. This makes it very simple to switch back and forth between projects, and ensures that all the views are updated correctly whenever a switch occurs.
  • UndoView - This is a QUndoViewer, showing the current undo/redo stack, and lets one jump back/forward in time by clicking. Many PCL operations aren’t reversible (and take a long time to compute), so we store clouds in the stack, so undo/redo just amounts to switching the pointers in the cloud_item(s) which were modified.
  • Tools - this still just a blank toolbox, but will contain icons for the different plugins. More on plugins after the image.
../_images/PluginArchitecture.png

So that’s the plugin framework in a nutshell. None of that XML stuff I mentioned before, though that may come at a later date (i.e. after GSoC). Let’s go over what happens when a user clicks on a tool icon.

  1. Clicking on the tool once causes the tool parameter model to be displayed in the tool parameter view.
  2. Clicking on the tool again causes the tool action to trigger, unless it is something like a drag selector, in which case clicking in the cloud view triggers the action.
  3. The action triggering causes the tool’s factory to create a new instance of the tool and a new instance of one of the command types (merge, split, modify, delete, create). These commands objects are all defined in the GUI code, not inside of individual plugins. This is very important; plugins do not actually interact with the model, only commands do. This makes things like undo/redo and safe multithreading feasible.
  4. The cloud command is sent to the work queue, which is responsible for spawning threads where tools do their processing. It maintains a reference count of what items from the project model are currently being processed, and pops commands off the stack until the top command depends on data currently being worked on. The command object is responsible for making a copy of the item being worked on, and sending the copy and the tool object off to their thread to go do work.
  5. When a tool returns with a result, the command will push the old value onto the undo/redo stack, and send the resulting data back to the project model.

That’s about it for now... I’ll let you know when I’ve got a demo tool up and running; I’m starting with normal estimation. I’m sure some of this architecture will be changed as I come across things that aren’t possible, or think of better ways to do things. If any of you see something of that nature right now (won’t work, or a better way) please let me know! Also, you get a cookie for having managed to read all the way through this.

Design CUDA functions
Thursday, June 14, 2012
../_images/gsoc125.png

Design CUDA functions based on PF on PCL and Kinfu.

  • ParticleXYZRPY, PointXYZRGB -> should be changed
  • Octree for search nearest point.
  • Random number generator for sampling.
  • Reduction step for ‘update’ function
  • After complete one period, I need to think about more general PF.
Continue my work from NVSC.
Thursday, June 14, 2012
../_images/gsoc1211.png

I believe today I will be starting updating the blog on the GSoC side, and continue my work on the Tegra 3 + Android + OpenNI + PCL etc...

See http://www.pointclouds.org/blog/nvcs/raymondlo84/index.php for my previous posts on OpenNI ports and other tricks.

Now the leading issue is to get PCL running on my Tegra 3 tablet. I hope I can get a decent performance out from the box.

First few weeks in GSoC
Monday, June 11, 2012
../_images/gsoc129.png

I’ve been working on the project “Organized Point Cloud Data Operations”. I’m adding some functionality for 2D images in PCL. These functions can be used on the RGB information or any single-channel information contained in organized point clouds. I use a datatype vector< vector< float> > to represent images. This choice makes it easy to integrate it with the PCL datatypes. Also, because it uses such simple data-types and does not have any PCL-specific dependencies right now, one can just pluck this 2D library from PCL and use it anywhere.

I found that there were some image write functions in the pcl_io module, but no read functions! So, I created some simple image read/write functions using VTK, which I will later place in pcl_io.

Till now, I’ve implemented things like convolution, edge detection and morphological operations. I’ve compared the outputs to that obtained from OpenCV implenmentations. I used some synthetic images to check for corner cases. Everything seems to be working properly! I’m yet to write gTests to do a pixel-to-pixel level comparison.

A lot of energy actually went into adapting to the PCL style of coding, writing code comments and documenting my work. Initially I used uncrustify to check for styling errors before committing. Then I found a PCL styling file for Eclipse which made life much easier. Still adapting to the “PCL style” of coding which is quite different to my default style. I guess it will take some time :)

Will be adding a lot more functionality to this module in the coming few weeks. Hope to have some pretty pictures to add in the next blog entry.

Occluding & Occluded Boundary Edges
Friday, June 08, 2012
../_images/gsoc121.png

For the last couple of weeks, I have developed a boundary edge detection (pcl::OrganizedEdgeDetection) for an organized point cloud. It mainly searches for depth discontinuities with a given threshold value which is linearly adpated with respect to depth values. Since the point cloud is organized, operation is quite efficient. While pcl::BoundaryEstimation takes several seconds, my unoptimized code takes about 70 ms. It also returns edge labels: occluding, occluded, and boundary (i.e. neither occluding nor occluded) edges.

In Kinect or similar sensors, it happens that ‘nan’ points exist between occluding and occluded edges. So my algorithm searches for corresponding points across the ‘nan’ area. This search is done in an organized fashion, so it isn’t so time consuming.

Following images show the detected occluding (green), occluded (red), and boundary (blue)edges:

../_images/screenshot-1338586968.png ../_images/screenshot-1338586994.png ../_images/screenshot-1338587035.png
Model/View Framework for PCL
Sunday, June 03, 2012
../_images/gsoc126.png

So, I’ve been working on the basic GUI, and I’ve decided to go with a classic model/view architecture, using the Qt framework. In architecture, the data the user sees is encapsulated inside of a model, which can then be viewed using various GUI elements. This decouples the data objects from the viewing interface presented to the user, which increases flexibility and reuse. So, the architecture looks something like this:

../_images/CloudComposerArchitecture.png

The core idea here is that we have the CloudModel as the core object, maintaining references to the multiple clouds it may contain - multiple being necessary to allow things like registration, or segmentation of points into distinct clouds to be manipulated. The CloudModel maintains information about the clouds, which can be viewed in treeform in the CloudBrowser. This will look very much like the Pipeline Browser in Paraview. Additionally, clicking on an element in the browser will display further information about the selected element in the CloudInspector. Things like number of points, datatype, etc... It will also allow the adjustment of properties; say you have normals selected, it will allow you to adjust the radius used to calculate them.

There’s also the CloudToolSelector, which is an interface to the plugins which provide PCL functionality. As I said in my previous post, I’m still on the fence on how to implement the plugins. Ideally, I’d like them to be generated automatically based on XML description files, but it remains to be seen how difficult that will be, and if it is even possible due to the templated nature of all the PCL functions.

Finally, there’s the CloudViewer, which implements a QAbstractItemView, and contains a QVTKWidget - PCLVisualizer. The eventual plan is to have this be a tabbed view, with tabs switching between cloud projects, ie, switching which model is being viewed. That will come later though, lets get it working with one project first...

In any case, I’ll push this basic framework (minus the tools) to the SVN in the coming days. Let me know what you think, and if anyone out there sees any flaws in this architecture, please let me know. This is my first foray into the model/view world, and I’d appreciate finding out if I’m doing something wrong sooner rather than later!

GUI for Manipulating Point Clouds
Wednesday, May 30, 2012
../_images/gsoc126.png

Hello everyone, I just wanted to give a belated introduction to this project, and a quick status update on what I’ve been up to. To begin with, the goal of this project is to develop a GUI which acts as a graphical means of using the various modules of the PCL. The basic idea is to develop something is similar to Paraview (without the distributed part, that may come later). Basically one can load multiple clouds, or capture them from an OpenNI device, and then apply PCL functions to analyze them, modify them, or merge them. The interface is a pretty standard Qt design, with docks on each side containing tools, a list of clouds in the current window, and a bottom dock with text output. PCL calls are performed in separate threads of course. I have the basic application layout done, with basic functionality - loading/saving clouds and viewing them using the PCLVisualizer class. I’ll be pushing it to the server as soon as I get back from this review meeting in Denmark. I’d like to apologize for the slow start here, I haven’t been home in 3 weeks now thanks to conferences and meetings, and so all I’ve really been able to do is read. On that note, I’d like to discuss what I’ve been reading, and what I intend to do with it. Let’s start with what I, and I assume the community, wants. Namely, a GUI application which is easy to maintain and extend as the PCL code behind it evolves:

  • Changes in underlying algorithms should have no effect on the GUI.
  • Changes to module interfaces should require as little change in GUI code as possible.
  • Adding new functionality shouldn’t require editing the application code - when a programmer adds a new function, they should be able to add it to the GUI with minimal hassle and no without the possibility of breaking the rest of the app.

This leads us to a few conclusions. First of all, we need to isolate PCL functionality into a set of plugins. This could be one plugin per module, or one plugin per tool (ie FPFH calculation, SACModel calculation, Outlier removal, etc...), or any level of granularity in between. Next, the interface for these plugins should be as simple as possible, while still remaining flexible enough to allow all of the PCL functionality to pass through it. Finally, when someone adds a function to PCL, they should be able to add it as a tool in the GUI with minimal, if any coding. In my mind, this leaves us with two options:

  • A standard plugin system, where we define an interface, and then code a class for each tool, which performs the desired calls to PCL.
  • A meta-compiler system, where tools are specified in an XML format, and we either parse it at either run-time (to determine what to do when a tool is selected) or at compile time (to generate code which is used when the tool is selected).

The second option is obviously more complicated, but would be much easier to maintain in the long run, since the only code would be the compiler/parser. The XML specification of how to use the PCL would be relatively simple, which would make updating and adding tools as simple as changing/adding a few lines of XML (copied from a general template). In the first option, a new tool (or changes to a modules interface) would require editing the code of the plugin class. This means (imho) that tools would be much more prone to breaking. So, what am I reading?

  • Di Gennaro, Davide. Advanced C++ Metaprogramming, 2012.

Which is kind of blowing my mind. I’m feeling more confident about templates by the day, but I’m also beginning to think this may be overkill for the project. On the other hand, I’m not terribly interested in programming another standard plugin interface. That would mean I was basically spending the entire summer writing implementations which call PCL functions... which would be prone to breaking, and would require quite a bit of babysitting to keep the application running properly. I know how to do that, there’s nothing new there, and I’d just be making a clone of many other things which are already out there; just with a PCL backend. The XML version would be pretty novel (at least for me - Paraview does something somewhat similar), and would result in an application that would be very easy to extend as PCL evolves. On the other hand, the XML version is higher risk- it could result in a month of coding which fails miserably, followed by a frantic month of implementing the standard plugin class version.

Now if you’ve made it through all the text, I’d like to ask what do you guys think? Any suggestions, or advice? As I said, I’ll be at this review meeting until the end of the week, so I won’t be starting any serious coding of the plugin mechanism until next week. I would really appreciate a discussion of this over the next few days.

Oh, and what’s a good name for this beast? I’ve come up with the following two:

  • Cloud Composer
  • Cloud Jockey

Anyone have anything better?

Studying particle filter on PCL
Wednesday, May 30, 2012
../_images/gsoc125.png

Particle filter(PF) on PCL is little bit different with general algorithm.

PF on PCL it contains
  • resample - update each particles
  • weight - calc each particle weights based on coherence between reference cloud and query cloud
  • update - Update representative state of particles

In general PF, motion is given from system with uncertainty, but PF on PCL use camera motion calculated in prev frame.

PCL Datasets
Wednesday, May 30, 2012
../_images/gsoc1210.png

I finally convinced myself I had something to share and say. During the first week of the coding period I decided to tackle issue #682,. The current datasets when accessed online are a list (xml) generated directly though the svn. While it works and is very fast it is a bit simplistic and improvements can be made, for example utilizing the WebGL PCD viewer, auto-loading the readme files etc.

Long story short,

../_images/pcl_1.png

A lot of features have been added so far, including, binary compressed PCD support for the WebGL Viewer, search/filtering results, mirroring controls between pointclouds, loading the readme file instantly, drag n drop PCD files from your desktop etc - among others. This mini project will hopefully be online soon, and a detailed post with instructions and included features will be made then. You can preview it if you go to issue #682 and feedback would be greatly appreciated.

Apart from this, I have installed PCL, ran a couple of examples, and tested image encoding (jpeg/png) of depth/rgb data. Lossy image compression algorithms (jpeg) are optimized for images and not depth so unfortunately while the compression rate is phenomenal a lot of unwanted noise is added. Entropy encoding does not suffer from this of course. The advantage of using images and videos lies in the browser, see browsers support images and video natively (the internet would be very ascii-art-like if they didn’t) so no overhead is added in the Javascript for decoding, plus it is quite trivial to import images in the GLSL shaders for very fast analysis. I am still performing tests to find an optimal solution.

Fine reads,

Julius Kammerl “Development and Evaluation of Point Cloud Compression for the Point Cloud Library” Institute for Media Technology, TUM, Germany - May 2011,

Fabrizio Pece, Jan Kautz, Tim Weyrich “Adapting Standard Video Codecs for Depth Streaming” Department of Computer Science, University College London, UK, Joint Virtual Reality Conference of EuroVR - EGVE (2011)

2D Drawing APIs in PCL and VTK
Sunday, May 27, 2012
../_images/gsoc127.png

Played with PCLVisualizer and surprised to see that it already contains API (simple function calls like addCircle) to draw primitives like Circle, Cube, Sphere, etc. It supports many other visualization functions; no wonder it is very powerful, much more than the simple CloudViewer.

Also played with vtkContext2D and vtkChart and got overwhelmed with the amount of APIs (which bypasses the ‘unnecessary’ VTK pipeline) for 2D drawing and Charts. Next steps seems clear.

  • Extend these classes to form PCL flavored classes for 2D drawing and making charts.
  • Either make a new ‘visualizer’ by subclassing vtkContextView or make a way to connect those extended classes to PCLVisualizer.
../_images/chartdemo.png
‘Modern C++’
Friday, May 25, 2012
../_images/gsoc127.png

I was browsing the code of PCL, hoping for good old C++ with lots of pointers and clean readable inherited classes like that in VTK and QT (example). But instead, I found something like this and a light bulb went off in my head.

So, I understood that I must know and read about this ‘modern C++’style which heavily uses templates, ‘smart pointers’, boost, etc, etc.. before I proceed further to play with pcl code. After doing some googling, I found the following book to be the best and recommended for people like me:

  • C++ Templates: The Complete Guide / Vandevoorde & Josuttis

I have been reading this book and checking up boost, stl and pcl code since my last update.

Finally, I can now roughly understand pcl code and have started experimenting and exploring pcl::visualization::PCLVisualizer about which I will update soon.

Evolve from PCLVisualizer
Friday, May 25, 2012
../_images/gsoc1212.png

PCLVisualizer enclosed many useful functions for point cloud rendering, I planned to inherit from it but then found something has to be changed in the parent class to support what I want, so I just copied PCLVisualizer to PCLModeler for now, and let it evolve from there as I progress. It’s a simpler and faster solution since I can deal with PCLModeler only. They may be refactored to remove duplicate code if they do share a lot in the end.

Another thing I want to mention is that sometimes UI may have very different behaviors on different platform(Windows and Linux in my case), and it’s very painful to tune the code and then switch(restart!!) between them to make sure the code have consistent behaviors. Two computers are required in this case, store source code on one computer, make two builds, code on one platform(Windows in my case) and check results instantly on the other computer!

Testing existing boundary estimation in PCL
Thursday, May 24, 2012
../_images/gsoc121.png

As a first step for 3D edge detection, I would like to focus on edges from geometric shape (e.g. depth discontinuities or high curvature regions). Before I write codes, I have tested the existing functions in PCL which are pcl::NormalEstimation and pcl::BoundaryEstimation. Since the boundary estimation is designed for unorganized point clouds, it takes several seconds to process a Kinect frame. For an efficient boundary estimation, I will design a boundary estimation for organized point cloud (i.e. Kinect frame).

Following images are the curvature and boundary images from pcl::NormalEstimation and pcl::BoundaryEstimation.

  • Curvature

    ../_images/screenshot-1337804031.png
  • Boundaries

    ../_images/screenshot-1337804028.png
Testing functionalities for the existing keypoint detectors in PCL
Tuesday, May 22, 2012
../_images/gsoc124.png

I am currently developing a test program in order to visually compare the keypoints extracted by the existing detectors in PCL. The program looks like a tutorial and it is possible to set up the characteristic parameters of each detectors at the time the program executes. I chose to introduce this possibility since I have noted that in most of the cases the detector parameters are dependent from the features and the structure of the input point cloud.

In what follows, I show the keypoints extracted by the:

  • NARF keypoint detector

    ../_images/narf_screenshot041.png
  • SIFT keypoint detector

    ../_images/sift_screenshot034.png
  • keypoint detector based on the uniform sampling technique

    ../_images/uniform_screenshot056.png

while the examples related to the harris 2D, 3D and 6D detectors are still under development.

UI framework
Monday, May 21, 2012
../_images/gsoc1212.png

After some playing with vtk code, I am getting used to it. I’ve created an simple Qt based UI which supports load/save point cloud/project files and some utility functions. I will clean the code a bit, submit the diff for mentor reviewing and then make an initial commit.

Getting started
Monday, May 21, 2012
../_images/gsoc123.png

These past weeks I have been reading up on the two segmentation methods described in the papers (Mean-shift and Active Segmentation in Robotics) and in the meanwhile checking out the segmentation API and the available tutorials in PCL. I worked on setting up my environment and making everything work. My next steps will involve doing some research on the adaptation of Active Segmentation to 3D data as well as creating some draft classes and outlining the main steps of the method.

Initial arrangements
Wednesday, May 16, 2012
../_images/gsoc127.png

I was having difficulty in accessing repositories with svn+ssh as I was behind a proxy server in my university with port 22 blocked. So here is the solution: get a direct connection! (well not exactly, but you have to get port 22 open somehow to get ssh working without server’s help). Now I am in my home with everything working perfectly :).

Setting up environment
Wednesday, May 16, 2012
../_images/gsoc125.png

I prefer Windows and visual studio for development. I also prepared Ubuntu and development tools (cmake, code, build, subversion tools).

I’m reading papers and text about particle filter.

  • Min-An Chao; Chun-Yuan Chu; Chih-Hao Chao; An-Yeu Wu, “Efficient parallelized particle filter design on CUDA,” Signal Processing Systems (SiPS), 2010 IEEE Workshop on Digital Object Identifier, pp.299-304

TODOs

  • Analysing particle filter and CUDA API on PCL
  • Writing draft headers for particle filter
Reading days
Monday, May 14, 2012
../_images/gsoc124.png

My primary activities in the last weeks were setting up the working environment, doing some basic testing on the new 3D object recognition tutorial, and fixing feature #644 while solving some related bugs. Beyond these tasks, I also have been engaged in some important readings. I list them below, since they could be useful to someone who approaches to the object recognition field:

  • Salti, S.; Tombari, F.; Stefano, L.D.; , “A Performance Evaluation of 3D Keypoint Detectors,” 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 2011 International Conference on , vol., no., pp.236-243, 16-19 May 2011
  • Yu Zhong; , “Intrinsic shape signatures: A shape descriptor for 3D object recognition,” Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on , vol., no., pp.689-696, Sept. 27 2009-Oct. 4 2009
  • Tombari, F.; Di Stefano, L.; , “Object Recognition in 3D Scenes with Occlusions and Clutter by Hough Voting,” Image and Video Technology (PSIVT), 2010 Fourth Pacific-Rim Symposium on , vol., no., pp.349-355, 14-17 Nov. 2010

I want to do a final remarkable consideration, it is useful to those users who are approaching to the 3D object recogntition tutorial. To test the tutorial with a more extended dataset, you could refer to http://vision.deis.unibo.it/SHOT/ (scroll down till the dataset section is reached).

Setting up code structure
Friday, May 11, 2012
../_images/gsoc1212.png

I’ve prepared cmake files for placing PCLModeler under apps/ and set up a very simple UI for me to try and learn vtkPolyData. I think pcl_visualizer is a good reference for learning it.

Getting familiar to VTK
Monday, May 07, 2012
../_images/gsoc1212.png

I am new to VTK and after some digging I agreed the reputation that VTK has a steep learning curve seems to be true. However, it’s really versatile and for sure is competent for this project. I am still not very clear about the VTK pipeline, but I think it should be OK after I read more code examples.

Setting up development environment
Sunday, May 06, 2012
../_images/gsoc1212.png

I prefer to develop under Windows then double check and commit under Linux, and I’ve managed to get pcl compiled for both. Well, there are some errors for some modules, but I will leave it for now.

Test post
Sunday, April 25, 2010
../_images/gsoc128.png

Test post