PCL Developers blog

Ferenc Balint-Benczedi

email:ferenc.balint-benczedi@in.tum.de
project:Segmentation Algorithms
mentor:Koen Buys and Stefan Holzer

About me

I am a first year PhD candidate at the Intelligent Autonomous Systems Group of the Technical University of Munich, working in robotic perception. My main interests are computer vision, machine learning, AI systems.

Project Description

I will be working on adapting and implementing two segmentation algorithms into PCL: Mean-shift and Active Segmentation in robotics based on these two papers:

Roadmap

  • familiarizing with the two segmentation methods
  • implement Active Segmentation and testing
  • implement segmentation based on mean-shift clustering and testing it
  • find a method for obtaining interest points automatically for active segmentation

Recent status updates

reading code on MS
Saturday, August 04, 2012
../../_images/gsoc123.png

Tomorrow I am going on vacation for two weeks, I just hope I will be able to take the time to finish the API for MeanShift and get it up and running until the deadline. In my spare time I am checking out some other existing implementations of MeanShift, namely the Edison library and the one existing in OpenCV as well as some existing Matlab implementations. I’ve also wrote a first script doing segmentation based on color, but results were not exactly what I was hoping for.

starting work on mean shift segmentation
Wednesday, August 01, 2012
../../_images/gsoc123.png

I have started working on implementing mean shift. I am going to try to keep the API as modular as possible, thinking of the fact that mean shift can be used for a variety of things among which segmentation is only one. First I am just writing a script like program to check if the algorithm works, and then I’ll implement it in the API. (segmentation will be based on color space and on normals at a first try)

fixed problem
Wednesday, July 25, 2012
../../_images/gsoc123.png

I managed to fix the region growing, so now, if the seed point is on a plane paralell to the table top the method does not fail. Below you can see some screen shots of the

../../_images/screenshot-1344673807.png ../../_images/screenshot-1344674049.png ../../_images/screenshot-1344674128.png
fixing things
Tuesday, July 17, 2012
../../_images/gsoc123.png

This past week I have been fine tuning and rewriting the region growing part of the segmentation method in order to fix the eroneous segmentation shown in my last post.

adding normals and more
Sunday, July 08, 2012
../../_images/gsoc123.png

I’ve finaly managed to add the normals to the region growing algorithm and results are promising. I’ve conditioned adding of points to the region on the angle between the seed points’ normal and the current point. Results doing this are shown in the screen shot below.

../../_images/scene_better.png

Adding this condition helped and not at the same time. Although now growing stops when we reach the table top, parts of the object that are parallel to it don’t get added as well. This was expected of course. Still...better then the first try.

Thanks to a friend of mine I found out about this theory:

The interesting part for me is that according to this theory every object can be broken down into piece primitives and there are only 32 kinds of these primitive shapes and all of them are convex, meaning that complex objects can be separated into these parts at their concavenesses.

The following is an extract from the book From Fragments to Objects - Segmentation and Grouping in Vision T.F. Shipley and P.J. Kellman (Editors) 2001 Elsevier Science B.V. All rights reserved.

“The simplest shapes are convex shapes—whose outlines have positive curvature throughout (see, e.g., Rosin, 2000, for the role of convexity in parsing). If the outline of a shape has regions of negative curvature, especially if these regions contain salient negative minima of curvature, this usually indicates that the shape can be further parsed to give simpler subshapes. [...] Three main geometrical factors determine the perceptual salience of a part (Hoffman & Singh, 1997): (1) its protrusion, (2) its relative area, and (3) the strength of its boundaries. Salience of a part increases as its protrusion, relative area, or boundary strength increases. In this section we briefly consider protrusion and relative area, and then discuss in more detail the strength of part boundaries. We restrict attention to 2D shapes; the theory for 3D shapes is more complex and discussed elsewhere (Hoffman & Singh, 1997).”

“Hypothesis of normalized curvature: The salience of a part boundary increases as the magnitude of normalized curvature at the boundary increases.”

“Hypothesis of Turning Angle: The salience of a negative-minimum boundary increases as the magnitude of the turning angle around the boundary increases.”

Based on these theories I introduces another constraint to my region growing: when adding a point the angle between the line connecting that point to the fixation point and the points normal is checked. If this angle is concave it means that the point currently being verified belongs to the object. Doing so resulted in the following results (blue points are the ones segmented out and the green patch on the objects is the neighborhood of the fixation point):

../../_images/scene_1.png ../../_images/scene_2.png ../../_images/scene_3.png ../../_images/scene_4.png ../../_images/scene_bad_2.png

As it can be observed introducing that extra condition improved results a lot.The last scene is not a good result. My next step will involve investigating the cause of that erroneous segmentation. As a final observation, it was clear for me since the beginning, that the whole of this method is dependent on having a good fixation point. If this is not the case, the algorithm would not work at the moment. One of my next steps will be to investigate the method recently proposed by the authors of the original paper for automatic fixation point estimation.

With this occasion I would like to thank Zoltan-Cs. Marton for coming up with the idea of using convex vs. concave angles, it helped me a lot:). THX m8:)