PCL Developers blog
Code Sprints
Q: What is a code sprint?
A code sprint is an accelerated research and development program, where we pair talented developers with senior researchers and engineers for 3-6 months of extended programming work. The model is inspired by the Google Summer Of Code initiative, and is meant to create open source solutions for interesting 2D/3D computer perception problems. Each code sprint is financially supported by a different institution (see below).
Code sprints are made possible through the involvement of our host organization, Open Perception. Please see our mission page for more information.
Q: What does this blog represent?
The Point Cloud Library (PCL) developer blog represents a great way to keep track of daily updates in PCL sponsored code sprints. Our developers are writing about their experiences and progress updates while working on exciting PCL projects.
Active Sprints
The following shows the current list of ongoing code sprints. Click on any of the logos below to go to the respective sprint blog page.
Completed Sprints
The list of code sprints which have been completed is:
We would like to thank our financial sponsors for their generous support. For details on how to start a new code sprint or contribute to PCL, please visit http://www.pointclouds.org/about and our organization’s web site at http://www.openperception.org/get-involved/.
Latest 15 blog updates
A few words about the project: the goal is to detect a drivable area (continuously flat area, segmented by a height gap such as curb). As an input we have two rectified images from the cameras on the car’s roof and a disparity map. An example of such images is below.
The point cloud, that was computed from them:
The point cloud is converted into the Digital Elevation Map (DEM) format to simplify the task. DEM is a grid in column-disparity space with heights associated to each node. A projection of DEM onto the left image is illustrated below.
On the following image you can see that despite of a low resolution of the DEM it is still possible to distinguish the road from the sidewalk.
A front view of the DEM in 3D space (nodes without corresponding points, i.e. the disparity map had no points, that should be projected onto this node, are marked with red):
DEM as a point cloud:
As a starting point we are going to implement an algorithm: J. Siegemund, U. Franke, and W. Forstner, ?A temporal filter approach for detection and reconstruction of curbs and road surfaces based on conditional random fields,? in Proc. IEEE Intelligent Vehicles Symp., 2011, pp. 637?642.
All source code related to the project can be found here.
The first milestone of the PCL Spectrolab Code Sprint has been completed. I received the Spectroscan 3D two weeks ago and I have used it to write a driver for the camera. In addition, I have created a simple 3D viewer for the camera which streams the data to a PCLVisualizer. All of the code for the driver (and the rest of the code sprint) can be found here. There are no Spectroscan3D units commericially available right now, but by the time I am done there will be a good set of open source tools to visualize and manipulate the data.
Here is an example scan from the camera.
An amplitude colorized view of an interior door.
An amplitude image of the interior door scan.
The Spectroscan3D driver is split into two components. The first component encapsulates the communication with the lidar camera. It speaks the lidar’s UDP protocol and handles receiving the raw range/amplitude images from the scanner. This component is meant to be independent of PCL. This way future developers can use it with other software systems as well. The second layer wraps the driver in pcl grabber interface and transforms the range image into a point cloud. You can register for either PointXYZ or PointXYZI point cloud events from the scanner.
The next stage of the code sprint will be writing a visualizer capable of controlling the camera like a video camera. It will be able to watch a live stream, record the stream, or play back an old stream from disk. Stay tuned for more updates!
The project “Stereo-based road area detection” has been started!
This is the first post of the Spectrolab PCL Code Sprint. Spectrolab is a Boeing company that has developed a new LIDAR camera. This camera is meant for industrial and robotics uses in both outdoor and indoor environments. It uses a scanning time of flight laser to generate a 256x128 range image at 5-6 hertz. It is still in development right now, but it promises to be a valuable tool for robotics and perception work. Spectrolab has sponsored integration of the camera into PCL so that when it hits the market, it can take advantage of the wealth of tools and knowledge within the PCL community.
As apart of the code sprint, I will be developing a new PCL grabber driver for the Spectrolab camera and a 3D viewer interface. As I complete the driver and software, I will post screen shots and code explanations to this blog.
Test
Here’s a video demonstrating the RobotEye Viewer application with the Ocular Robotics RE05 lidar sensor. See my previous blog post for a description of the application.
Last two weeks I’ve finished the NPP implementation of the Haar Cascade detector, it is now fully ported to PCL. I’ve included Alex Trevor his organized multi-plane segmentation to people api as well, so that people can do background subtraction based on the planes found in the image. This week I wrote the final report summarizing the work done as well as how to use the module.
I’ve finished developing a new pcl::Grabber called the pcl::RobotEyeGrabber, and a visualization application called RobotEye Viewer. The grabber uses the boost asio library to do asynchronous I/O on a UDP socket using a separate thread. The RobotEye sensor sends lidar data in UDP data packets which are converted by the grabber into pcl::PointCloud data. The RobotEye Viewer is a Qt application that uses the QVTKWidget to embed a pcl::visualization::PCLVisualizer visualization window.
I developed the code using MacOSX and Ubuntu Linux. I ssh to a machine that is located at the Ocular Robotics lab and has a live sensor connected to it. With this machine, I was able to run the RobotEye Viewer application remotely using the RE05 lidar sensor. Ocular Robotics setup a webcam that points at the RobotEye sensor so I can see it in action as I run the application remotely. This setup worked out very well. Here’s a screenshot of the application, stay tuned for a video.
The PCL Dinast Grabber Framework
At PCL 1.7 we offer a new driver for Dinast Cameras making use of the generic grabber interface that is present since PCL 1.0. This tutorial shows, in a nutshell, how to set up the pcl grabber to obtain data from the cameras.
So far it has been currently tested with the IPA-1110, Cyclopes II and the IPA-1002 ng T-Less NG but it is meant to work properly on the rest of the Dinast devices, since manufacturer specifications has been taken into account.
Small example
As the Dinast Grabber implements the generic grabber interface you will see high usage similarities with other pcl grabbers. In applications you can find a small example that contains the code required to set up a pcl::PointCloud<XYZI> callback to a Dinast camera device.
Here you can see a screenshot of the PCL Cloud Viewer showing the data from a cup laying on a table obtained through the Dinast Grabber interface:
And this is a video of the PCL Cloud Viewer showing the point cloud data corresponding to a face:
Dinast Grabber currently offer this data type, as is the one currently available from Dinast devices:
- void (const boost::shared_ptr<const pcl::PointCloud<pcl::PointXYZI> >&)
The code
The code from apps/src/dinast_grabber_example.cpp will be used for this tutorial:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | #include <pcl/common/time.h>
#include <pcl/point_types.h>
#include <pcl/io/dinast_grabber.h>
#include <pcl/visualization/cloud_viewer.h>
template <typename PointType>
class DinastProcessor
{
public:
typedef pcl::PointCloud<PointType> Cloud;
typedef typename Cloud::ConstPtr CloudConstPtr;
DinastProcessor(pcl::Grabber& grabber) : interface(grabber), viewer("Dinast Cloud Viewer") {}
void
cloud_cb_ (CloudConstPtr cloud_cb)
{
static unsigned count = 0;
static double last = pcl::getTime ();
if (++count == 30)
{
double now = pcl::getTime ();
std::cout << "Average framerate: " << double(count)/double(now - last) << " Hz" << std::endl;
count = 0;
last = now;
}
if (!viewer.wasStopped())
viewer.showCloud(cloud_cb);
}
int
run ()
{
boost::function<void (const CloudConstPtr&)> f =
boost::bind (&DinastProcessor::cloud_cb_, this, _1);
boost::signals2::connection c = interface.registerCallback (f);
interface.start ();
while (!viewer.wasStopped())
{
boost::this_thread::sleep (boost::posix_time::seconds (1));
}
interface.stop ();
return(0);
}
pcl::Grabber& interface;
pcl::visualization::CloudViewer viewer;
};
int
main ()
{
pcl::DinastGrabber grabber;
DinastProcessor<pcl::PointXYZI> v (grabber);
v.run ();
return (0);
}
|
The explanation
At first, when the constructor of DinastProcessor gets called, the Grabber and CloudViewer Classes are also initialized:
DinastProcessor(pcl::Grabber& grabber) : interface(grabber), viewer("Dinast Cloud Viewer") {}
At the run function what we first have is actually the callback and its registration:
boost::function<void (const CloudConstPtr&)> f =
boost::bind (&DinastProcessor::cloud_cb_, this, _1);
boost::signals2::connection c = interface.registerCallback (f);
We create a boost::bind object with the address of the callback cloud_cb_, we pass a reference to our DinastProcessor and the argument place holder _1. The bind then gets casted to a boost::function object which is templated on the callback function type, in this case void (const CloudConstPtr&). The resulting function object is then registered with the DinastGrabber interface.
The registerCallback call returns a boost::signals2::connection object, which we do not use in the this example. However, if you want to interrupt or cancel one or more of the registered data streams, you can call disconnect the callback without stopping the whole grabber:
boost::signals2::connection = interface (registerCallback (f));
// ...
if (c.connected ())
c.disconnect ();
After the callback is set up we start the interface. Then we loop until the viewer is stopped. Finally interface is stopped although this is not actually needed since the destructor takes care of that.
On the callback function cloud_cb_ we just do some framerate calculations and we show the obtained point cloud through the CloudViewer.
Testing the code
We will test the grabber with the previous example. Write down the whole code to a file called dinast_grabber.cpp at your preferred location. Then add this as a CMakeLists.txt file:
1 2 3 4 5 6 7 8 9 10 11 12 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
project(dinast_grabber)
find_package(PCL 1.7 REQUIRED)
include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})
add_executable (dinast_grabber dinast_grabber.cpp)
target_link_libraries (dinast_grabber ${PCL_LIBRARIES})
|
Then just proceed as a usual cmake compilation:
$ cd /PATH/TO/DINAST_EXAMPLE
$ mkdir build
$ cd build
$ cmake
$ make
If everything went as expected you should now have a binary to test your Dinast device. Go ahead, run it and you should be able to see the point cloud data from the camera:
$ ./dinast_grabber
Troubleshooting
Q: When I run the application I get an error similar to this one:
$ ./dinast_grabber
libusb: 0.000000 error [op_open] libusb couldn't open USB device /dev/bus/usb/002/010: Permission denied.
libusb: 0.009155 error [op_open] libusb requires write access to USB device nodes.
Where the last numbers of the /dev/bus/usb/... might vary.
A: This means you do not have permission to access the device. You can do a quick fix on the permissions of that specific device:
$ sudo chmod 666 /dev/bus/usb/002/010
Or you can make this changes permanent for all future Dinast devices writing a rule for udev. In debian-like systems it is usually done writing this:
# make dinast device mount with writing permissions (default is read only for unknown devices)
SUBSYSTEM=="usb", ATTR{idProduct}=="1402", ATTR{idVendor}=="18d1", MODE:="0666", OWNER:="root", GROUP:="video"
to a file like /etc/udev/rules.d/60-dinast-usb.rules.
If you still have problems you can always use the users mailing list: pcl-users@pointclouds.org to find some extra help.
Conclusions
With this new grabber a new kind of short-range sensors are available through the PCL Grabber interface. It is now a breeze to connect and obtain data from Dinast devices as you do with the rest of devices supported at PCL.
If you have any development suggestions on these or new devices you can contact us through pcl-developers@pointclouds.org.
It has been fast time since VLCS starting. Keven announces “pcl_hdl_simple_viewer” in Dec. 2012. Now, it supports live mode (network) and recorded file mode (PCAP file) with several visualization format (XYZ, XYZI, XYZRGB). He mainly develops under Linux-Fedora. I have tested and supported under Linux-Ubuntu and Windows 7.
For windows implementation, Hdl_simple_viewer needs WinPcap
- WinPcap user version, WinPcap development version (with environment setting PCAPDIR). It will be explained on document later.
- WinPcap Windows Installer : http://www.winpcap.org/install/default.htm
- WinPcap Developer’s Pack : http://www.winpcap.org/devel.htm
Unfortunately, PCAP is not stable under Windows 64bits. It takes a long time to know about it. So, I build up 32-bit development environment.
Ubuntu and Windows test image are shown as belows
Recording file mode(PCAP file) - road.pcap from velodyne website.
Ubuntu XYZ / XYZI / XYZRGB
Windows XYZ / XYZI / XYZRGB
Live mode (Live mode also provides three visualization format).
Ubuntu / windows
Ocular Robotics sent me some lidar scans to get started with. The data is stored in binary files with azimuth, elevation, range, and intensity fields. I wrote some code to load the binary data and convert it to the pcl::PointCloud data structure. From there, I saved the data in the pcd file format and opened it with ParaView using the PCL Plugin for ParaView. I used ParaView’s animation controls to create a video of the lidar scan:
I’m pleased to announce that PCL now supports the Velodyne High Definition Laser (HDL) -32 and -64 lasers. The interface is provided as a Grabber (HDL_Grabber) and accepts packets from the network (live) or PCAP file (recorded). Two sample programs, pcl_hdl_grabber and pcl_hdl_viewer_simple are provided to demonstrate the capabilities of the system. libpcap-devel is required to build the PCAP reading portions of code.
Sample PCAP Files are provided by Velodyne: http://velodyne.com/lidar/doc/Sample%20sets/HDL-32/
The image below came from the Building-Side.pcap.
I’m excited to start the PCL - Ocular Robotics code sprint. This week I’ll have a kickoff meeting over Skype with engineers from Ocular Robotics. In the meantime, I’m reading about the RE0x sensors and updating my PCL build directories!
The Honda Research Code Sprint for ground segmentation from stereo has been completed. PCL now includes tools for generating disparity images and point clouds from stereo data courtesy of Federico Tombari, as well as tools for segmenting a ground surface from such point clouds from myself. Attached is a report detailing the additions to PCL and the results, as well as a video overview of the project. There is a demo available in trunk apps, as pcl_stereo_ground_segmentation.
It has been quite a long time since my last post so I will give a full update on all the developments that I have taken care of in this time.
After a first implementation of the pcl::DinastGrabber for Dinast cameras (IPA-1002, IPA-1110, IPA-2001) I did some testing with multiple cameras. For that I mounted two of them on a robotics arm. After calibration, I got the combination of two pointclouds in one single view. The following picture shows the result of me standing in front of the two cameras.
Also here is the setup of the cameras on the robotics manipulator:
Then we used the RRT implementation with the robotic arm for planning purposes. The calibrated cameras where also used to build a collision shield around the robot. First RRT was run to get the path to the goal. When the robot was moving, collision checking was performed using the information obtained from the cameras. When an object was detected, the robot was stopped and RRT was run again in order to obtain a new path to the goal avoiding the detected object. The pic below shows the replanning part of the whole testing, while trying to avoid a box that was in the path to the goal.
