PCLFusion – How to use Kinect Fusion in PCL

• English (United States)
• 日本語

Why will not use Kinfu?

Kinfu is clone project of Kinect Fusion that included in PCL.
It has been development since before Kinect Fusion was included in Kinect SDK.

Currently, Kinfu has not been maintained for a long time.
Unfortunately, PCL doesn’t have resource for maintain it.
Also, It will require a lot of effort to develop in latest version development environment, because Kinfu is depended on OpenNI older version.^*1

^{*1 OpenNI 1.x is not supports Visual Studio 2012 and later.}

I propose that using Kinect Fusion instead of Kinfu.
You can not make to changes for improve algorithms, because it is not open source.
However, You can using the reconstructed data with Kinect Fusion in PCL.

PCLFusion

PCLFusion is a sample program that real-time converts and displays data from reconstructed with Kinect Fusion.
Sample Program has published following.

• PCLFusion

Initialize

Please create pcl::PointCloud for push the vertex data of Kinect Fusion, and create pcl::visualization::PCLVisualizer for display it.
In addition, You will register keyboard callback function for manually reset reconfiguration process.

// Initialize Point Cloud
inline void Kinect::initializePointCloud()
{
    // Visualizer
    viewer = boost::make_shared<pcl::visualization::PCLVisualizer>( "Point Cloud Viewer" );
    viewer->registerKeyboardCallback( Kinect::keyboardCallback, this );

    // Point Cloud
    cloud = boost::make_shared<pcl::PointCloud<pcl::PointXYZRGBA>>();
    cloud->is_dense = false;
}

// Keyboard Callback Function
void Kinect::keyboardCallback( const pcl::visualization::KeyboardEvent& event, void* cookie )
{
    // Reset Reconstruction
    if( event.getKeySym() == "r" && event.keyDown() ){
        static_cast<Kinect*>( cookie )->reset();
    }
}

Update

Then, You retrieve mesh data from volume data that reconstructed with Kinect Fusion.

At this time, If you set smaller value in sampling step, The frame-rate is drops because convert process takes time.
I recommend set “3” and more value in step for real-time processing.
(It mean is processing to thin out data like a voxel grid filter is performed.)

And, You retrieve vertex and color information from mesh data, convert it to pcl::PointCloud.
You can process it using PCL!

// Update Point Cloud
inline void Kinect::updatePointCloud()
{
    // Calculate Mesh Data
    const UINT step = 3;
    ComPtr<INuiFusionColorMesh> mesh;
    HRESULT ret = reconstruction->CalculateMesh( step, &mesh );
    if( FAILED( ret ) ){
        return;
    }

    // Retrieve Vertex Count
    const unsigned int verticesCount = mesh->VertexCount();
    if( !verticesCount ){
        return;
    }

    // Retrieve Vertices
    const Vector3* vertices = nullptr;
    ERROR_CHECK( mesh->GetVertices( &vertices ) );

    // Retrieve Colors
    const int* colors = nullptr;
    ERROR_CHECK( mesh->GetColors( &colors ) );

    // ReSet Point Cloud
    cloud->clear();
    cloud->width = static_cast<uint32_t>( verticesCount );
    cloud->height = static_cast<uint32_t>( 1 );
    cloud->points.resize( cloud->width * cloud->height );

    // Convert Mesh to Point Cloud
    Concurrency::parallel_for( static_cast<unsigned int>( 0 ), verticesCount, [&]( const unsigned int index ){
        pcl::PointXYZRGBA point;

        const Vector3 vertex = vertices[index];
        point.x = vertex.x;
        point.y = -vertex.y;
        point.z = -vertex.z;

        const uint32_t color = colors[index];
        point.rgba = color;

        cloud->points[index] = point;
    } );
}

Draw and Show

You will draw and show Point Cloud in PCLVisualizer.

// Draw Point Cloud
inline void Kinect::drawPointCloud()
{
    // Update Point Cloud
    if( !viewer->updatePointCloud( cloud, "cloud" ) ){
        viewer->addPointCloud( cloud, "cloud" );
    }
}

// Show Point Cloud
inline void Kinect::showPointCloud()
{
    // Spin Viewer
    viewer->spinOnce();
}

Execution Result

If you execute this sample program, Point Cloud data that reconstruct with Kinect Fusion are displayed in real-time.
If frame-rate is drops, You should change settings of sampling step.