Interactive visualization

This tutorial introduces user interaction features of the visualizer window.

 27# examples/python/visualization/interactive_visualization.py
 28
 29import numpy as np
 30import copy
 31import open3d as o3d
 32
 33
 34def demo_crop_geometry():
 35    print("Demo for manual geometry cropping")
 36    print(
 37        "1) Press 'Y' twice to align geometry with negative direction of y-axis"
 38    )
 39    print("2) Press 'K' to lock screen and to switch to selection mode")
 40    print("3) Drag for rectangle selection,")
 41    print("   or use ctrl + left click for polygon selection")
 42    print("4) Press 'C' to get a selected geometry")
 43    print("5) Press 'S' to save the selected geometry")
 44    print("6) Press 'F' to switch to freeview mode")
 45    pcd_data = o3d.data.DemoICPPointClouds()
 46    pcd = o3d.io.read_point_cloud(pcd_data.paths[0])
 47    o3d.visualization.draw_geometries_with_editing([pcd])
 48
 49
 50def draw_registration_result(source, target, transformation):
 51    source_temp = copy.deepcopy(source)
 52    target_temp = copy.deepcopy(target)
 53    source_temp.paint_uniform_color([1, 0.706, 0])
 54    target_temp.paint_uniform_color([0, 0.651, 0.929])
 55    source_temp.transform(transformation)
 56    o3d.visualization.draw_geometries([source_temp, target_temp])
 57
 58
 59def pick_points(pcd):
 60    print("")
 61    print(
 62        "1) Please pick at least three correspondences using [shift + left click]"
 63    )
 64    print("   Press [shift + right click] to undo point picking")
 65    print("2) After picking points, press 'Q' to close the window")
 66    vis = o3d.visualization.VisualizerWithEditing()
 67    vis.create_window()
 68    vis.add_geometry(pcd)
 69    vis.run()  # user picks points
 70    vis.destroy_window()
 71    print("")
 72    return vis.get_picked_points()
 73
 74
 75def demo_manual_registration():
 76    print("Demo for manual ICP")
 77    pcd_data = o3d.data.DemoICPPointClouds()
 78    source = o3d.io.read_point_cloud(pcd_data.paths[0])
 79    target = o3d.io.read_point_cloud(pcd_data.paths[2])
 80    print("Visualization of two point clouds before manual alignment")
 81    draw_registration_result(source, target, np.identity(4))
 82
 83    # pick points from two point clouds and builds correspondences
 84    picked_id_source = pick_points(source)
 85    picked_id_target = pick_points(target)
 86    assert (len(picked_id_source) >= 3 and len(picked_id_target) >= 3)
 87    assert (len(picked_id_source) == len(picked_id_target))
 88    corr = np.zeros((len(picked_id_source), 2))
 89    corr[:, 0] = picked_id_source
 90    corr[:, 1] = picked_id_target
 91
 92    # estimate rough transformation using correspondences
 93    print("Compute a rough transform using the correspondences given by user")
 94    p2p = o3d.pipelines.registration.TransformationEstimationPointToPoint()
 95    trans_init = p2p.compute_transformation(source, target,
 96                                            o3d.utility.Vector2iVector(corr))
 97
 98    # point-to-point ICP for refinement
 99    print("Perform point-to-point ICP refinement")
100    threshold = 0.03  # 3cm distance threshold
101    reg_p2p = o3d.pipelines.registration.registration_icp(
102        source, target, threshold, trans_init,
103        o3d.pipelines.registration.TransformationEstimationPointToPoint())
104    draw_registration_result(source, target, reg_p2p.transformation)
105    print("")
106
107
108if __name__ == "__main__":
109    demo_crop_geometry()
110    demo_manual_registration()

This script executes two applications of user interaction: demo_crop_geometry and demo_manual_registration.

Crop geometry

37        "1) Press 'Y' twice to align geometry with negative direction of y-axis"
38    )
39    print("2) Press 'K' to lock screen and to switch to selection mode")
40    print("3) Drag for rectangle selection,")
41    print("   or use ctrl + left click for polygon selection")
42    print("4) Press 'C' to get a selected geometry")
43    print("5) Press 'S' to save the selected geometry")
44    print("6) Press 'F' to switch to freeview mode")
45    pcd_data = o3d.data.DemoICPPointClouds()
46    pcd = o3d.io.read_point_cloud(pcd_data.paths[0])
47    o3d.visualization.draw_geometries_with_editing([pcd])
48
49
50def draw_registration_result(source, target, transformation):
51    source_temp = copy.deepcopy(source)

This function simply reads a point cloud and calls draw_geometries_with_editing. This function provides vertex selection and cropping.

Note

Open3D has a VisualizerWithEditing class that inherits Visualizer class. It adds graphic user interaction features. Likewise examples in Customized visualization, VisualizerWithEditing() can be explicitly used instead of draw_geometries_with_editing([pcd]).

Once a geometry is displayed, press Y twice to align geometry with negative direction of y-axis. After adjusting viewing orientation, press K to lock screen and to switch to the selection mode.

../../_images/crop_lock.png

Tip

The practical step for selecting area is to align the geometry with arbitrary axis using orthographic projection model. This trick makes selection easier, because it avoids self-occlusion hassle due to perspective projection.

To select a region, use either mouse drag (rectangle selection) or ctrl + left mouse click (polygon selection). The below example shows a selected area using a polygon.

../../_images/crop_selection.png

Note that the selected area is dark shaded. To keep the selected area and discard the rest, press C. A dialog box appears, which can be used to save the cropped geometry. The cropping result is shown after saving.

../../_images/crop_save.png ../../_images/crop_result.png

To finish selection mode, press F to switch to freeview mode.

../../_images/crop_freeview.png

Manual registration

Select correspondences

The following script registers two point clouds using point-to-point ICP. It gets initial alignment via user interaction.

61    print(
62        "1) Please pick at least three correspondences using [shift + left click]"
63    )
64    print("   Press [shift + right click] to undo point picking")
65    print("2) After picking points, press 'Q' to close the window")
66    vis = o3d.visualization.VisualizerWithEditing()
67    vis.create_window()
68    vis.add_geometry(pcd)
69    vis.run()  # user picks points
70    vis.destroy_window()
71    print("")
72    return vis.get_picked_points()
73
74
75def demo_manual_registration():
76    print("Demo for manual ICP")

The script reads two point clouds, and visualizes the point clouds before alignment.

../../_images/manual_icp_initial.png
52    target_temp = copy.deepcopy(target)
53    source_temp.paint_uniform_color([1, 0.706, 0])
54    target_temp.paint_uniform_color([0, 0.651, 0.929])
55    source_temp.transform(transformation)
56    o3d.visualization.draw_geometries([source_temp, target_temp])
57
58
59def pick_points(pcd):
60    print("")

The function pick_points(pcd) makes an instance of VisualizerWithEditing. To mimic draw_geometries, it creates windows, adds the geometry, visualizes the geometry, and then terminates. A novel interface function from VisualizerWithEditing is get_picked_points() that returns the indices of user-picked vertices.

To pick a vertex, press shift + left click on a window. If a vertex is selected, the visualizer window overlays a sphere on a selected vertex. For example, after picking three vertices in the source point cloud, it shows:

../../_images/manual_icp_source.png

This will print:

Picked point #58481 (2.14, 1.56, 1.53) to add in queue.
Picked point #77321 (2.86, 1.92, 1.09) to add in queue.
Picked point #42639 (3.28, 1.53, 1.45) to add in queue.

Press Q to close the window. The next step is to pick the same correspondences in the target point cloud. The color of the sphere helps to identify the same correspondence.

../../_images/manual_icp_target.png

This will print:

Picked point #54028 (1.62, 1.81, 1.23) to add in queue.
Picked point #97115 (2.45, 2.19, 1.11) to add in queue.
Picked point #47467 (2.75, 1.71, 1.45) to add in queue.

Tip

To get a good registration result, try to pick more than three points that are well-distributed in the scene. Using a vertex in the corner region is a good way to easily pick the right correspondence.

Registration using user correspondences

 77    pcd_data = o3d.data.DemoICPPointClouds()
 78    source = o3d.io.read_point_cloud(pcd_data.paths[0])
 79    target = o3d.io.read_point_cloud(pcd_data.paths[2])
 80    print("Visualization of two point clouds before manual alignment")
 81    draw_registration_result(source, target, np.identity(4))
 82
 83    # pick points from two point clouds and builds correspondences
 84    picked_id_source = pick_points(source)
 85    picked_id_target = pick_points(target)
 86    assert (len(picked_id_source) >= 3 and len(picked_id_target) >= 3)
 87    assert (len(picked_id_source) == len(picked_id_target))
 88    corr = np.zeros((len(picked_id_source), 2))
 89    corr[:, 0] = picked_id_source
 90    corr[:, 1] = picked_id_target
 91
 92    # estimate rough transformation using correspondences
 93    print("Compute a rough transform using the correspondences given by user")
 94    p2p = o3d.pipelines.registration.TransformationEstimationPointToPoint()
 95    trans_init = p2p.compute_transformation(source, target,
 96                                            o3d.utility.Vector2iVector(corr))
 97
 98    # point-to-point ICP for refinement
 99    print("Perform point-to-point ICP refinement")
100    threshold = 0.03  # 3cm distance threshold
101    reg_p2p = o3d.pipelines.registration.registration_icp(
102        source, target, threshold, trans_init,
103        o3d.pipelines.registration.TransformationEstimationPointToPoint())
104    draw_registration_result(source, target, reg_p2p.transformation)
105    print("")
106
107
108if __name__ == "__main__":
109    demo_crop_geometry()
110    demo_manual_registration()

The later part of the demo computes an initial transformation based on the user-provided correspondences. This script builds pairs of correspondences using Vector2iVector(corr). It utilizes TransformationEstimationPointToPoint.compute_transformation to compute the initial transformation from the correspondences. The initial transformation is refined using registration_icp.

The registration result is as follows:

../../_images/manual_icp_alignment.png