[Documentation] [TitleIndex] [WordIndex

Proposed changes for next RViz release

RViz currently has a plugin system, but it is not documented and is fairly limited. I am proposing to:

Existing Plugin API

Display subclasses

Subclasses of Display can be defined in a plugin. They are described in the plugin's .yaml file and are connected to the running RViz by calls to TypeRegistry.registerDisplay() in the plugin's rvizPluginInit() function.

For example in rviz/src/rviz/default_plugin/init.cpp, we have:

   1 extern "C" void rvizPluginInit(rviz::TypeRegistry* reg)
   2 {
   3   reg->registerDisplay<AxesDisplay>("rviz::AxesDisplay");
   4   reg->registerDisplay<CameraDisplay>("rviz::CameraDisplay");

Help text and human-readable names are stored in the plugin's .yaml file. plugin.cpp associates the registered classes with the data in the yaml file.

The constructor for Display looks like this:

   1   Display( const std::string& name, VisualizationManager* manager );

registerDisplay<MyClass>("name") instantiates a DisplayCreatorT<MyClass> instance which has a matching create function:

   1   Display* create(const std::string& name, VisualizationManager* manager);

which calls the proper constructor with the name and manager arguments.

Display subclasses are instantiated by RViz when a new Display is added via the "Add" button in the Displays window.

Arbitrary classes

Arbitrary classes with any superclass you like can also be plugged in, given that they have a default constructor. These are registered like so:

   1 extern "C" void rvizPluginInit(rviz::TypeRegistry* reg)
   2 {
   3   reg->registerClass<XYZPCTransformer>("rviz::PointCloudTransformer",
   4                                        "rviz::XYZPCTransformer",
   5                                        "XYZ");
   6   reg->registerClass<IntensityPCTransformer>("rviz::PointCloudTransformer",
   7                                              "rviz::IntensityPCTransformer",
   8                                              "Intensity");

However their ClassCreator::create() function takes no arguments, so it is impossible to give them a VisualizationManager. Also, there is no pre-specified time when these classes will be instantiated, as that must be written into a Display subclass.

The VisualizationManager* argument is critical, as it is the primary point of interaction with RViz internals.

Classes in RViz which a plugin will access

In addition to documenting the way Displays and other classes can be defined in plugins, I will also need to document VisualizationManager and other classes which the plugins access in order to do their work.

Changes to Plugin API


The same functionality offered by registerDisplay could be achieved by using registerClass. As this is a more general and cleaner approach, the API should be changed to use the former mechanism instead.

This implies a broad set of changes:


Current state: RViz has a Tool superclass which defines a mouse-interaction mode, and which can be chosen by a button which is placed in the tool bar area.

VizualizationManager keeps track of all Tools and handles Keyboard events to activate them. It offers a C++ API to add Tools and change the currently active Tool. It does not provide the option to remove a Tool. VizualizationFrame creates the GUI Toolbar holding buttons for all Tools and also creates the default set of Tools (Move Camera, Select, Pose estimate, Navigation goal).

Proposed changes: Add the capability to add Tools via the class registry. VizualizationManager would have to listen to the loading signal of all plugins and look for subclasses of rviz::Tool when a plugin is loaded. It already listens to the unloading signal, where it currently only refreshes the property manager. It would have to remove all Tools of the plugin being unloaded, notify VisualizationFrame, which would have to remove the corresponding button, and refresh the property manager tool_property_manager_. ToolPropertiesPanel would also have to listen to a newly introduced "tool removed" signal of VisualizationManager to sync with the changes.

Registering a new tool in a plugin would look like this:

   1   reg->registerClass<MyTool>("rviz::Tool", "my_plugin::MyTool", "My Tool");

As with displays, the Tools would have to provide a method initialize(std::string name, char shortcut_key, VisualizationManager*).

There will need to be a mechanism to assign shortcut keys to tools, e.g. trying to use the first letter and a dialog where the user can change the automatically assigned shortcuts. The key assignments will have to be saved in one of the config files.

The current set of Tools should then also become part of default_plugin.

View Controllers

RViz has a ViewController superclass, subclasses of which define various ways of changing the viewpoint in response to mouse motion.

Making view controllers pluggable will be very similar to doing it for tools, except that ViewsPanel provides the GUI widget for selecting them and needs to keep that list in sync (similar to the Tools panel).


The subcomponents of the RViz GUI are mostly subclasses of wxPanel, which can be docked and undocked from the main window frame. These can currently be added by plugins, but again only through Displays. Sometimes this is perfectly appropriate, such as for a video camera display. Sometimes it would be better for the new panel to simply appear when the plugin loads, and not create a new panel for each new instance of the display. For example, if a new "reset view" button was desired, it could be added in a panel plugin.

Similar to tools and view controllers, panels registered with registerClass() would be created and added to the GUI on plugin load and removed on unload. They would also be added and removed from the "view" menu.

For this to work, all such panels have to inherit from a common base class, say rviz::Panel. However, that poses a problem:

To make sure all such Panels inherit from wxWindow, one would usually make rviz::Panel a subclass of this. However, panels are ususally designed using wxFormBuilder, which creates classes which also inherit from wxWindow. Having a class which inherits from rviz::Panel and some generated class thus creates ambiguity (the diamond problem).

A way out of this would be to include a method wxWindow* getPane() into the interface of rviz::Panel, which would have to be implemented in derived classes and could return the object itself, or to require the Panels to register themselves by calling VisualizationManager::getWindowManager()->addPane(this).

Access Restrictions

It would probably make sense to put a layer of abstraction between RViz plugins and the implementation of RViz internals, to restrict access to internals. Currently there are very few such restrictions. The best example is WindowManagerInterface which is an abstract class describing the interface to the window manager. The implementor is VisualizationFrame, but the plugins do not see that.

However the manifest.xml file for RViz exports something like {{{-I rviz/src}}} in its cpp flags, meaning that all the internals of RViz are made available to plugin developers anyway. Binary packages of RViz also have the entire source code directory.

A little cleanup would not be hard and is probably a good idea, to prevent future changes to RViz internals from breaking plugins.

Interactive Objects

Multiple users have expressed interest in implementing controls directly in the 3D scene which the user can interact with via the mouse. For the pr2_interactive_manipulation project David Gossow has written just such a control, but it was a lot of work. With a new API within RViz I think we can make it easier and cleaner.

There is already a Pixel-perfect picking mechanism implemented in SelectionManager which requires everything that is added to the Ogre Scene to be registered (otherwise it will be rendered as-is into the selection buffer, which could cause confusion with pick colors used by other objects). The way this is done is that the creator has to create a SelectionHandler, which can handle multiple Ogre::Entity objects. SelectionManager takes care of creating a new Ogre::Technique named "Pick", which renders the entity in its pick color and is used to render the selection buffer.

SelectionHandler contains a set of virtual functions which will be called when the object is selected or deselected. One of these functions is bool needsAdditionalRenderPass, which will trigger an additional render pass when it returns true. This is used by point clouds, which will render all their points with a single pick color in the first pass and with different colors in the second pass. The reason behind this is the limitation on the number of pick colors used.

The only interaction it supports, however, is the one performed by the selection tool: selected objects create a set of properties which will be displayed in the Selection panel.

For supporting interactive objects, two things have to be added to this framework:

There also needs to be a new "Interact" tool, which on every mouse move

Extension: Interactive Markers

One of the most successful features of rviz are markers. The reason for this, I believe, is that they can be implemented with much less effort than a full-blown plugin and need no knowledge of the rviz and Ogre systems.

I propose to add an Interactive Marker message and a display, that allows to send a Marker (or collection of Markers) together with information about the possible user interaction on them to rviz. The Interactive Marker display would build on the new rviz infrastructure for interactive elements. The results of the user interaction would be channeled back in a defined way so whatever sent the markers can react on that user input.

Examples of this could be:

Implementation decisions:

This is only a very rough sketch of my ideas so far. Comments are welcome.

Adaption of ROS coordinate conventions

Current state

By default, Ogre uses the convention that x points to the right, y up and z to the back so it is a right-handed system. ROS also uses a right-handed system, however the axes are swapped so x points forward, y to the left and z up.

Working around this issue could simply be done by making all cameras face the positive x axis and having the z axis as their up vector. As both systems are right-handed, backface culling would still work as expected without changes.

Rviz however uses a complicated and non-intuitive workaround where the default coordinate convention of Ogre is used, but all Vectors and Quaternions coming from ROS and going back need to be converted using rviz::robotToOgre and rviz::ogreToRobot respectively (see rviz/common.h). Also, this is documented nowhere. What they do is basically this (though expressed more complicated in the source code):

Proposed changes

Possible risks are hidden in code that doesn't use the conversion functions but hardcoded conversions (which even exist within rviz, see CameraDisplay). Apart from that, this would mean changing small code pieces in many places within rviz, but would also be pretty straightforward, considering the above formulae.

I've created a ticket containing a patch for the current rviz trunk.

Question / concerns / comments

Enter your thoughts on the API and any questions / concerns you have here. Please sign your name.

Josh F

David G

Dave H

Results from meeting Dave / David

Action items:

2024-06-15 13:47