Proposed changes for next RViz release
Contents
RViz currently has a plugin system, but it is not documented and is fairly limited. I am proposing to:
- Document the existing API
- Extend it to allow 3 new types of plugins:
- Tools
- View Controllers
- Panels
- Add built-in support for mouse interaction with 3D objects
Existing Plugin API
Display subclasses
Subclasses of Display can be defined in a plugin. They are described in the plugin's .yaml file and are connected to the running RViz by calls to TypeRegistry.registerDisplay() in the plugin's rvizPluginInit() function.
For example in rviz/src/rviz/default_plugin/init.cpp, we have:
Help text and human-readable names are stored in the plugin's .yaml file. plugin.cpp associates the registered classes with the data in the yaml file.
The constructor for Display looks like this:
1 Display( const std::string& name, VisualizationManager* manager );
registerDisplay<MyClass>("name") instantiates a DisplayCreatorT<MyClass> instance which has a matching create function:
1 Display* create(const std::string& name, VisualizationManager* manager);
which calls the proper constructor with the name and manager arguments.
Display subclasses are instantiated by RViz when a new Display is added via the "Add" button in the Displays window.
Arbitrary classes
Arbitrary classes with any superclass you like can also be plugged in, given that they have a default constructor. These are registered like so:
However their ClassCreator::create() function takes no arguments, so it is impossible to give them a VisualizationManager. Also, there is no pre-specified time when these classes will be instantiated, as that must be written into a Display subclass.
The VisualizationManager* argument is critical, as it is the primary point of interaction with RViz internals.
Classes in RViz which a plugin will access
In addition to documenting the way Displays and other classes can be defined in plugins, I will also need to document VisualizationManager and other classes which the plugins access in order to do their work.
Changes to Plugin API
Displays
The same functionality offered by registerDisplay could be achieved by using registerClass. As this is a more general and cleaner approach, the API should be changed to use the former mechanism instead.
This implies a broad set of changes:
Display: has to offer a default constructor and a method void initialize(VisualizationManager* manager), which will be called after creation by DisplayWrapper/ClassCreator.
Plugin: Currently contains a specialized API for Displays, which would be removed. Instead, it would allow to access all Displays through getClassTypeInfoList(const std::string& base_class). This would affect:
NewDisplayDialog: call getClassTypeInfoList instead of getDisplayTypeInfoList
DisplayWrapper: currently calls createDisplay and getDisplayTypeInfo. The first one would have to be replaced by a call getClassTypeInfoList, then finding the ClassTypeInfo and calling its ClassCreator.
VizualisationManager: currently calls getDisplayTypeInfoByDisplayName
DisplayTypeInfo and DisplayCreator: become obsolete, instead add attribute help_description to ClassTypeInfo & use ClassCreator.
TypeRegistry: remove registerDisplay, getDisplayEntries.
- Add the functionality to have description text for all classes, which could be done in two ways:
Having all meta-info for the user in the YAML file. This would mean changing the YAML parsing code in Plugin::loadDescription. This would also introduce a conflict with the current registerClass interface, which provides a parameter for the "human-readable" name of a class, an information which would be duplicated in the YAML file. This parameter should thus be removed, requiring an entry for all classes in the YAML file.
Passing all info to the type registry in the init code, making the YAML file obsolete. registerClass}} would need an additional parameter {{{description_text then.
Tools
Current state: RViz has a Tool superclass which defines a mouse-interaction mode, and which can be chosen by a button which is placed in the tool bar area.
VizualizationManager keeps track of all Tools and handles Keyboard events to activate them. It offers a C++ API to add Tools and change the currently active Tool. It does not provide the option to remove a Tool. VizualizationFrame creates the GUI Toolbar holding buttons for all Tools and also creates the default set of Tools (Move Camera, Select, Pose estimate, Navigation goal).
Proposed changes: Add the capability to add Tools via the class registry. VizualizationManager would have to listen to the loading signal of all plugins and look for subclasses of rviz::Tool when a plugin is loaded. It already listens to the unloading signal, where it currently only refreshes the property manager. It would have to remove all Tools of the plugin being unloaded, notify VisualizationFrame, which would have to remove the corresponding button, and refresh the property manager tool_property_manager_. ToolPropertiesPanel would also have to listen to a newly introduced "tool removed" signal of VisualizationManager to sync with the changes.
Registering a new tool in a plugin would look like this:
1 reg->registerClass<MyTool>("rviz::Tool", "my_plugin::MyTool", "My Tool");
As with displays, the Tools would have to provide a method initialize(std::string name, char shortcut_key, VisualizationManager*).
There will need to be a mechanism to assign shortcut keys to tools, e.g. trying to use the first letter and a dialog where the user can change the automatically assigned shortcuts. The key assignments will have to be saved in one of the config files.
The current set of Tools should then also become part of default_plugin.
View Controllers
RViz has a ViewController superclass, subclasses of which define various ways of changing the viewpoint in response to mouse motion.
Making view controllers pluggable will be very similar to doing it for tools, except that ViewsPanel provides the GUI widget for selecting them and needs to keep that list in sync (similar to the Tools panel).
Panels
The subcomponents of the RViz GUI are mostly subclasses of wxPanel, which can be docked and undocked from the main window frame. These can currently be added by plugins, but again only through Displays. Sometimes this is perfectly appropriate, such as for a video camera display. Sometimes it would be better for the new panel to simply appear when the plugin loads, and not create a new panel for each new instance of the display. For example, if a new "reset view" button was desired, it could be added in a panel plugin.
Similar to tools and view controllers, panels registered with registerClass() would be created and added to the GUI on plugin load and removed on unload. They would also be added and removed from the "view" menu.
For this to work, all such panels have to inherit from a common base class, say rviz::Panel. However, that poses a problem:
To make sure all such Panels inherit from wxWindow, one would usually make rviz::Panel a subclass of this. However, panels are ususally designed using wxFormBuilder, which creates classes which also inherit from wxWindow. Having a class which inherits from rviz::Panel and some generated class thus creates ambiguity (the diamond problem).
A way out of this would be to include a method wxWindow* getPane() into the interface of rviz::Panel, which would have to be implemented in derived classes and could return the object itself, or to require the Panels to register themselves by calling VisualizationManager::getWindowManager()->addPane(this).
Access Restrictions
It would probably make sense to put a layer of abstraction between RViz plugins and the implementation of RViz internals, to restrict access to internals. Currently there are very few such restrictions. The best example is WindowManagerInterface which is an abstract class describing the interface to the window manager. The implementor is VisualizationFrame, but the plugins do not see that.
However the manifest.xml file for RViz exports something like {{{-I rviz/src}}} in its cpp flags, meaning that all the internals of RViz are made available to plugin developers anyway. Binary packages of RViz also have the entire source code directory.
A little cleanup would not be hard and is probably a good idea, to prevent future changes to RViz internals from breaking plugins.
Interactive Objects
Multiple users have expressed interest in implementing controls directly in the 3D scene which the user can interact with via the mouse. For the pr2_interactive_manipulation project David Gossow has written just such a control, but it was a lot of work. With a new API within RViz I think we can make it easier and cleaner.
There is already a Pixel-perfect picking mechanism implemented in SelectionManager which requires everything that is added to the Ogre Scene to be registered (otherwise it will be rendered as-is into the selection buffer, which could cause confusion with pick colors used by other objects). The way this is done is that the creator has to create a SelectionHandler, which can handle multiple Ogre::Entity objects. SelectionManager takes care of creating a new Ogre::Technique named "Pick", which renders the entity in its pick color and is used to render the selection buffer.
SelectionHandler contains a set of virtual functions which will be called when the object is selected or deselected. One of these functions is bool needsAdditionalRenderPass, which will trigger an additional render pass when it returns true. This is used by point clouds, which will render all their points with a single pick color in the first pass and with different colors in the second pass. The reason behind this is the limitation on the number of pick colors used.
The only interaction it supports, however, is the one performed by the selection tool: selected objects create a set of properties which will be displayed in the Selection panel.
For supporting interactive objects, two things have to be added to this framework:
SelectionHandler needs a new function virtual void processMouseEvent( ViewportMouseEvent& event );, onFocus and onLoseFocus that have to be overwritten to react to mouse events. The latter two could e.g. be used to highlight an interactive object when the mouse passes over it.
SelectionManager needs to expose its pick function in the public interface, which returns all CollisionHandles in the specified image region.
There also needs to be a new "Interact" tool, which on every mouse move
- determines which object is under the mouse
- If the left mouse button goes down, switches to "dragging" mode until the mouse is released
if not in dragging mode, forwards all mouse events to the SelectionHandler under the mouse, if there is any, otherwise to the current view controller. Also, calls onFocus and onLoseFocus of the SelectionHandlers.
if in dragging mode, forward everything to the last SelectionHandler if any, otherwise to the ViewController. The same SelectionHandler or ViewController stays active until the mouse button is released.
Extension: Interactive Markers
One of the most successful features of rviz are markers. The reason for this, I believe, is that they can be implemented with much less effort than a full-blown plugin and need no knowledge of the rviz and Ogre systems.
I propose to add an Interactive Marker message and a display, that allows to send a Marker (or collection of Markers) together with information about the possible user interaction on them to rviz. The Interactive Marker display would build on the new rviz infrastructure for interactive elements. The results of the user interaction would be channeled back in a defined way so whatever sent the markers can react on that user input.
Examples of this could be:
- A marker together with 6-DOF control, as in the current version of the cartesian gripper control plugin, or any subset of these DOFs.
- Maybe something which is streamlined to be moved along one axis or within a plane.
- A marker which can be clicked like a button
- A marker which shows a Pop-Up menu with discrete entries when the user clicks on it (e.g. an object point cloud with a menu entry "grab this")
Implementation decisions:
Will there be one InteractiveMarker message type which can handle all options or multiple types?
- Will the feedback be given through a return message or will this use the Action server/client model, or maybe even dynamic_reconfigure?
- What types of widgets are needed? How generic should the format be, i.e. where will this be on the ease-of-use vs. more generic axis. E.g. do we want
- - a pre-defined configurable widget for 6-dof control - or should there only be 1-dof-controls you can combine in some way. - Should there even be any type of logic, e.g. for dragging things around, in the gui, or do mouse events with additional 3D data get forwarded to the back channel and the external node would make things move? - We could provide both options.
This is only a very rough sketch of my ideas so far. Comments are welcome.
Adaption of ROS coordinate conventions
Current state
By default, Ogre uses the convention that x points to the right, y up and z to the back so it is a right-handed system. ROS also uses a right-handed system, however the axes are swapped so x points forward, y to the left and z up.
Working around this issue could simply be done by making all cameras face the positive x axis and having the z axis as their up vector. As both systems are right-handed, backface culling would still work as expected without changes.
Rviz however uses a complicated and non-intuitive workaround where the default coordinate convention of Ogre is used, but all Vectors and Quaternions coming from ROS and going back need to be converted using rviz::robotToOgre and rviz::ogreToRobot respectively (see rviz/common.h). Also, this is documented nowhere. What they do is basically this (though expressed more complicated in the source code):
- ogreToRos(x,y,z) = (-z,-x,y)
- rosToOgre(x,y,z) = (-y,z,-x)
Proposed changes
- Change the cameras in Ogre to follow the ROS convention. This would basically mean adapting the default view controllers.
- Make the conversion functions do nothing but output a compiler or runtime warning that says: "Don't include me, I'm deprecated". This way, little code would be broken.
Possible risks are hidden in code that doesn't use the conversion functions but hardcoded conversions (which even exist within rviz, see CameraDisplay). Apart from that, this would mean changing small code pieces in many places within rviz, but would also be pretty straightforward, considering the above formulae.
I've created a ticket containing a patch for the current rviz trunk.
Question / concerns / comments
Enter your thoughts on the API and any questions / concerns you have here. Please sign your name.
Josh F
There's no reason you can't pass a VisualizationManager to an arbitrary class plugin, you just need a virtual initialization method -- my plan was actually to remove the need for the plugin manager to know anything about Displays at all, instead requiring a default constructor and an initialize(name, display manager) constructor. I would refrain from adding anything rviz specific to the plugin code (for example, adding tool code to Plugin::load/unload), instead registering callbacks for plugin load/unload in the VisualizationManager.
- This also means Tools, View Controllers and Panels don't need special functionality in the plugin system
- As far as the coordinate conventions proposal goes, in the first week or so of rviz development I discovered a couple of places in Ogre itself that assume certain coordinate conventions such as Y-up, and -Z-forward. These may have been fixed in 1.6/1.7, and unfortunately I don't remember where they were (seem to remember something in the camera). So: definitely for it, but be careful.
- Completely agree with the separation of the plugin API. rviz was not designed to be plugin-based, which is why it has never had a public plugin API.
David G
I strongly agree with Josh on the registerClass vs. registerDisplay etc. issue. This would be a much nicer approach. Thanks for the hint. (An example of this approach would be the already existing PointCloudBase::loadTransformers(Plugin* plugin) btw.).
Regarding the coordinate transform issue, I've tried it out (see rviz patch) and didn't see any problems, probably the Ogre developers figured this out in the mean time. The code I wrote for directly setting the camera matrix in the Camera display wouldn't have worked perfectly in an earlier version of Ogre as far as I've read, so I'm not surprised to hear that there have been issues like that. If modifying the camera parameters would confuse Ogre, as a workaround you could still attach all cameras to a SceneNode which does the coordinate transform trick.
Dave H
- Thanks Josh and David. I agree with all your points. I think David G will be rolling these changes into this document in a bit.
Results from meeting Dave / David
Action items:
- make the default plugin treated like the other plugins, in a separate package
- put everything that is pluggable into default_plugin
- sort rviz main directory into subdirs: panels,
- reorganize code so public header files go into the include directory