Only released in EOL distros:   fuerte

Overview

RoboFrameNet is a system that grounds natural-language input (in the form of speech or text) into robot actions, using the concept of semantic frames as an intermediary. (The concept of semantic frames for robots was inspired by FrameNet; the author has no connection to or affiliation with this project.)

Quickstart Guide

To checkout, move to a directory within the ROS package path, then:

$ hg clone https://kforge.ros.org/appmanandroid/roboframenet roboframenet

To compile:

$ rosmake roboframenet --rosdep-install   # On electric

$ apt-get install ros-fuerte-pr2-props-*  # On fuerte
$ rosdep install                          

To run a test suite that is executable on a desktop:

$ roslaunch roboframenet_bringup roboframenet_desktop.launch

Demo commands can now be sent to the system:

$ rostopic pub command std_msgs/String "Speak."
$ rostopic pub command std_msgs/String "Ping."
$ rostopic pub command std_msgs/String "Loop."
$ rostopic pub command std_msgs/String "Stop."

Speak should play a mooing sound over the speakers. Ping displays "Ping!" in the terminal. Loop displays "Loop [n]." in the same way; however, instead of terminating, it simply counts upwards and waits to be preempted. Stop does nothing on its own but can take advantage of the preemption offered by the executor to preempt and thus halt the previously-running program. (This can be used, for instance, to halt Loop.)

Details

RoboFrameNet Pipeline

RoboFrameNet grounds natural language in robot actions through a processing pipeline. An illustration of this process is below:

• voice_command converts speech to text using the Android voice recognition software. To be released.

• stanford_parser_ros parses text and outputs a parse tree and a set of dependencies (eg, direct object, indirect object, etc.).

• semantic_framer fits the parsed text to semantic frames via lexical units. It outputs a filled lexical unit, which then directly fills in a semantic frame in frame_registrar.

• frame_registrar determines which registered actions are able to perform the scene described by a filled semantic frame. Because related semantic frames are linked in a graph, natural language can be grounded in one semantic frame, and servers registered with other semantic frames may be called to execute the concept. A list of sequences of actions is output.

• arbitrator determines which action sequences to execute and in what order by creating a state machine.

• executor takes this state machine and physically instantiates the actions on a robot.

Terminology

In the development of RoboFrameNet, many pieces of terminology were borrowed from FrameNet. The most pertinent ones are defined below, in terms of how they are used in the RoboFrameNet project. (This may differ slightly from their use in FrameNet.)

• A semantic frame describes a scene being played out. For instance, "an actor giving a recipient an object" may describe the semantic frame "giving". This is more general than "a PR2 giving Bill a ball" but less general than "an actor, recipient, and object interacting".

• Lexical units provide a mapping from parsed natural language to semantic frames. A lexical unit, as a word, is said to evoke a semantic frame. In the simplest case, used here, a verb describes a lexical unit. However, a verb alone is not particularly useful. (eg, "Give." is not a useful command.) Thus, lexical units contain the evoking word's dependencies, also known as frame elements. For instance, the lexical unit defined by "give" might have dependencies for the direct object (the object to be given) and the indirect object (the recipient).

• Frame elements are the roles that objects play in a semantic frame. For instance, the actor, recipient, and object in the "giving" semantic frame are known as frame elements (defined below).

Other Launch Options

You can also run launches specially designed for the PR2:

$ roslaunch roboframenet roboframenet_pr2.launch

as well as a Turtlebot:

$ roslaunch roboframenet roboframenet_turtlebot.launch

Registering Nodes, Adding Semantic Frames, and Adding Lexical Units

RFNServer was developed to enable users to quickly integrate their work with RoboFrameNet. Please see the RFNServer package for more details.

Report a Bug

<<TracLink(REPO COMPONENT)>>