Package proposal for Mechanical Turk package.

Overview

Mech Turk package contains tools for task-independent communication with annotation server. It supports raw data transfer to and from the server. It implements server authentication scheme. The package provides command line tools for sending and retrieving raw session data from the server.

Details

Communication API

The package provides communication helpers with built-in user authentication. The RPC-based and HTTP-based access is supported by mech_turk.rpc_communication and mech_turk.http_communication. The main role of helpers is to allow other packages to easily create a connection to the server using simple "connect(server_alias)" call. The server alias is automatically expanded into server name and user credentials. The communication API provides calls to post data (e.g. image or video) to be stored on the server.

Session command line operations

The package provides command line tools to submit data for annotation and download raw session results. These tools are used when the application generates and stores all necessary work units in server-defined format. The command line tools include submit_work_units.py for sending the data to the server and get_raw_session_results.py to download the data from the server.

Tests

The tests cover 2 core areas: XML-RPC-based API for server access and command line access to the server. The tests cover connection, session creation, posting tasks, posting submissions, posting and downloading the results through command-line tools.

Authentication and server names

Much of the server API requires an authenticated user. To avoid typing the passwords all the time, the mech_turk package uses a config file ("~/.ros/.mech_turk/auth.txt") to keep the user credentials. The file has very simple format:

server_alias:{
  server: server_name[:port],
  user:   user_name,
  pwd:    user_password
}
...

e.g.

test: {
  server: vm6.willowgarage.com,
  user: mt_tester,
  pwd: some-not-important-password
}
default: {
  server: mech_turk.willowgarage.com,
  user: mt_worker,
  pwd: some-not-important-password
}

After a server alias is declared in the auth.txt, it can be used in all mech turk commands using "--server=ALIAS". For convenience, "--server=default" is assumed as a default value.

Using command line tools

To submit raw data to the session, we can use submit_work_units.py:

rosrun mech_turk submit_work_units.py --session=test_session data/work_unit_*.xml

Once the work is completed, we can download the submissions and the original work units through get_raw_session_results.py:

rosrun mech_turk get_raw_session_results.py --session=test_session --saveto=results/all_results --filter=none

If we have Q/A enabled, we can restrict the download to only good results:

rosrun mech_turk get_raw_session_results.py --session=test_session --saveto=results/all_results --filter=none