CoSTAR on the DVRK

One of our students has been working on putting CoSTAR on the Da Vinci surgical robot. Maybe some day we will be able to use it to create an intelligent robotic assistant for surgeons?

You can check out a very, very early video below:

CoSTAR Paper

Anyone interested in CoSTAR: you can look up an early version of this material here. It’s a description of how we designed the system and the various issues we ran into when building our framework for end-user instruction of industrial robots.

  title={CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision},
  author={Paxton, Chris and Hundt, Andrew and Jonathan, Felix and Guerin, Kelleher and Hager, Gregory D},
  journal={arXiv preprint arXiv:1611.06145},

CoSTAR Instructor

The Instructor BT Editor

As of today, CoSTAR’s Instructor UI is available on GitHub. You can also find it with the rest of our project.

Getting started with Git

It has come to my attention that a lot of our new and incoming students have not really used Git before. Git’s an incredibly useful tool for software engineering; it’s particularly important for working on group projects.

There are a couple reasons for this: - It tracks user contributions and file history, making it easier to figure out when bugs were introduced - It allows us to easily maintain different versions of a project with features that are currently in development - It makes it easy to combine different subsets of our code.

Let’s get started. On Ubuntu you can install git like so:

sudo apt-get install git

On Windows, the process is barely more complicated. Download Git for windows and add it to your system path. Quick windows command line refresher: to bring up the command prompt, press windows+R, then type “cmd” and press enter. Everything else will be the same after this.

Now, there are a couple million guides that are a Google search away if you want to learn how to use Git with a new repository (see, for example, this one or this one or all these. In this example, I’m going to talk about how to specifically start contributing to one of my projects as an example for how to get things running.

What follows is a brief, high-level guide to how to get started using Git. This is not meant to be comprehensive, and those well-acquainted with git may not love how I use some of these terms. The point here is to get the high-level concepts across for people who have not yet dug into all the details.

Set Up Your Account

You’ll need to create an account on GitHub or whatever other site you are using. For GitHub, create an SSH key as per the instructions here. For GitLab (which we use internally at JHU), the instructions are similar.

Clone The Project

We use git clone to download the project and its history. To clone CoSTAR, for example:

git clone

Now, CoSTAR is a ROS project, which means you clone it into your Catkin workspace. For something else, like this java game, you can clone it wherever seems appropriate. For MATLAB code like this, you would normally clone it into your MATLAB workspace, for example.


Git lets us track code in different branches. For example, if I cd costar_stack, then I can run git branch:

chris@blunderbuss:~/catkin_ws/src/costar_stack$ git branch
* master

This means I am currently on branch master.

When you are starting out, you probably want to create your own branch for whatever feature you will be working on. For example, if I am adding support for the Barrett WAM arm to CoSTAR, I might do this:

git checkout -b barrett-wam

This creates a new branch called barret-wam and moves me into that branch.

In the future, we can switch between branches with git checkout:

git checkout master
git checkout barrett-wam

Making Changes and Pushing

Let’s say I add a new file: In this case, I use the git add command:

git add

This tells Git it should add this file to the set of files it’s currently tracking.

Now I want to commit this change. This actually adds it to my history.

git commit -a

A message will pop up to tell you to enter a message. Try to keep these messages short and sweet, and keep your commits atomic. What that means is that I only change essentially one thing at a time.

We’re not done! Now I’ve added this file to my local history, but I still need to share with others. To do that, I push:

git push origin barret-wam

Substitute barret-wam with whatever your branch is called.

Stay Up To Date

Use the git fetch command to keep your local information up to date. Use git pull to download changes others have made.

Hopefully this helps you get started.

Building a Behavior Tree with CoSTAR

CoSTAR is a framework we have been developing for authoring robot task plans. It is still very much a work in progress, but we are excited about how well it is coming a long.

A crucial part of letting end-users interact and program robots is giving them a framework that is powerful enough to solve their own problems, but adaptable enough that solutions won’t fall apart at the slightest deviation. We achieve all this through our Behavior Tree-based user interface, which you can see in the video below.

One of the key parts of this interface is our SmartMove operation. SmartMoves are special operations that take a list of predicate constraints on what constitutes a valid goal. In layman’s terms, you can describe the qualities you want an object to have before you pick it up: something like “get me the next piece from the right side of the table,” as show here.