Papers

Papers

Notes on Selected Work

Prospection: Interpretable Plans from Language by Predicting the Future

High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before taking it. We demonstrate the fidelity of plans generated by our framework when interpreting real, crowd-sourced natural language commands for a robot in simulated scenes.

@article{paxton2019prospection,
  author    = {Chris Paxton and
               Yonatan Bisk and
               Jesse Thomason and
               Arunkumar Byravan and
               Dieter Fox},
  title     = {Prospection: Interpretable Plans From Language By Predicting the Future},
  journal   = {International Conference on Robotics and Automation (ICRA), 2019 IEEE/RSJ International Conference on},
  year      = {2019}
  url       = {http://arxiv.org/abs/1903.08309},
}

Presented at ICRA 2019 in Montreal. Link: Prospection

Visual Robot Task Planning

We propose an approach that allows us to visualize intermediate goals and learn to plan complex activity from visual information.

@article{paxton2019visual,
  author    = {Chris Paxton and
               Yotam Barnoy and
               Kapil D. Katyal and
               Raman Arora and
               Gregory D. Hager},
  title     = {Visual Robot Task Planning},
  journal   = {International Conference on Robotics and Automation (ICRA), 2019 IEEE/RSJ International Conference on},
  year      = {2019},
  url       = {http://arxiv.org/abs/1804.00062},
}

Presented at ICRA 2019 in Montreal. Link: Visual Robot Task Planning

Evaluating Methods for End-User Creation of Robot Task Plans

How can we enable users to create effective, perception-driven task plans for collaborative robots? We conducted a 35-person user study with CoSTAR to find out.

@article{paxton2018evaluating,
  author    = {Chris Paxton and
               Felix Jonathan and
               Andrew Hundt and
               Bilge Mutlu and
               Gregory D. Hager},
  title     = {Evaluating Methods for End-User Creation of Robot Task Plans},
  journal   = {Intelligent Robots and Systems (IROS), 2018 IEEE/RSJ International Conference on},
  year      = {2018},
  url       = {http://arxiv.org/abs/1811.02690},
}

Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environments

Tree search based task and motion planning, combining multiple low-level policies learned via deep reinforcement learning and using them to create high-level task plans that navigate through intersections.

@article{paxton2017combining,
  title={Combining neural networks and tree search for task and motion planning in challenging environments},
  author={Paxton, Chris and Raman, Vasumathi and Hager, Gregory D and Kobilarov, Marin},
  journal={Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on},
  note={Available as arXiv preprint arXiv:1703.07887},
  year={2017}
}

Presented at IROS 2017 in Vancouver, Canada, as well as a poster presentation at RLDM in Ann Arbor, Michigan, USA. This work was partially supported by Zoox, Inc.

Arxiv link: Neural nets and tree search

CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision

Our new and improved CoSTAR system, with 3D pose recognition. This paper describes how we built a cross-platform system for authoring complex robot task plans with behavior trees. Winner of the KUKA innovation award.

Citation:

@article{paxton2017costar,
  title={Co{STAR}: Instructing Collaborative Robots with Behavior Trees and Vision},
  author={Paxton, Chris and Hundt, Andrew and Jonathan, Felix and Guerin, Kelleher and Hager, Gregory D},
  journal={Robotics and Automation (ICRA), 2017 IEEE International Conference on},
  note={Available as arXiv preprint arXiv:1611.06145},
  year={2017}
}

Presented at ICRA 2017 in Singapore.

Do What I Want, Not What I Did: Imitation of Skills by Planning Sequences of Actions

Sampling-based task and motion planning using skills learned from expert demonstrations.

Citation:

@inproceedings{paxton2016want,
  title={Do what I want, not what I did: Imitation of skills by planning sequences of actions},
  author={Paxton, Chris and Jonathan, Felix and Kobilarov, Marin and Hager, Gregory D},
  booktitle={Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on},
  pages={3778--3785},
  year={2016},
  organization={IEEE}
}

Presented at IROS 2016 in Daejeon, Korea.

Other Robotics Papers

  • Paxton, C., Bisk, Y., Thomason, J., Byravan, A., and Fox, D. (2019) Prospection: Interpretable Plans from Language by Predicting the Future. At IEEE International Conference on Robotics and Automation (ICRA 2019). arXiv.

  • Paxton, C., Barnoy, Y., Katyal, K., Arora, R., and Hager, G.D. (2019) Visual Robot Task Planning. At IEEE Conference on Robotics and Automation (ICRA 2019). arXiv

  • Katyal, K., Popek, K., Paxton, C., Burlina, P., and Hager, G.D. (2019) Uncertainty-Aware Occupancy Map Prediction Using Generative Networks for Robot Navigation. At IEEE Conference on Robotics and Automation (ICRA 2019).

  • Paxton, C., Jonathan, F., Hundt, A., Mutlu, B., and Hager, G.D. (2018) Evaluating Methods for End-User Creation of Robot Task Plans. At IEEE Conference on Intelligent Robots and Systems (IROS 2018). arXiv video

  • Paxton, C., Raman, V., Hager, G.D., and Kobilarov, M. (2017) Combining Neural Nets and Tree Search for Task and Motion Planning in Complex Environments. At IEEE Conference on Intelligent Robots and Systems (IROS 2017). Also presented at Reinforcement Learning and Decision Making (RLDM 2017). arXiv video

  • Paxton, C., Hundt, A., Jonathan, F., Guerin, K., and Hager, G.D. (2017) CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision. At IEEE Conference on Robotics and Automation (ICRA 2017). arXiv video

  • Paxton, C., Jonathan, F., Kobilarov, M., and Hager, G.D. (2016) Do What I Want, Not What I Did: Imitation of Skills by Planning Sequences of Actions. At IEEE Conference on Intelligent Robots and Systems (IROS 2016). arXiv video 1 video 2

  • Bohren, J., Paxton, C., Howarth, R., Hager, G.D., and Whitcomb, L. (2016). Enabling Semi-Autonomous Teleoperated Assembly over High-Latency Networks. At 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2016). Nominated for best technical contribution to human-robot interaction.

  • Ghalamzan E., A. M., Paxton, C., Hager, G.D., & Bascetta, L. (2015). An Incremental Approach to Learning Generalizable Robot Tasks from Human Demonstration. In IEEE International Conference on Robotics and Automation (ICRA).

  • Guerin, K., Lea, C., Paxton, C., & Hager, G.D. (2015). A Framework for End-User Instruction of a Robot Assistant for Manufacturing. In IEEE International Conference on Robotics and Automation (ICRA).

Workshop Papers

  • Paxton, C., Kobilarov, M., & Hager, G.D. (2015). Towards Robot Task Planning from Probabilistic Representations of Human Skills. At AAAI 2016 Workshop on Planning for Hybrid Systems. pdf arXiv

  • Ghalamzan E., A. M., Paxton, C., Hager, G.D., & Bascetta, L. (2014). Learning How to Avoid an Obstacle from Human Demonstration. At RSS 2014 workshop on Learning Plans with Context from Human Signals.

  • Paxton, C., Bohren, J., and Hager, G. D. (2014). Standards for Grounding Symbols for Robotic Task Ontologies. At IROS 2014 workshop on Standardized Knowledge Representation and Ontologies for Robotics and Automation.

Previous Work

When working on my Masters, I did some research in early prediction of sepsis before switching directions completely and starting to work on learning task representations for robot motion planning.

  • Henry, K., Paxton, C., Kim, K.S., Pham, J., & Saria, S. (2014). REWS: Real-time Early Warning Score for Septic Shock. In Critical Care Medicine (Vol. 42, Issue 12, p. A1384). Society for Critical Care Medicine. Winner of Annual Scientific Award.

  • Paxton, C., Niculescu-Mizil, A., & Saria, S. (2013). Developing Predictive Models Using Electronic Medical Records: Challenges and Pitfalls. In AMIA Annual Symposium Proceedings (Vol. 2013, p. 1109). American Medical Informatics Association.

  • Vedula, S.S., Malpani, A.O., Tao, L., Chen, G., Gao, Y., Poddar, P., Ahmidi, N., Paxton, C., Vidal, R., Khudanpur, S., Hager, G.D., and Chen, C.C.G. (2016). Analysis of the Structure of Surgical Activity for a Suturing and Knot-Tying Task. PLoS ONE 11(3): e0149174. doi:10.1371/journal.pone.0149174

Find me on Twitter, Github, or Mastodon!