Collaborative Open-source Manipulation Performance Assessment for Robotics Enhancement (COMPARE) Ecosystem
Funded by the National Science Foundation, Pathways to Enable Open-Source Ecosystems (POSE)
Awards TI-2346069 and TI-2229577
Improving the modularity of open-source components in the robot manipulation pipeline for more effective performance benchmarking
Major limitations to robot manipulation include reliable grasping of objects and a lack of quantitative, informative performance evaluation
Researchers typically only contribute to one novel subsystem component at a time while the other components used are generally not standardized
Bottlenecks include lack of relevant comparable benchmarks, issues integrating open-source products, and truly modular software, among others; additional information gathered during Phase I of this project which motivates the current Phase II effort can be found on the Background page
The COMPARE ecosystem aims to improve the compatibility and benchmarking of open-source products for robot manipulation
Goals of the COMPARE Ecosystem
Create a greater cohesion between open-source products by generating community-driven standards for components of software pipelines that increase modularity and enable implementation that allows for greater performance quantification
Unite the existing community of users and developers to build upon and integrate existing open-source products as well as improve and evolve the future of robot manipulation research
Facilitate sufficient commonality in hardware and software that allows for better quantitative evaluation of research and eases the implementation of the complex robot manipulation pipeline
Structure of the COMPARE Ecosystem
Steering committee for roadmap development and prioritization of ecosystem efforts
Advisory committee for oversight and ecosystem sustainability counsel
Community facilitator for community interaction and representation to the steering committee
Working groups comprised of developers and users to establish guidelines and standards
Ecosystem infrastructure (bottom) will support the development of open-source products and conducting performance evaluations of robot manipulation capabilities (top)
Functions of the COMPARE Ecosystem
The ecosystem will produce:
Standards and guidelines for topics including, but not limited to: open-source product development, procedures for conducting benchmarking, and reporting of performance results.
Online resources such as: discussion channels and forums, repositories of open-source products for robot manipulation, and leaderboards of benchmarking results.
Test and evaluation procedures to validate the open-source products and their compatibility with others in the software pipeline, including: reviews of their adherence to the developed standards and guidelines, performance assessments of their functionality in multiple contexts, and usability analyses of product integration.
A series of events will be conducted to activate the open-source ecosystem by engaging the community to share their research, develop best practices, and integrate open-source products together, including:
Conference workshops on benchmarking protocols, datasets, and software pipelines,
Seasonal schools to develop and integrate the open-source products, and
Competitions to propel the development of the open-source products and generate new benchmarks for the community.
Principal Investigators
Holly Yanco
University of Massachusetts Lowell
Adam Norton
University of Massachusetts Lowell
Kostas Bekris
Rutgers University
Berk Calli
Worcester Polytechnic University
Aaron Dollar
Yale University
Yu Sun
University of South Florida
Contact
If you are interested in contributing to the development of the COMPARE open-source ecosystem or have any questions or comments, please e-mail Adam Norton at adam_norton@uml.edu