• Major limitations to robot manipulation include reliable grasping of objects and a lack of quantitative, informative performance evaluation

• Researchers typically only contribute to one novel subsystem component at a time while the other components used are generally not standardized

• Bottlenecks include lack of relevant comparable benchmarks, issues integrating open-source products, and truly modular software, among others; additional information gathered during Phase I of this project which motivates the current Phase II effort can be found on the Background page

• The COMPARE ecosystem aims to improve the compatibility and benchmarking of open-source products for robot manipulation

Functions of the COMPARE Ecosystem

The ecosystem will produce:

• Standards and guidelines for topics including, but not limited to: open-source product development, procedures for conducting benchmarking, and reporting of performance results.

• Online resources such as: discussion channels and forums, repositories of open-source products for robot manipulation, and leaderboards of benchmarking results.

• Test and evaluation procedures to validate the open-source products and their compatibility with others in the software pipeline, including: reviews of their adherence to the developed standards and guidelines, performance assessments of their functionality in multiple contexts, and usability analyses of product integration.

A series of events will be conducted to activate the open-source ecosystem by engaging the community to share their research, develop best practices, and integrate open-source products together, including:

• Conference workshops on benchmarking protocols, datasets, and software pipelines,

• Seasonal schools to develop and integrate the open-source products, and 

• Competitions to propel the development of the open-source products and generate new benchmarks for the community.