Benchmarking Robot Manipulation:
Improving Interoperability and Modularity

Relevant COMPARE Slack Channel: #2025-rss-workshop

Speakers

Participation

Participants will be asked to fill out post-it notes to a poster board to highlight contributions they can make to open-source and benchmarking for robot manipulation research (what they can “give”) and those that they seek others to contribute to fill gaps to support their research (what they want to “get”). This exercise will be held during the coffee breaks to prepare for the breakout discussions to facilitate networking, potential matchmaking by the workshop organizers during these periods, and such that we can characterize the feedback received. 

During breakout sessions, we will divide into smaller groups to discuss improvements we can make as a community to the interoperability and modularity of the manipulation pipeline to enable more effective benchmarking, around topics such as (1) standards and guidelines on component/pipeline structures (e.g., input/output formats), (2) test and evaluation methods to ensure compatibility and usability, and (3) online resources to connect and support researchers. The workshop organizers will facilitate discussions followed by a review session where each group will provide a high-level summary. We will compile these results into key takeaways and post the workshop outputs on this webpage, similar to what has been done during our previous workshops (e.g., ICRA 2023).

Contributions

Short papers (2-4 pages following RSS guidelines; anonymization is not required) are sought to be presented as lightning talks followed by poster presentations. The workshop is structured around two main topics: 

• Interoperability and Modularity: Even though research labs often focus on a specific aspect of robotic manipulation (e.g., perception, planning, learning), they still need to implement the whole software pipeline to conduct experimental analysis. We will discuss how to reduce such a development burden via modular coding practices that would allow for manipulation pipelines that are reusable across research contexts. 

• Benchmarking Robot Manipulation: Protocols and tools that are widely accepted by the community are needed to enable systematic experimental analysis. We will discuss how to enable reliable quantified performance analysis in manipulation research and examine successful examples both in robotics and neighboring fields.

Contributions may be in the form of position papers, proposals for new efforts, or reporting of new results, discussing issues faced, successes achieved, and/or analyses of the current landscape of robotic manipulation when implementing open-source assets, improving modularity of software components, and conducting benchmarking. It is expected that authors of accepted papers will provide a lightning talk presentation, poster, and participate in topic discussions at the workshop. At least one author of each accepted submission must register for the workshop and attend in-person; remote presentation will not be allowed.

• March 31, 2025: Call for submissions open

• April 25, 2025, 23:59 Anywhere on Earth: Early submission deadline for short papers to ensure decision by RSS 2025 early registration deadline (April 30)

• April 28, 2025: Notification of acceptance of early workshop submissions

• May 23, 2025, 23:59 Anywhere on Earth: Submission deadline for short papers to ensure decision by RSS 2025 standard registration deadline (June 5)

• June 2, 2025: Notification of acceptance for workshop submissions

• June 25, 2025: Date of workshop at RSS 2025

Submissions should be e-mailed to compare.ecosystem@gmail.com with the text “[RSS 2025 Workshop]” in the subject line. Authors of accepted submissions are encouraged to upload their papers to arXiv.org; they will also be hosted on a publicly accessible Google Drive folder and linked on the workshop website along with presentation slides and videos of presentations.

All questions regarding the workshop can be e-mailed to compare.ecosystem@gmail.com 

Organizers