IROS 2021 Workshop - Benchmarking of robotic grasping and manipulation: protocols, metrics and data analysis


IROS 2021 logo


This workshop will be held online on 27th September 2021 between 16:10 and 18:10 CEST.  You will be able to join us via the IROS 2021 conference platform.

This workshop has been endorsed by the RAS Technical Committee on Performance Evaluation & Benchmarking of Robotic and Automation Systems and the RAS Technical Committee on Robotic Hands, Grasping and Manipulation.


Object manipulation is still a difficult task in both academia and industry. There have been promising advances in individual components:

  • developments in robot hardware have provided capable robot arms, tactile sensors, and anthropomorphic robot hands 
  • deep learning provide powerful vision systems for object recognition, segmentation and pose estimation with remarkable performance, 
  • Data driven approaches have led to better algorithms for grasp pose computation and motion planning.


Despite this, robust object manipulation, especially in unstructured environments, remains an unsolved problem. Systematic benchmarks are crucial, in order to design systems that combine these different components effectively. However, evaluating a single component is very challenging mainly due to the large range of tasks that can be performed and the variety of hardware and objects used to carry out the experiments. Benchmark protocols have been proposed, ensuring the repeatability of experiments, i.e. making sure to reduce variations in the experimental conditions across the trials. Other works have proposed empirical or theoretical metrics to assess the quality of grasping or manipulation. Using these tools, combined with statistical analysis, allows the research community to quantify the performance of components for a given task and gives insight into the shortcomings that must be addressed in future work. 

Therefore, we propose in this workshop to discuss the ongoing benchmarks, metrics, analysis and tools that can be used to improve the evaluation of robotic grasping and manipulation systems.

Topic of interest include (but not limited to):

  • Benchmarking and dataset for grasping and manipulation
  • Protocols or methods to evaluate components for grasping and manipulation
  • Tools or frameworks simplifying deployment of specific components on different robot systems
  • Technology transfer from academia to industry
  • New design of robot hands and grippers

Call for papers

Deadline: 27 August 2021

Notification of acceptance: 1 September 2021

We invite researchers  to submit their recent work around the topics of interest mentioned above.  Expected contributions should be submitted in the form of extended abstracts (2-3 pages max) in IEEE Conference paper format (PDF), and not simultaneously submitted to another journal or conference.  All submissions will be reviewed using a single-blind review process by an independent panel of reviewers. Accepted contributions will be asked to prepare a 5 minutes presentation that is going to be made available before the day of the workshop. Authors will be able to answer questions during the workshop event. Accepted contributions will be shared on this website.

All submissions should be addressed to, indicating IROS 2021 Workshop contribution in the email subject.

Keynote speakers

Dr Máximo Roa   Máximo Roa (German Aerospace Center - DLR)

   Title: "Influence of benchmark protocols in the design of a novel end-effector"

   Presentation (to be uploaded early September)


Dr Adam Norton   Adam Norton (University of Massachusetts Lowell NERVE Center)

   Title: "Remote Experimentation of Manipulation for Online Test and Evaluation (REMOTE)"

   Presentation (to be uploaded early September)


Dr Jurgen Leitner   Juxi Leitner (LYRO Robotics)

   Title: "What's the matter with metrics? And does the industry care?"

   Presentation (to be uploaded early September)


Dr Yasemin Bekiroglu   Yasemin Bekiroglu (Chalmers University of Technology)

   Title: "A Benchmarking protocol for comparing performances of grasp planning algorithms"

   Presentation (to be uploaded early September)


Dr Jelizaveta Konstantinova   Jelizaveta Konstantinova (Ocado Technology)

   Title: "Robotic grasping of delicate grocery items: benchmarking and challenges"

   Presentation (to be uploaded early September)


Dr Joseph Falco   Joseph Falco (National Institute of Standards and Technology - NIST)

   Title: "Performance Metrics and Test Methods to Support the Development of Next-Generation Robotic

                 Assembly Systems"

   Presentation (to be uploaded early September)


Rich Walker   Rich Walker (The Shadow Robot Company)

   Title: "Using benchmarks in an industrial R&D context"

   Presentation (to be uploaded early September)



Jeremy Fishel   Jeremy Fishel (Tangible Research)

   Title: "Performance Benchmarking in Telerobotics"

   Presentation (to be uploaded early September)


Panel members

In addition to the invited keynote speakers, two additional members will join the round table discussion:

  • Christian Schlegel (Technische Hochschule Ulm - THU)
  • Stjin de Looijer (Vanderlande)


According to the IROS 2021 organization committee, this event will be fully held online. A 2-hour slot between 16:10 and 18:10 (CEST) on September 27th will allow us to have interactive sessions on the conference platform. This is why all the presentations from the keynote speakers will be made available on the conference platform as well as on this website prior to the workshop. The main purpose of these 2 hours is to maximize the time researchers can interact with each others, even if not in person. For this reason, you will be able to post questions that are going to be addressed during one of the two interactive round tables. However, if you have questions regarding a specific presentation (e.g. technical detail), please privately address them via email, as the questions should encourage a common discussion involving most of the panel.


Due to the different time zones and availability of the speakers, we have set two interactive round table sessions in order to make sure they can attend at least one. The event will take place as follows:


16:10 - 16:15 - Welcome

16:15 - 17:00 - Interactive round table: How to improve the evaluation of robotic grasping and manipulation systems?

  • Jelizaveta Konstantinova
  • Máximo Roa
  • Juxi Leitner
  • Joseph Falco

17:00 - 17:20 - Interactive online poster session

17:20 - 18:05 - Interactive round table: How to make benchmarking robotic manipulation easier?

  • Yasemin Bekiroglu
  • Adam Norton
  • Rich Walker
  • Jeremy Fishel

18:05-18:10 - Closing remark



Brice Denoun   Brice Denoun

   Queen Mary University of London (QMUL) / The Shadow Robot Company


Beatriz Leon   Beatriz León

   The Shadow Robot Company


Gerhard Neumann   Gerhard Neumann

   Karlsruhe Institute of Technology (KIT)


Lorenzo Natale   Lorenzo Natale

   Istituto Italiano di Tecnologia (IIT)


Lorenzo Jamone   Lorenzo Jamone

   Queen Mary University of London (QMUL)