  As a representative cyber-physical system (CPS), robotic manipulator has been
widely adopted in various academic research and industrial processes,
indicating its potential to act as a universal interface between the cyber and
the physical worlds. Recent studies in robotics manipulation have started
employing artificial intelligence (AI) approaches as controllers to achieve
better adaptability and performance. However, the inherent challenge of
explaining AI components introduces uncertainty and unreliability to these
AI-enabled robotics systems, necessitating a reliable development platform for
system design and performance assessment. As a foundational step towards
building reliable AI-enabled robotics systems, we propose a public industrial
benchmark for robotics manipulation in this paper. It leverages NVIDIA
Omniverse Isaac Sim as the simulation platform, encompassing eight
representative manipulation tasks and multiple AI software controllers. An
extensive evaluation is conducted to analyze the performance of AI controllers
in solving robotics manipulation tasks, enabling a thorough understanding of
their effectiveness. To further demonstrate the applicability of our benchmark,
we develop a falsification framework that is compatible with physical
simulators and OpenAI Gym environments. This framework bridges the gap between
traditional testing methods and modern physics engine-based simulations. The
effectiveness of different optimization methods in falsifying AI-enabled
robotics manipulation with physical simulators is examined via a falsification
test. Our work not only establishes a foundation for the design and development
of AI-enabled robotics systems but also provides practical experience and
guidance to practitioners in this field, promoting further research in this
critical academic and industrial domain.
