  Benchmarking digital watermarking algorithms is not an easy task because
different applications of digital watermarking often have very different sets
of requirements and trade-offs between conflicting requirements. While there
have been some general-purpose digital watermarking benchmarking systems
available, they normally do not support complicated benchmarking tasks and
cannot be easily reconfigured to work with different watermarking algorithms
and testing conditions. In this paper, we propose OR-Benchmark, an open and
highly reconfigurable general-purpose digital watermarking benchmarking
framework, which has the following two key features: 1) all the interfaces are
public and general enough to support all watermarking applications and
benchmarking tasks we can think of; 2) end users can easily extend the
functionalities and freely configure what watermarking algorithms are tested,
what system components are used, how the benchmarking process runs, and what
results should be produced. We implemented a prototype of this framework as a
MATLAB software package and used it to benchmark a number of digital
watermarking algorithms involving two types of watermarks for content
authentication and self-restoration purposes. The benchmarking results
demonstrated the advantages of the proposed benchmarking framework, and also
gave us some useful insights about existing image authentication and
self-restoration watermarking algorithms which are an important but less
studied topic in digital watermarking.
