  Cardinality estimation (CardEst) plays a significant role in generating
high-quality query plans for a query optimizer in DBMS. In the last decade, an
increasing number of advanced CardEst methods (especially ML-based) have been
proposed with outstanding estimation accuracy and inference latency. However,
there exists no study that systematically evaluates the quality of these
methods and answer the fundamental problem: to what extent can these methods
improve the performance of query optimizer in real-world settings, which is the
ultimate goal of a CardEst method. In this paper, we comprehensively and
systematically compare the effectiveness of CardEst methods in a real DBMS. We
establish a new benchmark for CardEst, which contains a new complex real-world
dataset STATS and a diverse query workload STATS-CEB. We integrate multiple
most representative CardEst methods into an open-source database system
PostgreSQL, and comprehensively evaluate their true effectiveness in improving
query plan quality, and other important aspects affecting their applicability,
ranging from inference latency, model size, and training time, to update
efficiency and accuracy. We obtain a number of key findings for the CardEst
methods, under different data and query settings. Furthermore, we find that the
widely used estimation accuracy metric(Q-Error) cannot distinguish the
importance of different sub-plan queries during query optimization and thus
cannot truly reflect the query plan quality generated by CardEst methods.
Therefore, we propose a new metric P-Error to evaluate the performance of
CardEst methods, which overcomes the limitation of Q-Error and is able to
reflect the overall end-to-end performance of CardEst methods. We have made all
of the benchmark data and evaluation code publicly available at
https://github.com/Nathaniel-Han/End-to-End-CardEst-Benchmark.
