  Edge video analytics is becoming the solution to many safety and management
tasks. Its wide deployment, however, must first address the tension between
inference accuracy and resource (compute/network) cost. This has led to the
development of video analytics pipelines (VAPs), which reduce resource cost by
combining DNN compression/speedup techniques with video processing heuristics.
Our measurement study on existing VAPs, however, shows that today's methods for
evaluating VAPs are incomplete, often producing premature conclusions or
ambiguous results. This is because each VAP's performance varies substantially
across videos and time (even under the same scenario) and is sensitive to
different subsets of video content characteristics.
  We argue that accurate VAP evaluation must first characterize the complex
interaction between VAPs and video characteristics, which we refer to as VAP
performance clarity. We design and implement Yoda, the first VAP benchmark to
achieve performance clarity. Using primitive-based profiling and a carefully
curated benchmark video set, Yoda builds a performance clarity profile for each
VAP to precisely define its accuracy/cost tradeoff and its relationship with
video characteristics. We show that Yoda substantially improves VAP evaluations
by (1) providing a comprehensive, transparent assessment of VAP performance and
its dependencies on video characteristics; (2) explicitly identifying
fine-grained VAP behaviors that were previously hidden by large performance
variance; and (3) revealing strengths/weaknesses among different VAPs and new
design opportunities.
