  The GPU has emerged as the go-to accelerator for high throughput and parallel
workloads, spanning scientific simulations to AI, thanks to its performance and
power efficiency. Given that 6 out of the top 10 fastest supercomputers in the
world use NVIDIA GPUs and many AI companies each employ 10,000's of NVIDIA
GPUs, an accurate understanding of GPU power consumption is essential for
making progress to further improve its efficiency. Despite the limited
documentation and the lack of understanding of its mechanisms, NVIDIA GPUs'
built-in power sensor, providing easily accessible power readings via the
nvidia-smi interface, is widely used in energy efficient computing research on
GPUs. Our study seeks to elucidate the internal mechanisms of the power
readings provided by nvidia-smi and assess the accuracy of the power and energy
consumption data. We have developed a suite of micro-benchmarks to profile the
behaviour of nvidia-smi power readings and have evaluated them on over 70
different GPUs from all architectural generations since power measurement was
first introduced in the 'Fermi' generation. We have identified several
unforeseen problems in terms of power/energy measurement using nvidia-smi, for
example on the A100 and H100 GPUs only 25% of the runtime is sampled for power
consumption, during the other 75% of the time, the GPU can be using drastically
different power and nvidia-smi and results presented by it are unaware of this.
This along with other findings can lead to a drastic under/overestimation of
energy consumed, especially when considering data centres housing tens of
thousands of GPUs. We proposed several good practices that help to mitigate
these problems. By comparing our results to those measured from an external
power-meter, we have reduced the error in the energy measurement by an average
of 35% and in some cases by as much as 65% in the test cases we present.
