  We introduce an invariant of a hyperbolic knot which is a map $\alpha\mapsto
\mathbf{Phi}_\a(h)$ from $\mathbb{Q}/\mathbb{Z}$ to matrices with entries in
$\overline{\mathbb{Q}}[[h]]$ and with rows and columns indexed by the boundary
parabolic $SL_2(\mathbb{C})$ representations of the fundamental group of the
knot. These matrix invariants have a rich structure: (a) their
$(\sigma_0,\sigma_1)$ entry, where $\sigma_0$ is the trivial and $\sigma_1$ the
geometric representation, is the power series expansion of the Kashaev
invariant of the knot around the root of unity $e^{2 \pi i \alpha}$ as an
element of the Habiro ring, and the remaining entries belong to generalized
Habiro rings of number fields; (b) the first column is given by the
perturbative power series of Dimofte--Garoufalidis; (c)~the columns of
$\mathbf{Phi}$ are fundamental solutions of a linear $q$-difference equation;
(d)~the matrix defines an $SL_2(\mathbb{Z})$-cocycle $W_{\gamma}$ in
matrix-valued functions on $\mathbb{Q}$ that conjecturally extends to a smooth
function on $\mathbb{R}$ and even to holomorphic functions on suitable complex
cut planes, lifting the factorially divergent series $\mathbf{Phi}(h)$ to
actual functions. The two invariants $\mathbf{Phi}$ and $W_{\gamma}$ are
related by a refined quantum modularity conjecture which we illustrate in
detail for the three simplest hyperbolic knots, the $4_1$, $5_2$ and $(-2,3,7)$
pretzel knots. This paper has two sequels, one giving a different realization
of our invariant as a matrix of convergent $q$-series with integer coefficients
and the other studying its Habiro-like arithmetic properties in more depth.
