  Let $\mathcal E: y^2 = x^3 + A(T)x + B(T)$ be a nontrivial one-parameter
family of elliptic curves over $\mathbb{Q}(T)$, with $A(T), B(T) \in \mathbb
Z(T)$, and consider the $k$\textsuperscript{th} moments $A_{k,\mathcal{E}}(p)
:= \sum_{t (p)} a_{\mathcal{E}_t}(p)^k$ of the Dirichlet coefficients
$a_{\mathcal{E}_t}(p) := p + 1 - |\mathcal{E}_t (\mathbb{F}_p)|$. Rosen and
Silverman proved a conjecture of Nagao relating the first moment
$A_{1,\mathcal{E}}(p)$ to the rank of the family over $\mathbb{Q}(T)$, and
Michel proved that if $j(T)$ is not constant then the second moment is equal to
$A_{2,\mathcal{E}}(p) = p^2 + O(p^{3/2})$. Cohomological arguments show that
the lower order terms are of sizes $p^{3/2}, p, p^{1/2}$, and $1$. In every
case we are able to analyze in closed form, the largest lower order term in the
second moment expansion that does not average to zero is on average negative,
though numerics suggest this may fail for families of moderate rank. We prove
this Bias Conjecture for several large classes of families, including families
with rank, complex multiplication, and constant $j(T)$-invariant. We also study
the analogous Bias Conjecture for families of Dirichlet characters, holomorphic
forms on GL$(2)/\mathbb{Q}$, and their symmetric powers and Rankin-Selberg
convolutions. We identify all lower order terms in large classes of families,
shedding light on the arithmetic objects controlling these terms. The negative
bias in these lower order terms has implications toward the excess rank
conjecture and the behavior of zeros near the central point in families of
$L$-functions.
