  Let $S = (S_1, \ldots, S_d)$ denote the compression of the $d$-shift to the
complement of a homogeneous ideal $I$ of $\mathbb{C}[z_1, \ldots, z_d]$.
Arveson conjectured that $S$ is essentially normal. In this paper, we establish
new results supporting this conjecture, and connect the notion of essential
normality to the theory of the C*-envelope and the noncommutative Choquet
boundary.
  The unital norm closed algebra $\mathcal{B}_I$ generated by $S_1,\ldots,S_d$
modulo the compact operators is shown to be completely isometrically isomorphic
to the uniform algebra generated by polynomials on $\overline{V} :=
\overline{\mathcal{Z}(I) \cap \mathbb{B}_d}$, where $\mathcal{Z}(I)$ is the
variety corresponding to $I$. Consequently, the essential norm of an element in
$\mathcal{B}_I$ is equal to the sup norm of its Gelfand transform, and the
C*-envelope of $\mathcal{B}_I$ is identified as the algebra of continuous
functions on $\overline{V} \cap \partial \mathbb{B}_d$, which means it is a
complete invariant of the topology of the variety determined by $I$ in the
ball.
  Motivated by this determination of the C*-envelope of $\mathcal{B}_I$, we
suggest a new, more qualitative approach to the problem of essential normality.
We prove the tuple $S$ is essentially normal if and only if it is hyperrigid as
the generating set of a C*-algebra, which is a property closely connected to
Arveson's notion of a boundary representation.
  We show that most of our results hold in a much more general setting. In
particular, for most of our results, the ideal $I$ can be replaced by an
arbitrary (not necessarily homogeneous) invariant subspace of the $d$-shift.
