  An automorphism of a graph is called quasi-semiregular if it fixes a unique
vertex of the graph and its remaining cycles have the same length. This kind of
symmetry of graphs was first investigated by Kutnar, Malni\v{c}, Mart\'{i}nez
and Maru\v{s}i\v{c} in 2013, as a generalization of the well-known semiregular
automorphism of a graph. Symmetric graphs of valency three or four, admitting a
quasi-semiregular automorphism, have been classified in recent two papers.
  Let $p\geq 5$ be a prime and $\Gamma$ a connected symmetric graph of valency
$p$ admitting a quasi-semiregular automorphism. In this paper, we first prove
that either $\Gamma$ is a connected Cayley graph $\rm{Cay}(M,S)$ such that $M$
is a $2$-group admitting a fixed-point-free automorphism of order $p$ with $S$
as an orbit of involutions, or $\Gamma$ is a normal $N$-cover of a
$T$-arc-transitive graph of valency $p$ admitting a quasi-semiregular
automorphism, where $T$ is a non-abelian simple group and $N$ is a nilpotent
group. Then in case $p=5$, we give a complete classification of such graphs
$\Gamma$ such that either $\rm{Aut}(\Gamma)$ has a solvable arc-transitive
subgroup or $\Gamma$ is $T$-arc-transitive with $T$ a non-abelian simple group.
We also construct the first infinite family of symmetric graphs that have a
quasi-semiregular automorphism and an insolvable full automorphism group.
