  In view of the performance limitations of fully-decoupled designs for neural
architectures and accelerators, hardware-software co-design has been emerging
to fully reap the benefits of flexible design spaces and optimize neural
network performance. Nonetheless, such co-design also enlarges the total search
space to practically infinity and presents substantial challenges. While the
prior studies have been focusing on improving the search efficiency (e.g., via
reinforcement learning), they commonly rely on co-searches over the entire
architecture-accelerator design space. In this paper, we propose a
\emph{semi}-decoupled approach to reduce the size of the total design space by
orders of magnitude, yet without losing optimality. We first perform neural
architecture search to obtain a small set of optimal architectures for one
accelerator candidate. Importantly, this is also the set of (close-to-)optimal
architectures for other accelerator designs based on the property that neural
architectures' ranking orders in terms of inference latency and energy
consumption on different accelerator designs are highly similar. Then, instead
of considering all the possible architectures, we optimize the accelerator
design only in combination with this small set of architectures, thus
significantly reducing the total search cost. We validate our approach by
conducting experiments on various architecture spaces for accelerator designs
with different dataflows. Our results highlight that we can obtain the optimal
design by only navigating over the reduced search space. The source code of
this work is at \url{https://github.com/Ren-Research/CoDesign}.
