  With the continuous increase in the size and complexity of machine learning
models, the need for specialized hardware to efficiently run such models is
rapidly growing. To address such a need, silicon-photonic-based neural network
(SP-NN) accelerators have recently emerged as a promising alternative to
electronic accelerators due to their lower latency and higher energy
efficiency. Not only can SP-NNs alleviate the fan-in and fan-out problem with
linear algebra processors, their operational bandwidth can match that of the
photodetection rate (typically 100 GHz), which is at least over an order of
magnitude faster than electronic counterparts that are restricted to a clock
rate of a few GHz. Unfortunately, the underlying silicon photonic devices in
SP-NNs suffer from inherent optical losses and crosstalk noise originating from
fabrication imperfections and undesired optical couplings, the impact of which
accumulates as the network scales up. Consequently, the inferencing accuracy in
an SP-NN can be affected by such inefficiencies -- e.g., can drop to below 10%
-- the impact of which is yet to be fully studied. In this paper, we
comprehensively model the optical loss and crosstalk noise using a bottom-up
approach, from the device to the system level, in coherent SP-NNs built using
Mach-Zehnder interferometer (MZI) devices. The proposed models can be applied
to any SP-NN architecture with different configurations to analyze the effect
of loss and crosstalk. Such an analysis is important where there are
inferencing accuracy and scalability requirements to meet when designing an
SP-NN. Using the proposed analytical framework, we show a high power penalty
and a catastrophic inferencing accuracy drop of up to 84% for SP-NNs of
different scales with three known MZI mesh configurations (i.e., Reck,
Clements, and Diamond) due to accumulated optical loss and crosstalk noise.
