  Spiking neural networks (SNNs) have attracted much attention for their high
energy efficiency and for recent advances in their classification performance.
However, unlike traditional deep learning approaches, the analysis and study of
the robustness of SNNs to adversarial examples remain relatively
underdeveloped. In this work, we focus on advancing the adversarial attack side
of SNNs and make three major contributions. First, we show that successful
white-box adversarial attacks on SNNs are highly dependent on the underlying
surrogate gradient technique, even in the case of adversarially trained SNNs.
Second, using the best surrogate gradient technique, we analyze the
transferability of adversarial attacks on SNNs and other state-of-the-art
architectures like Vision Transformers (ViTs) and Big Transfer Convolutional
Neural Networks (CNNs). We demonstrate that the adversarial examples created by
non-SNN architectures are not misclassified often by SNNs. Third, due to the
lack of an ubiquitous white-box attack that is effective across both the SNN
and CNN/ViT domains, we develop a new white-box attack, the Auto Self-Attention
Gradient Attack (Auto-SAGA). Our novel attack generates adversarial examples
capable of fooling both SNN and non-SNN models simultaneously. Auto-SAGA is as
much as $91.1\%$ more effective on SNN/ViT model ensembles and provides a
$3\times$ boost in attack effectiveness on adversarially trained SNN ensembles
compared to conventional white-box attacks like Auto-PGD. Our experiments and
analyses are broad and rigorous covering three datasets (CIFAR-10, CIFAR-100
and ImageNet), five different white-box attacks and nineteen classifier models
(seven for each CIFAR dataset and five models for ImageNet).
