  This non-conventional paper represents the first attempt to uncover a
possible vulnerability in some proposals for optical network designs and
performance comparisons. While optical network designs and planning lie at the
heart of achieving fiber capacity efficiency and/or operational efficiency, its
combinatorial nature makes it computationally hard to reach optimal solutions
for realistic scenarios. Therefore, the well-established way that have been
taken for granted by not-so-small number of research papers is that an
optimization model based on mixed integer linear programming (MILP) is first
proposed and then due to the intractability of such combinatorial model, an
heuristic algorithm is offered as an approximation. The solution-quality
comparison between the MILP and heuristic is then carried out on small-scale
instances including topologies and traffic tests to verify the efficacy of the
proposed heuristic and the next step is to use such allegedly verified
heuristic for optical network designs of realistic scenarios. This approach may
nevertheless leave a critical vulnerability as there is no guarantee that one
performs well in small tests will generalize adequately for large-scale cases,
a common pitfall widely referred as the peril of extrapolation and/or
overfitting. Besides, it is not uncommon that in some research works, for
benchmarking purpose, the comparison between a new design proposal whose
performance is obtained from on one heuristic and a reference design based on
another heuristic is carried out. As the result of missing solution quality
check, such performance comparison relied merely on heuristic solutions may be
equally vulnerable as its results can be distorted and thus, be far from the
possibly achieved zones. In this work, we pinpoint those issues and provide a
realistic case study to highlight and demonstrate the impact of such
vulnerabilities.
