  Motivation: While the majority of gene histories found in a clade of
organisms are expected to be generated by a common process (e.g. the coalescent
process), it is well-known that numerous other coexisting processes (e.g.
horizontal gene transfers, gene duplication and subsequent
neofunctionalization) will cause some genes to exhibit a history quite distinct
from those of the majority of genes. Such "outlying" gene trees are considered
to be biologically interesting and identifying these genes has become an
important problem in phylogenetics.
  Results: We propose and implement KDETREES, a nonparametric method of
estimating distributions of phylogenetic trees, with the goal of identifying
trees which are significantly different from the rest of the trees in the
sample. Our method compares favorably with a similar recently-published method,
featuring an improvement of one polynomial order of computational complexity
(to quadratic in the number of trees analyzed), with simulation studies
suggesting only a small penalty to classification accuracy. Application of
KDETREES to a set of Apicomplexa genes identified several unreliable sequence
alignments which had escaped previous detection, as well as a gene
independently reported as a possible case of horizontal gene transfer. We also
analyze a set of Epichloe genes, fungi symbiotic with grasses, successfully
identifying a contrived instance of paralogy.
  Availability: Our method for estimating tree distributions and identifying
outlying trees is implemented as the R package KDETREES, and is available for
download from CRAN.
