text: We describe a new instance-based learning algorithm called the Boundary Forest (BF) algorithm, that can be used for supervised and unsupervised learning. The algorithm builds a forest of trees whose nodes store previously seen examples. It can be shown data points one at a time and updates itself incrementally, hence it is naturally online. Few instance-based algorithms have this property while being simultaneously fast, which the BF is. This is crucial for applications where one needs to respond to input data in real time. The number of children of each node is not set beforehand but obtained from the training procedure, which makes the algorithm very flexible with regards to what data manifolds it can learn. We test its generalization performance and speed on a range of benchmark datasets and detail in which settings it outperforms the state of the art. Empirically we find that training time scales as O(DNlog(N)) and testing as O(Dlog(N)), where D is the dimensionality and N the amount of data,
  @cite_8: 
  @cite_9: Given user data, one often wants to find approximate matches in a large database. A good example of such a task is finding images similar to a given image in a large collection of images. We focus on the important and technically diffcult case where each data element is high dimensional, or more generally, is represented by a point in a large metric spaceand distance calculations are computationally expensive. In this paper we introduce a data structure to solve this problem called a GNAT Geometric Near-neighbor Access Tree. It is based on the philosophy that the data structure should act as a hierarchical geometrical model of the data as opposed to a simple decomposition of the data that does not use its intrinsic geometry. In experiments, we find that GNAT's outperform previous data structures in a number of applications. Keywords near neighbor, metric space, approximate queries, data mining, Dirichlet domains, Voronoi regions
  @cite_3: 
  @cite_0: This paper presents the NV-Tree (Nearest Vector Tree). It addresses the specific, yet important, problem of efficiently and effectively finding the approximate k-nearest neighbors within a collection of a few billion high-dimensional data points. The NV-Tree is a very compact index, as only six bytes are kept in the index for each high-dimensional descriptor. It thus scales extremely well when indexing large collections of high-dimensional descriptors. The NV-Tree efficiently produces results of good quality, even at such a large scale that the indices cannot be kept entirely in main memory any more. We demonstrate this with extensive experiments using a collection of 2.5 billion SIFT (Scale Invariant Feature Transform) descriptors.
  @cite_16: Balltrees are simple geometric data structures with a wide range of practical applica­ tions to geometric ·learning tasks. In this report we compare 5 different algorithms for . constructing ball trees from data. We study the trade-off between construction time and the quality of the constructed tree. Two of the algorithms are on-line, two construct the structures from the data set in a top down fashion, and one uses a bottom up approach. We empirically study the algorithms on random data drawn from eight different probability distributions representing smooth, clustered, and curve distributed data in different ambient space dimen­ sions. We find that the bottom up approach usually produces the best trees but has the longest construction time. The other approaches have uses in specific circumstances. 1. IntemauonaI.Computer Science Institute, Berkeley, CA.
  @cite_11: Multidimensional data is data that exists and changes in more than one dimension, by time, or spatially, or both, sometimes dynamically. Think here of tracking hurricane data in order to project the storm's path, for just one example. As spatial and other multidimensional data structures become increasingly important for the applications in game programming, data mining, bioinformatics, and many other areas--including astronomy, geographic information systems, physics, etc., the need for a comprehensive book on the subject is paramount. This book is truly a life's work by the author who is clearly the best person for the job.
summary: There are several existing methods, including KD-trees @cite_3 , Geometric Near-neighbor Access Trees @cite_9 , and Nearest Vector trees @cite_0 that build tree search structures on large datasets (see @cite_11 for an extensive bibliography). These algorithms typically need batch access to the entire dataset before constructing their trees, in which case they may outperform the BF, however we are interested in an online setting. Two well known tree-based algorithms that allow online insertion are cover trees @cite_8 and ball trees. The ball tree online insertion algorithm @cite_16 is rather costly, requiring a volume minimizing step at each addition. The cover tree, on the other, has a cheap online insertion algorithm, and it comes with guarantees of query time scaling as @math where N is the amount of data and c the so-called expansion constant, which is related to the intrinsic dimensionality of the data. We will compare to cover trees below. Note that @math in fact depends on @math as it is defined as a worse case computation over the data set. It can also diverge from adding a single point.