text: Recently a deterministic method, frequent directions (FD) is proposed to solve the high dimensional low rank approximation problem. It works well in practice, but experiences high computational cost. In this paper, we establish a fast frequent directions algorithm for the low rank approximation problem, which implants a randomized algorithm, sparse subspace embedding (SpEmb) in FD. This new algorithm makes use of FD's natural block structure and sends more information through SpEmb to each block in FD. We prove that our new algorithm produces a good low rank approximation with a sketch of size linear on the rank approximated. Its effectiveness and efficiency are demonstrated by the experimental results on both synthetic and real world datasets, as well as applications in network analysis.
  @cite_24: A sketch of a matrix A is another matrix B which is significantly smaller than A but still approximates it well. Finding such sketches efficiently is an important building block in modern algorithms for approximating, for example, the PCA of massive matrices. This task is made more challenging in the streaming model, where each row of the input matrix can only be processed once and storage is severely limited. In this paper we adapt a well known streaming algorithm for approximating item frequencies to the matrix sketching setting. The algorithm receives n rows of a large matrix A e ℜ n x m one after the other in a streaming fashion. It maintains a sketch B ℜ l x m containing only l This gives a streaming algorithm whose error decays proportional to 1 l using O(ml) space. For comparison, random-projection, hashing or sampling based algorithms produce convergence bounds proportional to 1 √l. Sketch updates per row in A require amortized O(ml) operations and the algorithm is perfectly parallelizable. Our experiments corroborate the algorithm's scalability and improved convergence rate. The presented algorithm also stands out in that it is deterministic, simple to implement and elementary to prove.
  @cite_43: 
  @cite_47: We describe a new algorithm called FrequentDirections for deterministic matrix sketching in the row-update model. The algorithm is presented an arbitrary input matrix A R ^ n d one row at a time. It performs O(d ) operations per row and maintains a sketch matrix B R ^ d such that for any k < , |A^TA - B^TB |_2 |A - A_k |_F^2 ( -k) ;and ; |A - B_k (A) |_F^2 (1 + k -k ) |A-A_k |_F^2. Here, A_k stands for the minimizer of |A - A_k |_F over all rank k matrices (similarly for B_k) and B_k (A) is the rank k matrix resulting from projecting A on the row span of B_k. We show that both of these bounds are the best possible for the space allowed. The summary is mergeable and hence trivially parallelizable. Moreover, FrequentDirections outperforms exemplar implementations of existing streaming algorithms in the space-error tradeoff. This paper combines, simplifies, and extends the results of Liberty [Proceedings of the 1...
  @cite_6: Matrices have become essential data representations for many large-scale problems in data analytics, and hence matrix sketching is a critical task. Although much research has focused on improving the error size tradeoff under various sketching paradigms, the many forms of error bounds make these approaches hard to compare in theory and in practice. This paper attempts to categorize and compare the most known methods under row-wise streaming updates with provable guarantees, and then to tweak some of these methods to gain practical improvements while retaining guarantees. For instance, we observe that a simple heuristic iSVD, with no guarantees, tends to outperform all known approaches in terms of size error trade-off. We modify the best performing method with guarantees, FrequentDirections , under the size error trade-off to match the performance of iSVD and retain its guarantees. We also demonstrate some adversarial datasets where iSVD performs quite poorly. In comparing techniques in the time error trade-off, techniques based on hashing or sampling tend to perform better. In this setting, we modify the most studied sampling regime to retain error guarantee but obtain dramatic improvements in the time error trade-off. Finally, we provide easy replication of our studies on APT, a new testbed which makes available not only code and datasets, but also a computing platform with fixed environmental settings.
summary: Compared to most randomized algorithms, FD approaches the matrix sketching problem from a different perspective. It is deterministic, space efficient and produces more accurate estimates given a fixed sketch size. It was introduced by Liberty @cite_24 , later extended by @cite_47 with a comprehensive analysis, and a few variations @cite_6 @cite_43 have been developed since then. The algorithm extends the idea of frequent items for item frequency approximation problem to a general matrix. Given an input matrix @math and space parameter @math , at first, the algorithm considers the first @math rows in @math and shrinks its top @math orthogonal vectors by the same amount to obtain an @math matrix; then combines them with the next @math rows in @math for the next iteration, and repeat the procedure. It is illustrated in Fig. . The detailed algorithm is given in Algorithm .