text: Deep Learning has recently become hugely popular in machine learning, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level DP applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).
  @cite_26: 
  @cite_96: In 1977 Tore Dalenius articulated a desideratum for statistical databases: nothing about @PARASPLIT an individual should be learnable from the database that cannot be learned without access to the @PARASPLIT database. We give a general impossibility result showing that a natural formalization of Dalenius’ @PARASPLIT goal cannot be achieved if the database is useful. The key obstacle is the side information that @PARASPLIT may be available to an adversary. Our results hold under very general conditions regarding the @PARASPLIT database, the notion of privacy violation, and the notion of utility. @PARASPLIT Contrary to intuition, a variant of the result threatens the privacy even of someone not in @PARASPLIT the database. This state of affairs motivated the notion of differential privacy [15, 16], a strong @PARASPLIT ad omnia privacy which, intuitively, captures the increased risk to one’s privacy incurred by @PARASPLIT participating in a database.
  @cite_3: Anonymizing private data before release is not enough to reliably protect privacy, as Netflix and AOL have learned to their cost. Recent research on differential privacy opens a way to obtain robust, provable privacy guarantees, and systems like PINQ and Airavat now offer convenient frameworks for processing arbitrary userspecified queries in a differentially private way. However, these systems are vulnerable to a variety of covertchannel attacks that can be exploited by an adversarial querier. We describe several different kinds of attacks, all feasible in PINQ and some in Airavat. We discuss the space of possible countermeasures, and we present a detailed design for one specific solution, based on a new primitive we call predictable transactions and a simple differentially private programming language. Our evaluation, which relies on a proof-of-concept implementation based on the Caml Light runtime, shows that our design is effective against remotely exploitable covert channels, at the expense of a higher query completion time.
summary: Covert channels, however, can be used to defeat DP-protected databases as shown in the work of @cite_3 . In general, privacy cannot be guaranteed if auxiliary information (outside the DP model) is accessible to the adversary @cite_96 . At NDSS'16, it was shown by @cite_26 that DP at a certain granularity is not effective in real-life scenarios where data such as social data, mobile data, or medical records have strong correlations with each other. Note that it's a matter of setting DP granularity right and DP is not being violated at all.