Umeå University's logo

The research dedicated to the aggregation of classification trees and general trees (hierarchical structure of objects) has made enormous progress in the past decade. The problem statement for aggregation of classification trees or general trees is as follows: Given k classification or general trees for a set of objects, we aim to build a consensus tree (classification or general). That is, a representative tree for the given trees. In this paper, we explore different perspectives for the motivation to construct a single tree from multiple trees given by researchers. The survey presents the approaches for the aggregation of both the classification trees as well as general trees. We bifurcate our study of the aggregation approaches into two categories: Selecting a single tree from multiple trees and merging trees. We will discuss these categories and the aggregation approaches under these categories in the paper comprehensively. We also discuss the privacy aspects of tree aggregation approaches and the possible directions for new research like using the technique of aggregating decision trees in the field of Federated Learning, which is a booming topic.

We propose a privacy-preserving framework using Mondrian k-anonymity with decision trees in a Federated Learning (FL) setting for the horizontally partitioned data. Data heterogeneity in FL makes the data non-IID (Non-Independent and Identically Distributed). We use a novel approach to create non-IID partitions of data by solving an optimization problem. In this work, each device trains a decision tree classifier. Devices share the root node of their trees with the aggregator. The aggregator merges the trees by choosing the most common split attribute and grows the branches based on the split values of the chosen split attribute. This recursive process stops when all the nodes to be merged are leaf nodes. After the merging operation, the aggregator sends the merged decision tree to the distributed devices. Therefore, we aim to build a joint machine learning model based on the data from multiple devices while offering k-anonymity to the participants.

Attacking machine learning models is one of the many ways to measure the privacy of machine learning models. Therefore, studying the performance of attacks against machine learning techniques is essential to know whether somebody can share information about machine learning models, and if shared, how much can be shared? In this work, we investigate one of the widely used dimensionality reduction techniques Principal Component Analysis (PCA). We refer to a recent paper that shows how to attack PCA using a Membership Inference Attack (MIA). When using membership inference attacks against PCA, the adversary gets access to some of the principal components and wants to determine if a particular record was used to compute those principal components. We assume that the adversary knows the distribution of training data, which is a reasonable and useful assumption for a membership inference attack. With this assumption, we show that the adversary can make a data reconstruction attack, which is a more severe attack than the membership attack. For a protection mechanism, we propose that the data guardian first generate synthetic data and then compute the principal components. We also compare our proposed approach with Differentially Private Principal Component Analysis (DPPCA). The experimental findings show the degree to which the adversary successfully attempted to recover the users’ original data. We obtained comparable results with DPPCA. The number of principal components the attacker intercepted affects the attack’s outcome. Therefore, our work aims to answer how much information about machine learning models is safe to disclose while protecting users’ privacy.

The disclosure risk of synthetic/artificial data is still being determined. Studies show that synthetic data generation techniques generate similar data to the original data and sometimes even the exact original data. Therefore, publishing synthetic datasets can endanger the privacy of users. In our work, we study the synthetic data generated from different synthetic data generation techniques, including the most recent diffusion models. We perform a disclosure risk assessment of synthetic datasets via an attribute inference attack, in which an attacker has access to a subset of publicly available features and at least one synthesized dataset, and the aim is to infer the sensitive features unknown to the attacker. We also compute the predictive accuracy and F1 score of the random forest classifier trained on several synthetic datasets. For sensitive categorical features, we show that Attribute Inference Attack is not highly feasible or successful. In contrast, for continuous attributes, we can have an approximate inference. This holds true for the synthetic datasets derived from Diffusion models, GANs, and DPGANs, which shows that we can only have approximated Attribute Inference, not the exact Attribute Inference.

A lot of research in federated learning is ongoing ever since it was proposed. Federated learning allows collaborative learning among distributed clients without sharing their raw data to a central aggregator (if it is present) or to other clients in a peer to peer architecture. However, each client participating in the federation shares their model information learned from their data with other clients participating in the FL process, or with the central aggregator. This sharing of information, however, makes this approach vulnerable to various attacks, including data reconstruction attacks. Our research specifically focuses on Principal Component Analysis (PCA), as it is a widely used dimensionality technique. For performing PCA in a federated setting, distributed clients share local eigenvectors computed from their respective data with the aggregator, which then combines and returns global eigenvectors. Previous studies on attacks against PCA have demonstrated that revealing eigenvectors can lead to membership inference and, when coupled with knowledge of data distribution, result in data reconstruction attacks. Consequently, our objective in this work is to augment privacy in eigenvectors while sustaining their utility. To obtain protected eigenvectors, we use k-anonymity, and generative networks. Through our experimentation, we did a complete privacy, and utility analysis of original and protected eigenvectors. For utility analysis, we apply HIERARCHICAL CLUSTERING, RANDOM FOREST regressor, and RANDOM FOREST classifier on the protected, and original eigenvectors. We got interesting results, when we applied HIERARCHICAL CLUSTERING on the original, and protected datasets, and eigenvectors. The height at which the clusters are merged declined from 250 to 150 for original, and synthetic version of CALIFORNIA-HOUSING data, respectively. For the k-anonymous version of CALIFORNIA-HOUSING data, the height lies between 150, and 250. To evaluate the privacy risks of the federated PCA system, we act as an attacker, and conduct a data reconstruction attack.

Federated learning is a distributed machine learning framework, in which each client participating to the federation trains a machine learning model on its data, and shares the trained model information with a central server, which aggregates, and sends the aggregated information back to the distributed clients. The machine learning model we choose to work with is decision trees, due to their simplicity, and interpretability. On that note, we propose a full-fledged federated pipeline, which includes discretization  and learning with decision trees for the horizontally partitioned data. Our federated discretization approach can be plugged-in as a prepossessing step before any other federated learning algorithm. During discretization, we ensure that each client creates the number of discrete bins, according to their own data/choice. Hence, our approach is both federated and personalized. After discretization, we propose to apply the post randomization method to protect the discretized data with differential privacy guarantees. After protecting its database,  each client trains a decision tree classifier on its protected database locally, and shares the nodes, containing the split attribute, and the split value with the central server. The central server obtains the most occurred split attribute, and combines the split values. This process goes on until all the nodes to be merged are leaf nodes. The central server then shares the merged tree with the distributed clients. Hence, our proposed framework performs personalized, privacy-preserving federated learning with decision trees by discretizing continuous attributes, and then masking them prior to the training stage. We call our proposed framework DISCOLEAF.

In Federated Learning (FL), two or more devices collaboratively learn a model without sharing the data. However, it has been studied that sharing model parameters can lead to substantial privacy leakage. We study an FL framework for training Gradient Boosting Decision Trees (GBDT), which is Weighted Gradient Boosting Decision Trees (WGBDT), where an aggregated gradient of all similar records from different devices is used for updating the gradients for building trees for each device. One of the approaches for determining similar records among different devices is the Locality Sensitive Hashing (LSH), which places similar records in the same bucket and dissimilar records in different buckets. Hence, WGBDT uses the aggregated gradient of similar samples while training a GBDT model for each distributed device. WGBDT assumes that each device knows the hash values of other devices' records. Using two data reconstruction attacks, we show that this assumption compromises individual privacy. When the number of LSH functions (L) is bigger than the data dimension (d), we can reconstruct more than 90% of the training data. We also show that even when L < d, significant data reconstruction is still possible. We propose Rakshit, which adds another layer of anonymization using Mondrian. We show that the reconstruction accuracy is less in Rakshit. However, some reconstruction is still possible. Our findings provide useful insights for research in privacy-preserving FL for training GBDT.