Trustee Trustee

Welcome to Trustee’s documentation. Get started with installation and then get an overview with the quickstart. The rest of the docs describe each component of Trustee in detail, with a full reference in the API section.

Overview#

Trustee is a framework to extract decision tree explanation from black-box ML models.

Trustee Flowchart

Standard AI/ML development pipeline extended by Trustee.#

Getting Started#

This section contains basic information and instructions to get started with Trustee.

Python Version#

Trustee supports Python >=3.7.

Install Trustee#

Use the following command to install Trustee:

$ pip install trustee

Sample Code#

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

from trustee import ClassificationTrustee

X, y = datasets.load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30)

clf = RandomForestClassifier(n_estimators=100)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)

trustee = ClassificationTrustee(expert=clf)
trustee.fit(X_train, y_train, num_iter=50, num_stability_iter=10, samples_size=0.3, verbose=True)
dt, pruned_dt, agreement, reward = trustee.explain()
dt_y_pred = dt.predict(X_test)

print("Model explanation global fidelity report:")
print(classification_report(y_pred, dt_y_pred))
print("Model explanation score report:")
print(classification_report(y_test, dt_y_pred))

Other Use Cases#

For other examples and use cases of how Trustee can used to scrutinize ML models, listed in the table below, please check our Use Cases repository.

Use Case

Description

heartbleed_case/

Trustee application to a Random Forest Classifier for an Intrustion Detection System, trained with CIC-IDS-2017 dataset pre-computed features.

kitsune_case/

Trustee application to Kitsune model for anomaly detection in network traffic, trained with features extracted from Kitsune’s Mirai attack trace.

iot_case/

Trustee application to Random Forest Classifier to distguish IoT devices, trained with features extracted from the pcaps from the UNSW IoT Dataset.

moon_star_case/

Trustee application to Neural Network Moon and Stars Shortcut learning toy example.

nprint_ids_case/

Trustee application to the nPrintML AutoGluon Tabular Predictor for an Intrustion Detection System, also trained using pcaps from the CIC-IDS-2017 dataset.

nprint_os_case/

Trustee application to the nPrintML AutoGluon Tabular Predictor for OS Fingerprinting, also trained using with pcaps from the CIC-IDS-2017 dataset.

pensieve_case/

Trustee application to the Pensieve RL model for adaptive bit-rate prediction, and comparison to related work Metis.

vpn_case/

Trustee application the 1D-CNN trained to detect VPN traffic trained with the ISCX VPN-nonVPN dataset.

Supported AI/ML Libraries#

Library

Supported

scikit-learn

Keras

Tensorflow

PyTorch

AutoGluon

API Reference#

If you are looking for information on a specific function, class or method, this part of the documentation is for you.

Citing Us#

@inproceedings{Jacobs2022,
  title        = {AI/ML and Network Security: The Emperor has no Clothes},
  author       = {A. S. Jacobs and R. Beltiukov and W. Willinger and R. A. Ferreira and A. Gupta and L. Z. Granville},
  year         = 2022,
  booktitle    = {Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security},
  location     = {Los Angeles, CA, USA},
  publisher    = {Association for Computing Machinery},
  address      = {New York, NY, USA},
  series       = {CCS '22}
}