notebooks/Benchmark.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Benchmark of various outlier detection models\n",
"\n",
"### The models are evaluated by ROC, Precision @ n and execution time on 17 benchmark datasets. All datasets are split (60% for training and 40% for testing). The full result by averaging 10 indepent trials can be found [here](https://pyod.readthedocs.io/en/latest/benchmark.html).\n",
"\n",
"**[PyOD](https://github.com/yzhao062/pyod)** is a comprehensive **Python toolkit** to **identify outlying objects** in \n",
"multivariate data with both unsupervised and supervised approaches.\n",
"The model covered in this example includes:\n",
"\n",
" 1. Linear Models for Outlier Detection:\n",
" 1. **PCA: Principal Component Analysis** use the sum of\n",
" weighted projected distances to the eigenvector hyperplane \n",
" as the outlier outlier scores)\n",
" 2. **MCD: Minimum Covariance Determinant** (use the mahalanobis distances \n",
" as the outlier scores)\n",
" 3. **OCSVM: One-Class Support Vector Machines**\n",
" \n",
" 2. Proximity-Based Outlier Detection Models:\n",
" 1. **LOF: Local Outlier Factor**\n",
" 2. **CBLOF: Clustering-Based Local Outlier Factor**\n",
" 3. **kNN: k Nearest Neighbors** (use the distance to the kth nearest \n",
" neighbor as the outlier score)\n",
" 4. **HBOS: Histogram-based Outlier Score**\n",
" \n",
" 3. Probabilistic Models for Outlier Detection:\n",
" 1. **ABOD: Angle-Based Outlier Detection**\n",
" \n",
" 4. Outlier Ensembles and Combination Frameworks\n",
" 1. **Isolation Forest**\n",
" 2. **Feature Bagging**\n",
"\n",
" \n",
"Corresponding file could be found at /examples/compare_all_models.py"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from __future__ import division\n",
"from __future__ import print_function\n",
"\n",
"import os\n",
"import sys\n",
"from time import time\n",
"\n",
"# temporary solution for relative imports in case pyod is not installed\n",
"# if pyod is installed, no need to use the following line\n",
"sys.path.append(\n",
" os.path.abspath(os.path.join(os.path.dirname(\"__file__\"), '..')))\n",
"# supress warnings for clean output\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(\"ignore\")\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn.model_selection import train_test_split\n",
"from scipy.io import loadmat\n",
"\n",
"from pyod.models.abod import ABOD\n",
"from pyod.models.cblof import CBLOF\n",
"from pyod.models.feature_bagging import FeatureBagging\n",
"from pyod.models.hbos import HBOS\n",
"from pyod.models.iforest import IForest\n",
"from pyod.models.knn import KNN\n",
"from pyod.models.lof import LOF\n",
"from pyod.models.mcd import MCD\n",
"from pyod.models.ocsvm import OCSVM\n",
"from pyod.models.pca import PCA\n",
"\n",
"from pyod.utils.utility import standardizer\n",
"from pyod.utils.utility import precision_n_scores\n",
"from sklearn.metrics import roc_auc_score"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"... Processing arrhythmia.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.7687, precision @ rank n:0.3571, execution time: 1.126s\n",
"Cluster-based Local Outlier Factor ROC:0.7824, precision @ rank n:0.4643, execution time: 0.9786s\n",
"Feature Bagging ROC:0.7796, precision @ rank n:0.4643, execution time: 0.6076s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.8511, precision @ rank n:0.5714, execution time: 0.8914s\n",
"Isolation Forest ROC:0.8639, precision @ rank n:0.6071, execution time: 0.2507s\n",
"K Nearest Neighbors (KNN) ROC:0.782, precision @ rank n:0.5, execution time: 0.1133s\n",
"Local Outlier Factor (LOF) ROC:0.7787, precision @ rank n:0.4643, execution time: 0.0742s\n",
"Minimum Covariance Determinant (MCD) ROC:0.8228, precision @ rank n:0.4286, execution time: 1.4578s\n",
"One-class SVM (OCSVM) ROC:0.7986, precision @ rank n:0.5, execution time: 0.0481s\n",
"Principal Component Analysis (PCA) ROC:0.7997, precision @ rank n:0.5, execution time: 0.0642s\n",
"\n",
"... Processing cardio.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.5892, precision @ rank n:0.1918, execution time: 0.3579s\n",
"Cluster-based Local Outlier Factor ROC:0.973, precision @ rank n:0.7945, execution time: 0.0983s\n",
"Feature Bagging ROC:0.6385, precision @ rank n:0.1781, execution time: 0.8683s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.8373, precision @ rank n:0.4521, execution time: 0.005s\n",
"Isolation Forest ROC:0.9502, precision @ rank n:0.6027, execution time: 0.2767s\n",
"K Nearest Neighbors (KNN) ROC:0.734, precision @ rank n:0.3562, execution time: 0.1905s\n",
"Local Outlier Factor (LOF) ROC:0.588, precision @ rank n:0.1507, execution time: 0.0993s\n",
"Minimum Covariance Determinant (MCD) ROC:0.8195, precision @ rank n:0.411, execution time: 0.5835s\n",
"One-class SVM (OCSVM) ROC:0.9478, precision @ rank n:0.5342, execution time: 0.0822s\n",
"Principal Component Analysis (PCA) ROC:0.9616, precision @ rank n:0.6849, execution time: 0.003s\n",
"\n",
"... Processing glass.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.6951, precision @ rank n:0.25, execution time: 0.0361s\n",
"Cluster-based Local Outlier Factor ROC:0.7957, precision @ rank n:0.25, execution time: 0.0251s\n",
"Feature Bagging ROC:0.7073, precision @ rank n:0.25, execution time: 0.0351s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.7073, precision @ rank n:0.0, execution time: 0.005s\n",
"Isolation Forest ROC:0.7134, precision @ rank n:0.25, execution time: 0.1584s\n",
"K Nearest Neighbors (KNN) ROC:0.8384, precision @ rank n:0.25, execution time: 0.013s\n",
"Local Outlier Factor (LOF) ROC:0.7043, precision @ rank n:0.25, execution time: 0.003s\n",
"Minimum Covariance Determinant (MCD) ROC:0.8293, precision @ rank n:0.0, execution time: 0.0481s\n",
"One-class SVM (OCSVM) ROC:0.6585, precision @ rank n:0.25, execution time: 0.001s\n",
"Principal Component Analysis (PCA) ROC:0.686, precision @ rank n:0.25, execution time: 0.001s\n",
"\n",
"... Processing ionosphere.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.9181, precision @ rank n:0.8431, execution time: 0.0602s\n",
"Cluster-based Local Outlier Factor ROC:0.795, precision @ rank n:0.549, execution time: 0.0381s\n",
"Feature Bagging ROC:0.9303, precision @ rank n:0.8039, execution time: 0.0642s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.6052, precision @ rank n:0.3922, execution time: 0.006s\n",
"Isolation Forest ROC:0.8486, precision @ rank n:0.5882, execution time: 0.1785s\n",
"K Nearest Neighbors (KNN) ROC:0.932, precision @ rank n:0.8824, execution time: 0.0251s\n",
"Local Outlier Factor (LOF) ROC:0.9227, precision @ rank n:0.7843, execution time: 0.007s\n",
"Minimum Covariance Determinant (MCD) ROC:0.9669, precision @ rank n:0.8627, execution time: 0.0622s\n",
"One-class SVM (OCSVM) ROC:0.8257, precision @ rank n:0.6863, execution time: 0.004s\n",
"Principal Component Analysis (PCA) ROC:0.7941, precision @ rank n:0.5686, execution time: 0.003s\n",
"\n",
"... Processing letter.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.8783, precision @ rank n:0.4375, execution time: 0.365s\n",
"Cluster-based Local Outlier Factor ROC:0.5301, precision @ rank n:0.0312, execution time: 0.1052s\n",
"Feature Bagging ROC:0.8947, precision @ rank n:0.4062, execution time: 0.7309s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.6063, precision @ rank n:0.0938, execution time: 0.007s\n",
"Isolation Forest ROC:0.6201, precision @ rank n:0.0625, execution time: 0.2416s\n",
"K Nearest Neighbors (KNN) ROC:0.8573, precision @ rank n:0.3125, execution time: 0.1905s\n",
"Local Outlier Factor (LOF) ROC:0.8765, precision @ rank n:0.3438, execution time: 0.0922s\n",
"Minimum Covariance Determinant (MCD) ROC:0.8061, precision @ rank n:0.1875, execution time: 1.3225s\n",
"One-class SVM (OCSVM) ROC:0.5927, precision @ rank n:0.125, execution time: 0.0973s\n",
"Principal Component Analysis (PCA) ROC:0.5216, precision @ rank n:0.125, execution time: 0.005s\n",
"\n",
"... Processing lympho.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.9831, precision @ rank n:0.0, execution time: 0.0261s\n",
"Cluster-based Local Outlier Factor ROC:1.0, precision @ rank n:1.0, execution time: 0.0321s\n",
"Feature Bagging ROC:1.0, precision @ rank n:1.0, execution time: 0.0261s\n",
"Histogram-base Outlier Detection (HBOS) ROC:1.0, precision @ rank n:1.0, execution time: 0.005s\n",
"Isolation Forest ROC:1.0, precision @ rank n:1.0, execution time: 0.1795s\n",
"K Nearest Neighbors (KNN) ROC:1.0, precision @ rank n:1.0, execution time: 0.011s\n",
"Local Outlier Factor (LOF) ROC:1.0, precision @ rank n:1.0, execution time: 0.003s\n",
"Minimum Covariance Determinant (MCD) ROC:1.0, precision @ rank n:1.0, execution time: 0.0531s\n",
"One-class SVM (OCSVM) ROC:1.0, precision @ rank n:1.0, execution time: 0.002s\n",
"Principal Component Analysis (PCA) ROC:1.0, precision @ rank n:1.0, execution time: 0.001s\n",
"\n",
"... Processing mnist.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.7628, precision @ rank n:0.3367, execution time: 6.8502s\n",
"Cluster-based Local Outlier Factor ROC:0.8204, precision @ rank n:0.3605, execution time: 0.9004s\n",
"Feature Bagging ROC:0.7157, precision @ rank n:0.3741, execution time: 44.9604s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.5766, precision @ rank n:0.1361, execution time: 0.0411s\n",
"Isolation Forest ROC:0.7939, precision @ rank n:0.2721, execution time: 1.6943s\n",
"K Nearest Neighbors (KNN) ROC:0.8498, precision @ rank n:0.432, execution time: 6.6327s\n",
"Local Outlier Factor (LOF) ROC:0.7195, precision @ rank n:0.3673, execution time: 5.9395s\n",
"Minimum Covariance Determinant (MCD) ROC:0.8713, precision @ rank n:0.2653, execution time: 4.1711s\n",
"One-class SVM (OCSVM) ROC:0.854, precision @ rank n:0.3946, execution time: 4.3626s\n",
"Principal Component Analysis (PCA) ROC:0.8534, precision @ rank n:0.3878, execution time: 0.1433s\n",
"\n",
"... Processing musk.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.2161, precision @ rank n:0.1, execution time: 1.8951s\n",
"Cluster-based Local Outlier Factor ROC:0.9899, precision @ rank n:0.65, execution time: 0.2998s\n",
"Feature Bagging ROC:0.473, precision @ rank n:0.125, execution time: 12.2746s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.9999, precision @ rank n:0.975, execution time: 0.0582s\n",
"Isolation Forest ROC:1.0, precision @ rank n:1.0, execution time: 1.0528s\n",
"K Nearest Neighbors (KNN) ROC:0.8009, precision @ rank n:0.175, execution time: 1.7376s\n",
"Local Outlier Factor (LOF) ROC:0.4629, precision @ rank n:0.125, execution time: 1.5411s\n",
"Minimum Covariance Determinant (MCD) ROC:1.0, precision @ rank n:1.0, execution time: 21.114s\n",
"One-class SVM (OCSVM) ROC:1.0, precision @ rank n:1.0, execution time: 1.1922s\n",
"Principal Component Analysis (PCA) ROC:1.0, precision @ rank n:1.0, execution time: 0.1504s\n",
"\n",
"... Processing optdigits.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.4894, precision @ rank n:0.0152, execution time: 2.243s\n",
"Cluster-based Local Outlier Factor ROC:0.5329, precision @ rank n:0.0, execution time: 0.3911s\n",
"Feature Bagging ROC:0.5062, precision @ rank n:0.0303, execution time: 11.9945s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.8774, precision @ rank n:0.2121, execution time: 0.0271s\n",
"Isolation Forest ROC:0.6735, precision @ rank n:0.0303, execution time: 0.8623s\n",
"K Nearest Neighbors (KNN) ROC:0.406, precision @ rank n:0.0, execution time: 1.9502s\n",
"Local Outlier Factor (LOF) ROC:0.5277, precision @ rank n:0.0303, execution time: 1.7256s\n",
"Minimum Covariance Determinant (MCD) ROC:0.3822, precision @ rank n:0.0, execution time: 1.6714s\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"One-class SVM (OCSVM) ROC:0.5171, precision @ rank n:0.0, execution time: 1.4729s\n",
"Principal Component Analysis (PCA) ROC:0.526, precision @ rank n:0.0, execution time: 0.0521s\n",
"\n",
"... Processing pendigits.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.667, precision @ rank n:0.0526, execution time: 1.3767s\n",
"Cluster-based Local Outlier Factor ROC:0.9172, precision @ rank n:0.1579, execution time: 0.1955s\n",
"Feature Bagging ROC:0.4889, precision @ rank n:0.0526, execution time: 3.6236s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.9348, precision @ rank n:0.2632, execution time: 0.008s\n",
"Isolation Forest ROC:0.9376, precision @ rank n:0.3333, execution time: 0.6768s\n",
"K Nearest Neighbors (KNN) ROC:0.7371, precision @ rank n:0.0702, execution time: 0.8352s\n",
"Local Outlier Factor (LOF) ROC:0.4965, precision @ rank n:0.0702, execution time: 0.5886s\n",
"Minimum Covariance Determinant (MCD) ROC:0.8204, precision @ rank n:0.0877, execution time: 2.274s\n",
"One-class SVM (OCSVM) ROC:0.9235, precision @ rank n:0.3158, execution time: 0.9646s\n",
"Principal Component Analysis (PCA) ROC:0.9309, precision @ rank n:0.3158, execution time: 0.007s\n",
"\n",
"... Processing pima.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.7163, precision @ rank n:0.5253, execution time: 0.1484s\n",
"Cluster-based Local Outlier Factor ROC:0.7661, precision @ rank n:0.6061, execution time: 0.0752s\n",
"Feature Bagging ROC:0.6448, precision @ rank n:0.4444, execution time: 0.0982s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.711, precision @ rank n:0.5354, execution time: 0.002s\n",
"Isolation Forest ROC:0.6818, precision @ rank n:0.5152, execution time: 0.2176s\n",
"K Nearest Neighbors (KNN) ROC:0.7395, precision @ rank n:0.5859, execution time: 0.0572s\n",
"Local Outlier Factor (LOF) ROC:0.6574, precision @ rank n:0.4646, execution time: 0.011s\n",
"Minimum Covariance Determinant (MCD) ROC:0.7175, precision @ rank n:0.5152, execution time: 0.0491s\n",
"One-class SVM (OCSVM) ROC:0.6561, precision @ rank n:0.5051, execution time: 0.01s\n",
"Principal Component Analysis (PCA) ROC:0.6762, precision @ rank n:0.5354, execution time: 0.001s\n",
"\n",
"... Processing satellite.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.5653, precision @ rank n:0.3962, execution time: 1.6945s\n",
"Cluster-based Local Outlier Factor ROC:0.5548, precision @ rank n:0.345, execution time: 0.3288s\n",
"Feature Bagging ROC:0.572, precision @ rank n:0.4, execution time: 7.0868s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.7486, precision @ rank n:0.57, execution time: 0.017s\n",
"Isolation Forest ROC:0.6825, precision @ rank n:0.5825, execution time: 0.5966s\n",
"K Nearest Neighbors (KNN) ROC:0.6853, precision @ rank n:0.4988, execution time: 1.1901s\n",
"Local Outlier Factor (LOF) ROC:0.572, precision @ rank n:0.395, execution time: 0.9465s\n",
"Minimum Covariance Determinant (MCD) ROC:0.8055, precision @ rank n:0.6762, execution time: 2.5738s\n",
"One-class SVM (OCSVM) ROC:0.6478, precision @ rank n:0.5225, execution time: 1.3165s\n",
"Principal Component Analysis (PCA) ROC:0.5923, precision @ rank n:0.465, execution time: 0.022s\n",
"\n",
"... Processing satimage-2.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.8432, precision @ rank n:0.2333, execution time: 1.3987s\n",
"Cluster-based Local Outlier Factor ROC:0.9783, precision @ rank n:0.6667, execution time: 0.2396s\n",
"Feature Bagging ROC:0.5235, precision @ rank n:0.1667, execution time: 5.8847s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.9784, precision @ rank n:0.6, execution time: 0.015s\n",
"Isolation Forest ROC:0.9952, precision @ rank n:0.8667, execution time: 0.5154s\n",
"K Nearest Neighbors (KNN) ROC:0.9515, precision @ rank n:0.4333, execution time: 0.9897s\n",
"Local Outlier Factor (LOF) ROC:0.5257, precision @ rank n:0.1667, execution time: 0.7229s\n",
"Minimum Covariance Determinant (MCD) ROC:0.9963, precision @ rank n:0.6667, execution time: 2.0875s\n",
"One-class SVM (OCSVM) ROC:0.9997, precision @ rank n:0.9, execution time: 1.0097s\n",
"Principal Component Analysis (PCA) ROC:0.9816, precision @ rank n:0.7333, execution time: 0.016s\n",
"\n",
"... Processing shuttle.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.6171, precision @ rank n:0.2003, execution time: 13.0948s\n",
"Cluster-based Local Outlier Factor ROC:0.6273, precision @ rank n:0.2025, execution time: 0.7941s\n",
"Feature Bagging ROC:0.4725, precision @ rank n:0.0257, execution time: 73.3649s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.9871, precision @ rank n:0.9985, execution time: 0.015s\n",
"Isolation Forest ROC:0.9976, precision @ rank n:0.9596, execution time: 3.2025s\n",
"K Nearest Neighbors (KNN) ROC:0.6507, precision @ rank n:0.212, execution time: 9.6808s\n",
"Local Outlier Factor (LOF) ROC:0.5556, precision @ rank n:0.1548, execution time: 10.6754s\n",
"Minimum Covariance Determinant (MCD) ROC:0.99, precision @ rank n:0.7395, execution time: 10.3525s\n",
"One-class SVM (OCSVM) ROC:0.9934, precision @ rank n:0.956, execution time: 44.1575s\n",
"Principal Component Analysis (PCA) ROC:0.9915, precision @ rank n:0.9516, execution time: 0.032s\n",
"\n",
"... Processing vertebral.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.5366, precision @ rank n:0.2143, execution time: 0.0431s\n",
"Cluster-based Local Outlier Factor ROC:0.3937, precision @ rank n:0.0, execution time: 0.0311s\n",
"Feature Bagging ROC:0.5279, precision @ rank n:0.1429, execution time: 0.029s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.3506, precision @ rank n:0.0, execution time: 0.002s\n",
"Isolation Forest ROC:0.3772, precision @ rank n:0.0, execution time: 0.1575s\n",
"K Nearest Neighbors (KNN) ROC:0.4573, precision @ rank n:0.0714, execution time: 0.015s\n",
"Local Outlier Factor (LOF) ROC:0.4983, precision @ rank n:0.1429, execution time: 0.003s\n",
"Minimum Covariance Determinant (MCD) ROC:0.4103, precision @ rank n:0.0714, execution time: 0.0331s\n",
"One-class SVM (OCSVM) ROC:0.4686, precision @ rank n:0.0714, execution time: 0.002s\n",
"Principal Component Analysis (PCA) ROC:0.4085, precision @ rank n:0.0, execution time: 0.001s\n",
"\n",
"... Processing vowels.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.9616, precision @ rank n:0.6316, execution time: 0.2336s\n",
"Cluster-based Local Outlier Factor ROC:0.6496, precision @ rank n:0.1053, execution time: 0.0662s\n",
"Feature Bagging ROC:0.9365, precision @ rank n:0.3684, execution time: 0.2577s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.6876, precision @ rank n:0.1579, execution time: 0.003s\n",
"Isolation Forest ROC:0.8209, precision @ rank n:0.1579, execution time: 0.2116s\n",
"K Nearest Neighbors (KNN) ROC:0.9734, precision @ rank n:0.4737, execution time: 0.1113s\n",
"Local Outlier Factor (LOF) ROC:0.9398, precision @ rank n:0.3684, execution time: 0.0301s\n",
"Minimum Covariance Determinant (MCD) ROC:0.7243, precision @ rank n:0.1053, execution time: 0.755s\n",
"One-class SVM (OCSVM) ROC:0.8163, precision @ rank n:0.2632, execution time: 0.0311s\n",
"Principal Component Analysis (PCA) ROC:0.6297, precision @ rank n:0.1579, execution time: 0.002s\n",
"\n",
"... Processing wbc.mat ...\n",
"Angle-based Outlier Detector (ABOD) ROC:0.921, precision @ rank n:0.375, execution time: 0.0632s\n",
"Cluster-based Local Outlier Factor ROC:0.8906, precision @ rank n:0.375, execution time: 0.0421s\n",
"Feature Bagging ROC:0.9271, precision @ rank n:0.375, execution time: 0.0792s\n",
"Histogram-base Outlier Detection (HBOS) ROC:0.9479, precision @ rank n:0.5, execution time: 0.005s\n",
"Isolation Forest ROC:0.9436, precision @ rank n:0.5, execution time: 0.1564s\n",
"K Nearest Neighbors (KNN) ROC:0.9444, precision @ rank n:0.5, execution time: 0.0271s\n",
"Local Outlier Factor (LOF) ROC:0.9227, precision @ rank n:0.375, execution time: 0.007s\n",
"Minimum Covariance Determinant (MCD) ROC:0.9288, precision @ rank n:0.5, execution time: 0.0632s\n",
"One-class SVM (OCSVM) ROC:0.9358, precision @ rank n:0.375, execution time: 0.005s\n",
"Principal Component Analysis (PCA) ROC:0.9262, precision @ rank n:0.375, execution time: 0.001s\n"
]
}
],
"source": [
"# Define data file and read X and y\n",
"mat_file_list = ['arrhythmia.mat',\n",
" 'cardio.mat',\n",
" 'glass.mat',\n",
" 'ionosphere.mat',\n",
" 'letter.mat',\n",
" 'lympho.mat',\n",
" 'mnist.mat',\n",
" 'musk.mat',\n",
" 'optdigits.mat',\n",
" 'pendigits.mat',\n",
" 'pima.mat',\n",
" 'satellite.mat',\n",
" 'satimage-2.mat',\n",
" 'shuttle.mat',\n",
" 'vertebral.mat',\n",
" 'vowels.mat',\n",
" 'wbc.mat']\n",
"\n",
"# Define nine outlier detection tools to be compared\n",
"random_state = np.random.RandomState(42)\n",
"\n",
"df_columns = ['Data', '#Samples', '# Dimensions', 'Outlier Perc',\n",
" 'ABOD', 'CBLOF', 'FB', 'HBOS', 'IForest', 'KNN', 'LOF', 'MCD',\n",
" 'OCSVM', 'PCA']\n",
"roc_df = pd.DataFrame(columns=df_columns)\n",
"prn_df = pd.DataFrame(columns=df_columns)\n",
"time_df = pd.DataFrame(columns=df_columns)\n",
"\n",
"\n",
"for mat_file in mat_file_list:\n",
" print(\"\\n... Processing\", mat_file, '...')\n",
" mat = loadmat(os.path.join('data', mat_file))\n",
"\n",
" X = mat['X']\n",
" y = mat['y'].ravel()\n",
" outliers_fraction = np.count_nonzero(y) / len(y)\n",
" outliers_percentage = round(outliers_fraction * 100, ndigits=4)\n",
"\n",
" # construct containers for saving results\n",
" roc_list = [mat_file[:-4], X.shape[0], X.shape[1], outliers_percentage]\n",
" prn_list = [mat_file[:-4], X.shape[0], X.shape[1], outliers_percentage]\n",
" time_list = [mat_file[:-4], X.shape[0], X.shape[1], outliers_percentage]\n",
"\n",
" # 60% data for training and 40% for testing\n",
" X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,\n",
" random_state=random_state)\n",
"\n",
" # standardizing data for processing\n",
" X_train_norm, X_test_norm = standardizer(X_train, X_test)\n",
"\n",
" classifiers = {'Angle-based Outlier Detector (ABOD)': ABOD(\n",
" contamination=outliers_fraction),\n",
" 'Cluster-based Local Outlier Factor': CBLOF(\n",
" contamination=outliers_fraction, check_estimator=False,\n",
" random_state=random_state),\n",
" 'Feature Bagging': FeatureBagging(contamination=outliers_fraction,\n",
" random_state=random_state),\n",
" 'Histogram-base Outlier Detection (HBOS)': HBOS(\n",
" contamination=outliers_fraction),\n",
" 'Isolation Forest': IForest(contamination=outliers_fraction,\n",
" random_state=random_state),\n",
" 'K Nearest Neighbors (KNN)': KNN(contamination=outliers_fraction),\n",
" 'Local Outlier Factor (LOF)': LOF(\n",
" contamination=outliers_fraction),\n",
" 'Minimum Covariance Determinant (MCD)': MCD(\n",
" contamination=outliers_fraction, random_state=random_state),\n",
" 'One-class SVM (OCSVM)': OCSVM(contamination=outliers_fraction),\n",
" 'Principal Component Analysis (PCA)': PCA(\n",
" contamination=outliers_fraction, random_state=random_state),\n",
" }\n",
"\n",
" for clf_name, clf in classifiers.items():\n",
" t0 = time()\n",
" clf.fit(X_train_norm)\n",
" test_scores = clf.decision_function(X_test_norm)\n",
" t1 = time()\n",
" duration = round(t1 - t0, ndigits=4)\n",
" time_list.append(duration)\n",
"\n",
" roc = round(roc_auc_score(y_test, test_scores), ndigits=4)\n",
" prn = round(precision_n_scores(y_test, test_scores), ndigits=4)\n",
"\n",
" print('{clf_name} ROC:{roc}, precision @ rank n:{prn}, '\n",
" 'execution time: {duration}s'.format(\n",
" clf_name=clf_name, roc=roc, prn=prn, duration=duration))\n",
"\n",
" roc_list.append(roc)\n",
" prn_list.append(prn)\n",
"\n",
" temp_df = pd.DataFrame(time_list).transpose()\n",
" temp_df.columns = df_columns\n",
" time_df = pd.concat([time_df, temp_df], axis=0)\n",
"\n",
" temp_df = pd.DataFrame(roc_list).transpose()\n",
" temp_df.columns = df_columns\n",
" roc_df = pd.concat([roc_df, temp_df], axis=0)\n",
"\n",
" temp_df = pd.DataFrame(prn_list).transpose()\n",
" temp_df.columns = df_columns\n",
" prn_df = pd.concat([prn_df, temp_df], axis=0)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time complexity\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Data</th>\n",
" <th>#Samples</th>\n",
" <th># Dimensions</th>\n",
" <th>Outlier Perc</th>\n",
" <th>ABOD</th>\n",
" <th>CBLOF</th>\n",
" <th>FB</th>\n",
" <th>HBOS</th>\n",
" <th>IForest</th>\n",
" <th>KNN</th>\n",
" <th>LOF</th>\n",
" <th>MCD</th>\n",
" <th>OCSVM</th>\n",
" <th>PCA</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>arrhythmia</td>\n",
" <td>452</td>\n",
" <td>274</td>\n",
" <td>14.6018</td>\n",
" <td>1.126</td>\n",
" <td>0.9786</td>\n",
" <td>0.6076</td>\n",
" <td>0.8914</td>\n",
" <td>0.2507</td>\n",
" <td>0.1133</td>\n",
" <td>0.0742</td>\n",
" <td>1.4578</td>\n",
" <td>0.0481</td>\n",
" <td>0.0642</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>cardio</td>\n",
" <td>1831</td>\n",
" <td>21</td>\n",
" <td>9.6122</td>\n",
" <td>0.3579</td>\n",
" <td>0.0983</td>\n",
" <td>0.8683</td>\n",
" <td>0.005</td>\n",
" <td>0.2767</td>\n",
" <td>0.1905</td>\n",
" <td>0.0993</td>\n",
" <td>0.5835</td>\n",
" <td>0.0822</td>\n",
" <td>0.003</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>glass</td>\n",
" <td>214</td>\n",
" <td>9</td>\n",
" <td>4.2056</td>\n",
" <td>0.0361</td>\n",
" <td>0.0251</td>\n",
" <td>0.0351</td>\n",
" <td>0.005</td>\n",
" <td>0.1584</td>\n",
" <td>0.013</td>\n",
" <td>0.003</td>\n",
" <td>0.0481</td>\n",
" <td>0.001</td>\n",
" <td>0.001</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>ionosphere</td>\n",
" <td>351</td>\n",
" <td>33</td>\n",
" <td>35.8974</td>\n",
" <td>0.0602</td>\n",
" <td>0.0381</td>\n",
" <td>0.0642</td>\n",
" <td>0.006</td>\n",
" <td>0.1785</td>\n",
" <td>0.0251</td>\n",
" <td>0.007</td>\n",
" <td>0.0622</td>\n",
" <td>0.004</td>\n",
" <td>0.003</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>letter</td>\n",
" <td>1600</td>\n",
" <td>32</td>\n",
" <td>6.25</td>\n",
" <td>0.365</td>\n",
" <td>0.1052</td>\n",
" <td>0.7309</td>\n",
" <td>0.007</td>\n",
" <td>0.2416</td>\n",
" <td>0.1905</td>\n",
" <td>0.0922</td>\n",
" <td>1.3225</td>\n",
" <td>0.0973</td>\n",
" <td>0.005</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>lympho</td>\n",
" <td>148</td>\n",
" <td>18</td>\n",
" <td>4.0541</td>\n",
" <td>0.0261</td>\n",
" <td>0.0321</td>\n",
" <td>0.0261</td>\n",
" <td>0.005</td>\n",
" <td>0.1795</td>\n",
" <td>0.011</td>\n",
" <td>0.003</td>\n",
" <td>0.0531</td>\n",
" <td>0.002</td>\n",
" <td>0.001</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>mnist</td>\n",
" <td>7603</td>\n",
" <td>100</td>\n",
" <td>9.2069</td>\n",
" <td>6.8502</td>\n",
" <td>0.9004</td>\n",
" <td>44.9604</td>\n",
" <td>0.0411</td>\n",
" <td>1.6943</td>\n",
" <td>6.6327</td>\n",
" <td>5.9395</td>\n",
" <td>4.1711</td>\n",
" <td>4.3626</td>\n",
" <td>0.1433</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>musk</td>\n",
" <td>3062</td>\n",
" <td>166</td>\n",
" <td>3.1679</td>\n",
" <td>1.8951</td>\n",
" <td>0.2998</td>\n",
" <td>12.2746</td>\n",
" <td>0.0582</td>\n",
" <td>1.0528</td>\n",
" <td>1.7376</td>\n",
" <td>1.5411</td>\n",
" <td>21.114</td>\n",
" <td>1.1922</td>\n",
" <td>0.1504</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>optdigits</td>\n",
" <td>5216</td>\n",
" <td>64</td>\n",
" <td>2.8758</td>\n",
" <td>2.243</td>\n",
" <td>0.3911</td>\n",
" <td>11.9945</td>\n",
" <td>0.0271</td>\n",
" <td>0.8623</td>\n",
" <td>1.9502</td>\n",
" <td>1.7256</td>\n",
" <td>1.6714</td>\n",
" <td>1.4729</td>\n",
" <td>0.0521</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>pendigits</td>\n",
" <td>6870</td>\n",
" <td>16</td>\n",
" <td>2.2707</td>\n",
" <td>1.3767</td>\n",
" <td>0.1955</td>\n",
" <td>3.6236</td>\n",
" <td>0.008</td>\n",
" <td>0.6768</td>\n",
" <td>0.8352</td>\n",
" <td>0.5886</td>\n",
" <td>2.274</td>\n",
" <td>0.9646</td>\n",
" <td>0.007</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>pima</td>\n",
" <td>768</td>\n",
" <td>8</td>\n",
" <td>34.8958</td>\n",
" <td>0.1484</td>\n",
" <td>0.0752</td>\n",
" <td>0.0982</td>\n",
" <td>0.002</td>\n",
" <td>0.2176</td>\n",
" <td>0.0572</td>\n",
" <td>0.011</td>\n",
" <td>0.0491</td>\n",
" <td>0.01</td>\n",
" <td>0.001</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>satellite</td>\n",
" <td>6435</td>\n",
" <td>36</td>\n",
" <td>31.6395</td>\n",
" <td>1.6945</td>\n",
" <td>0.3288</td>\n",
" <td>7.0868</td>\n",
" <td>0.017</td>\n",
" <td>0.5966</td>\n",
" <td>1.1901</td>\n",
" <td>0.9465</td>\n",
" <td>2.5738</td>\n",
" <td>1.3165</td>\n",
" <td>0.022</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>satimage-2</td>\n",
" <td>5803</td>\n",
" <td>36</td>\n",
" <td>1.2235</td>\n",
" <td>1.3987</td>\n",
" <td>0.2396</td>\n",
" <td>5.8847</td>\n",
" <td>0.015</td>\n",
" <td>0.5154</td>\n",
" <td>0.9897</td>\n",
" <td>0.7229</td>\n",
" <td>2.0875</td>\n",
" <td>1.0097</td>\n",
" <td>0.016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>shuttle</td>\n",
" <td>49097</td>\n",
" <td>9</td>\n",
" <td>7.1511</td>\n",
" <td>13.0948</td>\n",
" <td>0.7941</td>\n",
" <td>73.3649</td>\n",
" <td>0.015</td>\n",
" <td>3.2025</td>\n",
" <td>9.6808</td>\n",
" <td>10.6754</td>\n",
" <td>10.3525</td>\n",
" <td>44.1575</td>\n",
" <td>0.032</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>vertebral</td>\n",
" <td>240</td>\n",
" <td>6</td>\n",
" <td>12.5</td>\n",
" <td>0.0431</td>\n",
" <td>0.0311</td>\n",
" <td>0.029</td>\n",
" <td>0.002</td>\n",
" <td>0.1575</td>\n",
" <td>0.015</td>\n",
" <td>0.003</td>\n",
" <td>0.0331</td>\n",
" <td>0.002</td>\n",
" <td>0.001</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>vowels</td>\n",
" <td>1456</td>\n",
" <td>12</td>\n",
" <td>3.4341</td>\n",
" <td>0.2336</td>\n",
" <td>0.0662</td>\n",
" <td>0.2577</td>\n",
" <td>0.003</td>\n",
" <td>0.2116</td>\n",
" <td>0.1113</td>\n",
" <td>0.0301</td>\n",
" <td>0.755</td>\n",
" <td>0.0311</td>\n",
" <td>0.002</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>wbc</td>\n",
" <td>378</td>\n",
" <td>30</td>\n",
" <td>5.5556</td>\n",
" <td>0.0632</td>\n",
" <td>0.0421</td>\n",
" <td>0.0792</td>\n",
" <td>0.005</td>\n",
" <td>0.1564</td>\n",
" <td>0.0271</td>\n",
" <td>0.007</td>\n",
" <td>0.0632</td>\n",
" <td>0.005</td>\n",
" <td>0.001</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Data #Samples # Dimensions Outlier Perc ABOD CBLOF FB \\\n",
"0 arrhythmia 452 274 14.6018 1.126 0.9786 0.6076 \n",
"0 cardio 1831 21 9.6122 0.3579 0.0983 0.8683 \n",
"0 glass 214 9 4.2056 0.0361 0.0251 0.0351 \n",
"0 ionosphere 351 33 35.8974 0.0602 0.0381 0.0642 \n",
"0 letter 1600 32 6.25 0.365 0.1052 0.7309 \n",
"0 lympho 148 18 4.0541 0.0261 0.0321 0.0261 \n",
"0 mnist 7603 100 9.2069 6.8502 0.9004 44.9604 \n",
"0 musk 3062 166 3.1679 1.8951 0.2998 12.2746 \n",
"0 optdigits 5216 64 2.8758 2.243 0.3911 11.9945 \n",
"0 pendigits 6870 16 2.2707 1.3767 0.1955 3.6236 \n",
"0 pima 768 8 34.8958 0.1484 0.0752 0.0982 \n",
"0 satellite 6435 36 31.6395 1.6945 0.3288 7.0868 \n",
"0 satimage-2 5803 36 1.2235 1.3987 0.2396 5.8847 \n",
"0 shuttle 49097 9 7.1511 13.0948 0.7941 73.3649 \n",
"0 vertebral 240 6 12.5 0.0431 0.0311 0.029 \n",
"0 vowels 1456 12 3.4341 0.2336 0.0662 0.2577 \n",
"0 wbc 378 30 5.5556 0.0632 0.0421 0.0792 \n",
"\n",
" HBOS IForest KNN LOF MCD OCSVM PCA \n",
"0 0.8914 0.2507 0.1133 0.0742 1.4578 0.0481 0.0642 \n",
"0 0.005 0.2767 0.1905 0.0993 0.5835 0.0822 0.003 \n",
"0 0.005 0.1584 0.013 0.003 0.0481 0.001 0.001 \n",
"0 0.006 0.1785 0.0251 0.007 0.0622 0.004 0.003 \n",
"0 0.007 0.2416 0.1905 0.0922 1.3225 0.0973 0.005 \n",
"0 0.005 0.1795 0.011 0.003 0.0531 0.002 0.001 \n",
"0 0.0411 1.6943 6.6327 5.9395 4.1711 4.3626 0.1433 \n",
"0 0.0582 1.0528 1.7376 1.5411 21.114 1.1922 0.1504 \n",
"0 0.0271 0.8623 1.9502 1.7256 1.6714 1.4729 0.0521 \n",
"0 0.008 0.6768 0.8352 0.5886 2.274 0.9646 0.007 \n",
"0 0.002 0.2176 0.0572 0.011 0.0491 0.01 0.001 \n",
"0 0.017 0.5966 1.1901 0.9465 2.5738 1.3165 0.022 \n",
"0 0.015 0.5154 0.9897 0.7229 2.0875 1.0097 0.016 \n",
"0 0.015 3.2025 9.6808 10.6754 10.3525 44.1575 0.032 \n",
"0 0.002 0.1575 0.015 0.003 0.0331 0.002 0.001 \n",
"0 0.003 0.2116 0.1113 0.0301 0.755 0.0311 0.002 \n",
"0 0.005 0.1564 0.0271 0.007 0.0632 0.005 0.001 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print('Time complexity')\n",
"time_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Analyze the performance of ROC and Precision @ n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ROC Performance\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Data</th>\n",
" <th>#Samples</th>\n",
" <th># Dimensions</th>\n",
" <th>Outlier Perc</th>\n",
" <th>ABOD</th>\n",
" <th>CBLOF</th>\n",
" <th>FB</th>\n",
" <th>HBOS</th>\n",
" <th>IForest</th>\n",
" <th>KNN</th>\n",
" <th>LOF</th>\n",
" <th>MCD</th>\n",
" <th>OCSVM</th>\n",
" <th>PCA</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>arrhythmia</td>\n",
" <td>452</td>\n",
" <td>274</td>\n",
" <td>14.6018</td>\n",
" <td>0.7687</td>\n",
" <td>0.7824</td>\n",
" <td>0.7796</td>\n",
" <td>0.8511</td>\n",
" <td>0.8639</td>\n",
" <td>0.782</td>\n",
" <td>0.7787</td>\n",
" <td>0.8228</td>\n",
" <td>0.7986</td>\n",
" <td>0.7997</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>cardio</td>\n",
" <td>1831</td>\n",
" <td>21</td>\n",
" <td>9.6122</td>\n",
" <td>0.5892</td>\n",
" <td>0.973</td>\n",
" <td>0.6385</td>\n",
" <td>0.8373</td>\n",
" <td>0.9502</td>\n",
" <td>0.734</td>\n",
" <td>0.588</td>\n",
" <td>0.8195</td>\n",
" <td>0.9478</td>\n",
" <td>0.9616</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>glass</td>\n",
" <td>214</td>\n",
" <td>9</td>\n",
" <td>4.2056</td>\n",
" <td>0.6951</td>\n",
" <td>0.7957</td>\n",
" <td>0.7073</td>\n",
" <td>0.7073</td>\n",
" <td>0.7134</td>\n",
" <td>0.8384</td>\n",
" <td>0.7043</td>\n",
" <td>0.8293</td>\n",
" <td>0.6585</td>\n",
" <td>0.686</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>ionosphere</td>\n",
" <td>351</td>\n",
" <td>33</td>\n",
" <td>35.8974</td>\n",
" <td>0.9181</td>\n",
" <td>0.795</td>\n",
" <td>0.9303</td>\n",
" <td>0.6052</td>\n",
" <td>0.8486</td>\n",
" <td>0.932</td>\n",
" <td>0.9227</td>\n",
" <td>0.9669</td>\n",
" <td>0.8257</td>\n",
" <td>0.7941</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>letter</td>\n",
" <td>1600</td>\n",
" <td>32</td>\n",
" <td>6.25</td>\n",
" <td>0.8783</td>\n",
" <td>0.5301</td>\n",
" <td>0.8947</td>\n",
" <td>0.6063</td>\n",
" <td>0.6201</td>\n",
" <td>0.8573</td>\n",
" <td>0.8765</td>\n",
" <td>0.8061</td>\n",
" <td>0.5927</td>\n",
" <td>0.5216</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>lympho</td>\n",
" <td>148</td>\n",
" <td>18</td>\n",
" <td>4.0541</td>\n",
" <td>0.9831</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>mnist</td>\n",
" <td>7603</td>\n",
" <td>100</td>\n",
" <td>9.2069</td>\n",
" <td>0.7628</td>\n",
" <td>0.8204</td>\n",
" <td>0.7157</td>\n",
" <td>0.5766</td>\n",
" <td>0.7939</td>\n",
" <td>0.8498</td>\n",
" <td>0.7195</td>\n",
" <td>0.8713</td>\n",
" <td>0.854</td>\n",
" <td>0.8534</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>musk</td>\n",
" <td>3062</td>\n",
" <td>166</td>\n",
" <td>3.1679</td>\n",
" <td>0.2161</td>\n",
" <td>0.9899</td>\n",
" <td>0.473</td>\n",
" <td>0.9999</td>\n",
" <td>1</td>\n",
" <td>0.8009</td>\n",
" <td>0.4629</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>optdigits</td>\n",
" <td>5216</td>\n",
" <td>64</td>\n",
" <td>2.8758</td>\n",
" <td>0.4894</td>\n",
" <td>0.5329</td>\n",
" <td>0.5062</td>\n",
" <td>0.8774</td>\n",
" <td>0.6735</td>\n",
" <td>0.406</td>\n",
" <td>0.5277</td>\n",
" <td>0.3822</td>\n",
" <td>0.5171</td>\n",
" <td>0.526</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>pendigits</td>\n",
" <td>6870</td>\n",
" <td>16</td>\n",
" <td>2.2707</td>\n",
" <td>0.667</td>\n",
" <td>0.9172</td>\n",
" <td>0.4889</td>\n",
" <td>0.9348</td>\n",
" <td>0.9376</td>\n",
" <td>0.7371</td>\n",
" <td>0.4965</td>\n",
" <td>0.8204</td>\n",
" <td>0.9235</td>\n",
" <td>0.9309</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>pima</td>\n",
" <td>768</td>\n",
" <td>8</td>\n",
" <td>34.8958</td>\n",
" <td>0.7163</td>\n",
" <td>0.7661</td>\n",
" <td>0.6448</td>\n",
" <td>0.711</td>\n",
" <td>0.6818</td>\n",
" <td>0.7395</td>\n",
" <td>0.6574</td>\n",
" <td>0.7175</td>\n",
" <td>0.6561</td>\n",
" <td>0.6762</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>satellite</td>\n",
" <td>6435</td>\n",
" <td>36</td>\n",
" <td>31.6395</td>\n",
" <td>0.5653</td>\n",
" <td>0.5548</td>\n",
" <td>0.572</td>\n",
" <td>0.7486</td>\n",
" <td>0.6825</td>\n",
" <td>0.6853</td>\n",
" <td>0.572</td>\n",
" <td>0.8055</td>\n",
" <td>0.6478</td>\n",
" <td>0.5923</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>satimage-2</td>\n",
" <td>5803</td>\n",
" <td>36</td>\n",
" <td>1.2235</td>\n",
" <td>0.8432</td>\n",
" <td>0.9783</td>\n",
" <td>0.5235</td>\n",
" <td>0.9784</td>\n",
" <td>0.9952</td>\n",
" <td>0.9515</td>\n",
" <td>0.5257</td>\n",
" <td>0.9963</td>\n",
" <td>0.9997</td>\n",
" <td>0.9816</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>shuttle</td>\n",
" <td>49097</td>\n",
" <td>9</td>\n",
" <td>7.1511</td>\n",
" <td>0.6171</td>\n",
" <td>0.6273</td>\n",
" <td>0.4725</td>\n",
" <td>0.9871</td>\n",
" <td>0.9976</td>\n",
" <td>0.6507</td>\n",
" <td>0.5556</td>\n",
" <td>0.99</td>\n",
" <td>0.9934</td>\n",
" <td>0.9915</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>vertebral</td>\n",
" <td>240</td>\n",
" <td>6</td>\n",
" <td>12.5</td>\n",
" <td>0.5366</td>\n",
" <td>0.3937</td>\n",
" <td>0.5279</td>\n",
" <td>0.3506</td>\n",
" <td>0.3772</td>\n",
" <td>0.4573</td>\n",
" <td>0.4983</td>\n",
" <td>0.4103</td>\n",
" <td>0.4686</td>\n",
" <td>0.4085</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>vowels</td>\n",
" <td>1456</td>\n",
" <td>12</td>\n",
" <td>3.4341</td>\n",
" <td>0.9616</td>\n",
" <td>0.6496</td>\n",
" <td>0.9365</td>\n",
" <td>0.6876</td>\n",
" <td>0.8209</td>\n",
" <td>0.9734</td>\n",
" <td>0.9398</td>\n",
" <td>0.7243</td>\n",
" <td>0.8163</td>\n",
" <td>0.6297</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>wbc</td>\n",
" <td>378</td>\n",
" <td>30</td>\n",
" <td>5.5556</td>\n",
" <td>0.921</td>\n",
" <td>0.8906</td>\n",
" <td>0.9271</td>\n",
" <td>0.9479</td>\n",
" <td>0.9436</td>\n",
" <td>0.9444</td>\n",
" <td>0.9227</td>\n",
" <td>0.9288</td>\n",
" <td>0.9358</td>\n",
" <td>0.9262</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Data #Samples # Dimensions Outlier Perc ABOD CBLOF FB \\\n",
"0 arrhythmia 452 274 14.6018 0.7687 0.7824 0.7796 \n",
"0 cardio 1831 21 9.6122 0.5892 0.973 0.6385 \n",
"0 glass 214 9 4.2056 0.6951 0.7957 0.7073 \n",
"0 ionosphere 351 33 35.8974 0.9181 0.795 0.9303 \n",
"0 letter 1600 32 6.25 0.8783 0.5301 0.8947 \n",
"0 lympho 148 18 4.0541 0.9831 1 1 \n",
"0 mnist 7603 100 9.2069 0.7628 0.8204 0.7157 \n",
"0 musk 3062 166 3.1679 0.2161 0.9899 0.473 \n",
"0 optdigits 5216 64 2.8758 0.4894 0.5329 0.5062 \n",
"0 pendigits 6870 16 2.2707 0.667 0.9172 0.4889 \n",
"0 pima 768 8 34.8958 0.7163 0.7661 0.6448 \n",
"0 satellite 6435 36 31.6395 0.5653 0.5548 0.572 \n",
"0 satimage-2 5803 36 1.2235 0.8432 0.9783 0.5235 \n",
"0 shuttle 49097 9 7.1511 0.6171 0.6273 0.4725 \n",
"0 vertebral 240 6 12.5 0.5366 0.3937 0.5279 \n",
"0 vowels 1456 12 3.4341 0.9616 0.6496 0.9365 \n",
"0 wbc 378 30 5.5556 0.921 0.8906 0.9271 \n",
"\n",
" HBOS IForest KNN LOF MCD OCSVM PCA \n",
"0 0.8511 0.8639 0.782 0.7787 0.8228 0.7986 0.7997 \n",
"0 0.8373 0.9502 0.734 0.588 0.8195 0.9478 0.9616 \n",
"0 0.7073 0.7134 0.8384 0.7043 0.8293 0.6585 0.686 \n",
"0 0.6052 0.8486 0.932 0.9227 0.9669 0.8257 0.7941 \n",
"0 0.6063 0.6201 0.8573 0.8765 0.8061 0.5927 0.5216 \n",
"0 1 1 1 1 1 1 1 \n",
"0 0.5766 0.7939 0.8498 0.7195 0.8713 0.854 0.8534 \n",
"0 0.9999 1 0.8009 0.4629 1 1 1 \n",
"0 0.8774 0.6735 0.406 0.5277 0.3822 0.5171 0.526 \n",
"0 0.9348 0.9376 0.7371 0.4965 0.8204 0.9235 0.9309 \n",
"0 0.711 0.6818 0.7395 0.6574 0.7175 0.6561 0.6762 \n",
"0 0.7486 0.6825 0.6853 0.572 0.8055 0.6478 0.5923 \n",
"0 0.9784 0.9952 0.9515 0.5257 0.9963 0.9997 0.9816 \n",
"0 0.9871 0.9976 0.6507 0.5556 0.99 0.9934 0.9915 \n",
"0 0.3506 0.3772 0.4573 0.4983 0.4103 0.4686 0.4085 \n",
"0 0.6876 0.8209 0.9734 0.9398 0.7243 0.8163 0.6297 \n",
"0 0.9479 0.9436 0.9444 0.9227 0.9288 0.9358 0.9262 "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print('ROC Performance')\n",
"roc_df"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Precision @ n Performance\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Data</th>\n",
" <th>#Samples</th>\n",
" <th># Dimensions</th>\n",
" <th>Outlier Perc</th>\n",
" <th>ABOD</th>\n",
" <th>CBLOF</th>\n",
" <th>FB</th>\n",
" <th>HBOS</th>\n",
" <th>IForest</th>\n",
" <th>KNN</th>\n",
" <th>LOF</th>\n",
" <th>MCD</th>\n",
" <th>OCSVM</th>\n",
" <th>PCA</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>arrhythmia</td>\n",
" <td>452</td>\n",
" <td>274</td>\n",
" <td>14.6018</td>\n",
" <td>0.3571</td>\n",
" <td>0.4643</td>\n",
" <td>0.4643</td>\n",
" <td>0.5714</td>\n",
" <td>0.6071</td>\n",
" <td>0.5</td>\n",
" <td>0.4643</td>\n",
" <td>0.4286</td>\n",
" <td>0.5</td>\n",
" <td>0.5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>cardio</td>\n",
" <td>1831</td>\n",
" <td>21</td>\n",
" <td>9.6122</td>\n",
" <td>0.1918</td>\n",
" <td>0.7945</td>\n",
" <td>0.1781</td>\n",
" <td>0.4521</td>\n",
" <td>0.6027</td>\n",
" <td>0.3562</td>\n",
" <td>0.1507</td>\n",
" <td>0.411</td>\n",
" <td>0.5342</td>\n",
" <td>0.6849</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>glass</td>\n",
" <td>214</td>\n",
" <td>9</td>\n",
" <td>4.2056</td>\n",
" <td>0.25</td>\n",
" <td>0.25</td>\n",
" <td>0.25</td>\n",
" <td>0</td>\n",
" <td>0.25</td>\n",
" <td>0.25</td>\n",
" <td>0.25</td>\n",
" <td>0</td>\n",
" <td>0.25</td>\n",
" <td>0.25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>ionosphere</td>\n",
" <td>351</td>\n",
" <td>33</td>\n",
" <td>35.8974</td>\n",
" <td>0.8431</td>\n",
" <td>0.549</td>\n",
" <td>0.8039</td>\n",
" <td>0.3922</td>\n",
" <td>0.5882</td>\n",
" <td>0.8824</td>\n",
" <td>0.7843</td>\n",
" <td>0.8627</td>\n",
" <td>0.6863</td>\n",
" <td>0.5686</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>letter</td>\n",
" <td>1600</td>\n",
" <td>32</td>\n",
" <td>6.25</td>\n",
" <td>0.4375</td>\n",
" <td>0.0312</td>\n",
" <td>0.4062</td>\n",
" <td>0.0938</td>\n",
" <td>0.0625</td>\n",
" <td>0.3125</td>\n",
" <td>0.3438</td>\n",
" <td>0.1875</td>\n",
" <td>0.125</td>\n",
" <td>0.125</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>lympho</td>\n",
" <td>148</td>\n",
" <td>18</td>\n",
" <td>4.0541</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>mnist</td>\n",
" <td>7603</td>\n",
" <td>100</td>\n",
" <td>9.2069</td>\n",
" <td>0.3367</td>\n",
" <td>0.3605</td>\n",
" <td>0.3741</td>\n",
" <td>0.1361</td>\n",
" <td>0.2721</td>\n",
" <td>0.432</td>\n",
" <td>0.3673</td>\n",
" <td>0.2653</td>\n",
" <td>0.3946</td>\n",
" <td>0.3878</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>musk</td>\n",
" <td>3062</td>\n",
" <td>166</td>\n",
" <td>3.1679</td>\n",
" <td>0.1</td>\n",
" <td>0.65</td>\n",
" <td>0.125</td>\n",
" <td>0.975</td>\n",
" <td>1</td>\n",
" <td>0.175</td>\n",
" <td>0.125</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>optdigits</td>\n",
" <td>5216</td>\n",
" <td>64</td>\n",
" <td>2.8758</td>\n",
" <td>0.0152</td>\n",
" <td>0</td>\n",
" <td>0.0303</td>\n",
" <td>0.2121</td>\n",
" <td>0.0303</td>\n",
" <td>0</td>\n",
" <td>0.0303</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>pendigits</td>\n",
" <td>6870</td>\n",
" <td>16</td>\n",
" <td>2.2707</td>\n",
" <td>0.0526</td>\n",
" <td>0.1579</td>\n",
" <td>0.0526</td>\n",
" <td>0.2632</td>\n",
" <td>0.3333</td>\n",
" <td>0.0702</td>\n",
" <td>0.0702</td>\n",
" <td>0.0877</td>\n",
" <td>0.3158</td>\n",
" <td>0.3158</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>pima</td>\n",
" <td>768</td>\n",
" <td>8</td>\n",
" <td>34.8958</td>\n",
" <td>0.5253</td>\n",
" <td>0.6061</td>\n",
" <td>0.4444</td>\n",
" <td>0.5354</td>\n",
" <td>0.5152</td>\n",
" <td>0.5859</td>\n",
" <td>0.4646</td>\n",
" <td>0.5152</td>\n",
" <td>0.5051</td>\n",
" <td>0.5354</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>satellite</td>\n",
" <td>6435</td>\n",
" <td>36</td>\n",
" <td>31.6395</td>\n",
" <td>0.3962</td>\n",
" <td>0.345</td>\n",
" <td>0.4</td>\n",
" <td>0.57</td>\n",
" <td>0.5825</td>\n",
" <td>0.4988</td>\n",
" <td>0.395</td>\n",
" <td>0.6762</td>\n",
" <td>0.5225</td>\n",
" <td>0.465</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>satimage-2</td>\n",
" <td>5803</td>\n",
" <td>36</td>\n",
" <td>1.2235</td>\n",
" <td>0.2333</td>\n",
" <td>0.6667</td>\n",
" <td>0.1667</td>\n",
" <td>0.6</td>\n",
" <td>0.8667</td>\n",
" <td>0.4333</td>\n",
" <td>0.1667</td>\n",
" <td>0.6667</td>\n",
" <td>0.9</td>\n",
" <td>0.7333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>shuttle</td>\n",
" <td>49097</td>\n",
" <td>9</td>\n",
" <td>7.1511</td>\n",
" <td>0.2003</td>\n",
" <td>0.2025</td>\n",
" <td>0.0257</td>\n",
" <td>0.9985</td>\n",
" <td>0.9596</td>\n",
" <td>0.212</td>\n",
" <td>0.1548</td>\n",
" <td>0.7395</td>\n",
" <td>0.956</td>\n",
" <td>0.9516</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>vertebral</td>\n",
" <td>240</td>\n",
" <td>6</td>\n",
" <td>12.5</td>\n",
" <td>0.2143</td>\n",
" <td>0</td>\n",
" <td>0.1429</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0.0714</td>\n",
" <td>0.1429</td>\n",
" <td>0.0714</td>\n",
" <td>0.0714</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>vowels</td>\n",
" <td>1456</td>\n",
" <td>12</td>\n",
" <td>3.4341</td>\n",
" <td>0.6316</td>\n",
" <td>0.1053</td>\n",
" <td>0.3684</td>\n",
" <td>0.1579</td>\n",
" <td>0.1579</td>\n",
" <td>0.4737</td>\n",
" <td>0.3684</td>\n",
" <td>0.1053</td>\n",
" <td>0.2632</td>\n",
" <td>0.1579</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>wbc</td>\n",
" <td>378</td>\n",
" <td>30</td>\n",
" <td>5.5556</td>\n",
" <td>0.375</td>\n",
" <td>0.375</td>\n",
" <td>0.375</td>\n",
" <td>0.5</td>\n",
" <td>0.5</td>\n",
" <td>0.5</td>\n",
" <td>0.375</td>\n",
" <td>0.5</td>\n",
" <td>0.375</td>\n",
" <td>0.375</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Data #Samples # Dimensions Outlier Perc ABOD CBLOF FB \\\n",
"0 arrhythmia 452 274 14.6018 0.3571 0.4643 0.4643 \n",
"0 cardio 1831 21 9.6122 0.1918 0.7945 0.1781 \n",
"0 glass 214 9 4.2056 0.25 0.25 0.25 \n",
"0 ionosphere 351 33 35.8974 0.8431 0.549 0.8039 \n",
"0 letter 1600 32 6.25 0.4375 0.0312 0.4062 \n",
"0 lympho 148 18 4.0541 0 1 1 \n",
"0 mnist 7603 100 9.2069 0.3367 0.3605 0.3741 \n",
"0 musk 3062 166 3.1679 0.1 0.65 0.125 \n",
"0 optdigits 5216 64 2.8758 0.0152 0 0.0303 \n",
"0 pendigits 6870 16 2.2707 0.0526 0.1579 0.0526 \n",
"0 pima 768 8 34.8958 0.5253 0.6061 0.4444 \n",
"0 satellite 6435 36 31.6395 0.3962 0.345 0.4 \n",
"0 satimage-2 5803 36 1.2235 0.2333 0.6667 0.1667 \n",
"0 shuttle 49097 9 7.1511 0.2003 0.2025 0.0257 \n",
"0 vertebral 240 6 12.5 0.2143 0 0.1429 \n",
"0 vowels 1456 12 3.4341 0.6316 0.1053 0.3684 \n",
"0 wbc 378 30 5.5556 0.375 0.375 0.375 \n",
"\n",
" HBOS IForest KNN LOF MCD OCSVM PCA \n",
"0 0.5714 0.6071 0.5 0.4643 0.4286 0.5 0.5 \n",
"0 0.4521 0.6027 0.3562 0.1507 0.411 0.5342 0.6849 \n",
"0 0 0.25 0.25 0.25 0 0.25 0.25 \n",
"0 0.3922 0.5882 0.8824 0.7843 0.8627 0.6863 0.5686 \n",
"0 0.0938 0.0625 0.3125 0.3438 0.1875 0.125 0.125 \n",
"0 1 1 1 1 1 1 1 \n",
"0 0.1361 0.2721 0.432 0.3673 0.2653 0.3946 0.3878 \n",
"0 0.975 1 0.175 0.125 1 1 1 \n",
"0 0.2121 0.0303 0 0.0303 0 0 0 \n",
"0 0.2632 0.3333 0.0702 0.0702 0.0877 0.3158 0.3158 \n",
"0 0.5354 0.5152 0.5859 0.4646 0.5152 0.5051 0.5354 \n",
"0 0.57 0.5825 0.4988 0.395 0.6762 0.5225 0.465 \n",
"0 0.6 0.8667 0.4333 0.1667 0.6667 0.9 0.7333 \n",
"0 0.9985 0.9596 0.212 0.1548 0.7395 0.956 0.9516 \n",
"0 0 0 0.0714 0.1429 0.0714 0.0714 0 \n",
"0 0.1579 0.1579 0.4737 0.3684 0.1053 0.2632 0.1579 \n",
"0 0.5 0.5 0.5 0.375 0.5 0.375 0.375 "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print('Precision @ n Performance')\n",
"prn_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}