Skip to content

Automate preprocessing of tabular data for anomaly detection methods. This pipeline handles data cleaning, normalization, and transformation, making your anomaly detection process efficient and accurate.

Notifications You must be signed in to change notification settings

JAdelhelm/Automated-Anomaly-Detection-Pipeline

Repository files navigation

AADP - Automated Anomaly Detection Pipeline

Structure of Pipeline (Click)

Abstract View - Project

alt text

Example

import numpy as np
import pandas as pd

from pipelines.control import AADP
from pipelines.defaults import initialize_autoencoder, initialize_autoencoder_modified
from pipelines.defaults import dummy_data
pd.set_option("display.max_columns", None)
from pyod.models.pca import PCA



if __name__ == "__main__":
    df_data = pd.read_csv("./temperature_USA.csv")



    # clf_if = IForest(n_jobs=-1)
    clf_pca = PCA()

    anomaly_detection_pipeline = AADP(
        deactivate_pattern_recognition=True,
        exclude_columns_no_variance=True,
        mark_anomalies_pct_data=0.005
    )

    X_output = anomaly_detection_pipeline.unsupervised_pipeline(
        X_train=df_data,
        clf=clf_pca,
        dump_model=False,
    )

    X_output.to_csv("temperatures_anomalies.csv", index=False)

Output

alt text


Highlights ⭐

πŸ“Œ BinaryEncoder instead of OneHotEncoder for nominal columns / Big Data and Performance

Newest research shows similar results for encoding nominal columns with significantly fewer dimensions.

  • (John T. Hancock and Taghi M. Khoshgoftaar. "Survey on categorical data for neural networks." In: Journal of Big Data 7.1 (2020), pp. 1–41.)
    • Tables 2, 4
  • (Diogo Seca and JoΓ£o Mendes-Moreira. "Benchmark of Encoders of Nominal Features for Regression." In: World Conference on Information Systems and Technologies. 2021, pp. 146–155.)
    • P. 151

πŸ“Œ Implementation of univariate methods / Detection of univariate anomalies

Both methods (MOD Z-Value and Tukey Method) are resilient against outliers, ensuring that the position measurement will not be biased. They also support multivariate anomaly detection algorithms in identifying univariate anomalies.

πŸ“Œ Transformation of time series data and standardization of data with RobustScaler / Normalization for better prediction results

πŸ“Œ Labeling of NaN values in an extra column instead of removing them / No loss of information



Pipeline - Logic

alt text


I used sklearn's Pipeline and Transformer concept to create this preprocessing pipeline

Feel free to contribute πŸ™‚

Reference