Automated pipeline for building a fraud detection training dataset

Abstract

The study addresses the problem of preparing training data for machine learning-based fraud detection systems in e-commerce transactions. Due to the strict confidentiality of real transaction data, researchers often rely on publicly available datasets that typically suffer from limited attribute schemas, anonymized features, and a focus on specific national markets. An analysis of existing open datasets revealed the necessity of creating a specialized dataset, as none of the available sources provide a sufficient combination of realistic fraud labels, semantic transparency of features, and domain-specific attributes required for training a multi-component fraud detection system.

An automated pipeline for integrating three open Kaggle datasets (IEEE-CIS, Credit Card Transactions Fraud Detection Dataset, Fraudulent E-Commerce) is proposed. The pipeline preserves authentic fraud labels and original transaction amounts while enriching records with synthetic attributes adapted to the specifics of the Ukrainian payment market. The methods developed include: uniform normalization of timestamps based on quantile rank transformation to eliminate dataset shift artifacts while preserving intra-day patterns, synthetic generation of authentication attributes according to the EMV 3D-Secure 2.0 standard with payment network distributions based on National Bank of Ukraine statistics, formation of aggregated client behavioral profiles, and generation of “entity-IP” pairs for IP Insights model training. Both auxiliary datasets are derived exclusively from the training subset to prevent data leakage.

The resulting dataset comprises 500000 transactions spanning 24 months with a fraud rate of 3.04%, designed for training a model pipeline that includes LightGBM, an autoencoder, and IP Insights. The chronological split simulates real-world deployment conditions where models are trained on historical events and evaluated on future ones.