leakr: Data Leakage Detection for Machine Learning in R

Description

leakr: Data Leakage Detection for Machine Learning in R

Details

The leakr package provides tools to automatically detect common data leakage patterns in machine learning workflows for tabular data. It identifies train/test contamination, target leakage, and duplicate rows with clear diagnostic reports and visualisations.

Key Features

• Train/Test Contamination: Detects ID overlaps and distributional shifts between training and test sets

• Target Leakage: Identifies features with suspicious correlations to the target variable

• Duplication Detection: Finds exact and near-duplicate rows

• Clear Reports: Generates severity-ranked diagnostics with actionable recommendations

• Visualisations: Creates diagnostic plots to highlight issues

Main Functions

• leakr_audit: Main function for comprehensive leakage detection

• leakr_summarise: Generate human-readable summaries

• leakr_plot: Create diagnostic visualisations

Built-in Detectors

• train_test_contamination: Checks for overlap between train/test sets

• target_leakage: Identifies suspicious feature-target relationships

• duplication_detection: Finds duplicate rows in datasets

Data Compatibility

Accepts data.frame, tibble, and data.table objects.

Quick Start

# Audit a dataset for leakage
library(leakr)
report <- leakr_audit(my_data, target = "outcome")

# View summary of issues found
leakr_summarise(report)

# Create diagnostic plots
leakr_plot(report)

Author(s)

Maintainer: Cheryl Isabella Lim cheryl.academic@gmail.com

See Also

• https://github.com/cherylisabella/leakr

• Report bugs at https://github.com/cherylisabella/leakr/issues

Initialise built-in detectors

Description

Initialise built-in detectors

Usage

.onLoad(libname, pkgname)

Enhanced column name cleaning with better robustness

Description

Enhanced column name cleaning with better robustness

Usage

clean_column_names(names)

Arguments

Value

Cleaned column names

Enhanced report compilation with numeric severity scores

Description

This function compiles a report with enhanced sorting, severity scoring, and detailed metadata, including configuration information.

Usage

compile_report(
  results,
  audit_data,
  config,
  show_config = FALSE,
  top_n = 10,
  report = "default"
)

Arguments

Value

A leakr_report object containing the summary, evidence, and metadata for the report.

Enhanced date detection handling multiple formats and data types

Description

Enhanced date detection handling multiple formats and data types

Usage

detect_and_convert_dates_enhanced(data, verbose)

Arguments

Value

data.frame with converted dates

Detect file format from extension and content

Description

Detect file format from extension and content

Usage

detect_file_format(file_path, verbose = TRUE)

Arguments

Value

Character string indicating detected format

Registry-based Detector System

Description

This section of the package manages a registry for various data leakage detectors. Detectors are stored in the .detector_registry environment and are accessible by name. The system allows for easy registration of detectors, providing their descriptions and registration times. Detectors can be queried by name or listed.

Usage

.detector_registry

Format

An object of class environment of length 2.

Determine risk level and CSS class from severity counts.

Description

Determine risk level and CSS class from severity counts.

Usage

determine_risk_level(severity_counts)

Arguments

Value

List with 'level' and CSS 'class'.

Helper function to return an empty snapshot info dataframe

Description

Helper function to return an empty snapshot info dataframe

Usage

empty_snapshot_info()

Value

Empty data.frame with correct structure

Export data with consistent messaging

Description

Export data with consistent messaging

Usage

export_data_internal(data, file_path, format, verbose, ...)

Arguments

Value

Path to exported file

Format detector names for display.

Description

Format detector names by converting them to title case and separating words by spaces.

Usage

format_detector_name(detector_name)

Arguments

Value

A title-cased, space-separated string.

Generate diagnostic plots for a leakr_report

Description

Generate diagnostic plots for a leakr_report

Usage

generate_diagnostic_plots(report)

Arguments

Value

A named list of ggplot objects (currently empty stub)

Generate evidence section with format-specific handling and DRY logic.

Description

Generate evidence section with format-specific handling and DRY logic.

Usage

generate_evidence_section(report, format)

Arguments

Value

Formatted evidence section string.

Report generator

Description

Generate an executive summary text for the leakage audit report.

Usage

generate_executive_summary_text(report)

Arguments

Value

Formatted summary string (Markdown/HTML-friendly).

Generate detailed issues section with output formatting and truncation.

Description

Generate detailed issues section with output formatting and truncation.

FIX ME

Usage

generate_issues_section(report, format)

Arguments

Value

Formatted issues section string.

Generate actionable recommendations based on report findings.

Description

This function generates actionable recommendations based on the findings in a leakr_report object.

Usage

generate_recommendations(report)

Arguments

Value

A character vector of recommendations.

Examples

## Not run: 
# Requires a leakr_report object
report <- leakr_audit(iris, target = "Species")
recommendations <- generate_recommendations(report)

## End(Not run)

Format recommendations for output.

Description

Format recommendations for output.

Usage

generate_recommendations_section(report, format)

Arguments

Value

Formatted recommendation section string.

Get detector information

Description

Retrieves information about detectors, optionally filtering by the detector name.

Usage

get_detector_info(name = NULL)

Arguments

Value

A list with detector information, including description and registration date.

Examples

# Get information for all detectors
get_detector_info()

# Get information for specific detectors that actually exist
get_detector_info("file_format")

Null-coalescing operator for clean default value handling

Description

Null-coalescing operator for clean default value handling

Usage

x %||% y

Arguments

Value

x if not NULL, otherwise y

Import CSV files with robust parsing

Description

Import CSV files with robust parsing

Usage

import_csv(file_path, encoding, verbose, ...)

Arguments

Value

data.frame

Import Excel files with enhanced sheet support

Description

Import Excel files with enhanced sheet support

Usage

import_excel(file_path, sheet, verbose, ...)

Arguments

Value

data.frame

Import JSON files with better structure handling

Description

Import and process JSON files, converting them into a standardized data.frame.

Usage

import_json(file_path, verbose = FALSE, ...)

Arguments

Value

A data.frame with the content from the JSON file, flattened.

Import Parquet files

Description

Import and process Parquet files into a standardized data.frame.

Usage

import_parquet(file_path, verbose = FALSE, ...)

Arguments

Value

A data.frame with the content from the Parquet file.

Import RDS files with validation

Description

Import RDS files with validation

Usage

import_rds(file_path, verbose, ...)

Arguments

Value

data.frame

Import TSV files with robust parsing

Description

Import TSV files with robust parsing

Usage

import_tsv(file_path, encoding, verbose, ...)

Arguments

Value

data.frame

Audit dataset for data leakage

Description

This function audits a dataset for potential data leakage, running a series of predefined detectors and generating a comprehensive report with detailed findings.

Usage

leakr_audit(
  data,
  target = NULL,
  split = NULL,
  id = NULL,
  detectors = NULL,
  config = list()
)

Arguments

Value

A leakr_report object containing the audit results, including summary, evidence, and metadata.

Examples

# Basic audit on iris dataset
report <- leakr_audit(iris, target = "Species")
print(report)

Create data snapshots with improved metadata handling

Description

Save data and metadata for reproducible leakage analysis with optimised performance.

Usage

leakr_create_snapshot(
  data,
  output_dir = file.path(tempdir(), "leakr_snapshots"),
  snapshot_name = NULL,
  metadata = list(),
  sample_for_hash = TRUE
)

Arguments

Value

Path to snapshot directory

Export data in various formats

Description

Save processed data to different file formats with consistent behaviour.

Usage

leakr_export_data(data, file_path, format = "csv", verbose = TRUE, ...)

Arguments

Value

Path to exported file (invisibly)

Convert caret training objects to standard format

Description

Extract data from caret train objects for leakage analysis.

Usage

leakr_from_caret(train_obj, original_data = NULL, target_name = "target")

Arguments

Value

List with data and metadata

Convert mlr3 Task objects to standard format

Description

Extract data from mlr3 Task objects for leakage analysis.

Usage

leakr_from_mlr3(task, include_target = TRUE)

Arguments

Value

List with data, target, and metadata

Convert tidymodels workflow to standard format

Description

Extract data from tidymodels workflows for leakage analysis.

Usage

leakr_from_tidymodels(workflow, data)

Arguments

Value

List with data and metadata

Import data from various sources for leakage analysis

Description

Flexible data import function supporting multiple formats with automatic format detection and preprocessing for leakage analysis.

Usage

leakr_import(
  source,
  format = "auto",
  preprocessing = list(),
  encoding = "UTF-8",
  sheet = NULL,
  verbose = TRUE,
  ...
)

Arguments

Value

Standardised data.frame suitable for leakage analysis

A standardized data.frame suitable for leakage analysis.

List available snapshots with enhanced information

Description

Display comprehensive information about available data snapshots.

Usage

leakr_list_snapshots(
  snapshots_dir = file.path(tempdir(), "leakr_snapshots"),
  include_metadata = TRUE
)

Arguments

Value

Data.frame with snapshot information

Load data snapshot with enhanced validation

Description

Restore data from a previously created snapshot with integrity checking.

Usage

leakr_load_snapshot(snapshot_path, format = "rds", verify_integrity = TRUE)

Arguments

Value

Data.frame from snapshot

Plot leakage detection results

Description

Plot leakage detection results

Usage

leakr_plot(x, ...)

Arguments

Value

A ggplot object

Fast import with default preprocessing

Description

Minimal quick import for typical user workflows. Uses leakr_import internally.

Usage

leakr_quick_import(source, ...)

Arguments

Value

Standardised data.frame

Enhanced summarise with better formatting

Description

This function provides a formatted summary of the leakage audit report. It displays a summary of the leakage issues, including the severity and top issues detected. Optionally, it can also display configuration details used for the audit.

Usage

leakr_summarise(
  report,
  top_n = 10,
  show_config = FALSE,
  config = NULL,
  audit_data = NULL,
  detectors = NULL,
  libname = NULL,
  pkgname = NULL
)

Arguments

Value

An invisible data.frame summarizing the top n issues detected.

Examples

# Create and summarise a report
report <- leakr_audit(iris, target = "Species")
leakr_summarise(report, top_n = 5)

List Registered Detectors

Description

Returns the names of all detectors currently registered in the system. This is useful for checking which detectors are available.

Usage

list_registered_detectors()

Value

A character vector containing the names of all registered detectors.

Examples

list_registered_detectors()

Create a new temporal detector

Description

Create a new temporal detector

Usage

new_temporal_detector(time_col, lookahead_window = 1)

Arguments

Value

A temporal_detector object

A temporal_detector object

Create a new train-test detector

Description

Create a new train-test detector

Usage

new_train_test_detector(threshold = 0.1)

Arguments

Value

A train_test_detector object

Plot a detector_result object

Description

Plot a detector_result object

Plot a detector_result object

Usage

## S3 method for class 'detector_result'
plot(x, palette = NULL, ...)

Arguments

Value

A ggplot object, invisibly. Printed if interactive

A ggplot object, invisibly. Printed if interactive

Plot a udld_report object

Description

This function generates a bar plot of leakage issues detected by different detectors. The plot displays the count of issues by severity level for each detector in a udld_report object.

Usage

## S3 method for class 'udld_report'
plot(x, palette = NULL, ...)

Arguments

Value

A ggplot object, invisibly. The plot is printed if the session is interactive.

Enhanced data preparation with robust preprocessing

Description

This function performs robust data preprocessing and prepares the data for leakage detection. It handles intelligent sampling, adjusts for the presence of a target variable, and structures the data for further audit and analysis.

Usage

prepare_audit_data(data, target, split, id, config)

Arguments

Value

A list of class audit_data containing preprocessed data along with metadata, such as:

• data: The processed data.

• target: The target variable name.

• split: The split vector or column name.

• n_rows: The number of rows in the data.

• n_cols: The number of columns in the data.

• was_sampled: A logical indicating whether sampling was performed.

Examples

## Not run: 
audit_data <- prepare_audit_data(data, target = "target_column",
                                 split = "train_test_split",
                                 id = "id_column",
                                 config = list(sample_size = 50000))

## End(Not run)

Enhanced preprocessing with better performance and robustness

Description

A preprocessing function to handle common data issues, such as removing empty rows/columns, handling dates, and converting character columns to factors. This function improves data quality before further analysis.

Usage

preprocess_imported_data(data, preprocessing = list(), verbose = FALSE)

Arguments

Value

A preprocessed data.frame.

Print method for leakr_report

Description

Print method for leakr_report

Usage

## S3 method for class 'leakr_report'
print(x, ...)

Arguments

Register a new detector

Description

Register a new data leakage detector function

Usage

register_detector(name, fun, description = "")

Arguments

Value

Invisibly returns registration status

Run a detector on data

Description

Run a detector on data

Usage

run_detector(detector, data, split = NULL, id = NULL, config = list())

Arguments

Value

A detector result object

A detector result object

Run multiple detectors on audit data

Description

This function runs multiple leakage detectors on the provided audit data and returns the results for each detector.

Usage

run_detectors(detectors, audit_data, config)

Arguments

Value

A list where each element contains the results of running a detector. If a detector fails, an error message is included in the result.

Examples

## Not run: 
detectors <- list(
  temporal = list(func = temporal_detector_func),
  train_test = new_train_test_detector()
)
results <- run_detectors(detectors, audit_data = iris, config = list(sample_size = 50000))

## End(Not run)

Stratified sampling helper

Description

This function performs stratified sampling based on the provided target vector. The sampling is done proportionally to the distribution of values in the target vector.

Usage

stratified_sample(target_vec, n_sample)

Arguments

Value

A vector of indices representing the sampled observations.

Robust data validation and preprocessing

Description

This function performs data validation and preprocessing for audit purposes. It checks the validity of the input data, ensures that the target and ID columns exist, and handles empty or problematic columns.

Usage

validate_and_preprocess_data(data, target, split, id)

Arguments

Value

The validated and preprocessed data.

Examples

## Not run: 
# Example data
data <- data.frame(target = rnorm(100), id = 1:100)
target <- "target"
id <- "id"
validated_data <- validate_and_preprocess_data(data, target, NULL, id)

## End(Not run)

Enhanced data validation with better error messages

Description

Enhanced data validation with better error messages

Usage

validate_imported_data(data, source)

Arguments

Value

TRUE (invisibly) if validation passes