Data-Centric Machine Learning: A Framework for Feature Reliability Analysis

This article introduces a data-centric machine learning framework that focuses on four dimensions: feature reliability, stability, drift behavior, and consistency of feature importance, providing quality assurance for ML systems in production environments. This framework responds to the Data-Centric AI movement advocated by Andrew Ng, emphasizing the decisive impact of data quality on model performance, shifting from the traditional model-centric paradigm to a new direction of systematically improving data quality.

Paradigm Shift

The field of machine learning is evolving from model-centric to data-centric. Traditionally, focus was on model-level optimizations such as algorithm selection and hyperparameter tuning, but practice shows that data quality has a greater impact on the final outcome.

Key Value of Feature Reliability

Features are the core of model inputs, and their quality determines the upper limit of model performance. In practical applications, features often face issues such as missing values, outliers, distribution shifts, and evolve over time (e.g., changes in data sources), which are key guarantees for the continuous stability of production ML systems.

Core Function Dimensions

The framework focuses on four dimensions:

1. Feature Reliability: Measure the credibility of feature values, detect errors and noise;
2. Stability: Evaluate the statistical consistency of feature distribution over time;
3. Drift Behavior: Analyze changes in the relationship between features and target variables, detect concept drift;
4. Consistency of Feature Importance: Verify the stability of feature importance across different models/time points.

Key Technical Implementation Points

• Statistical tests need to be customized based on feature types (numerical/categorical);
• Drift detection balances sensitivity and false positive rate;
• Ensure computational efficiency for large-scale data;
• Result visualization to aid understanding;
• Configurability to support parameter adjustments based on domain knowledge.

Impact of Feature Drift

Feature drift is an invisible killer in production ML. For example, changes in the way user age is collected in recommendation systems can lead to model performance degradation, and the framework can detect this in time and trigger retraining or alerts.

Multi-Scenario Applications

• Financial Risk Control: Monitor the quality of borrower features to ensure the reliability of scorecards;
• Recommendation Systems: Track user behavior feature drift and adjust recommendation strategies;
• Industrial Predictive Maintenance: Verify the reliability of sensor features to avoid false positives and false negatives;
• Medical AI: Ensure the consistency of clinical features to protect patient safety.

Data-Centric ML Engineering Practices

The framework embodies best practices: continuous monitoring of dynamic data quality, system-level feature quality analysis, quantitative indicator management, and integration of automated pipelines.

Integration with MLOps

Can be embedded into MLOps workflows: ensure input quality during data validation, screen reliable features before training, monitor drift during service, and guide feature engineering during retraining.

Summary and Outlook

This framework promotes the evolution of ML systems from 'working' to 'working reliably'. In the future, more tools and methodologies will help establish a data-centric ML culture, supporting reliable applications in key business areas.