Elastic ML Lifecycle Automation: A Practical Guide from Manual Modeling to Intelligent Workflows

This is an open-source workshop project that demonstrates how to integrate Data Frame Analytics (DFA), AI Agent Builder, and Workflows in Elastic Stack to automate the entire ML process, solving the last-mile problem in engineering. Using the fictional LendPath company scenario, it showcases a dual design of manual modeling (to understand underlying mechanisms) and automated paths (to improve efficiency), covering the complete lifecycle from data exploration to real-time inference.

After model development, issues like deployment, real-time inference, performance monitoring, and automatic retraining need to be addressed. Elastic Stack's DFA feature allows model training within ES, but manual operation of DFA jobs, version management, and pipeline coordination are still cumbersome and error-prone.

Project Goal: Convert mortgage platform audit logs into fraud detection models. Dual-path design: Manual path (step-by-step DFA task creation via Dev Tools) and automated path (AI Agent + Workflows intelligent process). Multi-source data fusion: Integrate three types of heterogeneous data sources—IAM audit (PingOne), database audit (Oracle), and internal audit (custom system); the data generator uses correlation rules (e.g., risk scoring of abnormal events, feature associations like off_hours) to ensure the model learns business patterns.

Cross-index Mapping Consistency

Implement IaC via bootstrap-classification.py: Create explicit data streams, consistent mapping templates, fix discrepancies, and generate Kibana data views.

Model Training and Deployment

• Manual path: Explore data → ES|QL check class balance → Create DFA task → Monitor training → Analyze results → Confirm model → Deploy ingestion pipeline → Bind index
• Automated path: Enable Workflows → Build ML Readiness Analyst agent → Dialogue to auto-discover schema/features → Create automated workflow → Execute the entire process automatically

AI Agent Role

Automate EDA steps (index list, schema analysis, class balance evaluation) and focus on expert decision-making stages.

Time Pattern Modeling

Carefully consider transaction volume and risk weights for workdays/weekends, holidays, and peak hours to improve fraud detection accuracy.

Application Scenarios

• SOC automation: Real-time risk assessment, priority ranking, adaptive learning, cross-source correlation
• Compliance audit: 100% event coverage, replacing sampling audits

• Technical dependencies: Requires Elastic 9.2+ or Serverless version; Workflows/Agent need to be explicitly enabled
• Cost: Production-level ML features require corresponding subscriptions
• Interpretability: Limited ability to explain individual predictions from DFA decision tree ensembles
• Synthetic data: Cannot fully replicate the noise and edge cases of real data

Key Takeaways

1. Three-tier automation ladder: Manual → Semi-automated → Fully automated; choose as needed
2. Data engineering first: A good infrastructure is a prerequisite for effective algorithms
3. Reproducibility: Core requirement of MLOps

Project Value

As a runnable template, it demonstrates Elastic's evolution from a log platform to an intelligent data platform, covering the entire ML lifecycle and providing a panoramic view.