DataFreeOpen Source

SYNAPSEML

Distributed ML on Apache Spark made simple

MIT

ABOUT

Scaling ML workflows across large datasets typically requires moving data out of Spark into separate training frameworks (PyTorch, TensorFlow, Scikit-learn), creating complex data pipelines that break on large data and add engineering overhead. SynapseML (formerly MMLSpark) brings ML directly into the Apache Spark ecosystem. It provides scalable estimators for classification, regression, anomaly detection, and deep learning that run as native Spark ML Pipelines, eliminating data movement. It also integrates Azure Cognitive Services, ONNX model serving, and LightGBM, allowing data scientists to build end-to-end pipelines — from feature engineering to distributed training to model serving — entirely within Spark.

INSTALL

pip install synapseml

INTEGRATION GUIDE

1. Train XGBoost or LightGBM models on terabytes of Spark DataFrames without exporting data to separate frameworks 2. Build a scalable OCR pipeline that processes millions of PDFs using Spark to orchestrate Azure Computer Vision calls 3. Deploy deep learning models (CNNs, BERT) for inference across a Spark cluster with ONNX runtime integration 4. Create a distributed anomaly detection system that scores streaming telemetry against historical baselines 5. Orchestrate multi-step ML workflows where data processing, training, and serving all stay inside Spark

SYNAPSEML

ABOUT

INTEGRATION GUIDE

TAGS