← Apache Spark learning pathApache Spark · Module 8 of 10
Machine learning with MLlib
Train scalable ML models on Spark.
📖 Key ideas
- MLlib provides algorithms: regression, classification, clustering, recommendation.
- Pipelines chain transformers and estimators.
- VectorAssembler combines features into a vector.
- Evaluate with BinaryClassificationEvaluator etc.
- Use when data is too large for single-machine ML.
💡 Real-life example: An MLlib pipeline is an assembly line: clean, convert, assemble, train, in a fixed order.
PipelineSequence of ML stages
TransformerConverts a DataFrame
EstimatorLearns from data to produce a model
VectorAssemblerCombines columns into a feature vector