Use Python model libraries instead of MLlib

pyspark.ml (MLlib) is not available, although Spark Connect 4.x supports it. Train and score models with Python libraries (scikit-learn, PyTorch, ONNX Runtime) inside mapInArrow or a pandas UDF, or outside the engine.

Check targets: pysail==0.7.3 with pyspark-client==4.1.3

The Python assertions below check this behavior. Details attributed to external documentation are outside those assertions. Newer releases may behave differently.

Run the check

Follow the test setup, then run from the repository root (sail-examples/). The harness applies the configuration declared in this check.

Sail

uv run pytest 'tests/test_facts.py::test_fact[mllib-unavailable[sail]]'

Apache Spark (requires Java 17)

uv run pytest 'tests/test_facts.py::test_fact[mllib-unavailable[spark]]'

Python code

mllib-unavailable.py25 lines

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25FACT = {
    "id": "mllib-unavailable",
    "title": "Use Python model libraries instead of MLlib",
    "category": "missing-features",
    "core": True,
    "text": """`pyspark.ml` (MLlib) is not available, although Spark Connect 4.x supports it. Train and
        score models with Python libraries (scikit-learn, PyTorch, ONNX Runtime) inside `mapInArrow` or a
        pandas UDF, or outside the engine.""",
    "engines": ["sail", "spark"],
    "order": 1,
}


def check(spark, engine):
    from pyspark.ml.feature import VectorAssembler

    df = spark.createDataFrame([(1.0, 2.0), (3.0, 4.0)], ["a", "b"])
    try:
        rows = VectorAssembler(inputCols=["a", "b"], outputCol="features").transform(df).collect()
    except Exception as error:  # noqa: BLE001
        assert engine == "sail", f"pyspark.ml failed on Spark: {error}"
        assert "ML relation" in str(error) or "ml command" in str(error).lower(), f"other failure: {error}"
        return {"error": type(error).__name__, "message": str(error)[:200]}
    assert engine == "spark", "pyspark.ml now works on Sail; update this fact"
    return {"rows": len(rows)}