Install Python dependencies on the server

spark.addArtifacts() is not supported, so Python files and packages cannot be shipped from the client at run time. Install every library your UDFs import into the Python environment the Sail server runs in.

Check targets: pysail==0.7.3 with pyspark-client==4.1.3

The Python assertions below check this behavior. Details attributed to external documentation are outside those assertions. Newer releases may behave differently.

Run the check

Follow the test setup, then run from the repository root (sail-examples/). The harness applies the configuration declared in this check.

Sail

uv run pytest 'tests/test_facts.py::test_fact[udf-dependencies-in-server-env[sail]]'

Python code

udf-dependencies-in-server-env.py25 lines

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25FACT = {
    "id": "udf-dependencies-in-server-env",
    "title": "Install Python dependencies on the server",
    "category": "python-udfs",
    "core": True,
    "text": """`spark.addArtifacts()` is not supported, so Python files and packages cannot be shipped
        from the client at run time. Install every library your UDFs import into the Python environment the
        Sail server runs in.""",
    "engines": ["sail"],
    "order": 5,
}

import tempfile
from pathlib import Path


def check(spark, engine):
    module = Path(tempfile.mkdtemp()) / "helper_module.py"
    module.write_text("VALUE = 1\n")
    try:
        spark.addArtifacts(str(module), pyfile=True)
    except Exception as error:  # noqa: BLE001
        assert "add artifacts" in str(error).lower(), f"failed for another reason: {str(error)[:300]}"
        return {"error": type(error).__name__}
    raise AssertionError("addArtifacts now works; update this fact")

Install Python dependencies on the server

Run the check

Sail

Python code

Related examples