Python UDFsView Python source

Python UDFs share an interpreter in local mode

In local mode (the default), Python UDFs run inside the Sail server process, in one Python interpreter shared by all tasks, while Apache Spark runs them in separate Python worker processes. Prefer built-in functions, and for CPU-heavy Python work prefer libraries that release the GIL (NumPy, PyArrow compute, ONNX Runtime, tokenizers) so tasks can overlap.

Check targets: pysail==0.7.3 with pyspark-client==4.1.3

The Python assertions below check this behavior. Details attributed to external documentation are outside those assertions. Newer releases may behave differently.

Run the check

Follow the test setup, then run from the repository root (sail-examples/). The harness applies the configuration declared in this check.

Sail

uv run pytest 'tests/test_facts.py::test_fact[udfs-share-one-interpreter[sail]]'

Apache Spark (requires Java 17)

uv run pytest 'tests/test_facts.py::test_fact[udfs-share-one-interpreter[spark]]'

Python code

udfs-share-one-interpreter.py32 lines

View udfs-share-one-interpreter.py on GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32FACT = {
    "id": "udfs-share-one-interpreter",
    "title": "Python UDFs share an interpreter in local mode",
    "category": "python-udfs",
    "core": True,
    "text": """In local mode (the default), Python UDFs run inside the Sail server process, in one Python
        interpreter shared by all tasks, while Apache Spark runs them in separate Python worker processes.
        Prefer built-in functions, and for CPU-heavy Python work prefer libraries that release the GIL
        (NumPy, PyArrow compute, ONNX Runtime, tokenizers) so tasks can overlap.""",
    "engines": ["sail", "spark"],
    "order": 1,
}

import os

from pyspark.sql import functions as F


def check(spark, engine):
    @F.udf("long")
    def pid(_):
        import os

        return os.getpid()

    pids = {r.pid for r in spark.range(0, 4096, 1, 32).select(pid("id").alias("pid")).distinct().collect()}
    if engine == "sail":
        # The harness starts Sail in-process, so the server process is this process.
        assert pids == {os.getpid()}, f"expected UDFs in the server process {os.getpid()}, saw {pids}"
    else:
        assert os.getpid() not in pids, "Spark ran a Python UDF in the client process"
    return {"distinct_udf_pids": len(pids), "in_server_process": pids == {os.getpid()}}

Related examples

• Choose a Python UDF style