Sail examples

Appearance

Run and connectView Python source

Start Sail and connect a PySpark session

Connect to Sail with a normal PySpark Spark Connect session. In a script, start Sail in-process with pysail.spark.SparkConnectServer(), call start(), read the port from server.listening_address, and connect with SparkSession.builder.remote(f"sc://localhost:{port}"). As a long-running service, run sail spark server (default 127.0.0.1:50051, per the Sail docs). SELECT version() returns the Sail version; spark.version returns the PySpark client's version.

Check targets: pysail==0.7.3 with pyspark-client==4.1.3

The Python assertions below check this behavior. Details attributed to external documentation are outside those assertions. Newer releases may behave differently.

Run the check

Follow the test setup, then run from the repository root (sail-examples/). The harness applies the configuration declared in this check.

Sail

uv run pytest 'tests/test_facts.py::test_fact[start-and-connect[sail]]'

Apache Spark (requires Java 17)

uv run pytest 'tests/test_facts.py::test_fact[start-and-connect[spark]]'

Python code

start-and-connect.py29 lines
View start-and-connect.py on GitHub 
FACT = {
    "id": "start-and-connect",
    "title": "Start Sail and connect a PySpark session",
    "category": "run-and-connect",
    "core": True,
    "text": """Connect to Sail with a normal PySpark Spark Connect session. In a script, start Sail
        in-process with `pysail.spark.SparkConnectServer()`, call `start()`, read the port from
        `server.listening_address`, and connect with `SparkSession.builder.remote(f"sc://localhost:{port}")`.
        As a long-running service, run `sail spark server` (default `127.0.0.1:50051`, per the Sail docs).
        `SELECT version()` returns the Sail version; `spark.version` returns the PySpark client's version.""",
    "engines": ["sail", "spark"],
    "order": 1,
}


def check(spark, engine):
    import pyspark

    engine_version = spark.sql("SELECT version()").collect()[0][0]
    assert spark.version == pyspark.__version__, (
        f"spark.version {spark.version} != client {pyspark.__version__}"
    )
    if engine == "sail":
        import pysail

        assert engine_version == pysail.__version__, f"version() returned {engine_version!r}"
    else:
        assert engine_version.startswith(pyspark.__version__), f"Spark version() returned {engine_version!r}"
    return {"version()": engine_version, "spark.version": spark.version}