What to use insteadView Python source

Use DataFrames and SQL instead of RDDs

spark.sparkContext and the RDD API are not available, as on any Spark Connect server. Use the DataFrame and SQL APIs instead, for example spark.createDataFrame([], schema) rather than sparkContext.emptyRDD().

Check targets: pysail==0.7.3 with pyspark-client==4.1.3

The Python assertions below check this behavior. Details attributed to external documentation are outside those assertions. Newer releases may behave differently.

Run the check

Follow the test setup, then run from the repository root (sail-examples/). The harness applies the configuration declared in this check.

Sail

uv run pytest 'tests/test_facts.py::test_fact[no-spark-context[sail]]'

Apache Spark (requires Java 17)

uv run pytest 'tests/test_facts.py::test_fact[no-spark-context[spark]]'

Python code

no-spark-context.py20 lines

View no-spark-context.py on GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20FACT = {
    "id": "no-spark-context",
    "title": "Use DataFrames and SQL instead of RDDs",
    "category": "missing-features",
    "core": True,
    "text": """`spark.sparkContext` and the RDD API are not available, as on any Spark Connect server.
        Use the DataFrame and SQL APIs instead, for example `spark.createDataFrame([], schema)` rather
        than `sparkContext.emptyRDD()`.""",
    "engines": ["sail", "spark"],
    "order": 2,
}


def check(spark, engine):
    try:
        spark.sparkContext  # noqa: B018
    except Exception as error:  # noqa: BLE001
        assert "JVM_ATTRIBUTE_NOT_SUPPORTED" in str(error), f"other failure: {error}"
        return {"error": type(error).__name__}
    raise AssertionError("spark.sparkContext is available; update this fact")

Related examples

• Hello Sail