> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Launch Notebook with Spark

> Launch a JupyterLab notebook with a dedicated Spark Connect cluster for interactive big data workloads.

TrueFoundry Spark Notebooks provide a JupyterLab environment with a dedicated [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html) server running alongside it. This gives you an interactive PySpark and Scala environment backed by a fully managed Spark cluster on Kubernetes — with no external infrastructure to set up.

Use Spark Notebooks when you need to:

* Explore and transform large datasets interactively
* Prototype Spark ETL pipelines before productionizing them
* Run distributed computations without managing Spark infrastructure

## Getting Started

To launch a Spark Notebook, select **Jupyter Notebook with Spark** as the workbench type in the deployment form and configure the Spark cluster settings.

<Frame caption="Select Jupyter Notebook with Spark as the workbench type">
  <img src="https://mintcdn.com/truefoundry/XtxgxhalGETYdfqO/images/spark-notebook-select-type.png?fit=max&auto=format&n=XtxgxhalGETYdfqO&q=85&s=3b95c6e558b40b70b900be2dbe9977f7" alt="" width="1024" height="596" data-path="images/spark-notebook-select-type.png" />
</Frame>

<Steps>
  <Step title="Create a new Notebook">
    Navigate to your workspace and click **New Notebook**. Select the **Jupyter Notebook with Spark** type.
  </Step>

  <Step title="Choose a Spark image">
    Select the pre-built Spark image or provide a custom extended image.

    <Frame caption="Image selection and notebook configuration">
      <img src="https://mintcdn.com/truefoundry/U0X9_nlHy-6thsfS/images/spark-notebook-image-config.png?fit=max&auto=format&n=U0X9_nlHy-6thsfS&q=85&s=dd526810c73c2104cd0a8c748a3d637b" alt="" width="1024" height="610" data-path="images/spark-notebook-image-config.png" />
    </Frame>
  </Step>

  <Step title="Configure Spark cluster resources">
    Set the driver resources, executor count (or dynamic scaling), and executor resources.

    <Frame caption="Spark Cluster Config — driver resources and executor instances">
      <img src="https://mintcdn.com/truefoundry/XtxgxhalGETYdfqO/images/spark-notebook-cluster-config.png?fit=max&auto=format&n=XtxgxhalGETYdfqO&q=85&s=33b4060faac75fd2167c130ce8f2008f" alt="" width="1024" height="688" data-path="images/spark-notebook-cluster-config.png" />
    </Frame>
  </Step>

  <Step title="Launch">
    Click deploy. The notebook and Spark Connect server will start together. A `SparkSession` is automatically available in every Python and Scala notebook cell.
  </Step>
</Steps>

## Pre-built Images

Multiple pre-built images are available, each aligned to a Databricks LTS runtime version:

| Image                                                              | Spark | Python | Scala | Delta Lake | Databricks LTS |
| ------------------------------------------------------------------ | ----- | ------ | ----- | ---------- | -------------- |
| `jupyter-spark:0.4.10-py3.12.3-sc2.13-spark4.0.2-delta4.0.1-sudo`  | 4.0.2 | 3.12   | 2.13  | 4.0.1      | 17.3           |
| `jupyter-spark:0.4.10-py3.11.14-sc2.12-spark3.5.7-delta3.3.2-sudo` | 3.5.7 | 3.11   | 2.12  | 3.3.2      | 16.4           |
| `jupyter-spark:0.4.10-py3.10.16-sc2.12-spark3.5.7-delta3.3.2-sudo` | 3.5.7 | 3.10   | 2.12  | 3.3.2      | 15.4           |
| `jupyter-spark:0.4.10-py3.10.16-sc2.12-spark3.5.7-delta3.1.0-sudo` | 3.5.7 | 3.10   | 2.12  | 3.1.0      | 14.3           |

<Info>
  All images are hosted under `public.ecr.aws/truefoundrycloud/`. For example, the full URI for the Databricks 17.3 LTS image is `public.ecr.aws/truefoundrycloud/jupyter-spark:0.4.10-py3.12.3-sc2.13-spark4.0.2-delta4.0.1-sudo`.
</Info>

<Info>
  All Jupyter Spark images are available at [https://gallery.ecr.aws/truefoundrycloud/jupyter-spark](https://gallery.ecr.aws/truefoundrycloud/jupyter-spark)
</Info>

The image includes:

* **PySpark** with Spark Connect client
* **Delta Lake** for ACID table operations
* **Scala kernel** (Almond) pre-configured with Spark Connect JARs
* **Conda** for managing multiple Python environments

## Using Spark in the Notebook

Spark is preconfigured in the notebook and available via the `spark` variable.

<CodeGroup>
  ```python Python theme={"dark"}
  # `spark` is already available — no setup needed
  df = spark.range(1000000).toDF("id")
  df.filter(df.id % 2 == 0).count()
  ```

  ```scala Scala theme={"dark"}
  // SparkSession is pre-initialized in the Scala kernel
  val df = spark.range(1000000).toDF("id")
  df.filter($"id" % 2 === 0).count()
  ```
</CodeGroup>

The notebook connects to the Spark Connect server via the `SPARK_CONNECT_URL` environment variable, which is automatically set to point to the co-located Spark Connect server.

<Note>
  The startup script retries the connection up to 5 times (configurable via `SPARK_INIT_RETRIES`). If the Spark Connect server hasn't started yet, the session will be created once it becomes available.
</Note>

### Using Delta Lake

Delta Lake is pre-installed, enabling ACID transactions on your data lake:

```python theme={"dark"}
df.write.format("delta").mode("overwrite").save("s3a://my-bucket/delta-table/")

delta_df = spark.read.format("delta").load("s3a://my-bucket/delta-table/")
delta_df.show()
```

## Spark Cluster Configuration

The Spark cluster is configured through the **Spark Cluster Config** section in the deployment form.

### Driver Resources

The Spark Connect server (driver) runs as a separate pod. Configure its resources based on the complexity of your query plans and the volume of data collected to the driver.

<ParamField body="cpu_request" type="number" default="1">
  Minimum CPU cores for the driver.
</ParamField>

<ParamField body="cpu_limit" type="number" default="3">
  Maximum CPU cores for the driver.
</ParamField>

<ParamField body="memory_request" type="number" default="4000">
  Minimum memory in MB for the driver.
</ParamField>

<ParamField body="memory_limit" type="number" default="6000">
  Maximum memory in MB for the driver.
</ParamField>

### Executor Instances

Choose between **Fixed** and **Dynamic** executor scaling:

<Tabs>
  <Tab title="Fixed Instances">
    A fixed number of executor pods are launched when the Spark cluster starts.

    | Parameter | Default | Description                      |
    | --------- | ------- | -------------------------------- |
    | `count`   | 2       | Number of executor pods to start |
  </Tab>

  <Tab title="Dynamic Scaling">
    Executors scale up and down based on workload. Idle executors are removed and new ones are added when tasks are queued.

    | Parameter | Default | Description                 |
    | --------- | ------- | --------------------------- |
    | `min`     | 1       | Minimum number of executors |
    | `max`     | 4       | Maximum number of executors |
  </Tab>
</Tabs>

### Executor Resources

Each executor pod gets its own resource allocation:

<ParamField body="cpu_request" type="number" default="2">
  CPU cores per executor.
</ParamField>

<ParamField body="memory_request" type="number" default="4000">
  Memory in MB per executor.
</ParamField>

<ParamField body="ephemeral_storage_request" type="number" default="5000">
  Ephemeral disk in MB per executor (used for shuffle data).
</ParamField>

### Spark Configuration Properties

Pass additional Spark configuration as key-value pairs. These are applied to the Spark Connect server and executors.

```
spark.sql.adaptive.enabled = true
spark.sql.shuffle.partitions = 200
spark.jars.packages = io.delta:delta-spark_2.12:3.3.2
```

<Note>
  Some internal configuration (e.g., `spark.jars.ivy`, connection timeouts, `spark.connect` packages) is managed automatically. User-supplied `spark.jars.packages` values are merged with the internal ones.
</Note>

### Spark Image

By default, the Spark Connect server and executors use the `apache/spark:4.0.2` image.

You can override this with a custom Spark image in the **Advanced** section of the Spark Cluster Config. The image must have Spark pre-installed and be compatible with the Kubernetes executor model.

## Environment Variables

The following environment variables are automatically set or can be overridden:

| Variable                 | Default        | Description                             |
| ------------------------ | -------------- | --------------------------------------- |
| `SPARK_CONNECT_URL`      | Auto-generated | gRPC URL of the Spark Connect server    |
| `SPARK_INIT_RETRIES`     | `5`            | Number of connection retries at startup |
| `SPARK_INIT_RETRY_DELAY` | `3`            | Seconds between retries                 |

You can also add custom environment variables (plain text or secret references) in the deployment form for your application code.

## Service Account

If your Spark jobs need to access cloud storage (S3, GCS, ADLS) or other cloud services, assign a Kubernetes service account with the appropriate IAM role to the notebook. The Spark Connect server and executors inherit this service account for cloud access.

Configure the service account in the **Advanced** section of the deployment form.

* AWS: [Authenticate to AWS services using IAM service account](/docs/use-aws-services-using-iam-serviceaccount)
* GCP: [Authenticate to GCP using IAM serviceaccount](/docs/authenticate-to-gcp-using-iam-serviceaccount)

## Custom Images

You can build custom Spark notebook images by extending the pre-built base images:

<CodeGroup>
  ```dockerfile Dockerfile theme={"dark"}
  FROM public.ecr.aws/truefoundrycloud/jupyter-spark:0.4.10-py3.11.14-sc2.12-spark3.5.7-delta3.3.2-sudo

  # Install additional pip packages
  RUN python3 -m pip install --no-cache-dir \
      koalas \
      mlflow \
      scikit-learn

  # Install apt packages
  USER root
  RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
      apt-get install -y --no-install-recommends graphviz && \
      apt-get clean && rm -rf /var/lib/apt/lists/*
  USER jovyan
  ```
</CodeGroup>

<Warning>
  Do not overwrite the ENTRYPOINT or CMD instructions. These are built into the base images and are critical for correct operation.
</Warning>

Build and push the image to a registry integrated with TrueFoundry, then select it as a custom image when creating the notebook.

## AI Coding Agents

Spark notebook images include AI coding agents (OpenCode and Claude Code) via Jupyter AI v3. These agents can help with Spark code, debug notebook errors, and interact with both Python and Scala kernels.

To enable AI agents, set environment variables on your Spark notebook deployment. See the [AI Coding Agents in Notebooks](/docs/launch-notebooks#ai-coding-agents-in-notebooks) section in the Jupyter Notebook guide for setup instructions.