Accelerate Spark batch workloads and sessions with Lightning Engine

Console

Use the Google Cloud console to enable Lightning Engine on a batch workload.

1. In the Google Cloud console:

   1. Go to Managed Service for Apache Spark Batches.
   2. Click Create to open the Create batch page.

2. Select and fill in the following fields:

   • Container:
     • Runtime version: Select 2.3.

   • Tier Configuration:

     • Select Premium. This automatically enables and checks "Enable LIGHTNING ENGINE to accelerate Spark performance".

     When you select the premium tier, the Driver Compute Tier and the Executor Compute Tier are set to Premium. These automatically set premium tier compute settings can't be overridden for batches using runtimes prior to 3.0.

     You can configure the Driver Disk Tier and the Executor Disk Tier to Premium or leave them at their default Standard tier value. If you choose a premium disk tier, you must select the disk size. For more information, see resource allocation properties.

   • Properties: Optional: Enter the following Key (property name) and Value pair if you want to select the Native Query Execution runtime:

     Key	Value
     spark.dataproc.lightningEngine.runtime	native

3. Fill in, select, or confirm other batch workloads settings. See Submit a Spark batch workload.

4. Click Submit to run the Spark batch workload.

gcloud

Set the following gcloud CLI gcloud dataproc batches submit spark command flags to enable a Lightning Engine on a batch workload.

gcloud dataproc batches submit spark \
    --project=PROJECT_ID \
    --region=REGION \
    --version=2.3 \
    --properties=dataproc.tier=premium \
    OTHER_FLAGS_AS_NEEDED

Notes:

• PROJECT_ID: Your Google Cloud project ID. Project IDs are listed in the Project info section on the Google Cloud console Dashboard.
• REGION: An available Compute Engine region to run the workload.

• --properties=dataproc.tier=premium. Setting the premium tier automatically sets the following properties on the batch workload:

  • spark.dataproc.engine=lightningEngine selects Lightning Engine for the batch workload.
  • spark.dataproc.driver.compute.tier and spark.dataproc.executor.compute.tier are set to premium (see resource allocation properties). These automatically set premium tier compute settings can't be overridden for batches using runtimes prior to 3.0.

• Other properties

  • Native Query Engine: spark.dataproc.lightningEngine.runtime=native Add this property if you want to select the Native Query Execution runtime.

  • Disk tiers and sizes: By default, driver and executor disk sizes are set to standard tiers and sizes. You can add properties to select premium disk tiers and sizes (in multiples of 375 GiB).
    For more information, see resource allocation properties.

• OTHER_FLAGS_AS_NEEDED: See Submit a Spark batch workload.

API

To enable Lightning Engine on a batch workload, as part of your batches.create request:

• Set RuntimeConfig.version to 2.3.

• Add "dataproc.tier":"premium" to RuntimeConfig.properties Setting the premium tier automatically sets the following properties on the batch workload:

  • spark.dataproc.engine=lightningEngine selects Lightning Engine for the batch workload.
  • spark.dataproc.driver.compute.tier and spark.dataproc.executor.compute.tier are set to premium (see resource allocation properties). These automatically set premium tier compute settings can't be overridden for batches using runtimes prior to 3.0.

Other RuntimeConfig.properties:

• Native Query Engine: spark.dataproc.lightningEngine.runtime:native. Add this property if you want to select the Native Query Execution runtime.

• Disk tiers and sizes: By default, driver and executor disk sizes are set to standard tiers and sizes. You can add properties to select premium tiers and sizes (in multiples of 375 GiB).
  For more information, see resource allocation properties.

See Submit a Spark batch workload to set other batch workload API fields.

Console

Use the Google Cloud console to enable Lightning Engine on a runtime profile (session template):

1. In the Google Cloud console, go to the Runtime Profiles page:

   Go to Runtime Profiles

2. Click Create to open the Create a runtime profile page.

3. Select and fill in the following fields:

   • Tier: verify that Premium (default) is selected, and verify that Enable Lightning Engine to accelerate performance (selected by default, which sets spark.dataproc.engine=lightningEngine) is selected.

   • Additional configuration: configure the following runtime and Spark property settings:

     • Runtime configuration: click Edit and verify that Spark runtime version is set to 2.3, and then click Save.

     • Other customizations: (optional) to select the Native Query Execution runtime, click Edit, click Add property under Spark properties, enter the following Key (property name) and Value pair, and then click Save:

       Key	Value
       spark.dataproc.lightningEngine.runtime	native

4. Fill in, select, or confirm other runtime profile settings. For more information about runtime profile settings, see Create a runtime profile.

5. Click Create.

gcloud

You can't directly create a Managed Service for Apache Spark session template (runtime profile) using the gcloud CLI. Instead, you can use the gcloud beta dataproc session-templates import command to import an existing session template, edit the imported template to enable the Lightning Engine and optionally the Native Query runtime, and then export the edited template using the gcloud beta dataproc session-templates export command.

API

To enable Lightning Engine on a session template (runtime profile), as part of your sessionTemplates.create request:

• Set RuntimeConfig.version to 2.3.
• Add "dataproc.tier":"premium" and "spark.dataproc.engine":"lightningEngine" to RuntimeConfig.properties.

Other RuntimeConfig.properties:

• Native Query Engine: spark.dataproc.lightningEngine.runtime:native: Add this property to RuntimeConfig.properties to select the Native Query Execution runtime.

For more information about setting other session template API fields, see Create a runtime profile (session template).

gcloud

Set the following gcloud CLI gcloud beta dataproc sessions create spark command flags to enable Lightning Engine on an interactive session.

gcloud beta dataproc sessions create spark \
    --project=PROJECT_ID \
    --location=REGION \
    --version=2.3 \
    --properties=dataproc.tier=premium,spark.dataproc.engine=lightningEngine \
    OTHER_FLAGS_AS_NEEDED

Notes:

• PROJECT_ID: Your Google Cloud project ID. Project IDs are listed in the Project info section on the Google Cloud console Dashboard.
• REGION: An available Compute Engine region to run the workload.

• --properties=dataproc.tier=premium,spark.dataproc.engine=lightningEngine. These properties enable Lightning Engine on the session.

• Other properties:

  • Native Query Engine: spark.dataproc.lightningEngine.runtime=native: Add this property to select the Native Query Execution runtime.

• OTHER_FLAGS_AS_NEEDED: See Create an interactive session.

API

To enable Lightning Engine on a session, as part of your sessions.create request:

• Set RuntimeConfig.version to 2.3.
• Add "dataproc.tier":"premium" and "spark.dataproc.engine":"lightningEngine" to RuntimeConfig.properties.

Other RuntimeConfig.properties:

• Native Query Engine: spark.dataproc.lightningEngine.runtime:native: Add this property to RuntimeConfig.properties if you want to select the Native Query Execution runtime.

See Create an interactive session to set other session template API fields.

BigQuery notebook

You can enable Lightning Engine when you create a session in a BigQuery Studio PySpark notebook.

from google.cloud.dataproc_spark_connect import DataprocSparkSession
from google.cloud.dataproc_v1 import Session
session = Session()

# Enable Lightning Engine.
session.runtime_config.properties["version"] = "2.3"
session.runtime_config.properties["dataproc.tier"] = "premium"
session.runtime_config.properties["spark.dataproc.engine"] = "lightningEngine"

# Enable THE Native Query Execution runtime.
session.runtime_config.properties["spark.dataproc.lightningEngine.runtime"] = "native"

# Create the Spark session.
spark = (
   DataprocSparkSession.builder
     .appName("APP_NAME")
     .dataprocSessionConfig(session)
     .getOrCreate())

# Add Spark application code here: