Recreate and update a Google Kubernetes Engine virtual cluster

gcloud

1. Set environment variables:

   CLUSTER=existing Managed Service for Apache Spark on GKE  cluster name \
     REGION=region
   

2. Export the existing Managed Service for Apache Spark on GKE cluster configuration to a YAML file.

   gcloud dataproc clusters export $CLUSTER \
       --region=$REGION > "${CLUSTER}-config.yaml"
   

3. Update the configuration.

   1. Remove the kubernetesNamespace field. Removing this field is necessary to avoid a namespace conflict when you create the updated cluster.

      Sample sed command to remove the kubernetesNamespace field:

      sed -E "s/kubernetesNamespace: .+$//g" ${CLUSTER}-config.yaml
      

   2. Make additional changes to update Managed Service for Apache Spark on GKE virtual cluster configuration settings, such as changing the Spark componentVersion.

4. Delete the existing Managed Service for Apache Spark on GKE virtual cluster if you will create a cluster that has the same name as the cluster it is updating (if you are replacing the original cluster).

5. Wait for the previous delete operation to finish, and then import the updated cluster configuration to create a new Managed Service for Apache Spark on GKE virtual cluster with the updated config settings.

   gcloud dataproc clusters import $CLUSTER \
       --region=$REGION \
       --source="${CLUSTER}-config.yaml"

API

1. Set environment variables:

   CLUSTER=existing Managed Service for Apache Spark on GKE  cluster name \
     REGION=region
   

2. Export the existing Managed Service for Apache Spark on GKE cluster configuration to a YAML file.

   curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)"  "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters/${CLUSTER}?alt=json" > "${CLUSTER}-config.json"

3. Update the configuration.

   1. Remove the kubernetesNamespace field. Removal of this field is necessary to avoid a namespace conflict when you create the updated cluster.

      Sample jq command to remove kubernetesNamespace field:

      jq 'del(.virtualClusterConfig.kubernetesClusterConfig.kubernetesNamespace)'
      

   2. Make additional changes to update Managed Service for Apache Spark on GKE virtual cluster configuration settings, such as changing the Spark componentVersion.

4. Delete the existing Managed Service for Apache Spark on GKE virtual cluster if you will create a cluster that has the same name as the cluster it is updating (if you are replacing the original cluster).

   curl -X DELETE -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters/${CLUSTER}"
   

5. Wait for the previous delete operation to finish, and then import the updated cluster configuration to create a new Managed Service for Apache Spark on GKE virtual cluster with the updated settings.

   curl -i -X POST  -H "Authorization: Bearer $(gcloud auth print-access-token)"  -H "Content-Type: application/json; charset=utf-8" -d "@${CLUSTER}-config.json" "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters?alt=json"
   

Console

The Google Cloud console does not support recreating a Managed Service for Apache Spark on GKE virtual cluster by importing an existing cluster's configuration.