Cloud Dataproc V1 API - Class Google::Cloud::Dataproc::V1::ClusterConfig (v1.18.0)

Reference documentation and code samples for the Cloud Dataproc V1 API class Google::Cloud::Dataproc::V1::ClusterConfig.

The cluster config.

Inherits

Object

Extended By

Google::Protobuf::MessageExts::ClassMethods

Includes

Google::Protobuf::MessageExts

Methods

#autoscaling_config

def autoscaling_config() -> ::Google::Cloud::Dataproc::V1::AutoscalingConfig

Returns

(::Google::Cloud::Dataproc::V1::AutoscalingConfig) — Optional. Autoscaling config for the policy associated with the cluster. Cluster does not autoscale if this field is unset.

#autoscaling_config=

def autoscaling_config=(value) -> ::Google::Cloud::Dataproc::V1::AutoscalingConfig

Parameter

value (::Google::Cloud::Dataproc::V1::AutoscalingConfig) — Optional. Autoscaling config for the policy associated with the cluster. Cluster does not autoscale if this field is unset.

Returns

(::Google::Cloud::Dataproc::V1::AutoscalingConfig) — Optional. Autoscaling config for the policy associated with the cluster. Cluster does not autoscale if this field is unset.

#auxiliary_node_groups

def auxiliary_node_groups() -> ::Array<::Google::Cloud::Dataproc::V1::AuxiliaryNodeGroup>

Returns

(::Array<::Google::Cloud::Dataproc::V1::AuxiliaryNodeGroup>) — Optional. The node group settings.

#auxiliary_node_groups=

def auxiliary_node_groups=(value) -> ::Array<::Google::Cloud::Dataproc::V1::AuxiliaryNodeGroup>

Parameter

value (::Array<::Google::Cloud::Dataproc::V1::AuxiliaryNodeGroup>) — Optional. The node group settings.

Returns

(::Array<::Google::Cloud::Dataproc::V1::AuxiliaryNodeGroup>) — Optional. The node group settings.

#cluster_tier

def cluster_tier() -> ::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterTier

Returns

(::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterTier) — Optional. The cluster tier.

#cluster_tier=

def cluster_tier=(value) -> ::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterTier

Parameter

value (::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterTier) — Optional. The cluster tier.

Returns

(::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterTier) — Optional. The cluster tier.

#cluster_type

def cluster_type() -> ::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterType

Returns

(::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterType) — Optional. The type of the cluster.

#cluster_type=

def cluster_type=(value) -> ::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterType

Parameter

value (::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterType) — Optional. The type of the cluster.

Returns

(::Google::Cloud::Dataproc::V1::ClusterConfig::ClusterType) — Optional. The type of the cluster.

#config_bucket

def config_bucket() -> ::String

Returns

(::String) — Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's staging bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

#config_bucket=

def config_bucket=(value) -> ::String

Parameter

value (::String) — Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's staging bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

Returns

(::String) — Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's staging bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

#dataproc_metric_config

def dataproc_metric_config() -> ::Google::Cloud::Dataproc::V1::DataprocMetricConfig

Returns

(::Google::Cloud::Dataproc::V1::DataprocMetricConfig) — Optional. The config for Dataproc metrics.

#dataproc_metric_config=

def dataproc_metric_config=(value) -> ::Google::Cloud::Dataproc::V1::DataprocMetricConfig

Parameter

value (::Google::Cloud::Dataproc::V1::DataprocMetricConfig) — Optional. The config for Dataproc metrics.

Returns

(::Google::Cloud::Dataproc::V1::DataprocMetricConfig) — Optional. The config for Dataproc metrics.

#encryption_config

def encryption_config() -> ::Google::Cloud::Dataproc::V1::EncryptionConfig

Returns

(::Google::Cloud::Dataproc::V1::EncryptionConfig) — Optional. Encryption settings for the cluster.

#encryption_config=

def encryption_config=(value) -> ::Google::Cloud::Dataproc::V1::EncryptionConfig

Parameter

value (::Google::Cloud::Dataproc::V1::EncryptionConfig) — Optional. Encryption settings for the cluster.

Returns

(::Google::Cloud::Dataproc::V1::EncryptionConfig) — Optional. Encryption settings for the cluster.

#endpoint_config

def endpoint_config() -> ::Google::Cloud::Dataproc::V1::EndpointConfig

Returns

(::Google::Cloud::Dataproc::V1::EndpointConfig) — Optional. Port/endpoint configuration for this cluster

#endpoint_config=

def endpoint_config=(value) -> ::Google::Cloud::Dataproc::V1::EndpointConfig

Parameter

value (::Google::Cloud::Dataproc::V1::EndpointConfig) — Optional. Port/endpoint configuration for this cluster

Returns

(::Google::Cloud::Dataproc::V1::EndpointConfig) — Optional. Port/endpoint configuration for this cluster

#engine

def engine() -> ::Google::Cloud::Dataproc::V1::ClusterConfig::Engine

Returns

(::Google::Cloud::Dataproc::V1::ClusterConfig::Engine) — Optional. The cluster engine.

#engine=

def engine=(value) -> ::Google::Cloud::Dataproc::V1::ClusterConfig::Engine

Parameter

value (::Google::Cloud::Dataproc::V1::ClusterConfig::Engine) — Optional. The cluster engine.

Returns

(::Google::Cloud::Dataproc::V1::ClusterConfig::Engine) — Optional. The cluster engine.

#gce_cluster_config

def gce_cluster_config() -> ::Google::Cloud::Dataproc::V1::GceClusterConfig

Returns

(::Google::Cloud::Dataproc::V1::GceClusterConfig) — Optional. The shared Compute Engine config settings for all instances in a cluster.

#gce_cluster_config=

def gce_cluster_config=(value) -> ::Google::Cloud::Dataproc::V1::GceClusterConfig

Parameter

value (::Google::Cloud::Dataproc::V1::GceClusterConfig) — Optional. The shared Compute Engine config settings for all instances in a cluster.

Returns

(::Google::Cloud::Dataproc::V1::GceClusterConfig) — Optional. The shared Compute Engine config settings for all instances in a cluster.

#initialization_actions

def initialization_actions() -> ::Array<::Google::Cloud::Dataproc::V1::NodeInitializationAction>

Returns

(::Array<::Google::Cloud::Dataproc::V1::NodeInitializationAction>) —
Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes. You can test a node's role metadata to run an executable on a master or worker node, as shown below using curl (you can also use wget):
```
ROLE=$(curl -H Metadata-Flavor:Google
http://metadata/computeMetadata/v1/instance/attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  ... master specific actions ...
else
  ... worker specific actions ...
fi
```

#initialization_actions=

def initialization_actions=(value) -> ::Array<::Google::Cloud::Dataproc::V1::NodeInitializationAction>

Parameter

value (::Array<::Google::Cloud::Dataproc::V1::NodeInitializationAction>) —
Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes. You can test a node's role metadata to run an executable on a master or worker node, as shown below using curl (you can also use wget):
```
ROLE=$(curl -H Metadata-Flavor:Google
http://metadata/computeMetadata/v1/instance/attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  ... master specific actions ...
else
  ... worker specific actions ...
fi
```

Returns

(::Array<::Google::Cloud::Dataproc::V1::NodeInitializationAction>) —
Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes. You can test a node's role metadata to run an executable on a master or worker node, as shown below using curl (you can also use wget):
```
ROLE=$(curl -H Metadata-Flavor:Google
http://metadata/computeMetadata/v1/instance/attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  ... master specific actions ...
else
  ... worker specific actions ...
fi
```

#lifecycle_config

def lifecycle_config() -> ::Google::Cloud::Dataproc::V1::LifecycleConfig

Returns

(::Google::Cloud::Dataproc::V1::LifecycleConfig) — Optional. Lifecycle setting for the cluster.

#lifecycle_config=

def lifecycle_config=(value) -> ::Google::Cloud::Dataproc::V1::LifecycleConfig

Parameter

value (::Google::Cloud::Dataproc::V1::LifecycleConfig) — Optional. Lifecycle setting for the cluster.

Returns

(::Google::Cloud::Dataproc::V1::LifecycleConfig) — Optional. Lifecycle setting for the cluster.

#master_config

def master_config() -> ::Google::Cloud::Dataproc::V1::InstanceGroupConfig

Returns

(::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for the cluster's master instance.

#master_config=

def master_config=(value) -> ::Google::Cloud::Dataproc::V1::InstanceGroupConfig

Parameter

value (::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for the cluster's master instance.

Returns

(::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for the cluster's master instance.

#master_flexible_config

def master_flexible_config() -> ::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig

Returns

(::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for the cluster's master instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration across node groups:
- For single-node clusters: master_flexible_config must be specified, and no worker configs should be set.
- For multi-node (standard) clusters: both master_flexible_config and worker_flexible_config must be specified. Mixing standard and flexible configs across node groups is not allowed.
Only one of master_config or master_flexible_config can be set.

#master_flexible_config=

def master_flexible_config=(value) -> ::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig

Parameter

value (::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for the cluster's master instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration across node groups:
- For single-node clusters: master_flexible_config must be specified, and no worker configs should be set.
- For multi-node (standard) clusters: both master_flexible_config and worker_flexible_config must be specified. Mixing standard and flexible configs across node groups is not allowed.
Only one of master_config or master_flexible_config can be set.

Returns

(::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for the cluster's master instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration across node groups:
- For single-node clusters: master_flexible_config must be specified, and no worker configs should be set.
- For multi-node (standard) clusters: both master_flexible_config and worker_flexible_config must be specified. Mixing standard and flexible configs across node groups is not allowed.
Only one of master_config or master_flexible_config can be set.

#metastore_config

def metastore_config() -> ::Google::Cloud::Dataproc::V1::MetastoreConfig

Returns

(::Google::Cloud::Dataproc::V1::MetastoreConfig) — Optional. Metastore configuration.

#metastore_config=

def metastore_config=(value) -> ::Google::Cloud::Dataproc::V1::MetastoreConfig

Parameter

value (::Google::Cloud::Dataproc::V1::MetastoreConfig) — Optional. Metastore configuration.

Returns

(::Google::Cloud::Dataproc::V1::MetastoreConfig) — Optional. Metastore configuration.

#secondary_worker_config

def secondary_worker_config() -> ::Google::Cloud::Dataproc::V1::InstanceGroupConfig

Returns

(::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for a cluster's secondary worker instances

#secondary_worker_config=

def secondary_worker_config=(value) -> ::Google::Cloud::Dataproc::V1::InstanceGroupConfig

Parameter

value (::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for a cluster's secondary worker instances

Returns

(::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for a cluster's secondary worker instances

#secondary_worker_flexible_config

def secondary_worker_flexible_config() -> ::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig

Returns

(::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for a cluster's secondary worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the secondary worker group and is mutable via cluster updates and autoscaling.

Only one of secondary_worker_config or secondary_worker_flexible_config can be set.

In High Obtainability (HO) mode, secondary workers are optional. If secondary workers are used in an HO cluster, they must be configured using secondary_worker_flexible_config (not secondary_worker_config).

#secondary_worker_flexible_config=

def secondary_worker_flexible_config=(value) -> ::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig

Parameter

value (::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for a cluster's secondary worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the secondary worker group and is mutable via cluster updates and autoscaling.

Only one of secondary_worker_config or secondary_worker_flexible_config can be set.

In High Obtainability (HO) mode, secondary workers are optional. If secondary workers are used in an HO cluster, they must be configured using secondary_worker_flexible_config (not secondary_worker_config).

Returns

(::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for a cluster's secondary worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the secondary worker group and is mutable via cluster updates and autoscaling.

Only one of secondary_worker_config or secondary_worker_flexible_config can be set.

In High Obtainability (HO) mode, secondary workers are optional. If secondary workers are used in an HO cluster, they must be configured using secondary_worker_flexible_config (not secondary_worker_config).

#security_config

def security_config() -> ::Google::Cloud::Dataproc::V1::SecurityConfig

Returns

(::Google::Cloud::Dataproc::V1::SecurityConfig) — Optional. Security settings for the cluster.

#security_config=

def security_config=(value) -> ::Google::Cloud::Dataproc::V1::SecurityConfig

Parameter

value (::Google::Cloud::Dataproc::V1::SecurityConfig) — Optional. Security settings for the cluster.

Returns

(::Google::Cloud::Dataproc::V1::SecurityConfig) — Optional. Security settings for the cluster.

#software_config

def software_config() -> ::Google::Cloud::Dataproc::V1::SoftwareConfig

Returns

(::Google::Cloud::Dataproc::V1::SoftwareConfig) — Optional. The config settings for cluster software.

#software_config=

def software_config=(value) -> ::Google::Cloud::Dataproc::V1::SoftwareConfig

Parameter

value (::Google::Cloud::Dataproc::V1::SoftwareConfig) — Optional. The config settings for cluster software.

Returns

(::Google::Cloud::Dataproc::V1::SoftwareConfig) — Optional. The config settings for cluster software.

#temp_bucket

def temp_bucket() -> ::String

Returns

(::String) — Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket. The default bucket has a TTL of 90 days, but you can use any TTL (or none) if you specify a bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

#temp_bucket=

def temp_bucket=(value) -> ::String

Parameter

value (::String) — Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket. The default bucket has a TTL of 90 days, but you can use any TTL (or none) if you specify a bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

Returns

(::String) — Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket. The default bucket has a TTL of 90 days, but you can use any TTL (or none) if you specify a bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

#worker_config

def worker_config() -> ::Google::Cloud::Dataproc::V1::InstanceGroupConfig

Returns

(::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for the cluster's worker instances.

#worker_config=

def worker_config=(value) -> ::Google::Cloud::Dataproc::V1::InstanceGroupConfig

Parameter

value (::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for the cluster's worker instances.

Returns

(::Google::Cloud::Dataproc::V1::InstanceGroupConfig) — Optional. The Compute Engine config settings for the cluster's worker instances.

#worker_flexible_config

def worker_flexible_config() -> ::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig

Returns

(::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for the cluster's worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the worker group and is mutable via cluster updates and autoscaling.

Only one of worker_config or worker_flexible_config can be set.

In High Obtainability (HO) mode, for multi-node (standard) clusters, both master_flexible_config and worker_flexible_config must be specified. Mixing standard and flexible configs across node groups is not allowed. Single-node clusters do not specify worker configs.

#worker_flexible_config=

def worker_flexible_config=(value) -> ::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig

Parameter

value (::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for the cluster's worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the worker group and is mutable via cluster updates and autoscaling.

Only one of worker_config or worker_flexible_config can be set.

In High Obtainability (HO) mode, for multi-node (standard) clusters, both master_flexible_config and worker_flexible_config must be specified. Mixing standard and flexible configs across node groups is not allowed. Single-node clusters do not specify worker configs.

Returns

(::Google::Cloud::Dataproc::V1::FlexibleInstanceGroupConfig) — Optional. The High Obtainability (HO) config settings for the cluster's worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the worker group and is mutable via cluster updates and autoscaling.

Only one of worker_config or worker_flexible_config can be set.

In High Obtainability (HO) mode, for multi-node (standard) clusters, both master_flexible_config and worker_flexible_config must be specified. Mixing standard and flexible configs across node groups is not allowed. Single-node clusters do not specify worker configs.

Cloud Dataproc V1 API - Class Google::Cloud::Dataproc::V1::ClusterConfig (v1.18.0) Stay organized with collections Save and categorize content based on your preferences.

Inherits

Extended By

Includes

Methods

#autoscaling_config

#autoscaling_config=

#auxiliary_node_groups

#auxiliary_node_groups=

#cluster_tier

#cluster_tier=

#cluster_type

#cluster_type=

#config_bucket

#config_bucket=

#dataproc_metric_config

#dataproc_metric_config=

#encryption_config

#encryption_config=

#endpoint_config

#endpoint_config=

#engine

#engine=

#gce_cluster_config

#gce_cluster_config=

#initialization_actions

#initialization_actions=

#lifecycle_config

#lifecycle_config=

#master_config

#master_config=

#master_flexible_config

#master_flexible_config=

#metastore_config

#metastore_config=

#secondary_worker_config

#secondary_worker_config=

#secondary_worker_flexible_config

#secondary_worker_flexible_config=

#security_config

#security_config=

#software_config

#software_config=

#temp_bucket

#temp_bucket=

#worker_config

#worker_config=

#worker_flexible_config

#worker_flexible_config=

Cloud Dataproc V1 API - Class Google::Cloud::Dataproc::V1::ClusterConfig (v1.18.0)