Managed Service for Apache Spark lets you store and process workload data by using the Hadoop Distributed File System (HDFS) and Cloud Storage in parallel. With the pre-installed Cloud Storage connector, you can transfer data into and out of your clusters while tailoring underlying VM disk storage to your job performance requirements.
Storage options and considerations
The following features and considerations can be important when selecting compute and data storage options for Managed Service for Apache Spark clusters and jobs:
- HDFS with Cloud Storage: Managed Service for Apache Spark uses the Hadoop Distributed File System (HDFS) for storage. Additionally, Managed Service for Apache Spark automatically installs the HDFS-compatible Cloud Storage connector, which enables the use of Cloud Storage in parallel with HDFS. Data can be moved in and out of a cluster through upload and download to HDFS or Cloud Storage.
- VM disks:
- By default, when no local SSDs are provided, HDFS data and intermediate shuffle data is stored on VM boot disks, which are Persistent Disks.
- If you use local SSDs, HDFS data and intermediate shuffle data is stored on the SSDs.
- Persistent disk (PD) size and type affect performance and VM size, whether using HDFS or Cloud Storage for data storage.
- VM Boot disks are deleted when the cluster is deleted.