Create pipeline alerts

Pipeline alerts in Cloud Data Fusion let you track execution status and monitor workflow health. These notifications help you detect pipeline failures immediately.

You can configure alerts using two different services depending on your deployment stage. Cloud Data Fusion alerts are set up during batch pipeline design and can't be added or edited after deployment. In contrast, you can create log-based alerts through Cloud Monitoring in the Google Cloud console at any time—either before or after you deploy the pipeline.

Before you begin

You need a Cloud Data Fusion instance. For more information, see Create a public instance.

Add an alert in Cloud Data Fusion

Open your instance.
1. In the Google Cloud console, go to the Cloud Data Fusion page.
2. To open the instance in the Cloud Data Fusion Studio, click Instances, and then click View instance.
  
  Go to Instances
Go to the Studio page where you're designing your pipeline.
Click Configure > Pipeline alerts > Add.
Select and configure the alert. For more information about configuring alerts, see Batch pipeline alerts.

Note: For email alerts, if your pipeline doesn't have a long-running Managed Service for Apache Spark cluster with Simple Mail Transfer Protocol (SMTP) server, enter the mail server or service's information, such as its host and port requirements.
After you configure the alert, click Validate to check for errors, and then click Confirm.

Create an alert in Monitoring

Enable Cloud Logging in your Cloud Data Fusion instance.
Create a log-based alert.
1. In the Google Cloud console, go to the Logs Explorer page.
  
  Go to the Logs Explorer
  
  This page lets you store, query, update, and delete data.
2. Enter the following query:
```
resource.type="cloud_dataproc_cluster"
logName="projects/PROJECT_ID/logs/datafusion-pipeline-logs"
resource.labels.project_id="PROJECT_ID"
severity=ERROR
jsonPayload.message=~"Pipeline '.*' failed."
```
  Replace PROJECT_ID with your project's ID.
3. Click Create alert and configure the alert.

An alert is only generated when the pipeline runs on a Managed Service for Apache Spark cluster and fails. For Managed Service for Apache Spark cluster creation failures, the alert is not generated.

What's next

Refer to the CDAP OS Batch Pipeline Alerts reference pages.