diff --git a/docs/sources/datasources/google-cloud-monitoring/_index.md b/docs/sources/datasources/google-cloud-monitoring/_index.md index 27bc7b390cd..55eb63a3dad 100644 --- a/docs/sources/datasources/google-cloud-monitoring/_index.md +++ b/docs/sources/datasources/google-cloud-monitoring/_index.md @@ -43,165 +43,126 @@ refs: destination: /docs/grafana//administration/provisioning/#data-sources - pattern: /docs/grafana-cloud/ destination: /docs/grafana//administration/provisioning/#data-sources + alerting: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/ + variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//dashboards/variables/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/visualizations/dashboards/variables/ + annotate-visualizations: + - pattern: /docs/grafana/ + destination: /docs/grafana//dashboards/build-dashboards/annotate-visualizations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/visualizations/dashboards/build-dashboards/annotate-visualizations/ + transformations: + - pattern: /docs/grafana/ + destination: /docs/grafana//panels-visualizations/query-transform-data/transform-data/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/visualizations/panels-visualizations/query-transform-data/transform-data/ + visualizations: + - pattern: /docs/grafana/ + destination: /docs/grafana//panels-visualizations/visualizations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/visualizations/panels-visualizations/visualizations/ + configure-gcm: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ + query-editor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + template-variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/template-variables/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/template-variables/ + annotations-gcm: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/annotations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/annotations/ + alerting-gcm: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/alerting/ + google-authentication: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/ + troubleshooting: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/troubleshooting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/troubleshooting/ --- # Google Cloud Monitoring data source -Grafana ships with built-in support for Google Cloud Monitoring. -This topic describes queries, templates, variables, and other configuration specific to the Google Cloud Monitoring data source. +Google Cloud Monitoring (formerly Stackdriver) is Google Cloud Platform's native monitoring and observability service that collects metrics, events, and metadata from GCP resources, hosted uptime probes, and application instrumentation. The Google Cloud Monitoring data source in Grafana allows you to query and visualize this data alongside metrics from other systems, creating unified dashboards for comprehensive infrastructure and application monitoring. -For instructions on how to add a data source to Grafana, refer to the [administration documentation](ref:data-source-management). -Only users with the organization administrator role can add data sources. +Grafana includes built-in support for Google Cloud Monitoring, so you don't need to install a plugin. -Once you've added the Google Cloud Monitoring data source, you can [configure it](#configure-the-data-source) so that your Grafana instance's users can create queries in its [query editor](query-editor/) and apply [annotations](#annotations) when they [build dashboards](ref:build-dashboards) and use [Explore](ref:explore). +## Get started -## Before you begin +The following documents will help you get started with the Google Cloud Monitoring data source: -Complete the following before you configure the data source. +- [Configure the data source](ref:configure-gcm) - Set up authentication and connect to Google Cloud +- [Query editor](ref:query-editor) - Create and edit Metric and SLO queries +- [Template variables](ref:template-variables) - Create dynamic dashboards with Google Cloud Monitoring variables +- [Annotations](ref:annotations-gcm) - Overlay Google Cloud Monitoring events on graphs +- [Alerting](ref:alerting-gcm) - Create alert rules based on GCP metrics and SLOs +- [Google authentication](ref:google-authentication) - Configure authentication methods for GCP access +- [Troubleshooting](ref:troubleshooting) - Solve common configuration and query errors -### Configure Google authentication +## Supported query types -Before you can request data from Google Cloud Monitoring, you must configure authentication. -All requests to Google APIs are performed on the server-side by the Grafana backend. +The Google Cloud Monitoring data source supports the following query types: -For authentication options and configuration details, refer to [Google authentication](google-authentication/). +| Query type | Description | +| ----------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| **Metrics** | Query time series data from GCP resources using the Monitoring Query Language (MQL) or the visual builder. | +| **Service Level Objectives (SLOs)** | Query SLO data defined in Google Cloud Monitoring to track service reliability and error budgets. | -When configuring Google authentication, keep in mind the following additional steps related to Google Cloud Monitoring. +## Additional features -#### Configure a GCP Service Account +After you configure the Google Cloud Monitoring data source, you can: -When you [create a Google Cloud Platform (GCP) Service Account and key file](google-authentication/#create-a-gcp-service-account-and-key-file), the Service Account must have the **Monitoring Viewer** role (**Role > Select a role > Monitoring > Monitoring Viewer**): +- Create a wide variety of [visualizations](ref:visualizations) using GCP metrics. +- Configure and use [template variables](ref:variables) for dynamic dashboards. +- Add [transformations](ref:transformations) to manipulate query results. +- Add [annotations](ref:annotate-visualizations) to overlay events on your graphs. +- Set up [alerting](ref:alerting) based on GCP metrics. +- Use [Explore](ref:explore) to investigate your Google Cloud data without building a dashboard. -{{< figure src="/static/img/docs/v71/cloudmonitoring_service_account_choose_role.png" max-width="600px" class="docs-image--no-shadow" caption="Choose role" >}} +## Pre-configured dashboards -#### Grant the GCE Default Service Account scope - -If Grafana is running on a Google Compute Engine (GCE) virtual machine, when you [Configure a GCE Default Service Account](google-authentication/#configure-a-gce-default-service-account), you must also grant that Service Account access to the "Cloud Monitoring API" scope. - -### Enable necessary Google Cloud Platform APIs - -Before you can request data from Google Cloud Monitoring, you must first enable necessary APIs on the Google end. - -1. Open the Monitoring and Cloud Resource Manager API pages: - - [Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com) - - [Cloud Resource Manager API](https://console.cloud.google.com/apis/library/cloudresourcemanager.googleapis.com) - -1. On each page, click the `Enable` button. - - {{< figure src="/static/img/docs/v71/cloudmonitoring_enable_api.png" max-width="450px" class="docs-image--no-shadow" caption="Enable GCP APIs" >}} - -## Configure the data source - -To configure basic settings for the data source, complete the following steps: - -1. Click **Connections** in the left-side menu. -1. Under Your connections, click **Data sources**. -1. Enter `Google Cloud Monitoring` in the search bar. -1. Click **Google Cloud Monitoring**. - - The **Settings** tab of the data source is displayed. - -1. Set the data source's basic configuration options: - - | Name | Description | - | ----------- | ------------------------------------------------------------------------ | - | **Name** | Sets the name you use to refer to the data source in panels and queries. | - | **Default** | Sets whether the data source is pre-selected for new panels. | - -### Provision the data source - -You can define and configure the data source in YAML files as part of the Grafana provisioning system. -For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources). - -#### Provisioning examples - -**Using the JWT (Service Account key file) authentication type:** - -```yaml -apiVersion: 1 - -datasources: - - name: Google Cloud Monitoring - type: stackdriver - access: proxy - jsonData: - tokenUri: https://oauth2.googleapis.com/token - clientEmail: stackdriver@myproject.iam.gserviceaccount.com - authenticationType: jwt - defaultProject: my-project-name - secureJsonData: - privateKey: | - -----BEGIN PRIVATE KEY----- - POSEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCb1u1Srw8ICYHS - ... - yA+23427282348234= - -----END PRIVATE KEY----- -``` - -**Using the JWT (Service Account private key path) authentication type:** - -```yaml -apiVersion: 1 - -datasources: - - name: Google Cloud Monitoring - type: stackdriver - access: proxy - jsonData: - tokenUri: https://oauth2.googleapis.com/token - clientEmail: stackdriver@myproject.iam.gserviceaccount.com - authenticationType: jwt - defaultProject: my-project-name - privateKeyPath: /etc/secrets/gce.pem -``` - -**Using GCE Default Service Account authentication:** - -```yaml -apiVersion: 1 - -datasources: - - name: Google Cloud Monitoring - type: stackdriver - access: proxy - jsonData: - authenticationType: gce -``` - -## Import pre-configured dashboards - -The Google Cloud Monitoring data source ships with pre-configured dashboards for some of the most popular GCP services. -These curated dashboards are based on similar dashboards in the GCP dashboard samples repository. +The Google Cloud Monitoring data source includes pre-configured dashboards for popular GCP services. These curated dashboards are based on similar dashboards in the GCP dashboard samples repository. {{< figure src="/static/img/docs/google-cloud-monitoring/curated-dashboards-7-4.png" class="docs-image--no-shadow" max-width="650px" caption="Curated dashboards for Google Cloud Monitoring" >}} -**To import curated dashboards:** +To import a pre-configured dashboard: -1. Navigate to the data source's [configuration page](#configure-the-data-source). -1. Select the **Dashboards** tab. +1. Go to **Connections** > **Data sources**. +1. Select your Google Cloud Monitoring data source. +1. Click the **Dashboards** tab. +1. Click **Import** next to the dashboard you want to use. - This displays the curated selection of importable dashboards. +The dashboards include a [template variable](ref:template-variables) populated with the projects accessible by the configured [service account](ref:google-authentication) each time you load the dashboard. After Grafana loads the dashboard, you can select a project from the dropdown list. -1. Select **Import** for the dashboard to import. +To customize an imported dashboard, save it under a different name. Otherwise, Grafana upgrades can overwrite your customizations with the new version. -The dashboards include a [template variable](template-variables/) populated with the projects accessible by the configured [Service Account](google-authentication/) each time you load the dashboard. -After Grafana loads the dashboard, you can select a project from the dropdown list. +## Related resources -**To customize an imported dashboard:** - -To customize one of these dashboards, we recommend that you save it under a different name. -If you don't, upgrading Grafana can overwrite the customized dashboard with the new version. - -## Query the data source - -The Google Cloud Monitoring query editor helps you build two types of queries: **Metric** and **Service Level Objective (SLO)**. - -For details, refer to the [query editor documentation](query-editor/). - -## Use template variables - -Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. -Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. -Grafana refers to such variables as template variables. - -For details, see the [template variables documentation](template-variables/). +- [Google Cloud Monitoring documentation](https://cloud.google.com/monitoring/docs) +- [Monitoring Query Language (MQL) reference](https://cloud.google.com/monitoring/mql/reference) +- [Google Cloud metrics list](https://cloud.google.com/monitoring/api/metrics_gcp) +- [Grafana community forum](https://community.grafana.com/) diff --git a/docs/sources/datasources/google-cloud-monitoring/alerting/index.md b/docs/sources/datasources/google-cloud-monitoring/alerting/index.md new file mode 100644 index 00000000000..77ab46d1da8 --- /dev/null +++ b/docs/sources/datasources/google-cloud-monitoring/alerting/index.md @@ -0,0 +1,264 @@ +--- +aliases: + - ../../data-sources/google-cloud-monitoring/alerting/ +description: Set up alerts using Google Cloud Monitoring data in Grafana +keywords: + - grafana + - google + - cloud + - monitoring + - alerting + - alerts + - metrics + - slo + - recording rules +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Alerting +title: Google Cloud Monitoring alerting +weight: 450 +refs: + alerting: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//alerting/ + alerting-fundamentals: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//alerting/fundamentals/ + create-alert-rule: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/ + configure-gcm: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ + query-editor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + troubleshoot: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/troubleshooting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/troubleshooting/ + recording-rules: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/create-recording-rules/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/ +--- + +# Google Cloud Monitoring alerting + +The Google Cloud Monitoring data source supports [Grafana Alerting](ref:alerting), allowing you to create alert rules based on GCP metrics and Service Level Objectives (SLOs). You can monitor your Google Cloud environment and receive notifications when specific conditions are met. + +## Before you begin + +Before you create alert rules, ensure the following: + +- You have appropriate permissions to create alert rules in Grafana. +- Your Google Cloud Monitoring data source is configured and working correctly. Refer to [Configure the data source](ref:configure-gcm). +- You're familiar with [Grafana Alerting concepts](ref:alerting-fundamentals). + +## Supported query types for alerting + +The following query types support alerting: + +| Query type | Use case | Notes | +| ---------------------------------- | ------------------------------------------------------ | -------------------------------------------------- | +| **Builder** | Threshold-based alerts on GCP resource metrics | Best suited for alerting; returns time-series data | +| **MQL** | Complex metric queries using Monitoring Query Language | Use for advanced filtering and aggregations | +| **Service Level Objectives (SLO)** | Alert on SLO compliance, error budgets, or burn rate | Monitor service reliability | +| **PromQL** | Prometheus-style queries on GCP metrics | Familiar syntax for Prometheus users | + +{{< admonition type="note" >}} +Alert queries must return numeric data that Grafana can evaluate against a threshold. Queries that return only text or non-numeric data can't be used directly for alerting. +{{< /admonition >}} + +## Authentication requirements + +Alerting rules run as background processes without a user context. Both supported authentication methods work with alerting: + +| Authentication method | Supported | +| --------------------------- | --------- | +| Google JWT File | ✓ | +| GCE Default Service Account | ✓ | + +## Create an alert rule + +To create an alert rule using Google Cloud Monitoring data: + +1. Go to **Alerting** > **Alert rules**. +1. Click **New alert rule**. +1. Enter a name for your alert rule. +1. In the **Define query and alert condition** section: + - Select your Google Cloud Monitoring data source. + - Configure your query (for example, a Builder query for CPU usage or an SLO query for error budget). + - Add a **Reduce** expression if your query returns multiple series. + - Add a **Threshold** expression to define the alert condition. +1. Configure the **Set evaluation behavior**: + - Select or create a folder and evaluation group. + - Set the evaluation interval (how often the alert is checked). + - Set the pending period (how long the condition must be true before firing). +1. Add labels and annotations to provide context for notifications. +1. Click **Save rule**. + +For detailed instructions, refer to [Create a Grafana-managed alert rule](ref:create-alert-rule). + +## Example: VM CPU usage alert + +This example creates an alert that fires when Compute Engine VM CPU utilization exceeds 80%: + +1. Create a new alert rule. +1. Configure the query: + - **Query type**: Builder + - **Project**: Select your GCP project + - **Service**: Compute Engine + - **Metric**: `instance/cpu/utilization` + - **Group by function**: mean +1. Add expressions: + - **Reduce**: Last (to get the most recent data point) + - **Threshold**: Is above 0.8 (CPU utilization is returned as a decimal) +1. Set evaluation to run every 1 minute with a 5-minute pending period. +1. Save the rule. + +## Example: SLO error budget alert + +This example alerts when an SLO's error budget remaining drops below 20%: + +1. Create a new alert rule. +1. Configure the query: + - **Query type**: Service Level Objectives (SLO) + - **Project**: Select your GCP project + - **Service**: Select your SLO service + - **SLO**: Select your SLO + - **Selector**: SLO Error Budget Remaining +1. Add expressions: + - **Reduce**: Last + - **Threshold**: Is below 0.2 (20% remaining) +1. Set evaluation to run every 5 minutes. +1. Save the rule. + +## Example: Cloud SQL memory alert + +This example alerts when Cloud SQL instance memory usage exceeds 90%: + +1. Create a new alert rule. +1. Configure the query: + - **Query type**: Builder + - **Project**: Select your GCP project + - **Service**: Cloud SQL + - **Metric**: `database/memory/utilization` + - **Filter**: Add a filter for specific database instances if needed +1. Add expressions: + - **Reduce**: Last + - **Threshold**: Is above 0.9 +1. Set evaluation to run every 1 minute. +1. Save the rule. + +## Best practices + +Follow these recommendations to create reliable and efficient alerts with Google Cloud Monitoring data. + +### Use appropriate query intervals + +- Set the alert evaluation interval to be greater than or equal to the minimum data resolution from Google Cloud Monitoring. +- Most GCP metrics have 1-minute granularity at minimum. +- Avoid very short intervals (less than 1 minute) as they may cause evaluation timeouts or miss data points. + +### Reduce multiple series + +When your query returns multiple time series (for example, CPU usage across multiple VMs), use the **Reduce** expression to aggregate them: + +- **Last**: Use the most recent value +- **Mean**: Average across all series +- **Max/Min**: Use the highest or lowest value +- **Sum**: Total across all series + +### Use appropriate alignment periods + +For alerting queries, ensure the alignment period provides enough data points: + +- Use "cloud monitoring auto" or "grafana auto" for most cases. +- For more precise control, set a fixed alignment period that matches your evaluation interval. + +### Handle no data conditions + +Configure what happens when no data is returned: + +1. In the alert rule, find **Configure no data and error handling**. +1. Choose an appropriate action: + - **No Data**: Keep the alert in its current state + - **Alerting**: Treat no data as an alert condition + - **OK**: Treat no data as a healthy state + +### Test queries before alerting + +Always verify your query returns expected data before creating an alert: + +1. Go to **Explore**. +1. Select your Google Cloud Monitoring data source. +1. Run the query you plan to use for alerting. +1. Confirm the data format and values are correct. +1. Verify the query returns numeric data suitable for threshold evaluation. + +## Recording rules + +The Google Cloud Monitoring data source supports [Grafana-managed recording rules](ref:recording-rules). Recording rules periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric. + +Use recording rules to: + +- Reduce query load on Google Cloud Monitoring by pre-computing complex aggregations. +- Create derived metrics from GCP data for use in alerts and dashboards. +- Import Google Cloud Monitoring data into a Prometheus-compatible database. + +{{< admonition type="note" >}} +Grafana-managed recording rules write results to a Prometheus-compatible database (such as Grafana Mimir or the Grafana Cloud managed Prometheus). You must configure a target data source for storing the recorded metrics. +{{< /admonition >}} + +For instructions on creating recording rules, refer to [Create recording rules](ref:recording-rules). + +## Troubleshooting + +If your Google Cloud Monitoring alerts aren't working as expected, use the following sections to diagnose and resolve common issues. + +### Alerts not firing + +- Check that the query returns numeric data in Explore. +- Ensure the evaluation interval allows enough time for data to be available. +- Verify the threshold is set correctly (remember that many GCP metrics return decimals, not percentages). +- Review the alert rule's health and any error messages in the Alerting UI. + +### Authentication errors in alert evaluation + +If you see authentication errors when alerts evaluate: + +- Verify the service account has the **Monitoring Viewer** role. +- If using a JWT key file, ensure it hasn't been deleted or revoked. +- Check that the required APIs (Monitoring API, Cloud Resource Manager API) are enabled. + +### Query timeout errors + +- Increase the alignment period to reduce the number of data points. +- Reduce the time range in the query. +- Simplify complex MQL queries. +- Add filters to narrow the result set. + +For additional troubleshooting help, refer to [Troubleshoot Google Cloud Monitoring](ref:troubleshoot). + +## Additional resources + +- [Grafana Alerting documentation](ref:alerting) +- [Create alert rules](ref:create-alert-rule) +- [Create recording rules](ref:recording-rules) +- [Google Cloud Monitoring query editor](ref:query-editor) diff --git a/docs/sources/datasources/google-cloud-monitoring/annotations/index.md b/docs/sources/datasources/google-cloud-monitoring/annotations/index.md new file mode 100644 index 00000000000..5c2d06830c0 --- /dev/null +++ b/docs/sources/datasources/google-cloud-monitoring/annotations/index.md @@ -0,0 +1,92 @@ +--- +aliases: + - ../../data-sources/google-cloud-monitoring/annotations/ +description: Use annotations to overlay Google Cloud Monitoring events on Grafana graphs +keywords: + - grafana + - google + - cloud + - monitoring + - annotations + - events +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Annotations +title: Google Cloud Monitoring annotations +weight: 400 +refs: + annotate-visualizations: + - pattern: /docs/grafana/ + destination: /docs/grafana//dashboards/build-dashboards/annotate-visualizations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//dashboards/build-dashboards/annotate-visualizations/ +--- + +# Google Cloud Monitoring annotations + +[Annotations](ref:annotate-visualizations) overlay rich event information on top of graphs. You can use annotations to mark important events, deployments, or incidents on your dashboards. + +## Before you begin + +Before you configure annotations, ensure you have the following: + +- A configured Google Cloud Monitoring data source. +- A dashboard where you want to add annotations. + +## Annotation limitations + +Keep the following limitations in mind when using annotations: + +- **Performance:** Rendering annotations is expensive. Limit the number of rows returned to maintain dashboard performance. +- **Native events:** There's no support for displaying Google Cloud Monitoring's native annotations and events. However, annotations work well with [custom metrics](https://cloud.google.com/monitoring/custom-metrics/) in Google Cloud Monitoring. + +## Add an annotation query + +To add an annotation query to a dashboard: + +1. Open the dashboard where you want to add annotations. +1. Click **Dashboard settings** (gear icon). +1. Select **Annotations** in the left menu. +1. Click **Add annotation query**. +1. Select your Google Cloud Monitoring data source. +1. Configure the annotation query using the query editor. + +## Configure the annotation query + +With the query editor for annotations, you can select a metric and filters to define which data points create annotations. + +The **Title** and **Text** fields support templating and can use data returned from the query. + +For example, the Title field could have the following text: + +`{{metric.type}} has value: {{metric.value}}` + +Example result: `monitoring.googleapis.com/uptime_check/http_status has this value: 502` + +## Annotation patterns + +Use the following patterns in the **Title** and **Text** fields to display metric data in your annotations: + +| Pattern format | Description | Example | Result | +| ------------------------ | --------------------------------- | -------------------------------- | ------------------------------------------------- | +| `{{metric.value}}` | Value of the metric/point. | `{{metric.value}}` | `555` | +| `{{metric.type}}` | Returns the full Metric Type. | `{{metric.type}}` | `compute.googleapis.com/instance/cpu/utilization` | +| `{{metric.name}}` | Returns the metric name part. | `{{metric.name}}` | `instance/cpu/utilization` | +| `{{metric.service}}` | Returns the service part. | `{{metric.service}}` | `compute` | +| `{{metric.label.xxx}}` | Returns the metric label value. | `{{metric.label.instance_name}}` | `grafana-1-prod` | +| `{{resource.label.xxx}}` | Returns the resource label value. | `{{resource.label.zone}}` | `us-east1-b` | + +## Example: Annotate uptime check failures + +To create annotations for uptime check failures: + +1. Add an annotation query using the Google Cloud Monitoring data source. +1. Select the `monitoring.googleapis.com/uptime_check/check_passed` metric. +1. Add a filter for `check_passed = false`. +1. Set the **Title** to: `Uptime check failed: {{metric.label.check_id}}` +1. Set the **Text** to: `Region: {{resource.label.zone}}` + +This creates an annotation marker on your graph each time an uptime check fails. diff --git a/docs/sources/datasources/google-cloud-monitoring/configure/index.md b/docs/sources/datasources/google-cloud-monitoring/configure/index.md new file mode 100644 index 00000000000..ba4dac08461 --- /dev/null +++ b/docs/sources/datasources/google-cloud-monitoring/configure/index.md @@ -0,0 +1,363 @@ +--- +aliases: + - ../../data-sources/google-cloud-monitoring/configure/ +description: This document provides configuration instructions for the Google Cloud Monitoring data source. +keywords: + - grafana + - google + - cloud + - monitoring + - stackdriver + - configure +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Configure +title: Configure the Google Cloud Monitoring data source +weight: 100 +refs: + provisioning-data-sources: + - pattern: /docs/grafana/ + destination: /docs/grafana//administration/provisioning/#data-sources + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//administration/provisioning/#data-sources + private-data-source-connect: + - pattern: /docs/grafana/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ + configure-pdc: + - pattern: /docs/grafana/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc + google-authentication: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/ + google-authentication-jwt: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/#create-a-gcp-service-account-and-key-file + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/#create-a-gcp-service-account-and-key-file + google-authentication-gce: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/#use-gce-default-service-account + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/#use-gce-default-service-account + query-editor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + template-variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/template-variables/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/template-variables/ + annotations: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/annotations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/annotations/ + alerting: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/alerting/ + explore: + - pattern: /docs/grafana/ + destination: /docs/grafana//explore/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//explore/ + gcm-index: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/ +--- + +# Configure the Google Cloud Monitoring data source + +This document provides instructions for configuring the Google Cloud Monitoring data source in Grafana. + +## Before you begin + +Before you begin, ensure you have the following: + +- **Grafana permissions:** You must have the `Organization administrator` role to configure data sources. +- **GCP project:** A Google Cloud Platform project. +- **GCP permissions:** Permissions to create a service account or configure GCE default service account settings in your GCP project. + +Grafana includes built-in support for Google Cloud Monitoring, so you don't need to install a plugin. + +## Set up GCP authentication + +Before you can request data from Google Cloud Monitoring, you must configure authentication. +All requests to Google APIs are performed on the server-side by the Grafana backend. + +For authentication options and configuration details, refer to [Google authentication](ref:google-authentication). + +When you configure Google authentication, note the following requirements specific to Google Cloud Monitoring. + +### Configure a GCP Service Account + +When you [create a Google Cloud Platform (GCP) Service Account and key file](ref:google-authentication-jwt), the Service Account must have the **Monitoring Viewer** role (**Role > Select a role > Monitoring > Monitoring Viewer**): + +{{< figure src="/static/img/docs/v71/cloudmonitoring_service_account_choose_role.png" max-width="600px" class="docs-image--no-shadow" caption="Choose role" >}} + +### Grant the GCE Default Service Account scope + +If Grafana is running on a Google Compute Engine (GCE) virtual machine, when you [configure a GCE Default Service Account](ref:google-authentication-gce), you must also grant that Service Account access to the "Cloud Monitoring API" scope. + +## Enable Google Cloud Platform APIs + +Before you can request data from Google Cloud Monitoring, you must enable the necessary APIs in your GCP project. + +1. Open the Monitoring and Cloud Resource Manager API pages: + - [Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com) + - [Cloud Resource Manager API](https://console.cloud.google.com/apis/library/cloudresourcemanager.googleapis.com) + +1. On each page, click **Enable**. + + {{< figure src="/static/img/docs/v71/cloudmonitoring_enable_api.png" max-width="450px" class="docs-image--no-shadow" caption="Enable GCP APIs" >}} + +## Add the data source + +To add the Google Cloud Monitoring data source: + +1. Click **Connections** in the left-side menu. +1. Click **Add new connection**. +1. Enter `Google Cloud Monitoring` in the search bar. +1. Select **Google Cloud Monitoring**. +1. Click **Add new data source** in the upper right. + +You're taken to the **Settings** tab where you configure the data source. + +## Configure the data source in the UI + +The following are configuration options for the Google Cloud Monitoring data source. + +| Setting | Description | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Name** | Sets the name you use to refer to the data source in panels and queries. | +| **Default** | Sets whether the data source is pre-selected for new panels. | +| **Universe Domain** | The universe domain to connect to. For more information, refer to the [Google Cloud universe domains documentation](https://cloud.google.com/docs/overview#universe_domains). Defaults to `googleapis.com`. | + +### Authentication + +Configure how Grafana authenticates with Google Cloud. + +| Setting | Description | +| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Authentication type** | Select the authentication method. Choose **Google JWT File** to use a service account key file, or **GCE Default Service Account** if Grafana is running on a GCE virtual machine. | + +### JWT Key Details + +These settings appear when you select **Google JWT File** as the authentication type. + +| Setting | Description | +| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **JWT token** | Upload or paste your Google JWT token. You can drag and drop a `.json` key file, click **Click to browse files** to upload, or use **Paste JWT Token** or **Fill In JWT Token manually**. | + +### Service account impersonation + +Use service account impersonation to have Grafana authenticate as a different service account than the one provided in the JWT token. + +| Setting | Description | +| ---------------------------------- | --------------------------------------------------------------------------------------------------- | +| **Enable** | Toggle to enable service account impersonation. | +| **Service account to impersonate** | Enter the email address of the service account to impersonate when making requests to Google Cloud. | + +### Private data source connect + +_Only available for Grafana Cloud._ + +Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud. For more information on how PDC works, refer to [Private data source connect](ref:private-data-source-connect). For steps on setting up a PDC connection, refer to [Configure Grafana private data source connect (PDC)](ref:configure-pdc). + +| Setting | Description | +| ------------------------------- | --------------------------------------------------------------------------- | +| **Private data source connect** | Select a PDC connection from the drop-down menu or create a new connection. | + +### Save and test + +Click **Save & test** to test the connection. A successful connection displays the following message: + +`Successfully queried the Google Cloud Monitoring API.` + +## Provision the data source + +You can define and configure the data source in YAML files as part of the Grafana provisioning system. +For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources). + +### Provisioning examples + +**Using the JWT (Service Account key file) authentication type:** + +```yaml +apiVersion: 1 + +datasources: + - name: Google Cloud Monitoring + type: stackdriver + access: proxy + jsonData: + tokenUri: https://oauth2.googleapis.com/token + clientEmail: stackdriver@myproject.iam.gserviceaccount.com + authenticationType: jwt + defaultProject: my-project-name + universeDomain: googleapis.com + secureJsonData: + privateKey: | + -----BEGIN PRIVATE KEY----- + POSEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCb1u1Srw8ICYHS + ... + yA+23427282348234= + -----END PRIVATE KEY----- +``` + +**Using the JWT (Service Account private key path) authentication type:** + +```yaml +apiVersion: 1 + +datasources: + - name: Google Cloud Monitoring + type: stackdriver + access: proxy + jsonData: + tokenUri: https://oauth2.googleapis.com/token + clientEmail: stackdriver@myproject.iam.gserviceaccount.com + authenticationType: jwt + defaultProject: my-project-name + universeDomain: googleapis.com + privateKeyPath: /etc/secrets/gce.pem +``` + +**Using GCE Default Service Account authentication:** + +```yaml +apiVersion: 1 + +datasources: + - name: Google Cloud Monitoring + type: stackdriver + access: proxy + jsonData: + authenticationType: gce + universeDomain: googleapis.com +``` + +## Provision the data source using Terraform + +You can provision the Google Cloud Monitoring data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). + +For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. + +### Terraform prerequisites + +Before you begin, ensure you have the following: + +- [Terraform](https://www.terraform.io/downloads) installed. +- Grafana Terraform provider configured with appropriate credentials. +- For Grafana Cloud: A [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) with data source permissions. + +### Provider configuration + +Configure the Grafana provider to connect to your Grafana instance: + +```hcl +terraform { + required_providers { + grafana = { + source = "grafana/grafana" + version = ">= 2.0.0" + } + } +} + +# For Grafana Cloud +provider "grafana" { + url = "" + auth = "" +} + +# For self-hosted Grafana +# provider "grafana" { +# url = "http://localhost:3000" +# auth = "" +# } +``` + +### Terraform examples + +The following examples show how to configure the Google Cloud Monitoring data source for each authentication method. + +**Using the JWT (Service Account key file) authentication type:** + +```hcl +resource "grafana_data_source" "google_cloud_monitoring" { + type = "stackdriver" + name = "Google Cloud Monitoring" + + json_data_encoded = jsonencode({ + tokenUri = "https://oauth2.googleapis.com/token" + clientEmail = "" + authenticationType = "jwt" + defaultProject = "" + universeDomain = "googleapis.com" + }) + + secure_json_data_encoded = jsonencode({ + privateKey = "" + }) +} +``` + +**Using the JWT (Service Account private key path) authentication type:** + +```hcl +resource "grafana_data_source" "google_cloud_monitoring" { + type = "stackdriver" + name = "Google Cloud Monitoring" + + json_data_encoded = jsonencode({ + tokenUri = "https://oauth2.googleapis.com/token" + clientEmail = "" + authenticationType = "jwt" + defaultProject = "" + universeDomain = "googleapis.com" + privateKeyPath = "/etc/secrets/gce.pem" + }) +} +``` + +**Using GCE Default Service Account authentication:** + +```hcl +resource "grafana_data_source" "google_cloud_monitoring" { + type = "stackdriver" + name = "Google Cloud Monitoring" + + json_data_encoded = jsonencode({ + authenticationType = "gce" + universeDomain = "googleapis.com" + }) +} +``` + +For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). + +## Next steps + +After you configure the Google Cloud Monitoring data source, you can: + +- [Query GCP metrics](ref:query-editor) using the visual Builder, MQL, SLO, or PromQL query types. +- [Create template variables](ref:template-variables) for dynamic, reusable dashboards. +- [Add annotations](ref:annotations) to overlay GCP events on your graphs. +- [Set up alerting](ref:alerting) to receive notifications based on GCP metrics and SLOs. +- [Explore your data](ref:explore) to investigate metrics without building a dashboard. +- [Import pre-configured dashboards](ref:gcm-index) for popular GCP services. diff --git a/docs/sources/datasources/google-cloud-monitoring/google-authentication/index.md b/docs/sources/datasources/google-cloud-monitoring/google-authentication/index.md index ba2472d3893..5a03fa75dee 100644 --- a/docs/sources/datasources/google-cloud-monitoring/google-authentication/index.md +++ b/docs/sources/datasources/google-cloud-monitoring/google-authentication/index.md @@ -1,75 +1,157 @@ --- aliases: - - /docs/grafana/next/datasources/cloudmonitoring/ -description: Google authentication + - ../../data-sources/google-cloud-monitoring/google-authentication/ +description: Configure authentication methods to connect Grafana to Google Cloud Monitoring keywords: - grafana - google + - cloud + - monitoring - authentication + - service account + - jwt + - gce labels: products: - cloud - enterprise - oss -title: Authentication -weight: 5 +menuTitle: Google authentication +title: Google authentication +weight: 200 +refs: + configure-gcm: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ --- -# Configure Google authentication +# Google authentication -Requests from a Grafana plugin to Google are made on behalf of an Identity and Access Management (IAM) role or IAM user. -The IAM user or IAM role must have the associated policies to perform certain API actions. -Since these policies are specific to each data source, refer to the data source documentation for details. +This document explains how to configure authentication between Grafana and Google Cloud Platform (GCP). You must configure authentication before you can use the Google Cloud Monitoring data source to query metrics and SLOs. All requests to Google APIs are performed on the server-side by the Grafana backend. -You can authenticate a Grafana plugin to Google by uploading a Google JSON Web Token (JWT) file, or by automatically retrieving credentials from the Google metadata server. -The latter option is available only when running Grafana on a GCE virtual machine. -## Use a Google Service Account key file +## Before you begin -To authenticate the Grafana plugin with the Google API, create a Google Cloud Platform (GCP) Service Account for the Project you want to show data. +Before you configure authentication, ensure you have the following: -Each Grafana data source integrates with one GCP Project. -To visualize data from multiple GCP Projects, create one data source per GCP Project. +- A Google Cloud Platform project with the Monitoring API enabled. +- Permissions to create service accounts or configure GCE instance settings in your GCP project. +- Access to the Grafana data source configuration page. -### Create a GCP Service Account and key file +## Supported authentication methods -1. Navigate to the [APIs and Services Credentials page](https://console.cloud.google.com/apis/credentials). -1. Click on the **Create credentials** dropdown and select the **Service account** option. +The Google Cloud Monitoring data source supports the following authentication methods: + +| Method | Use case | +| --------------------------------- | --------------------------------------------------------------------------------------------------- | +| **Google JWT File** | Use when Grafana runs outside of GCP, or when you need explicit control over credentials. | +| **GCE Default Service Account** | Use when Grafana runs on a Google Compute Engine VM with a configured service account. | +| **Service Account Impersonation** | Use when you need Grafana to act as a different service account than the one it authenticates with. | + +## Use a Google service account key file + +Use this method when Grafana runs outside of Google Cloud Platform, or when you need explicit control over which credentials are used. + +Each Grafana data source connects to one GCP project by default. To visualize data from multiple GCP projects, create one data source per project, or use service account impersonation. + +### Create a GCP service account and key file + +To create a service account and download its key file: + +1. Navigate to the [APIs and Services Credentials page](https://console.cloud.google.com/apis/credentials) in the GCP Console. +1. Click the **Create credentials** dropdown and select **Service account**. 1. In **Service account name**, enter a name for the account. -1. From the **Role** dropdown, choose the roles required by the specific plugin. -1. Click **Done**. -1. Use the newly created account to [create a service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console). - A JSON key file is created and downloaded to your computer. -1. Store the key file in a secure place, because it grants access to your Google data. -1. In the Grafana data source configuration page, upload the key file. - The file's contents are encrypted and saved in the Grafana database. - Remember to save the file after uploading. +1. Click **Create and continue**. +1. In the **Grant this service account access to project** section, select the **Monitoring Viewer** role from the **Role** dropdown. +1. Click **Continue**, then click **Done**. +1. In the service accounts list, click the service account you created. +1. Go to the **Keys** tab and click **Add key** > **Create new key**. +1. Select **JSON** and click **Create**. -#### Create a GCP service account for multiple projects + A JSON key file downloads to your computer. -You can create a service account and key file that can be used to access multiple projects. Follow steps 1-5 above, then: +1. Store the key file securely. It grants access to your Google Cloud data. -1. Note the email address of the service account, it will look a little strange like `foobar-478@main-boardwalk-90210.iam.gserviceaccount.com`. -1. Navigate to the other project(s) you want to access. -1. Add the service account email address to the IAM page of each project, and grant it the required roles. -1. Navigate back to the original project's service account and create a [service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console). A JSON key file is created and downloaded to your computer -1. Store the key file in a secure place, because it grants access to your Google data. -1. In the Grafana data source configuration page, upload the key file. - The file's contents are encrypted and saved in the Grafana database. - Remember to save the file after uploading. +### Upload the key file to Grafana -## Configure a GCE default service account +1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. +1. Under **Authentication type**, select **Google JWT File**. +1. Upload the JSON key file using one of the available methods (drag and drop, browse, or paste). +1. Click **Save & test** to verify the connection. -When Grafana is running on a Google Compute Engine (GCE) virtual machine, Grafana can automatically retrieve default credentials from the metadata server. As a result, there is no need to generate a private key file for the service account. You also do not need to upload the file to Grafana. The following preconditions must be met before Grafana can retrieve default credentials. +### Grant access to multiple projects -- You must create a Service Account for use by the GCE virtual machine. For more information, refer to [Create new service account](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#createanewserviceaccount). -- Verify that the GCE virtual machine instance is running as the service account that you created. For more information, refer to [setting up an instance to run as a service account](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#using). -- Allow access to the specified API scope. +You can configure a single service account to access multiple GCP projects: -For more information about creating and enabling service accounts for GCE instances, refer to [enabling service accounts for instances in Google documentation](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances). +1. Create a service account and key file following the steps above. +1. Note the service account email address (for example, `grafana-monitoring@my-project.iam.gserviceaccount.com`). +1. In each additional project you want to access: + 1. Navigate to **IAM & Admin** > **IAM**. + 1. Click **Grant access**. + 1. Enter the service account email address. + 1. Assign the **Monitoring Viewer** role. + 1. Click **Save**. +1. Use the original key file in Grafana. The service account can now access all configured projects. -### Service account impersonation +## Use GCE Default Service Account -You can also configure the plugin to use [service account impersonation](https://cloud.google.com/iam/docs/service-account-impersonation). -You need to ensure the service account used by this plugin has the `iam.serviceAccounts.getAccessToken` permission. This permission is in roles like the [Service Account Token Creator role](https://cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountTokenCreator) (roles/iam.serviceAccountTokenCreator). Also, the service account impersonated by this plugin needs [Monitoring Viewer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.viewer). +When Grafana runs on a Google Compute Engine (GCE) virtual machine, it can automatically retrieve credentials from the GCE metadata server. This method doesn't require you to create or manage key files. + +### Prerequisites for GCE authentication + +Before using this method, ensure the following: + +- Grafana is running on a GCE virtual machine. +- The VM has a service account assigned with the **Monitoring Viewer** role. +- The VM has the **Cloud Monitoring API** scope enabled. + +### Configure the GCE instance + +1. In the GCP Console, navigate to **Compute Engine** > **VM instances**. +1. Stop the VM if it's running (you can't change scopes on a running VM). +1. Click the VM name, then click **Edit**. +1. Under **Service account**, select a service account with the **Monitoring Viewer** role. +1. Under **Access scopes**, select **Set access for each API** and enable the **Cloud Monitoring API** (read-only). +1. Click **Save** and restart the VM. + +### Configure the data source + +1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. +1. Under **Authentication type**, select **GCE Default Service Account**. +1. Click **Save & test** to verify the connection. + +For more information about GCE service accounts, refer to the [Google documentation on service accounts for instances](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances). + +## Configure service account impersonation + +Service account impersonation allows Grafana to authenticate as one service account but act as a different service account when making API requests. This is useful for: + +- Accessing resources across multiple projects with a single configuration. +- Following the principle of least privilege by separating authentication from authorization. +- Auditing and tracking which service account accessed specific resources. + +### Prerequisites for impersonation + +The service account used by Grafana (the "caller") must have the following: + +- The `iam.serviceAccounts.getAccessToken` permission on the target service account. +- This permission is included in the **Service Account Token Creator** role (`roles/iam.serviceAccountTokenCreator`). + +The service account being impersonated (the "target") must have: + +- The **Monitoring Viewer** role on the projects you want to access. + +### Configure impersonation + +1. In the GCP Console, grant the caller service account the **Service Account Token Creator** role on the target service account. +1. Grant the target service account the **Monitoring Viewer** role on the relevant projects. +1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. +1. Configure authentication using either **Google JWT File** or **GCE Default Service Account**. +1. Enable **Service Account Impersonation**. +1. Enter the email address of the target service account. +1. Click **Save & test** to verify the connection. + +For more information, refer to the [Google documentation on service account impersonation](https://cloud.google.com/iam/docs/service-account-impersonation). diff --git a/docs/sources/datasources/google-cloud-monitoring/query-editor/index.md b/docs/sources/datasources/google-cloud-monitoring/query-editor/index.md index 5ce9a98c152..65d160a3168 100644 --- a/docs/sources/datasources/google-cloud-monitoring/query-editor/index.md +++ b/docs/sources/datasources/google-cloud-monitoring/query-editor/index.md @@ -39,17 +39,16 @@ refs: This topic explains querying specific to the Google Cloud Monitoring data source. For general documentation on querying data sources in Grafana, see [Query and transform data](ref:query-transform-data). -## Choose a query editing mode +## Query types -The Google Cloud Monitoring query editor helps you build queries for two types of data, which both return time series data: +The Google Cloud Monitoring query editor supports the following query types: -- [Metrics](#query-metrics) - - You can also create [Monitoring Query Language (MQL)](#use-the-monitoring-query-language) queries. - -- [Service Level Objectives (SLO)](#query-service-level-objectives) - -You also use the query editor when you [annotate](#apply-annotations) visualizations. +| Query type | Description | +| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | +| [**Builder**](#query-metrics) | Build metrics queries visually by selecting a service, metric, filters, and aggregation options. | +| [**MQL**](#use-the-monitoring-query-language) | Write queries using the Monitoring Query Language for advanced use cases. | +| [**Service Level Objectives (SLO)**](#query-service-level-objectives) | Query SLO data to track service reliability and error budgets. | +| [**PromQL**](#query-with-promql) | Write Prometheus-style queries against Google Cloud Monitoring metrics. | ## Query metrics @@ -59,7 +58,7 @@ The metrics query editor helps you select metrics, group and aggregate by labels ### Create a metrics query -1. Select the **Metrics** option in the **Query Type** dropdown. +1. Select **Builder** in the **Query type** dropdown. 1. Select a project from the **Project** dropdown. 1. Select a Google Cloud Platform service from the **Service** dropdown. 1. Select a metric from the **Metric** dropdown. @@ -234,9 +233,21 @@ To understand basic MQL concepts, refer to [Introduction to Monitoring Query Lan **To create an MQL query:** -1. Select the **Metrics** option in the **Query Type** dropdown. +1. Select **MQL** in the **Query type** dropdown. 1. Select a project from the **Project** dropdown. 1. Enter your MQL query in the text area. +1. _(Optional)_ Configure the **Graph period** setting. + +Press `Shift+Enter` to run the query. + +#### Configure MQL options + +The following options are available for MQL queries: + +| Setting | Description | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Alias by** | Control the format of legend keys. Refer to [Set alias patterns for MQL queries](#set-alias-patterns-for-mql-queries) for available patterns. | +| **Graph period** | Enable the toggle to override the default time period. Select a period from the dropdown to control the granularity of the returned time series data. | ### Set alias patterns for MQL queries @@ -255,7 +266,7 @@ To understand basic concepts in service monitoring, refer to the [Google Cloud M **To create an SLO query:** -1. Select the **Service Level Objectives (SLO)** option in the **Query Type** dropdown. +1. Select **Service Level Objectives (SLO)** in the **Query type** dropdown. 1. Select a project from the **Project** dropdown. 1. Select an [SLO service](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services) from the **Service** dropdown. 1. Select an [SLO](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services.serviceLevelObjectives) from the **SLO** dropdown. @@ -285,41 +296,46 @@ The **Alias By** field helps you control the format of legend keys for SLO queri SLO queries use the same alignment period functionality as [metric queries](#define-the-alignment-period). -### Create a Prometheus query +## Query with PromQL -**To create an Prometheus query:** +The PromQL query type allows you to query Google Cloud Monitoring metrics using Prometheus Query Language (PromQL) syntax. This is useful if you're familiar with PromQL from Prometheus or Grafana Mimir and want to use the same query syntax with Google Cloud Monitoring data. -1. Select the **PromQL** option in the **Query Type** dropdown. +For more information about PromQL support in Google Cloud Monitoring, refer to the [Google Cloud documentation on PromQL](https://cloud.google.com/monitoring/promql). + +### Create a PromQL query + +To create a PromQL query: + +1. Select **PromQL** in the **Query type** dropdown. 1. Select a project from the **Project** dropdown. -1. Enter your Prometheus query in the text area. -1. Enter a Min Step interval. The **Min step** setting defines the lower bounds on the interval between data points. For example, set this to `1h` to hint that measurements are taken hourly. This setting supports the `$__interval` and `$__rate_interval` macros. +1. Enter your PromQL query in the text area. -## Apply annotations +### Configure PromQL options -{{< figure src="/static/img/docs/google-cloud-monitoring/annotations-8-0.png" max-width= "400px" class="docs-image--right" >}} +The following options are available for PromQL queries: -[Annotations](ref:annotate-visualizations) overlay rich event information on top of graphs. -You can add annotation queries in the Dashboard menu's Annotations view. +| Setting | Description | +| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Min step** | Defines the lower bounds on the interval between data points. For example, set this to `1h` to hint that measurements are taken hourly. Supports the `$__interval` and `$__rate_interval` macros. | -Rendering annotations is expensive, and it's important to limit the number of rows returned. -There's no support for displaying Google Cloud Monitoring's annotations and events, but it works well with [custom metrics](https://cloud.google.com/monitoring/custom-metrics/) in Google Cloud Monitoring. +### PromQL query examples -With the query editor for annotations, you can select a metric and filters. -The `Title` and `Text` fields support templating and can use data returned from the query. +The following examples show common PromQL query patterns for Google Cloud Monitoring: -For example, the Title field could have the following text: +**Query CPU utilization for Compute Engine instances:** -`{{metric.type}} has value: {{metric.value}}` +```promql +compute_googleapis_com:instance_cpu_utilization +``` -Example result: `monitoring.googleapis.com/uptime_check/http_status has this value: 502` +**Filter by label:** -### Patterns for the annotation query editor +```promql +compute_googleapis_com:instance_cpu_utilization{instance_name="my-instance"} +``` -| Alias pattern format | Description | Alias pattern example | Example result | -| ------------------------ | --------------------------------- | -------------------------------- | ------------------------------------------------- | -| `{{metric.value}}` | Value of the metric/point. | `{{metric.value}}` | `555` | -| `{{metric.type}}` | Returns the full Metric Type. | `{{metric.type}}` | `compute.googleapis.com/instance/cpu/utilization` | -| `{{metric.name}}` | Returns the metric name part. | `{{metric.name}}` | `instance/cpu/utilization` | -| `{{metric.service}}` | Returns the service part. | `{{metric.service}}` | `compute` | -| `{{metric.label.xxx}}` | Returns the metric label value. | `{{metric.label.instance_name}}` | `grafana-1-prod` | -| `{{resource.label.xxx}}` | Returns the resource label value. | `{{resource.label.zone}}` | `us-east1-b` | +**Calculate the rate of a counter metric:** + +```promql +rate(logging_googleapis_com:log_entry_count[5m]) +``` diff --git a/docs/sources/datasources/google-cloud-monitoring/template-variables/index.md b/docs/sources/datasources/google-cloud-monitoring/template-variables/index.md index 6551b08da04..64745c825cf 100644 --- a/docs/sources/datasources/google-cloud-monitoring/template-variables/index.md +++ b/docs/sources/datasources/google-cloud-monitoring/template-variables/index.md @@ -48,7 +48,7 @@ For an introduction to templating and template variables, refer to the [Templati ## Use query variables Variables of the type _Query_ help you query Google Cloud Monitoring for various types of data. -The Google Cloud Monitoring data source plugin provides the following **Query Types**: +The Google Cloud Monitoring data source provides the following **Query Types**: | Name | List returned | | ---------------------------------- | --------------------------------------------------------------------- | diff --git a/docs/sources/datasources/google-cloud-monitoring/troubleshooting/index.md b/docs/sources/datasources/google-cloud-monitoring/troubleshooting/index.md new file mode 100644 index 00000000000..0313365e0d7 --- /dev/null +++ b/docs/sources/datasources/google-cloud-monitoring/troubleshooting/index.md @@ -0,0 +1,338 @@ +--- +aliases: + - ../../data-sources/google-cloud-monitoring/troubleshooting/ +description: Troubleshooting guide for the Google Cloud Monitoring data source in Grafana +keywords: + - grafana + - google + - cloud + - monitoring + - stackdriver + - troubleshooting + - errors + - authentication + - query +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Troubleshooting +title: Troubleshoot Google Cloud Monitoring data source issues +weight: 500 +refs: + configure-gcm: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/configure/ + google-authentication: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/google-authentication/ + template-variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/template-variables/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/template-variables/ + query-editor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/google-cloud-monitoring/query-editor/ + private-data-source-connect: + - pattern: /docs/grafana/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ +--- + +# Troubleshoot Google Cloud Monitoring data source issues + +This document provides solutions to common issues you may encounter when configuring or using the Google Cloud Monitoring data source. For configuration instructions, refer to [Configure Google Cloud Monitoring](ref:configure-gcm). + +## Authentication errors + +These errors occur when GCP credentials are invalid, missing, or don't have the required permissions. + +### "Permission denied" or "Access denied" + +**Symptoms:** + +- Save & test fails with permission errors +- Queries return authorization errors +- Projects, metrics, or labels don't load + +**Possible causes and solutions:** + +| Cause | Solution | +| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Service account missing required permissions | Assign the **Monitoring Viewer** role to the service account in the GCP Console under **IAM & Admin** > **IAM**. Refer to [Configure Google Cloud Monitoring](ref:configure-gcm) for details. | +| Incorrect service account key file | Verify the JSON key file was downloaded correctly and contains valid credentials. Generate a new key if necessary. | +| Service account key has been deleted | Check the service account in GCP Console under **IAM & Admin** > **Service Accounts**. If the key was deleted, create a new one. | +| Wrong project selected | Verify the default project in the data source configuration matches a project the service account has access to. | +| APIs not enabled | Enable the Monitoring API and Cloud Resource Manager API in the GCP Console. Refer to [Configure Google Cloud Monitoring](ref:configure-gcm) for links. | + +### "Invalid JWT" or "JWT token error" + +**Symptoms:** + +- Authentication fails when using Google JWT File +- Error message references invalid or malformed JWT + +**Solutions:** + +1. Verify you uploaded the complete JSON key file, not just the private key portion. +1. Check that the JSON file is properly formatted and not corrupted. +1. Ensure the key file contains all required fields: `type`, `project_id`, `private_key_id`, `private_key`, `client_email`, `client_id`, `auth_uri`, `token_uri`. +1. Generate a new service account key and re-upload it. + +### GCE Default Service Account not working + +**Symptoms:** + +- Data source test fails when using GCE Default Service Account +- Works with JWT but fails with GCE authentication + +**Solutions:** + +1. Verify Grafana is running on a Google Compute Engine (GCE) virtual machine. +1. Check that the GCE instance has the **Cloud Monitoring API** scope enabled. +1. Verify the GCE default service account has the **Monitoring Viewer** role. +1. If the VM was created without the required scope, you may need to stop the instance, edit it to add the scope, and restart. + +### Service account impersonation errors + +**Symptoms:** + +- Authentication fails when service account impersonation is enabled +- Error: "Unable to impersonate service account" + +**Solutions:** + +1. Verify the primary service account has the `roles/iam.serviceAccountTokenCreator` role on the target service account. +1. Check that the target service account email is entered correctly. +1. Ensure the target service account has the **Monitoring Viewer** role. +1. Verify both service accounts are in projects that have the required APIs enabled. + +## Connection errors + +These errors occur when Grafana cannot reach Google Cloud Monitoring endpoints. + +### "Request timed out" or connection failures + +**Symptoms:** + +- Data source test times out +- Queries fail with timeout errors +- Intermittent connection issues + +**Solutions:** + +1. Verify network connectivity from the Grafana server to Google Cloud endpoints (`monitoring.googleapis.com`). +1. Check firewall rules allow outbound HTTPS (port 443) to Google Cloud services. +1. For Grafana Cloud connecting to private resources, configure [Private data source connect](ref:private-data-source-connect). +1. Check if a corporate proxy is blocking connections to Google Cloud. + +### "SSL certificate problem" + +**Symptoms:** + +- SSL/TLS handshake errors +- Certificate verification failures + +**Solutions:** + +1. Ensure the system time is correct on the Grafana server. +1. Verify the Grafana server has up-to-date CA certificates installed. +1. Check if a corporate proxy is intercepting HTTPS traffic. + +## Metrics query errors + +These errors occur when querying Google Cloud Monitoring metrics. + +### "No data" or empty results + +**Symptoms:** + +- Query executes without error but returns no data +- Charts show "No data" message + +**Possible causes and solutions:** + +| Cause | Solution | +| ------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| Time range doesn't contain data | Expand the dashboard time range. GCP metrics have different retention periods based on the metric type. | +| Wrong project selected | Verify you've selected the correct project in the query editor. | +| Incorrect metric type | Verify the service, metric type, and metric are correct. Check available metrics in the GCP Console Metrics Explorer. | +| Missing labels/filters | Some metrics require specific label filters to return data. Try removing filters to see if data appears. | +| Resource not emitting metrics | Verify the resource exists and is actively emitting metrics. Some metrics only populate under certain conditions. | + +### Metrics don't appear in drop-down + +**Symptoms:** + +- Expected metrics don't appear in the query editor +- Metric drop-down is empty for a service + +**Solutions:** + +1. Verify the metric exists in the selected project and region. +1. Check that the service account has the **Monitoring Viewer** role. +1. Some metrics are only available for specific resource types. Check the [Google Cloud metrics list](https://cloud.google.com/monitoring/api/metrics_gcp). +1. Use the Query Inspector to verify the API request and response. + +### Label values not loading + +**Symptoms:** + +- Label value drop-down doesn't populate +- Filters can't be applied + +**Solutions:** + +1. Verify the service account has the **Monitoring Viewer** role. +1. Ensure a project, service, and metric are selected before label values can load. +1. Label values are populated from existing metric data. If no metrics match the current selection, no values appear. + +### MQL query errors + +**Symptoms:** + +- Monitoring Query Language (MQL) queries fail with syntax errors +- MQL query returns unexpected results + +**Solutions:** + +1. Validate your MQL syntax using the [MQL reference documentation](https://cloud.google.com/monitoring/mql/reference). +1. Check for typos in metric types, label names, or function names. +1. Ensure time range syntax is valid in your query. +1. Test the query in the GCP Console Metrics Explorer before using it in Grafana. + +### "Too many data points" or API throttling + +**Symptoms:** + +- Queries fail with quota errors +- Performance degrades with multiple panels + +**Solutions:** + +1. Increase the alignment period to reduce the number of data points. +1. Reduce the time range of your queries. +1. Use fewer metric queries per panel. +1. Request a quota increase in the GCP Console under **APIs & Services** > **Quotas**. +1. Enable query caching in Grafana to reduce API calls. + +## SLO query errors + +These errors are specific to Service Level Objective (SLO) queries. + +### SLO services don't appear + +**Symptoms:** + +- SLO service selector is empty +- Can't find expected SLO services + +**Solutions:** + +1. Verify SLOs are defined in Google Cloud Monitoring for the selected project. +1. Check that the service account has access to view SLOs. +1. Ensure the project has services configured in the Service Monitoring section of the GCP Console. + +### SLO query returns no data + +**Symptoms:** + +- SLO query executes but returns no data +- SLO values show as empty + +**Solutions:** + +1. Verify the SLO exists and is active in the GCP Console. +1. Check that the time range includes periods when the SLO had data. +1. Ensure the selected SLO selector (SLI value, compliance, error budget, etc.) is appropriate for the SLO type. +1. Some SLOs may not have data if the underlying service hasn't received traffic. + +## Template variable errors + +These errors occur when using template variables with the Google Cloud Monitoring data source. + +### Variables return no values + +**Symptoms:** + +- Variable drop-down is empty +- Dashboard fails to load with variable errors + +**Solutions:** + +1. Verify the data source connection is working. +1. Check that the service account has permissions to list the requested resources. +1. For dependent variables, ensure parent variables have valid selections. +1. Verify the project is selected correctly in the variable query. + +### Variables are slow to load + +**Symptoms:** + +- Dashboard takes a long time to load +- Variable selectors are slow to populate + +**Solutions:** + +1. Set variable refresh to **On dashboard load** instead of **On time range change**. +1. Reduce the scope of variable queries (filter by specific project or service). +1. Limit the number of dependent variables in a chain. + +For more information on template variables, refer to the [template variables documentation](ref:template-variables). + +## Pre-configured dashboard issues + +These issues occur with the bundled pre-configured dashboards. + +### Imported dashboards show no data + +**Symptoms:** + +- Imported dashboards show empty panels +- Template variables don't load + +**Solutions:** + +1. Verify the data source name in the dashboard matches your Google Cloud Monitoring data source. +1. Check that the service account has access to the projects shown in the project variable. +1. Ensure the resources (Compute Engine instances, Cloud SQL, etc.) exist and are emitting metrics. +1. Verify the required GCP services are enabled in your project. + +## Enable debug logging + +To capture detailed error information for troubleshooting: + +1. Set the Grafana log level to `debug` in the configuration file: + + ```ini + [log] + level = debug + ``` + +1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). +1. Look for Google Cloud Monitoring-specific entries that include request and response details. +1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. + +## Get additional help + +If you've tried the solutions above and still encounter issues: + +1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. +1. Review [Google Cloud Monitoring issues on GitHub](https://github.com/grafana/grafana/issues) for known bugs. +1. Consult the [Google Cloud Monitoring documentation](https://cloud.google.com/monitoring/docs) for service-specific guidance. +1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user. +1. When reporting issues, include: + - Grafana version + - GCP project (redact if sensitive) + - Error messages (redact sensitive information) + - Steps to reproduce + - Query configuration (redact credentials)