LogoDark mode logo
LogoDark mode logo
Contact SupportLoading Light/Dark Toggle

Data sources

data sourcesgoogle cloud platform gcp plugin

Google Cloud Platform (GCP) plugin

For more information about what this plugin does and the data streams it retrieves, see:

Google Cloud Platform

Monitor your GCP environment, including GKE, Hosts and more.

To add a data source click on the + next to Data Sources on the left-hand menu in SquaredUp. Search for the data source and click on it to open the Configure data source page.

Before you start

In the GCP console select your project and check that the following are enabled:

  • VMware Engine API, see GCP - Enabling the VMware Engine
  • Google Kubernetes Engine (GKE), see GCP - Enabling the Artifact Registry and Google Kubernetes Engine APIs
  • Cloud Monitoring API, see GCP - Enabling the Monitoring API

To view cost information in SquaredUp using the Cost data stream check the following:

  • Export your billing data to Google BigQuery: GCP - Export Cloud Billing data to BigQuery
  • Enable BigQuery API, see GCP - Enable the BigQuery API

GCP Service Account Configuration

  1. Create a new Service Account, or edit an existing account.
    See GCP - Creating and managing service accounts
  2. Ensure the account has the role Viewer by adding the role Basic > Viewer.
    To view cost information, the account also needs to have BigQuery Data Viewer role.
    See GCP - Grant a single role
  3. Create a new key for the Service Account using the key type JSON. Download the JSON file as you will need to copy information from this JSON file when adding the data source next.
    See GCP - Creating service account keys

    Make sure to store the key file securely, because it can be used to authenticate as your service account.

Configuring the data source

  1. Open the JSON file that you downloaded when creating the key.
  2. Copy and paste the clientEmail from the JSON file into the data source form.
  3. Copy and paste the private_key from the JSON file into the data source form (everything between the quotes).
  4. Copy and paste the project_ids from the JSON file into the data source form.
  5. Optionally, to use the Cost data stream, copy and paste in the billingProjectId, billingDataSetName and billingTableName.
  6. (Optional) Restrict access to this data source:
    You can enable this option if you only want certain users or groups to have access to the data source, or the permission to link it to new workspaces. See data source access control for more information.

    The term data source here really means data source instance. For example, a user may configure two instances of the AWS data source, one for their development environment and one for production. In that case, each data source instance has its own access control settings.

    By default, Restrict access to this data source is set to off. The data source can be viewed, edited and administered by anyone. If you would like to control who has access to this data source, switch Restrict access to this data source to on.

    Use the Restrict access to this data source dropdown to control who has access to the workspace:

    • By default, the user setting the permissions for the data source will be given Full Control and the Everyone group will be given Link to workspace permissions.
    • Tailor access to the data source, as required, by selecting individual users or user groups from the dropdown and giving them Link to workspace or Full Control permissions.
    • If the user is not available from the dropdown, you are able to invite them to the data source by typing in their email address and then clicking Add. The new user will then receive an email inviting them to create an account on SquaredUp. Once the account has been created, they will gain access to the organization.
    • At least one user or group must be given Full Control.
    • Admin users can edit the configuration, modify the Access Control List (ACL) and delete the data source, regardless of the ACL chosen.

    Access level
    Permissions
    Link to workspace
    • User can link the data source to any workspace they have at least Editor permissions for.
    • Data from the data source can then be viewed by anyone with any access to the workspace.
    • User can share the data source data with anyone they want.
    • User cannot configure the data source in any way, or delete it.
    Full Control
    • User can change the data source configuration, ACL, and delete the data source.

    See Access control for more information.

  7. Click Test and add to validate the data source configuration. SquaredUp will now attempt to connect to SquaredUp using the provided authentication method.

    • Testing passed – a success message will be displayed and then the configuration will be saved.
    • Testing passed with warnings – warnings will be listed and potential fixes suggested. You can still use the data source with warnings. Select Save with warnings if you believe that you can still use the data source as required with the warnings listed. Alternatively, address the issues listed and then select Rerun tests to validate the data source configuration again. If the validation now passes, click Save.
    • Testing Failed – errors will be listed and potential fixes suggested. You cannot use the data source with errors. You are able to select Save with errors if you believe that a system outside of SquaredUp is causing the error that you need to fix. Alternatively, address the issues listed and then select Rerun tests to validate the data source configuration again. If the validation now passes, click Save.

    You can edit any data source configurations at any time from Settings > Data Sources.

Next steps

Data streams

You can use these data streams to create new tiles to show data, or if there are preconfigured dashboards installed you can copy or edit those.

Data streams standardize data from all the different shapes and formats your tools use into a straightforward tabular format.

While creating a tile you can tweak data streams by grouping or aggregating specific columns.

Depending on the kind of data, SquaredUp will automatically suggest how to visualize the result, for example as a table or line graph.

Data streams can be either global or scoped:

  • Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").
  • A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").

See Data Streams for more information.

The following data streams are installed with this plugin.

Data stream
Description
Builds
Builds for the selected trigger.
Cost
Cost for the selected objects
CPU Number
Number of vCPUs reserved on the host of the instance
CPU Usage/sec
Delta vCPU usage for all vCPUs, in vCPU-seconds
CPU Utilization
CPU utilization of selected host
Disk Max Read Bytes/sec
Disk's maximum per-second read throughput over a period of time
Disk Max Read Operations/sec
Disk's maximum per-second read requests count over a period of time
Disk Max Write Bytes/sec
Disk's maximum per-second write throughput over a period of time
Disk Max Write Operations/sec
Disk's maximum per-second write throughput over a period of time
Disk Status
Status for the selected disks
Disk Total Read Bytes Count
Count of bytes read from disk
Disk Total Read Operations Count
Count of disk read IO operations
Disk Total Write Bytes Count
Count of bytes written to disk
Disk Total Write Operations Count
Count of disk write IO operations
Execution times
Distribution of workflow execution times
Finished execution count
Count of finished executions for the workflow
GKE Status
GKE status for the selected resources
Host Status
Status for the selected hosts
Instance Total Uptime
Elapsed time since the VM was started, in seconds
Instance Uptime
Delta of how long the VM has been running, in seconds
Memory Total RAM
The total amount of memory in the VM
Memory Used RAM
Memory currently used in the VM
Network Total In
Count of bytes received from the network
Network Total Out
Count of bytes sent over the network
Oldest unacked message age
Age (in seconds) of the oldest unacknowledged message (a.k.a. backlog message) in a subscription
Publish message size
Distribution of publish message sizes (in bytes)
Publish requests
Cumulative count of publish requests
SQL Instance Status
Status for the selected SQL instances
Unacked messages
Number of unacknowledged messages (a.k.a. backlog messages) in a subscription.
Workflow Executions (most recent)
Most recent executions for the selected workflow

Configurable data streams

The following data streams installed with this plugin have configurable Parameters.

Data stream
Description
Parameters
BigQuery
Returns the result of a BigQuery query.
  1. Query:
    Enter a BigQuery query.
  2. Project ID:
    Specify the project that you would like to run the BigQuery against.
Firestore Documents
Firestore databases can be queried for document data by specifying a Document Path
  1. Document Path:
    Specify the Document Path to a document or a collection. The path should be relative to the database (i.e. not include the project or database name). For example mycollection or mycollection/mydoc.
Firestore Structured Query
Firestore databases can be queried for document data by specifying a Firestore query.
  1. Structured Query:
    Enter the structured query JSON (See Firestore StructuredQuery schema for detailed information). For example:
    {
        "from": {
            "collectionId": "users"
        },
        "where": {
            "fieldFilter": {
                "field": {
                    "fieldPath": "firstName"
                },
                "op": "EQUAL",
                "value": {
                    "stringValue": "Jo"
                }
            }
        }
    }
Monitor Metric
Gets data for a specified monitor metric.
  1. Metric Name:
    Select the desired metric from the dropdown. The dropdown options will automatically update depending on the object that you have scoped to. For example, scope to a Host and select CPU usage from the Metric Name dropdown.
MQL Query
The GCP data source allows you to set up configurable data streams for any kind of query for Monitoring Query Language (MQL).
You are able to query any object from your Google Cloud, even if they are not indexed in the SquaredUp graph.
See Google Cloud: Using Monitoring Query Language for more information about MQL.
  1. Project:
    Select the project that you would like to run the MQL against from the dropdown.

    You will not need to select a project if you have already selected one in the scope.

  2. MQL Query:
    Enter a MQL query. See MQL query examples below for more information about queries.
  3. Automatically apply dashboard timeframe:
    By default, Automatically apply dashboard timeframe will be selected.

    If you do not wish to use a dashboard timeframe, you can set the timeframe in the MQL query by adding within to the query. If you deselect Automatically apply dashboard timeframe and do not specify a timeframe in the MQL query, the query will fail. To add a timeframe to a query, simply type | within (timeframe) at the end of your query. For example, to add a timeframe of 24 hours you enter: | within (24h)

MQL query examples

MQL query not using dashboard timeframe - CPU usage time in the last 20 minutes in the selected project:

fetch gce_instance | metric 'compute.googleapis.com/instance/cpu/usage_time' | within (20m)

MQL query not using dashboard timeframe - authentication events count in the last 24 hours:

fetch iam_service_account
| metric 'iam.googleapis.com/service_account/authn_events_count'
| align rate(1m)
| every 1m | within (24h)

MQL query using dashboard timeframe - uploaded bytes in BigQuery data set:

fetch bigquery_dataset
| metric 'bigquery.googleapis.com/storage/uploaded_bytes'
| align rate(1m)
| every 1m

Tips for using the GCP Cost data stream

The Cost data stream is part of the GCP data source and uses the GCP BigQuery API. The GCP data source needs some additional configuration to enable the Cost data stream see How to add a Google Cloud Platform data source

  • The GCP Cost data stream shows daily data.
  • By default the Cost data stream splits costs by service.
  • We recommend scopes used with the GCP Cost data stream contain only one type of object, for example, only accounts, projects or hosts. Mixed scopes may include duplicated data in the results, because a query is sent for each type of object in the scope, and the results are combined. For example, resulting data would show the cost of a host, as well as the cost of a project, even if the host is part of that project. It is useful to create a scope for your account(s), a scope of project(s) and a scope for hosts.

Use Cases for the GCP Cost data stream

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Account(s) or Project(s)

Visualization:

Line Graph is good for showing data for dashboard timeframes of 7 or 30 days. A line will be shown for each service.

As the GCP Cost data stream shows daily data, so dashboard timeframes of less than 7 days won't show a line.

The dashboard timeframe setting lets you choose the time span that a dashboard displays data for. For example, instead of showing data from the last 12 hours, you can choose to show data from the last 7 days.

Tiles on a dashboard inherit that dashboard's timeframe by default, meaning that the tile data shown changes whenever you change the dashboard timeframe. However, you can instead choose to set a fixed timeframe for a tile via its timeframe setting.

Fixed-timeframe tiles display their set timeframe in a bubble next to their title and the tile data does not change when the dashboard timeframe is changed.

If all the tiles on a dashboard use a fixed timeframe then the timeframe dropdown is disabled. Likewise, if a specific timeframe is unsupported then it is disabled in the dropdown.

You should indicate if a tile's timeframe can be changed in the name of a tile. For example, naming a tile Performance during the last week indicated that the tile always displays data for the last week.

Setting a default timeframe

To set the default timeframe for a dashboard, click the pin

next to an option in the timeframe dropdown. This sets the initial timeframe for all viewers of the dashboard, including shared dashboards.

Cache refresh intervals

As well as determining the period for the data that displays, the dashboard timeframe also controls the cache refresh frequency of that data. In other words, the timeframe you pick also affects how "fresh" that data is.

Dashboard timeframe
Cache expiry
1 hour
1 minute
12 hours
5 minutes
24 hours
15 minutes
7 days
6 hours
30 days
12 hours
Up to 6 months
24 hours
Over 6 months
7 days

When the dashboard timeframe covers more than one day the first column of a table shows the date. To only show each service once for the selected dashboard timeframe see Use Case: How much does each GCP service cost for a specific timeframe?.

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Account(s) or Project(s) or Hosts

Shaping:

The following settings will show how much each service costs you in total for the account(s) or project(s) in scope for the whole timeframe selected:

Group by
Service (For each service...)
Aggregate column
Amount (...show me the combined amount...)
Aggregation type
Total (... in total.)

Visualization:

Table, Donut or Bar Chart. Each bar or segment will show a service.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to 7 or 30 days.

The dashboard timeframe setting lets you choose the time span that a dashboard displays data for. For example, instead of showing data from the last 12 hours, you can choose to show data from the last 7 days.

Tiles on a dashboard inherit that dashboard's timeframe by default, meaning that the tile data shown changes whenever you change the dashboard timeframe. However, you can instead choose to set a fixed timeframe for a tile via its timeframe setting.

Fixed-timeframe tiles display their set timeframe in a bubble next to their title and the tile data does not change when the dashboard timeframe is changed.

If all the tiles on a dashboard use a fixed timeframe then the timeframe dropdown is disabled. Likewise, if a specific timeframe is unsupported then it is disabled in the dropdown.

You should indicate if a tile's timeframe can be changed in the name of a tile. For example, naming a tile Performance during the last week indicated that the tile always displays data for the last week.

Setting a default timeframe

To set the default timeframe for a dashboard, click the pin

next to an option in the timeframe dropdown. This sets the initial timeframe for all viewers of the dashboard, including shared dashboards.

Cache refresh intervals

As well as determining the period for the data that displays, the dashboard timeframe also controls the cache refresh frequency of that data. In other words, the timeframe you pick also affects how "fresh" that data is.

Dashboard timeframe
Cache expiry
1 hour
1 minute
12 hours
5 minutes
24 hours
15 minutes
7 days
6 hours
30 days
12 hours
Up to 6 months
24 hours
Over 6 months
7 days

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Hosts

Shaping:

The following settings will show how much each service costs you in total for the account(s) or project(s) in scope for the whole timeframe selected:

Group by
Object (For each host...)
Aggregate column
Amount (...show me the combined amount...)
Aggregation type
Total (... in total.)

Visualization:

Table, Donut or Bar Chart. If, for example, you've scoped to Hosts, then each bar will be the cost for one host.

When you choose to 'Group by' Object this means it will group by whatever you have scoped to. In this case it will group by Host, but if you've scoped to Project(s) it will group by projects and each segment or bar would show a project.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to 7 or 30 days.

The dashboard timeframe setting lets you choose the time span that a dashboard displays data for. For example, instead of showing data from the last 12 hours, you can choose to show data from the last 7 days.

Tiles on a dashboard inherit that dashboard's timeframe by default, meaning that the tile data shown changes whenever you change the dashboard timeframe. However, you can instead choose to set a fixed timeframe for a tile via its timeframe setting.

Fixed-timeframe tiles display their set timeframe in a bubble next to their title and the tile data does not change when the dashboard timeframe is changed.

If all the tiles on a dashboard use a fixed timeframe then the timeframe dropdown is disabled. Likewise, if a specific timeframe is unsupported then it is disabled in the dropdown.

You should indicate if a tile's timeframe can be changed in the name of a tile. For example, naming a tile Performance during the last week indicated that the tile always displays data for the last week.

Setting a default timeframe

To set the default timeframe for a dashboard, click the pin

next to an option in the timeframe dropdown. This sets the initial timeframe for all viewers of the dashboard, including shared dashboards.

Cache refresh intervals

As well as determining the period for the data that displays, the dashboard timeframe also controls the cache refresh frequency of that data. In other words, the timeframe you pick also affects how "fresh" that data is.

Dashboard timeframe
Cache expiry
1 hour
1 minute
12 hours
5 minutes
24 hours
15 minutes
7 days
6 hours
30 days
12 hours
Up to 6 months
24 hours
Over 6 months
7 days

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Account(s)

Shaping:

The following settings will show how much all the service cost you in total for the accounts in scope for the whole timeframe selected:

Group by
Object (For the account(s) in scope...)
Aggregate column
Amount (...show me the combined amount...)
Aggregation type
Total (... in total.)

When you choose to 'Group by' Object this means it will group by whatever you have scoped to. In this case it will group by Account, but if you've scoped to Project(s) it will group by projects.

Visualization:

If you only have one account in the scope you can choose Scalar, as you only want to see one number (total cost for all services).

If you have several accounts in the scope you can choose Table, to see the cost per account.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to 7 or 30 days.

The dashboard timeframe setting lets you choose the time span that a dashboard displays data for. For example, instead of showing data from the last 12 hours, you can choose to show data from the last 7 days.

Tiles on a dashboard inherit that dashboard's timeframe by default, meaning that the tile data shown changes whenever you change the dashboard timeframe. However, you can instead choose to set a fixed timeframe for a tile via its timeframe setting.

Fixed-timeframe tiles display their set timeframe in a bubble next to their title and the tile data does not change when the dashboard timeframe is changed.

If all the tiles on a dashboard use a fixed timeframe then the timeframe dropdown is disabled. Likewise, if a specific timeframe is unsupported then it is disabled in the dropdown.

You should indicate if a tile's timeframe can be changed in the name of a tile. For example, naming a tile Performance during the last week indicated that the tile always displays data for the last week.

Setting a default timeframe

To set the default timeframe for a dashboard, click the pin

next to an option in the timeframe dropdown. This sets the initial timeframe for all viewers of the dashboard, including shared dashboards.

Cache refresh intervals

As well as determining the period for the data that displays, the dashboard timeframe also controls the cache refresh frequency of that data. In other words, the timeframe you pick also affects how "fresh" that data is.

Dashboard timeframe
Cache expiry
1 hour
1 minute
12 hours
5 minutes
24 hours
15 minutes
7 days
6 hours
30 days
12 hours
Up to 6 months
24 hours
Over 6 months
7 days

Was this article helpful?


Have more questions or facing an issue?
Submit a ticket

On this page

  • Before you start
  • GCP Service Account Configuration
  • Configuring the data source
  • Next steps
  • Data streams
  • Configurable data streams
  • MQL query examples
  • Tips for using the GCP Cost data stream
  • Use Cases for the GCP Cost data stream

Footer

Sites

  • SquaredUp
  • SQUAREDUP DS
  • DOWNLOAD
  • COMMUNITY ANSWERS

Quick Links

  • Contact Support
  • Events
  • Careers

Small Print

  • Privacy Policy
  • Terms and Conditions
YoutubeX (Twitter)LinkedInBlueSky

© SquaredUp 2025