Google Cloud Platform (GCP) plugin

Data streams standardize data from all the different shapes and formats your tools use into a straightforward tabular format. While creating a tile you can tweak data streams by grouping or aggregating specific columns. Depending on the kind of data, SquaredUp will automatically suggest how to visualize the result, for example as a table or line graph.

Data streams can be either global or scoped:

• Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").
• A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").

See Data Streams for more information.

The GCP data source allows you to set up data streams for any kind of query for Google BigQuery.

1. In the tile editor, filter by the GCP data source, select BigQuery from the data stream list and then click Next.
2. Query:
   Enter a BigQuery query.
3. Project ID:
   Specify the project that you would like to run the BigQuery against.

The GCP data source allows you to set up configurable data streams for any kind of query for Monitoring Query Language (MQL). Use the + MQL Query data stream to enter your own MQL queries into SquaredUp.

You are able to query any object from your Google Cloud, even if they are not indexed in the SquaredUp graph. See Google Cloud: Using Monitoring Query Language for more information about MQL.

1. In the tile editor, filter by the GCP data source, select MQL Query from the data stream list and then click Next.

   You can either select the scoped MQL Query data stream or the global MQL Query data stream.

2. If you selected the scoped MQL Query data stream, select the objects you want to use and then click Next.

   You do not need to do this if you have selected the global MQL Query data stream.

3. Project:
   Select the project that you would like to run the MQL against from the dropdown.

   You will not need to select a project if you have already selected one in the scope.

4. MQL Query:
   Enter a MQL query.

   Mustache parameters are only supported if you have scoped to a GCP object and selected the + MQL Query. Mustache parameters are not supported if you have scoped to a project or the data source instance itself.

   
   See MQL query examples below for more information about queries. Supports mustache parameters supplied in an array

   A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.

   This data stream supplies scoped objects in an array for mustache parameters. When there are multiple objects in scope this data source will send the query once with all the objects in an array.

   When the scoped objects are supplied in an array the normal mustache syntax, for example {{name}}, must be contained between {{#.}} and {{/.}} (the full-stop indicates that the whole object should be used, in this case the array of objects in scope).

   For example, a query where clause might look like:

   Copy

   | where ComputerName in (  {{#.}}    '{{name}}',    {{/.}} '' )

   • The {{#.}} and {{/.}} indicate that what is contained within is expanded for each element in the array of objects.
   • You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.
   • '{{name}}', means that the name property is expanded inside single-quotes with a trailing comma.
   • The trailing single quotes '' are necessary to stop the query being rejected because a trailing comma is disallowed.
   • Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
5. Automatically apply dashboard timeframe:
   By default, Automatically apply dashboard timeframe will be selected.

   The dashboard timeframe is the current timeframe setting for a dashboard. Users can change the dashboard timeframe to see data for a different time span, for example, instead of showing data from "the last 12 hours" it can be changed to show data from "the last 7 days".

   Tiles can be configured to:

   • Use dashboard timeframe (default). For these tiles the data shown will change when the user changes the dashboard timeframe.
   • Use a fixed timeframe from the options available. These tiles show a clock icon and hovering shows the fixed timeframe configured. The data will not change when the dashboard timeframe is changed.

   If the dashboard timeframe is unavailable, such as when all the tiles on that dashboard are using a fixed timeframe, then the button is disabled. Likewise, if a specific timeframe is unsupported then it is disabled in the timeframe picker.

   Set a default dashboard timeframe

   To set the default dashboard timeframe, click the pin icon

   when using the dashboard timeframe picker. This sets the initial timeframe for all viewers of the dashboard, including shared dashboards.

   Tip: Indicate with the name of a tile if the tile's timeframe can be changed. For example, naming a tile "Performance during the last week" tells users that this tile always shows data for the last week. Naming a tile just "Performance" indicates to users that changing the dashboard timeframe will change the data.

   If you do not wish to use a dashboard timeframe, you can set the timeframe in the MQL query by adding within to the query. If you deselect Automatically apply dashboard timeframe and do not specify a timeframe in the MQL query, the query will fail. To add a timeframe to a query, simply type | within (timeframe) at the end of your query. For example, to add a timeframe of 24 hours you enter: | within (24h)

fetch gce_instance | metric 'compute.googleapis.com/instance/cpu/usage_time' | within (20m)

fetch iam_service_account
| metric 'iam.googleapis.com/service_account/authn_events_count'
| align rate(1m)
| every 1m | within (24h)

fetch bigquery_dataset
| metric 'bigquery.googleapis.com/storage/uploaded_bytes'
| align rate(1m)
| every 1m

fetch gce_instance
            | metric 'compute.googleapis.com/instance/cpu/usage_time'
            | filter (metadata.system_labels.name == '{{#.}}{{name}}{{/.}}')
            | align rate(1m)
            | every 1m

1. In the tile editor, filter by the GCP data source, select Monitor Metric from the data stream list and then click Next.
2. Select the objects that you want to use and then click Next.

   If you are selecting multiple objects, we suggest you only scope to objects of the same type. For example, the data will probably not be displayed in a helpful way if you were to select both Host objects and SQL objects in the same scope.

3. Metric Name:
   Select the desired metric from the dropdown. The dropdown options will automatically update depending on the object that you have scoped to. For example, scope to a Host and select CPU usage from the Metric Name dropdown.

Just like other services, Workflow is also supported by the GCP data source. You can monitor Workflow executions, execution times, and finished execution counts.

To set up a data stream for Workflow:

1. In the tile editor, filter by the GCP data source or select Workflow from the scope, then select your desired metrics (Workflow executions, Execution times, Finished execution count) from the list of data streams, and then click Next.
2. Select the Workflow you wish to scope to, then click Next.

The GCP data source also supports monitoring Builds from the Build Trigger service.

To set up a data stream for Build:

1. In the tile editor, filter by the GCP data source or select Build Trigger from the scope, then select Builds from the list of data streams, and then click Next.
2. Select the Build Trigger you wish to scope to, then click Next.

For PubSub service, you can monitor Publish requests, Publish message size, Oldest unacked message age, and Unacked messages.

To set up a data stream for PubSub:

1. In the tile editor, filter by the GCP data source or select Topic or Subscription from the scope (depending on what you want to monitor), then select your desired metrics from the list of data streams, and then click Next.
2. Select the Topic or Subscription you wish to scope to, then click Next.

Firestore databases can be queried for document data by specifying a Document Path to your.

1. In the tile editor, filter by the Google Cloud Platform data source or select Database from the scope.
2. Select Firestore Documents from the list of data streams, and then click Next.
3. Select the database you wish to scope to, then click Next.
4. Specify the Document Path to a document or a collection. The path should be relative to the database (i.e. not include the project or database name). For example mycollection or mycollection/mydoc.

Firestore databases can be queried for document data by specifying a Firestore query.

1. In the tile editor, filter by the Google Cloud Platform data source or select Database from the scope.
2. Select Firestore Structured Query from the list of data streams, and then click Next.
3. Select the database you wish to scope to, then click Next.
4. Enter the structured query JSON. For example:
   

   See the Firestore StructuredQuery schema documentation for detailed information on writing queries.

   Copy

   {
       "from": {
           "collectionId": "users"
       },
       "where": {
           "fieldFilter": {
               "field": {
                   "fieldPath": "firstName"
               },
               "op": "EQUAL",
               "value": {
                   "stringValue": "Jo"
               }
           }
       }
   }

Create BigQuery (Scoped) data stream

Code example:

{
  "name": "exampleScopedBigQuery",
  "dataSourceConfig": {
    "query": "SELECT project.name, service.description FROM `<project_id>.<data_set_name>.<table_name>` WHERE project.id in ('{{sourceId}}') GROUP BY 1,2 ORDER BY 1,2;"
  },
  "rowPath": [],
  "matches": {
    "sourceType.0": {
      "type": "oneOf",
      "values": [
        "GCP Project"
      ]
    }
  },
      "metadata": [
        { "name": "name", "displayName": "Project","shape": "string", "role": "label" },
        { "name": "description", "displayName": "Service", "shape": "string" },
      ]
}

{
    "name": "exampleBigQuery",
    "dataSourceConfig": {
        "useTimeFrame": true,
        "query": "SELECT <column_1> FROM `<project_id>.<data_set_name>.<table_name>` WHERE <dateColumn> BETWEEN @startDate AND @endDate"
    },
    "rowPath": [],
    "matches": "none",
    "metadata": []
}

//If you want to specify only one value of an object property//
"matches": {
	"ObjectProperty": {
		"type": "equals",
		"value": "ValueOfTheObjectProperty"
	}
},
 
//If you want to specify multiple values for an object property//
"matches": {
	"ObjectProperty": {
 		"type": "oneOf",
		"values": ["ValueOfTheObjectProperty1", "ValueOfTheObjectProperty2", "ValueOfTheObjectProperty3"]
	}
},

"matches": {
    "sourceName": {
      "type": "equals",
      "value": "AppDynamics"
    },
    "type": {
      "type": "equals",
      "value": "app"
    }
  },

"matches": [
    {
        "sourceName": {
            "type": "equals",
            "value": "AppDynamics"
        }
    },
    {
        "type": {
            "type": "equals",
            "value": "app"
        }
    }
]

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Account(s) or Project(s)

Visualization:

Line Graph is good for showing data for dashboard timeframes of 7 or 30 days. A line will be shown for each service.

When the dashboard timeframe covers more than one day the first column of a table shows the date. To only show each service once for the selected dashboard timeframe see Use Case: How much does each GCP service cost for a specific timeframe?.

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Account(s) or Project(s) or Hosts

Shaping:

The following settings will show how much each service costs you in total for the account(s) or project(s) in scope for the whole timeframe selected:

Visualization:

Table, Donut or Bar Chart. Each bar or segment will show a service.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to 7 or 30 days.

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Hosts

Shaping:

The following settings will show how much each service costs you in total for the account(s) or project(s) in scope for the whole timeframe selected:

Visualization:

Table, Donut or Bar Chart. If, for example, you've scoped to Hosts, then each bar will be the cost for one host.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to 7 or 30 days.

Create a new tile on a dashboard with the following settings:

Data stream:

Cost

Objects:

Account(s)

Shaping:

The following settings will show how much all the service cost you in total for the accounts in scope for the whole timeframe selected:

Visualization:

If you only have one account in the scope you can choose Scalar, as you only want to see one number (total cost for all services).

If you have several accounts in the scope you can choose Table, to see the cost per account.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to 7 or 30 days.