AWS Data Stream Parameters

The AWS plugin provides configurable data streams to fetch cost data and service metrics.

Each configurable data stream exposes a set of parameters that lets you filter the data you retrieve. The following information describes the available configurable data streams and their parameters in detail.

Athena Query

Returns the result of a query against an Athena table, for reporting over data held in S3.

Parameter
Description
Region
Specify the AWS region to target.
Workgroup
Specify the Athena workgroup to run the query in. Leave blank to use the primary workgroup.
Database
Specify the Glue database the query runs against. Leave blank if the query fully qualifies table names, for example "database"."table".
Query
Enter the Athena SQL query. Unqualified table names resolve against the Database field above. Any statement your workgroup permissions allow can run here, not just SELECT.

Examples

Count pass/fail rate by day

SELECT date_trunc('day', run_time) AS day,
       status,
       count(*) AS total
FROM "e2e_metrics"
GROUP BY 1, 2
ORDER BY 1

Top failing test groups in the last 24 hours

SELECT test_group, count(*) AS failures
FROM "e2e_metrics"
WHERE status = 'FAILED'
  AND run_time > date_add('hour', -24, now())
GROUP BY test_group
ORDER BY failures DESC

Required permissions

Depending on your IAM configuration, grant the plugin's AWS role the following permissions:

  • athena:StartQueryExecution
  • athena:GetQueryExecution
  • athena:GetQueryResults
  • athena:StopQueryExecution
  • athena:ListWorkGroups
  • glue:GetDatabases
  • glue:GetTable
  • S3 read access to the Athena query results and query bucket

Cost

Returns AWS account cost and usage, broken down by service.

Parameter
Description
Metric
Specify the metric name from the AWS Cost Explorer.
Granularity
Select the time interval for the data. Choose from Daily, Monthly, or Hourly.
Group by
Optionally, specify a GroupBy clause group your cost data by Dimension, Tag, or Cost Category to break down costs with more granularity. For example:
[
  {
    "Type": "TAG",
    "Key": "Environment"
  }
]
Filter by
Optionally, specify a filter expression to refine your cost data. For example: 
"Dimensions": {
      "Key": "SERVICE",
      "Values": ["Amazon Elastic Compute Cloud - Compute"]
    }

Example

The parameter configuration below produces the tile shown with it.

Metric: Amortized Cost | Granularity: Monthly | Group by: {"Key": "SERVICE", "Type": "DIMENSION"}

CloudWatch Metrics

Select a Query method to use. The parameters that display depend on your choice:

By resource

Choose metrics for named AWS resources, when you already know which resources the tile is about.

Parameter
Description
Resources
Specify the resources to filter by.
Metric
Specify the name of the CloudWatch metric you want to query.
Statistic
Specify statistics to aggregate data by. You can enter a CloudWatch statistic even if it is not available in the list. See CloudWatch statistics definitions for a description of each of the statistics.
Note

Which statistics are valid can vary depending on the metric. Refer to the AWS documentation page for clarification.

Period
Enter the aggregation period for the metric query, optionally using a unit suffix: s for seconds (default), m for minutes, h for hours.
For example 15m or 2h.
Use custom dimension
Select this check box if you want to apply a custom dimension to the query. Dimensions are filters that identify a specific metric stream.
For example, the metric CPUUtilization can exist for many resources. Specifying a dimensions tells CloudWatch which resource you want.
When selected, the following fields display:
  • Dimension name:
    Specify the CloudWatch dimension names to apply to the metric query. You can specify up to four dimensions, though not all metrics support every dimension combination.
  • Dimension value:
    Specify the value for each dimension. This is typically the identifier of the resource you want to query. The values must correspond to the dimension names provided above.

By namespace

Choose metrics across a whole CloudWatch namespace rather than named resources, so resources added later appear without editing the tile.

Parameter
Description
Region
Specify the AWS region where your CloudWatch metrics are located.
Cross-account observability
Choose which AWS account to read data from. Linked accounts can only be used if CloudWatch Cross-account observability is enabled for the AWS account. Choose from:
  • Primary AWS account: Uses the account same account used to authenticate AWS when configuring the data source.
  • Linked AWS account: Enables querying of metrics from the account you specify in the Linked account field.
Metric namespace
Specify a metric namespace. Standard AWS namespaces are listed and custom namespaces must be entered manually.
Metric name
Specify the name of the metric to query, such as CPUUtilization, Invocations, or DatabaseConnections
Statistic
Specify statistics to aggregate data by. You can enter a CloudWatch statistic even if it is not available in the list. See CloudWatch statistics definitions for a description of each of the statistics.
Note

Which statistics are valid can vary depending on the metric. Refer to the AWS documentation page for clarification.

Dimension name / Dimension value
Some metrics require dimensions to categorize, filter, or group metrics. Specify up to four Dimension name and Dimension value fields as needed.
The order of the dimensions does not matter, but not all combinations of dimensions return data.
Period
Enter the aggregation period for the metric query, optionally using a unit suffix: s for seconds (default), m for minutes, h for hours.
For example 15m or 2h.

Insights query

Return CloudWatch metrics using a Metrics Insights query, for aggregation the other two modes cannot express.

Parameter
Description
Region
Select or enter the AWS region in which the CloudWatch Metrics Insights query will be executed.
Query
Enter a CloudWatch Metrics Insights query. Metrics Insights uses a SQL-like syntax to query, filter, group, and aggregate CloudWatch metrics across multiple resources. For example:
SELECT AVG(CPUUtilization)
FROM SCHEMA("AWS/EC2", InstanceId)
GROUP BY InstanceId
ORDER BY AVG() DESC

CloudWatch Logs

Returns CloudWatch log data for the resources in scope.

Parameter
Description
Region
Specify the AWS region where your CloudWatch logs are located.
Log groups
Specify the name or ARN of each log group you want to query. Press Enter after each entry to add it to the list.
Limit
Enter the maximum number of results to return per query.
Query
Enter a CloudWatch Logs Insights query. Use the syntax supported on the CloudWatch > Logs Insights page in the AWS console. See the documentation on CloudWatch Logs Insights queries for reference.

Service Quotas

Returns the quotas applied to an AWS service, so a dashboard can show what is approaching a limit.

Parameter
Description
Service
Specify the AWS service to retrieve quotas for, such as EC2, Lambda, VPC, or RDS. The stream returns all quotas associated with the selected service in the chosen Region.
Region
Select the AWS region to query quotas from. Many AWS service quotas are regional and can differ between regions.
Quota type
Determines which quota values are returned. This can be useful when comparing your current limits against the AWS defaults or when combining quota data with usage metrics in SQL analytics:
  • Current: Returns the quotas currently applied to your account, including any approved quota increases.
  • Default: Returns the AWS default quota values before any account-specific increases have been applied.

Additional usage metrics

Some quotas have accesses to additional metric data. Where there are metrics available, you can use this data stream with SQL analytics to provide additional usage context.

In the following example, the CurrentQuotas and DefaultQuotas datasets are joined with the Metric dataset. This allows the metric values to be viewed alongside both the current applied quota and the default quota, providing additional context when analyzing service utilization and determining how close a service is to its configured limits.

WITH
    "current_default_quota" AS (
        SELECT
            "c"."QuotaName",
            "c"."QuotaCode",
            "c"."UsageMetric.MetricName" as "MetricName",
            "c"."Value" AS "CurrentQuota",
            "d"."Value" AS "DefaultQuota"
        FROM
            "CurrentQuotas" "c"
            JOIN "DefaultQuotas" "d" ON "c"."QuotaCode" = "d"."QuotaCode"
    )
SELECT
    "m"."MetricDataResults.Timestamps",
    "m"."MetricDataResults.Values" AS "MetricValue",
    "m"."MetricDataResults.Label" AS "QuotaName",
    "cd"."CurrentQuota",
    "cd"."DefaultQuota"
FROM
    "Metric" "m"
    LEFT JOIN "current_default_quota" "cd" ON "m"."MetricDataResults.Label" = "cd"."MetricName"
In this tile, you can see that the quota has been increased from the default value, giving the metric useful context and showing that the current usage remains well within the applied limit.

Required permissions

This data stream should function with the ServiceQuotasReadOnlyAccess managed policy attached. However, depending on your IAM configuration, you may also need to grant the following permissions:

  • servicequotas:ListServices
  • servicequotas:ListServiceQuotas
  • servicequotas:ListAWSDefaultServiceQuotas

Timestream Query

Returns the result of a query against a Timestream table, for time series data held in Timestream.

Parameter
Description
Region
Specify the AWS region to target.
Query
Enter a Timestream query. Queries must follow Timestream’s SQL syntax, including database, table, measure, and time range references.

Examples

Retrieve CPU utilization over the past hour

SELECT time, measure_value::double AS cpu_utilization
FROM "MetricsDB"."SystemMetrics"
WHERE measure_name = 'cpu_utilization'
  AND time > ago(1h)
ORDER BY time DESC

Aggregate average temperature by device for the last 24 hours

SELECT device_id,
       AVG(measure_value::double) AS avg_temperature
FROM "IoTDB"."SensorReadings"
WHERE measure_name = 'temperature'
  AND time BETWEEN ago(24h) AND now()
GROUP BY device_id

DynamoDB Query

Query items from a DynamoDB table by partition key, or scan the whole table.

Parameter
Description
Query type
Required. Select how the table is read. Choose from:
  • Query: Returns items matching a specific partition key.
  • Scan: Reads every item in the table.
Region
Required. Enter the AWS region to target.
Table
Required. Select the DynamoDB table to query.
Index name
Select the GSI or LSI to query. Leave empty to query the primary index.
Partition key value
Required. Enter the value of the index's partition key to query.
Sort key operator
Select the comparison operator for the sort key condition. Choose from:
  • Equal to
  • Less than or equal to
  • Less than
  • Greater than or equal to
  • Greater than
  • Between
  • Begins with
Sort key value
Enter the sort key value to compare against.
Sort key value (end)
Required. Enter the upper bound for the Between condition.
Sort direction
Required. Select the order results are returned in by sort key. Choose from:
  • Ascending
  • Descending
Projection
Enter a comma-separated list of attribute names to return. Leave empty to return all attributes.
Limit
Enter the maximum number of items to return per page.

The sort key, sort direction and projection settings appear only when Query type is set to Query. Sort key value (end) appears only when Sort key operator is set to Between.

Deprecated streams

The following data streams are flagged as deprecated but can still be accessed via the Data streams tab by selecting Filter > Visibility > Deprecated.

Cost (Global - Configurable)

Retrieves AWS account cost and usage by service across all regions. Superseded by Cost.

Parameter
Description
Metric
Required. Enter the metric name from Cost Explorer.
Granularity
Required. Select the granularity of the returned data.
Group by
Enter an expression grouping the cost request by DIMENSION, TAG or COST_CATEGORY.
Filter by
Enter an expression filtering the cost request.

S3 Metric

Retrieves CloudWatch metrics for one or more S3 buckets. Superseded by CloudWatch Metrics.

Parameter
Description
Metric
Required. Enter the metric name from CloudWatch.
Statistic
Required. Select the statistic to return. Valid statistics vary by metric, so check the AWS documentation for the metric you are querying.
Storage type
Required. Select the storage type used as a dimension in the CloudWatch query.
Other storage type
Enter a storage type to use as the dimension if it is not listed under Storage type.

DynamoDB Metric

Retrieves CloudWatch metrics for one or more DynamoDB tables. Superseded by CloudWatch Metrics.

Parameter
Description
Metric
Required. Enter the metric name from CloudWatch.
Statistic
Required. Select the statistic to return. Valid statistics vary by metric, so check the AWS documentation for the metric you are querying.
Operation
Enter the operation name from CloudWatch. Required only for table operation metrics.

CloudWatch Metrics by Query

Queries CloudWatch metrics by namespace and name, by a Metric Insights SQL query, or by raw metric source JSON. Superseded by CloudWatch Metrics.

Parameter
Description
Region
Required. Enter the AWS region to target.
Query type
Required. Select how the metric is specified.
Cross-account observability
Required. Enable to query metrics in linked accounts, where CloudWatch cross-account observability is configured for the AWS account.
Linked account
Required. Select the linked account to read metrics from. This applies only to monitoring accounts configured for cross-account observability.
Metric namespace
Required. Select or enter the metric namespace. Only standard AWS namespaces are listed, so enter custom namespaces manually.
Metric name
Required. Select or enter the metric name.
Statistic
Required. Select one or more statistics, or enter one that is not listed. Valid statistics vary by metric, so check the AWS documentation for the metric you are querying.
Dimension 1 to 4 name
Select or enter a dimension name. Some metrics require one or more dimensions. Order is not usually significant, but not every combination returns data.
Dimension 1 to 4 value
Select or enter the value for the matching dimension.
Period
Enter the aggregation period for the query. Add a unit suffix of s for seconds, m for minutes or h for hours, for example 15m or 2h.
Query
Required. Enter a CloudWatch Metric Insights SQL query, or paste the JSON from the CloudWatch metrics source tab.

CloudWatch Metrics by Resource

Retrieves CloudWatch metrics for the objects selected in the tile. Superseded by CloudWatch Metrics.

Parameter
Description
Metric
Required. Select or enter a CloudWatch metric name.
Statistic
Required. Select the statistic to return, or enter one that is not listed. Valid statistics vary by metric, so check the AWS documentation for the metric you are querying.
Period
Enter the aggregation period for the query. Add a unit suffix of s for seconds, m for minutes or h for hours, for example 15m or 2h.

Was this article helpful?

Have more questions or facing an issue?
Submit a ticket