AWS plugin

Data streams standardize data from all the different shapes and formats your tools use into a straightforward tabular format. While creating a tile you can tweak data streams by grouping or aggregating specific columns. Depending on the kind of data, SquaredUp will automatically suggest how to visualize the result, for example as a table or line graph.

Data streams can be either global or scoped:

• Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").

• A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").

See Data Streams for more information.

This data stream allows you to visualize and monitor Timestream data by querying one or more Timestream tables in the specified AWS region. The IAM user with programmatic access configured when adding the AWS data source must have the AmazonTimestreamReadOnlyAccess AWS managed policy.

Filter by the AWS data source in the tile editor, select Timestream Query, and then click Next

Region: Select the AWS region to target.

Query: Enter the Timestream query. Timestream database and table names may need to be enclosed in double quotes (e.g. FROM “DatabaseName”.”TableName”). It’s recommended to copy a valid and tested Timestream query from the Query editor within the AWS Timestream portal.

This three minute video shows how to visualize AWS Timestream data:

Play YouTube video

This data stream allows you to fetch logs from any log group and run queries on them.

Filter by the AWS data source in the tile editor, select CloudWatch Logs, and then click Next

Region: Select the AWS region to target.

Log group(s): Enter the names of the log groups you want to query/fetch logs from, separated by commas.

Limit: Enter a limit for the number of results you want to fetch per query.

Query: Enter the Log Insights query you want to run. Use Log Insights in CloudWatch logs in the AWS console to write your queries. For example, you can add a filter to they query | filter: @message like 'error'. See AWS: CloudWatch Logs Insights query syntax

Allows you to query any metric in AWS CloudWatch using the source from the CloudWatch Metrics explorer.

Filter by the AWS data source in the tile editor, select CloudWatch Metrics Query, and then click Next

Region: Select the AWS region to target.

Query: In CloudWatch Metrics you can either copy the JSON from the Source tab or the SQL Query from the Query Editor.

Warning: SquaredUp does not support the Minimize source option available in the CloudWatch console. Please disable this option before copying the query to ensure an error is not shown.

Filter by the AWS data source in the tile editor, select CloudWatch Metric, and then click Next

Scope to the objects you want to use, and then click Next.

The current resources supported are:

• API Gateway

• CloudFront

• CloudWatch RUM

• CloudWatch Synthetics

• Cognito UserPools

• DynamoDB

• ECS clusters

• EC2

• ElastiCache Memcached

• ElastiCache Redis

• Elastic load balancers (ELB)

• Lambda

• Route53

• RDS

• S3 buckets

• WAF WebACLs

Metric: Select the metric that you want to measure.

Statistic: Select the required statistic.

Available statistics vary depending on the metric that has been selected.

The following resources have their own configurable data streams as they have additional fields:

• DynamoDB Metric: optional operation type that is required to fetch certain metrics.

• S3 Metric: requires you to select a storage type depending on metric.

Explore costs occurred in AWS. Filter by the AWS data source in the tile editor, select Cost, and then click Next

Filters and grouping info at AWS: GetCostAndUsage

Metric: Select the cost metric from Cost Explorer e.g. UnblendedCost.

Granularity: Select how frequently you want to sample the data: Hourly, Daily, or Monthly.

Group By: Add grouping for the cost request by Dimension, Tag or Cost.

Filter: Enter the filter used to remove cost data from the query based on dimensions.

Cost examples:

Group by availability zone

[
  {
    "Type": "DIMENSION",
    "Key": "AZ"
  }
]

Group by service

[
  {
    "Type": "DIMENSION",
    "Key": "SERVICE"
  }
]

Filter to lambda cost only

{
  "Dimensions": {
    "Key": "SERVICE",
    "Values": [
      "AWS Lambda"
    ]
  }
}

Filter to remove tax and developer support

{
  "Not": {
    "Dimensions": {
      "Key": "SERVICE",
      "Values": [
        "Tax",
        "AWS Support (Developer)"
      ]
    }
  }
}

This Data Stream uses the AWS Cost Explorer to show you data about how much each of you AWS services cost you.

The Cost Data Stream returns the daily unblendedcosts per service. If you want to use a different granularity (like monthly costs) or a different cost metric (like amortized costs), you need to create a custom Data Stream (see Custom Data Streams). Costs will always be returned in the currency you set in your AWS account.

Note: You need the AWS Cost Explorer service to use the Cost Data Stream. This service collects the costs of your other services. Please be aware that AWS charges you for every request that the AWS Cost Explorer sends. Each tile that uses the AWS Cost Data Stream makes a request every time the dashboard reloads.

Tax costs

Line Graphs

Remember that the cost data from AWS is daily data. Therefore, Line Graphs will only show data when the dashboard timeframe is set to 7 days or 30 days. Any smaller timeframe will leave the Line Graph empty since there's not enough data.

+ What is the dashboard timeframe?

Scalars

Create a new tile on a dashboard with the following settings:

Data Stream:

Select the Cost Data Stream.

Objects:

Select a scope that contains your AWS account(s).

Visualization:

You could select a Table or a Line Graph to visualize your data. A Line Graph is probably the more convenient way to see what each service cost you over time.

Remember that the cost data from AWS is daily data. Therefore, Line Graphs will only show data when the dashboard timeframe is set to 7 days or 30 days. Any smaller timeframe will leave the Line Graph empty since there's not enough data.

+ What is the dashboard timeframe?

Create a new tile on a dashboard with the following settings:

Data Stream:

Select the Cost Data Stream.

Objects:

Select a scope that contains your AWS account(s).

Shaping:

The following settings will get you the result of how much all services combined cost you in total or on average:

Visualization:

You could select a Table or a Line Graph to visualize your data. A Line Graph is probably the more convenient way to see what all services combined cost you over time.

Remember that the cost data from AWS is daily data. Therefore, Line Graphs will only show data when the dashboard timeframe is set to 7 days or 30 days. Any smaller timeframe will leave the Line Graph empty since there's not enough data.

+ What is the dashboard timeframe?

Create a new tile on a dashboard with the following settings:

Data Stream:

Select the Cost Data Stream.

Objects:

Select a scope that contains your AWS account(s).

Shaping:

The following settings will get you the result of how much each service cost you:

Visualization:

There are different ways to visualize your data, try a Donut or a Table.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to today, 7 days or 30 days.

+ What is the dashboard timeframe?

Create a new tile on a dashboard with the following settings:

Scope:

Select a scope that contains your AWS account(s). If you haven't created a scope that contains your AWS account(s), you can create one by clicking on the + next to Scope.

Shaping:

The following settings will get you the result of how much each service cost you:

Visualization:

Since you only want to see one number (all costs for all services), you can choose a Scalar.

After you're done creating the tile, you can specify the timeframe by changing the dashboard timeframe to today, 7 days or 30 days.

+ What is the dashboard timeframe?

This Data Stream shows you the Cost Anomaly alerts from AWS. It returns the alerts you have set up manually as well as the AI generated ones (when AWS suspects you're paying too much for a service).

Which visualization should I use?

This Data Stream returns a list of alerts and a Table is the best way to display them. The Table will show you:

• The Date the alert was triggered

• The Service for which the alert was triggered

• The Anomaly Score, a AWS internal rating system which defines how severe the anomaly is

• The Total Impact, meaning how much more than expected you payed

• The Name of the monitor that triggered the alert

This custom Data Stream will enable you to get different granularity or metrics than the default Cost Data Stream for AWS.

Which entry point do I have to select from the dropdown?

Account Cost

Code example:

{
    "name": "account-cost",
    "dataSourceConfig": {
        "groupBy": [
            {
                "Type": "DIMENSION",
                "Key": "SERVICE"
            }
        ],
        "metrics": [
            "AmortizedCost"
        ],
        "granularity": "HOURLY",
        "filter": {
            "Dimensions": {
                "Key": "SERVICE",
                "Values": [
                    "AWS Cost Explorer"
                ]
            }
        },
        "rowPath": [
            "ResultsByTime",
            [
                "Groups"
            ]
        ],
        "matches": {
            "sourceType.0": {
                "type": "equals",
                "value": "AWS Account"
            }
        },
        "metadata": [
            {
                "name": "ResultsByTime.TimePeriod.Start",
                "displayName": "Date",
                "shape": "date"
            },
            {
                "name": "ResultsByTime.Groups.Keys.0",
                "displayName": "Service",
                "role": "label",
                "shape": "string"
            },
            {
                "name": "ResultsByTime.Groups.Metrics.AmortizedCost.Amount",
                "displayName": "Cost",
                "role": "value",
                "shape": "currency"
            },
            {
                "name": "ResultsByTime.Groups.Metrics.AmortizedCost.Unit",
                "displayName": "Currency",
                "shape": "string"
            },
            {
                "name": "ResultsByTime.Estimated",
                "displayName": "Estimated",
                "shape": "boolean"
            }
        ]
    }
}

Parameters

name

Mandatory

The internal name of the data stream. Can be used the refer to this data stream in a tile's JSON instead of using the data stream's internal ID.

dataSourceConfig Parameters

Note: You'll find a detailed description about what each parameter does and which values you can use in the AWS API docs: https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostAndUsage.html

matches Parameters

Note: Defining the matches parameter is mandatory.

With the matches parameter you define for which objects the data stream will be shown in SquaredUp. It works like this:
When you configure a tile, you have to choose its scope. If this scope contains objects you specified here in the matches parameter, the data stream will be shown in SquaredUp under Data Streams. If the scope doesn't contain objects specified here, the data stream will be hidden.
This keeps things clean and simple since you'll only see the data stream when it's relevant for the scope you chose. As a best practice you should limit the data stream to objects that make sense for the specific use case of this data stream.

Format for matches:

//If you want to specify only one value of an object property//
"matches": {
    "ObjectProperty": {
        "type": "equals",
        "value": "ValueOfTheObjectProperty"
    }
},
 
//If you want to specify multiple values for an object property//
"matches": {
    "ObjectProperty": {
         "type": "oneOf",
        "values": ["ValueOfTheObjectProperty1", "ValueOfTheObjectProperty2", "ValueOfTheObjectProperty3"]
    }
},

Example for limiting a data stream to objects:

If you are using multiple values for the object property, you can decide if you want the data stream to be visible for objects that match all of the criteria or at least one of the criteria.

Lets say you have two values you want objects to have in order for the data stream to be visible for them: 

• a SourceName property with the value AppDynamics (meaning objects that come from the AppDynamics data source)

• a type property with the value app (meaning application objects)

If you want the data stream to be visible only for objects that match both of the criteria, your code would look like this:

  "matches": {
    "sourceName": {
      "type": "equals",
      "value": "AppDynamics"
    },
    "type": {
      "type": "equals",
      "value": "app"
    }
  },

If you want the data stream to be visible for objects that match at least one of the criteria, your code would look like this:

"matches": [
    {
        "sourceName": {
            "type": "equals",
            "value": "AppDynamics"
        }
    },
    {
        "type": {
            "type": "equals",
            "value": "app"
        }
    }
]

Note: If you run into errors when configuring the matches parameter, check if you're dealing with a global entry point.

Data streams can be either global or scoped:

• Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").

• A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").

+ Identifying and configuring global entry points

rowPath Parameter

Optional

SquaredUp expects data in table form, and here's where you define how the table with your return data will be structured.

The rowPath will tell SquaredUp which items you want to convert into rows.

Example:

Let's say your return data looks like this:

{
    "generalInfo": "some info",
    "results": [
        {
            "name": "object 1",
            "tags": [
                "tag 1",
                "tag 2",
                "tag 3"
            ]
        },
        {
            "name": "object 2",
            "tags": [
                "tag 1",
                "tag 4"
            ]
        }
    ]
}

Now it depends on what data you want to base your table on, do you want rows per object or per tag?

If you want to see which objects have which tags, your rowPath would be results, and your table would look like this:

If you want to turn each tag into a row and see to which objects they are applied, your rowPath would be results.tags, and your table would look like this:

As you can see in the example, each parameter gets turned into a column and the items of the parameter you chose as the rowPath will be turned into rows.

You might also have noticed in the example that the parameter generalInfo was included in both cases. This is because after going through all parameters on the same level as your rowPath, SquaredUp will move through all levels above the rowPath and create columns from those parameters, too. Parameters on levels below the rowPath won't be included in the table.

metadata Parameter

Optional, but recommended

The metadata parameters are used to describe columns in order to tell SquaredUp what to do with them. You can do multiple things with the metadata parameters:

• Specify how SquaredUp should interpret the columns you return and - to an extent - how their content displayed. You do this by giving each column a shape.

  The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link.

• Filter out or just hide columns.

  Only the columns you define in metadata will be returned in the results. This helps you to filter out columns you don't need. If you need the content of a column but don't want to display it, you can use the visible parameter.

• Give columns a nicely readable display name.

• Assign a specific role to columns

  The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role value to the column that contains the actual value you want to use in your visualization.

Note: If you don't specify any metadata, all columns will be returned and SquaredUp will do its best to determine which columns should be used for which purpose. If you're returning pretty simple data, for example just a string and a number, this can work fine. But if you're returning two columns with numbers it gets trickier for SquaredUp to figure out which one is the value and which one is just an ID or some other number.

Parameters:

Tip: Before you start specifying metadata, leave them empty at first and get all the raw data with your new data stream once. In order to do this, finish creating your custom data stream without metadata and create a tile with this data stream. The Table visualization will show you all raw data.
This will give you an overview about all columns and their content and help you decide which columns you need and what their shapes and roles should be. It's also essential for getting the correct column name to reference in the name parameter.
Use this information to go back to the data stream configuration and specifying the metadata.

+ List of shapes

+ List of roles