Prometheus plugin

For more information about what this plugin does and the data streams it retrieves, see:

Prometheus

The Prometheus data source enables you to query data from your Prometheus environment for use in your dashboards, analytics and monitoring.

The data source supports the following types of data streams:

  • PromQL
  • Simple query (an easy-to-use alternative to PromQL for common queries)
  • Alerts
  • Status of the Prometheus instance

In addition to data streams, the data source can index labels in your Prometheus database as objects in SquaredUp. Indexing objects makes it easier to query data, correlate Prometheus data with other data sources, and explore the data interactively. Object indexing is explained further here.

The Prometheus data source is available as a Cloud data source and also as an On-prem data source. Use the On-prem data source if your Prometheus instance is on a private network and is not publicly accessible.

Object Indexing

Prometheus time series are described by a set of labels. These labels can be indexed by SquaredUp to make it easier to query, explore and correlate metrics.

The indexed labels are stored in SquaredUp as objects. For example, when (at a minimum) the job label is indexed an object is created for each unique value, such as prometheus-local.

If Index Open Telemetry Semantic Convention labels is selected when you Add a Prometheus data source to SquaredUp, then additional labels that conform to the Open Telemetry Semantic Conventions for resources are also indexed. For example, the k8s_pod_name label will be indexed as Kubernetes Pod objects. See https://opentelemetry.io/docs/specs/semconv/resource/ for more examples of resource labels.

These objects can then be selected in the tile editor when using the object-aware version of the PromQL and Simple Query data streams. When selected, the data streams are automatically scoped to the required label values, so there is no need to create "label matcher" expressions in PromQL.

They can also be viewed in the map, and you can even configure custom correlations between Prometheus objects and objects form other data sources, for example the Kubernetes Pod objects, with other data source objects, for example AWS EKS Kubernetes Pod objects.

To index custom labels that are not automatically indexed with Open Telemetry Semantic Conventions, use the Index custom labels option when you Add a Prometheus data source to SquaredUp.

How to add a Prometheus data source

Prerequisite for the On-prem version

The On-prem version requires an agent installed on a host that has access to your private network.

If you have already created an agent in SquaredUp that you can use for this data source, you can skip this step and choose the agent group you want to use while adding the data source.

Configure and deploy an agent

If you have already created an agent in SquaredUp that you can use for this data source, you can skip this step and choose the agent group you want to use while adding the data source.

Add a Prometheus data source to SquaredUp

  1. To add a data source click on the + next to Data Sources on the left-hand menu in SquaredUp. Search for the data source and click on it to open the Configure data source page.

  2. Display Name:
    Enter a display name for the data source. If connecting to multiple Prometheus instances, you may want to give this data source a more descriptive name such as Prometheus Prod.
  3. Agent Group:
    If configuring the On-prem version, select the Agent Group from the list.
  4. Prometheus URL:
    Enter the URL of your Prometheus instance. Use the On-prem version to access a Prometheus instance that is not publicly accessible.
  5. Authentication Type:
    Select one of the following authentication types, used to access your server, from the Query Type field:
    • None: No authentication is required.
    • API Token: You must enter the API authentication token in the API Token field.
    • Basic Authentication: You must enter a Username and Password in the corresponding fields.
  6. Ignore Certificate errors:

    If you activate this checkbox the data source will ignore certificate errors when accessing the server. This is useful if you have self-signed certificates.

  7. Import Semantic Conventions compatible labels:
    Select whether to index Open Telemetry Semantic Conventions labels. For example, the k8s_pod_name label will be indexed as Kubernetes Pod objects.
  8. Enable object indexing:
    Select whether to index labels as objects. Indexing labels as objects makes it easier to correlate, explore and query your Prometheus data (see Object indexing) Selecting this option will, at a minimum, index Prometheus jobs (the "job" label). Selecting this option reveals additional indexing options.
  9. Index Open Telemetry labels:
    Select whether to index Open Telemetry Semantic Conventions labels. For example, the k8s_pod_name label will be indexed as Kubernetes Pod objects. For more information on Open Telemetry Semantic Conventions, see https://opentelemetry.io/docs/specs/semconv/resource/.
  10. Index custom labels:
    Select whether to index custom labels. These are additional labels that will be indexed as objects in SquaredUp.
    1. Click Add a custom index.
    2. Enter one of more labels to index as an object. For example app_instance. Use commas to separate multiple labels, for example app_instance_namespace, app_instance_id.
    3. Enter the type of object that will be created by the index, for example App Instance.
    4. Click Add another custom index to add further indexes.
  11. Optionally, select whether you would like to restrict access to this data source instance. By default, restricted access is set to off.

  12. Click Test and add to validate the data source configuration.

Using the Prometheus data streams

The data source supports the following types of data streams:

  • PromQL
  • Simple queries (an easy-to-use alternative to PromQL for common queries)
  • Alerts
  • Status of the Prometheus instance

PromQL vs Simple Query

PromQL is commonly used for querying Prometheus and there are many resources available that include tutorials and example queries. PromQL can be difficult to get right however, even for experienced users. The Simple Query data streams offer an alternative, easier-to-use method of querying Prometheus data for common queries.

Simple Query is the recommended data stream for non-experience PromQL users.

Global vs object-aware

The PromQL and Simple Query data streams both have global and object-aware versions. Object-aware versions use the objects indexed from the database labels to make it easier to scope your queries to the required labels. For example, an object-aware simple query can be scoped to Kubernetes Nodes by browsing and selecting the nodes during the Objects step of the tile editor.

Object-aware versions are recommended.

Using object-aware data streams requires "Enable object indexing" to be configured.

Query results

With the exception of the Alerts data stream, all data streams return data in a tabular format, with the following columns:

  • Timestamp: timestamp of the metric value. If single values are requests (typically an ‘instant’ query) the the timestamp should be the current time. For ‘range’ queries, the timestamps will be at regular intervals across the configured timeframe.
  • Value: the metric value. This is displayed with 5 decimal places by default. Use the Columns step of the tile editor to change the number of decimal places.
  • Auto Label: this is a column automatically created by the data stream to be used as a series label on graphs and other visualizations. It is constructed from the labels that uniquely identify a distinct time series in the returned data. This columns is not stored in Prometheus and cannot be used for aggregations and filtering.
  • Labels: the remaining columns are all the labels for the Prometheus time series.

Troubleshooting query timeouts

It can be common for queries to timeout if they attempt to return a large dataset. To reduce the chance of timeouts, follow these best practices:

  • Scope queries to only the required labels. Use object selections (with object-aware data streams) and label filters to filter the query to the data of interest.
  • Aggregate by labels. A common cause of timeouts is a query that attempts to return a large number of unique time series. For example, a query for an HTTP response time may try to return a time series for each unique client IP address. Aggregating by the client IP address will reduce the number of time series returned
  • Increase the time interval. If queries are still causing timeouts, try increasing the time interval to reduce the number of data points returned.

If the query causes a timeout and you are not sure why, try changing the query to an "instant" or "single value" query that returns just the latest data point for each time series. Use this to filter and aggregate labels to reduce the number of time series returned by the query.

Data streams

The following data streams are installed with this plugin.

Was this article helpful?


Have more questions or facing an issue?