Elasticsearch plugin

For more information about what this plugin does and the data streams it retrieves, see:

Monitor any metrics from your Elasticsearch environment using custom QueryDSL, Lucene queries and API calls.

How to add the data source

To add a data source click on the + next to Data Sources on the left-hand menu in SquaredUp. Search for the data source and click on it to open the Configure data source page.

Configuring the data source

Display Name:
Enter a name for your data source. This helps you to identify this data source in the list of your data sources.
Instance URL:
Enter the URL for your Elasticsearch environment.
Where to find the Instance URL:
The URL depends on your Elasticsearch environment:
1. If you are using the Elastic cloud service, you need to enter the endpoint URL, which has the following format:
  https://CLUSTER_ID.REGION.CLOUD_PLATFORM.DOMAIN:PORT
2. If you are running Elasticsearch in a different environment, ask your Elastic administrator for the instance URL and port number.
Enter the username and password for your Elasticsearch environment.

Optionally, select whether you would like to restrict access to this data source instance. By default, restricted access is set to off.

The term data source here really means data source instance. For example, a user may configure two instances of the AWS data source, one for their development environment and one for production. In that case, each data source instance has its own access control settings.

By default, Restrict access to this data source is set to off. The data source can be viewed, edited and administered by anyone. If you would like to control who has access to this data source, switch Restrict access to this data source to on.

Use the Restrict access to this data source dropdown to control who has access to the workspace:

By default, the user setting the permissions for the data source will be given Full Control and the Everyone group will be given Link to workspace permissions.
Tailor access to the data source, as required, by selecting individual users or user groups from the dropdown and giving them Link to workspace or Full Control permissions.
If the user is not available from the dropdown, you are able to invite them to the data source by typing in their email address and then clicking Add. The new user will then receive an email inviting them to create an account on SquaredUp. Once the account has been created, they will gain access to the organization.
At least one user or group must be given Full Control.
Admin users can edit the configuration, modify the Access Control List (ACL) and delete the data source, regardless of the ACL chosen.

Access Level:

Link to workspace	User can link the data source to any workspace they have at least Editor permissions for. Data from the data source can then be viewed by anyone with any access to the workspace. User can share the data source data with anyone they want. User cannot configure the data source in any way, or delete it.
Full Control	User can change the data source configuration, ACL, and delete the data source.

See Access control for more information.

Click Add.
You can also add a data source from Settings > Data Sources > Add data source, but sample dashboards are not added when using this method.

Using the Elasticsearch data streams

Data streams standardize data from all the different shapes and formats your tools use into a straightforward tabular format. While creating a tile you can tweak data streams by grouping or aggregating specific columns. Depending on the kind of data, SquaredUp will automatically suggest how to visualize the result, for example as a table or line graph.

Data streams can be either global or scoped:

Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").
A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").

See Data Streams for more information.

Data streams

The following data streams are installed with this plugin.

Scope to an object from any data source
Select + Query from the data stream list
Enter a Display Name for the new data stream
The next two fields, JSON Query Body and JSON Query Parameters, are both optional, but you must enter details for either one or the other.
Optionally, enter a JSON Query Body, for example:
```
{
    "query": {
        "query_string": {
            "query": "managedEntityDisplayName:\"Microsoft Windows Server 2016 Datacenter\""
        }
    }
}
```
A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.
This data stream supplies scoped objects individually for mustache parameters. When there are multiple objects in scope this data source will send the query multiple times, once for each object. The results are then displayed together, for example in a single table.
You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.
Optionally, enter a set of JSON Query Parameters, for example:
```
{
    "q": "managedEntityDisplayName:\"Microsoft Windows Server 2016 Datacenter\""
}
```
A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.
This data stream supplies scoped objects individually for mustache parameters. When there are multiple objects in scope this data source will send the query multiple times, once for each object. The results are then displayed together, for example in a single table.
You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.

Scope to an object from any data source
Select + Lucene Query from the data stream list
Enter a Display Name for the new data stream
Enter a Lucene Query, for example:
```
managedEntityDisplayName:"Microsoft Windows Server 2016 Datacenter"
```
A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.
This data stream supplies scoped objects individually for mustache parameters. When there are multiple objects in scope this data source will send the query multiple times, once for each object. The results are then displayed together, for example in a single table.
You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.

Scope to an object from any data source
Select + Custom API Call from the data stream list
Enter a Display Name for the new data stream
Select an HTTP Method to use for the API call
The next three fields, Elasticsearch API Endpoint, JSON Headers and JSON Parameters, are all optional, but you must enter details for at least one.
Enter an Elasticsearch API Endpoint, for example:
{{name}}/_search
A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.
This data stream supplies scoped objects individually for mustache parameters. When there are multiple objects in scope this data source will send the query multiple times, once for each object. The results are then displayed together, for example in a single table.
You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.
Optionally, enter a set of JSON Headers
Optionally, enter a set of JSON Parameters
A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.
This data stream supplies scoped objects individually for mustache parameters. When there are multiple objects in scope this data source will send the query multiple times, once for each object. The results are then displayed together, for example in a single table.
You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.
Optionally, enter a JSON Body
A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.
This data stream supplies scoped objects individually for mustache parameters. When there are multiple objects in scope this data source will send the query multiple times, once for each object. The results are then displayed together, for example in a single table.
You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.

Writing a custom data stream (advanced users)

In SquaredUp, browse to Settings > Advanced > Data Streams.
Click Add custom data stream.
Add your custom data stream by entering the following settings:
1. Name:
  Enter a display name for your data stream.
  The display name is the name that you use to identify your data stream in SquaredUp. It has no technical impact and doesn't need to be referenced in the data stream's code.
2. Data source:
  Choose the data source this data stream is for.
  After you've chosen the data source the Entry Point field displays.
3. Entry Point:
  Specify the data stream entry point and enter the Code below.
  Each data stream uses an entry point, which can either be global (unscoped) or scoped, and this determines whether the data stream uses the tile scope.
  Data streams can be either global or scoped:
  - Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").
  - A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").
  To find out which entry point to select and get code examples for the Code field, see the help below.
Click Save to save your data stream.

You can use mustache parameters in your data stream. You can use them in your query as well as anywhere else in the data stream code (for example, to call an API endpoint depending on an object's name).

A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.

This data stream supplies scoped objects individually for mustache parameters. When there are multiple objects in scope this data source will send the query multiple times, once for each object. The results are then displayed together, for example in a single table.

You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.

For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.

Query Example:

{
  "name": "customQuery",
  "dataSourceConfig": {
    "queryBody": {
      "query": {
        "query_string": {
          "query": "managedEntityDisplayName:\"Microsoft Windows Server 2016 Datacenter\""
        }
      }
    },
    "queryParams": {}
  },
  "rowPath": [
    "hits",
    "hits"
  ],
  "matches": "all",
  "metadata": [
    {
      "name": "hits.hits._index",
      "displayName": "Index"
    },
    {
      "name": "hits.hits._source.managedEntityDisplayName",
      "displayName": "Entity Name"
    },
    {
      "name": "hits.hits._source.counter",
      "displayName": "Metric"
    },
    {
      "name": "hits.hits._source.value",
      "displayName": "Value"
    },
    {
      "name": "hits.hits._source.timestamp",
      "displayName": "Timestamp",
      "shape": "date"
    }
  ]
}

Parameters:

`name`	Mandatory	The internal name of the Data Stream. Can be used the refer to this Data Stream in a tile's JSON instead of using the Data Stream's internal ID.
`queryBody`	Mandatory	Insert your Query DSL here. Your query needs to replace lines 5 to 9 in the code example above.
`rowPath`	Mandatory	Depends on the data returned from your query. Enter the path to the row you want to use (the row path in the code example above is hits.hits).
`metadata`	Mandatory	Depends on the data returned from your query. Use the code example above as a template for the columns you want to return.

metadata Parameter:

Optional, but recommended

The metadata parameters are used to describe columns in order to tell SquaredUp what to do with them. You can do multiple things with the metadata parameters:

Specify how SquaredUp should interpret the columns you return and - to an extent - how their content displayed. You do this by giving each column a shape.
The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link.
Filter out or just hide columns.
Only the columns you define in metadata will be returned in the results. This helps you to filter out columns you don't need. If you need the content of a column but don't want to display it, you can use the visible parameter.
Give columns a nicely readable display name.
Assign a specific role to columns .
The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role value to the column that contains the actual value you want to use in your visualization.

If you don't specify any metadata, all columns will be returned and SquaredUp will do its best to determine which columns should be used for which purpose. If you're returning pretty simple data, for example just a string and a number, this can work fine. But if you're returning two columns with numbers it gets trickier for SquaredUp to figure out which one is the value and which one is just an ID or some other number.

Parameters:

Before you start specifying metadata, leave them empty at first and get all the raw data with your new data stream once.

In order to do this, finish creating your custom data stream without metadata and create a tile with this data stream. The Table visualization will show you all raw data.

This will give you an overview about all columns and their content and help you decide which columns you need and what their shapes and roles should be. It's also essential for getting the correct column name to reference in the name parameter.

Use this information to go back to the data stream configuration and specifying the metadata.

`name`	Mandatory	Enter the name of the column you are referencing here. To find the name of a column, get the data from this data stream once without any metadata. See the tip above for how to do that. You'll see the column name when you hover over the column in the Table.
`displayName`	Optional	Here you can give the column a user-friendly name
`shape`	Recommended	The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link. Note: Please refer to the list of shapes below this table to see available shapes.
`role`	Recommended	The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role `value` to the column that contains the actual value you want to use in your visualization. Note: Please refer to the list of roles below this table to see available roles.
`visible`	Optional	`true` or `false` Use this if you need a columns content but don't need to display the column itself. Example: Column A contains the full link to a ticket in your ticket system. Column B contains the ticket ID. You want to use the ticket ID as a label for the link, turning the long URL into a much nicer to read "Ticket 123". This is why you need the content of column B, to assign it as a label for column A. But since the URL is now displayed as the ticket ID, it would be redundant to still display column B. This is why you hide column B with `false`.

There are many different shapes you can use for your columns and the list of possible shapes gets expanded constantly:

Basic types, like: boolean, date, number, string
Currency types that get displayed with two decimal values and their currency symbol (for example $23,45), like: currency (generic currency), eur, gbp, usd
Data types, like: bytes, kilobytes, megabytes
Time types, like: seconds, milliseconds, timespan
The status type : state
Utility types, like:
customUnit
url (will be displayed as a link)

Tip:

Some shapes can be configured.

If a shape is configurable, you can edit how the shape displays data in SquaredUp.

See Configurable shapes

`id`	Used by data streams feeding the aggregate health stream to identify their Id column
`label`	A column containing user-friendly names. Line Graphs use this role to group data into series. so each label will get its own line in the Line Graph.
`link`	A column containing a link that can be used as a drilldown in Status Blocks.
`timestamp`	A column containing a date to use on the X -axis of a Line Graph.
`unitLabel`	A column containing user-friendly labels for data series, e.g. ‘Duration’. Line Graphs can use this role to label the Y-axis.
`value`	A column containing the numeric value you want to use in your visualization.

Lucene Query Example:

{
  "name": "luceneQuery",
  "dataSourceConfig": {
    "queryBody": {},
    "queryParams": {
      "q": "managedEntityDisplayName:\"Microsoft Windows Server 2016 Datacenter\""
    }
  },
  "rowPath": [
    "hits",
    "hits"
  ],
  "matches": "all",
  "metadata": [
    {
      "name": "hits.hits._index",
      "displayName": "Index"
    },
    {
      "name": "hits.hits._source.managedEntityDisplayName",
      "displayName": "Entity Name"
    },
    {
      "name": "hits.hits._source.counter",
      "displayName": "Metric"
    },
    {
      "name": "hits.hits._source.value",
      "displayName": "Value"
    },
    {
      "name": "hits.hits._source.timestamp",
      "displayName": "Timestamp",
      "shape": "date"
    }
  ]
}

Parameters:

`name`	Mandatory	The internal name of the Data Stream. Can be used the refer to this Data Stream in a tile's JSON instead of using the Data Stream's internal ID.
`q`	Mandatory	Insert your Lucene query here (line 6 in the code example above).
`rowPath`	Mandatory	Depends on the data returned from your query. Enter the path to the row you want to use (the row path in the code example above is hits.hits).
`metadata`	Mandatory	Depends on the data returned from your query. Use the code example above as a template for the columns you want to return.

metadata Parameter:

Optional, but recommended

The metadata parameters are used to describe columns in order to tell SquaredUp what to do with them. You can do multiple things with the metadata parameters:

Specify how SquaredUp should interpret the columns you return and - to an extent - how their content displayed. You do this by giving each column a shape.
The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link.
Filter out or just hide columns.
Only the columns you define in metadata will be returned in the results. This helps you to filter out columns you don't need. If you need the content of a column but don't want to display it, you can use the visible parameter.
Give columns a nicely readable display name.
Assign a specific role to columns .
The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role value to the column that contains the actual value you want to use in your visualization.

Parameters:

Before you start specifying metadata, leave them empty at first and get all the raw data with your new data stream once.

In order to do this, finish creating your custom data stream without metadata and create a tile with this data stream. The Table visualization will show you all raw data.

Use this information to go back to the data stream configuration and specifying the metadata.

`name`	Mandatory	Enter the name of the column you are referencing here. To find the name of a column, get the data from this data stream once without any metadata. See the tip above for how to do that. You'll see the column name when you hover over the column in the Table.
`displayName`	Optional	Here you can give the column a user-friendly name
`shape`	Recommended	The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link. Note: Please refer to the list of shapes below this table to see available shapes.
`role`	Recommended	The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role `value` to the column that contains the actual value you want to use in your visualization. Note: Please refer to the list of roles below this table to see available roles.
`visible`	Optional	`true` or `false` Use this if you need a columns content but don't need to display the column itself. Example: Column A contains the full link to a ticket in your ticket system. Column B contains the ticket ID. You want to use the ticket ID as a label for the link, turning the long URL into a much nicer to read "Ticket 123". This is why you need the content of column B, to assign it as a label for column A. But since the URL is now displayed as the ticket ID, it would be redundant to still display column B. This is why you hide column B with `false`.

There are many different shapes you can use for your columns and the list of possible shapes gets expanded constantly:

Basic types, like: boolean, date, number, string
Currency types that get displayed with two decimal values and their currency symbol (for example $23,45), like: currency (generic currency), eur, gbp, usd
Data types, like: bytes, kilobytes, megabytes
Time types, like: seconds, milliseconds, timespan
The status type : state
Utility types, like:
customUnit
url (will be displayed as a link)

Tip:

Some shapes can be configured.

If a shape is configurable, you can edit how the shape displays data in SquaredUp.

See Configurable shapes

`id`	Used by data streams feeding the aggregate health stream to identify their Id column
`label`	A column containing user-friendly names. Line Graphs use this role to group data into series. so each label will get its own line in the Line Graph.
`link`	A column containing a link that can be used as a drilldown in Status Blocks.
`timestamp`	A column containing a date to use on the X -axis of a Line Graph.
`unitLabel`	A column containing user-friendly labels for data series, e.g. ‘Duration’. Line Graphs can use this role to label the Y-axis.
`value`	A column containing the numeric value you want to use in your visualization.

Custom API Call Example:

{
  "name": "customApiCall",
  "dataSourceConfig": {
    "method": "POST",
    "endpoint": "{{name}}/_search",
    "headers": {},
    "params": {},
    "body": {
      "query": {
        "query_string": {
          "query": "managedEntityDisplayName:\"Microsoft Windows Server 2016 Datacenter\""
        }
      }
    }
  },
  "rowPath": [
    "hits",
    "hits"
  ],
  "matches": "all",
  "metadata": []
}

Parameters:

`name`	Mandatory	The internal name of the Data Stream. Can be used the refer to this Data Stream in a tile's JSON instead of using the Data Stream's internal ID.
`method`	Mandatory	Enter the method (POST or GET)
`endpoint`	Mandatory	Enter the API endpoint you want to call.
`headers`, `params`	Optional	Here you can add headers and parameters to your API call.
`body`	Mandatory or optional	Mandatory when you are using a POST `method` to insert your custom query. Optional when you are using a GET `method`.
`rowPath`	Mandatory	Depends on the data returned from your query. Enter the path to the row you want to use (the row path in the code example above is hits.hits).
`metadata`	Mandatory	Depends on the data returned from your query.

metadata Parameter

Optional, but recommended

The metadata parameters are used to describe columns in order to tell SquaredUp what to do with them. You can do multiple things with the metadata parameters:

Specify how SquaredUp should interpret the columns you return and - to an extent - how their content displayed. You do this by giving each column a shape.
The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link.
Filter out or just hide columns.
Only the columns you define in metadata will be returned in the results. This helps you to filter out columns you don't need. If you need the content of a column but don't want to display it, you can use the visible parameter.
Give columns a nicely readable display name.
Assign a specific role to columns .
The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role value to the column that contains the actual value you want to use in your visualization.

Parameters:

Before you start specifying metadata, leave them empty at first and get all the raw data with your new data stream once.

In order to do this, finish creating your custom data stream without metadata and create a tile with this data stream. The Table visualization will show you all raw data.

Use this information to go back to the data stream configuration and specifying the metadata.

`name`	Mandatory	Enter the name of the column you are referencing here. To find the name of a column, get the data from this data stream once without any metadata. See the tip above for how to do that. You'll see the column name when you hover over the column in the Table.
`displayName`	Optional	Here you can give the column a user-friendly name
`shape`	Recommended	The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link. Note: Please refer to the list of shapes below this table to see available shapes.
`role`	Recommended	The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role `value` to the column that contains the actual value you want to use in your visualization. Note: Please refer to the list of roles below this table to see available roles.
`visible`	Optional	`true` or `false` Use this if you need a columns content but don't need to display the column itself. Example: Column A contains the full link to a ticket in your ticket system. Column B contains the ticket ID. You want to use the ticket ID as a label for the link, turning the long URL into a much nicer to read "Ticket 123". This is why you need the content of column B, to assign it as a label for column A. But since the URL is now displayed as the ticket ID, it would be redundant to still display column B. This is why you hide column B with `false`.

There are many different shapes you can use for your columns and the list of possible shapes gets expanded constantly:

Basic types, like: boolean, date, number, string
Currency types that get displayed with two decimal values and their currency symbol (for example $23,45), like: currency (generic currency), eur, gbp, usd
Data types, like: bytes, kilobytes, megabytes
Time types, like: seconds, milliseconds, timespan
The status type : state
Utility types, like:
customUnit
url (will be displayed as a link)

Tip:

Some shapes can be configured.

If a shape is configurable, you can edit how the shape displays data in SquaredUp.

See Configurable shapes

`id`	Used by data streams feeding the aggregate health stream to identify their Id column
`label`	A column containing user-friendly names. Line Graphs use this role to group data into series. so each label will get its own line in the Line Graph.
`link`	A column containing a link that can be used as a drilldown in Status Blocks.
`timestamp`	A column containing a date to use on the X -axis of a Line Graph.
`unitLabel`	A column containing user-friendly labels for data series, e.g. ‘Duration’. Line Graphs can use this role to label the Y-axis.
`value`	A column containing the numeric value you want to use in your visualization.

Was this article helpful?

Have more questions or facing an issue?

Submit a ticket