Object indexing

Object indexing imports objects from your integration into the SquaredUp graph, making them available throughout the platform for searching, scoping, drilldowns, and RollUp-style dashboards.

How object indexing works

Each indexing definition contains one or more import steps. Each step:

1. Calls a data stream
2. Reads the returned rows
3. Maps those rows into graph objects
4. Imports those objects into SquaredUp

This allows you to transform almost any external dataset into indexed objects that can participate in the SquaredUp object graph.

Typical objects include hosts, virtual machines, applications, services, Kubernetes resources, and users.

Index definition location

Index definitions live in the indexDefinitions/ folder of your plugin.

By convention this file is named default.json, but any filename works: SquaredUp loads every .json file in the folder, and each becomes a separate import definition.

Example index definition

{
  "steps": [
    {
      "name": "Import vehicles",
      "dataStream": {
        "name": "vehicles",
        "id": "datastream-xxxxxxxxxx",
        "config": {}
      },
      "timeframe": "none",
      "objectMapping": {
        "id": "name",
        "name": "name",
        "type": { "value": "starwars-vehicle" },
        "properties": [
          "model",
          "manufacturer",
          "vehicle_class",
          { "vehicleId": "name" },
          { "crewCount": "crew" },
          { "maxSpeed": "max_atmosphering_speed" },
          { "owner": { "value": "Disney" } }
        ]
      }
    }
  ],
  "frequencyMinutes": 720
}

Configure object indexing

Configure object indexing in the Configure indexing window. To open it, click More options

Parameters

Import objects that depend on other objects

Some objects can only be fetched once you know their parent. For example, an API might list datasets per project, so you need the projects before you can ask for their datasets. Add dependsOn to a step to make it wait for other steps, then scope it to the objects those steps imported.

How dependent steps run

A step with no dependsOn is a root step and runs as soon as the import starts. A step that lists other steps in dependsOn waits until every one of them finishes with a status of succeeded or warning. A step can depend on several steps, and chains can be as deep as you need.

The dependent step uses scope.query to select the objects imported by the earlier step, and its data stream then runs for those objects. Reference the scoped object with {{object.rawId}} in the data stream's endpointPath, or with context.objects[0] in a script.

dependsOn controls run order and scope only. It doesn't create relationships between the parent and child objects.

Set up a dependent step

1. Add a step that imports the parent objects, such as projects.
2. Add a second step whose data stream fetches the child objects for one parent, using {{object.rawId}} in its endpointPath.
3. Set scope.query on the second step to select the parent objects, for example g.V().has("sourceType", "Project").
4. Set dependsOn on the second step to the name of the first step.

In this example, the datasets step waits for projects, then calls projects/{{object.rawId}}/datasets once for each imported project:

{
  "steps": [
    {
      "name": "projects",
      "dataStream": {
        "name": "listProjects"
      },
      "timeframe": "none",
      "objectMapping": {
        "id": "id",
        "name": "displayName",
        "type": {
          "value": "Project"
        },
        "properties": [
          "organizationId",
          "studioHost",
          "createdAt"
        ]
      }
    },
    {
      "name": "datasets",
      "dataStream": {
        "name": "datasets"
      },
      "scope": {
        "query": "g.V().has(\"sourceType\", \"Project\")"
      },
      "timeframe": "none",
      "objectMapping": {
        "id": "uid",
        "name": "displayName",
        "type": {
          "value": "Dataset"
        },
        "properties": [
          "datasetId",
          "aclMode",
          "createdAt",
          "createdByUserId",
          "projectId"
        ]
      },
      "optional": true,
      "dependsOn": [
        "projects"
      ]
    }
  ]
}

What happens when a step fails

If a step that another step depends on fails or is cancelled, the dependent step is skipped with the message Skipped: dependency '<name>' did not succeed. The skip cascades, so any steps that depend on the skipped step are skipped too.

Set optional to true on a step whose failure shouldn't stop the import. If an optional step fails, it's recorded as a warning instead of failing the import, and steps that depend on it still run.

Validation rules

SquaredUp checks the steps when the plugin loads, and rejects an index definition that has:

• Two steps with the same name
• A dependsOn entry naming a step that doesn't exist
• A step that depends on itself
• A cycle, such as two steps that depend on each other

Properties

ImportDefinition properties

ImportStepDefinition properties

objectMapping properties

Indexing tips

• Use multiple steps to import different object types, for example one step for hosts and one for services.
• Mark data streams used only for indexing with "visibility": { "type": "hidden" } so they don't appear in the tile editor.
• SquaredUp always builds each object's internal ID by combining its type and id from your objectMapping. Your original, unprefixed id value is kept automatically as a rawId property, and the plugin config ID is separately folded in to guarantee the object is unique across the whole organization. There's no need to add your own raw ID property.

Was this article helpful?