Change failure rate

This tile models change failure rate by correlating successful build runs with incident-level bugs. Successful builds represent changes that were eligible to reach production, while incident-priority bugs represent negative, user-impacting outcomes.

By comparing the volume of incidents to the volume of successful changes over the same timeframe, we can surface periods where change is disproportionately causing problems, helping teams identify elevated delivery risk without relying on explicit deployment-to-incident linkage.

We'll utilize two data sources in this tile, then combine the datasets using SQL Analytics:

• Azure DevOps: We'll use the Build Runs data stream to return successful build runs.
• Jira: We'll use the JQL data stream to return incident tickets that have been raised since the deployment. If you use a different tool for tracking incidents (e.g. JSM or ServiceNow), you can easily edit this tile to use a data stream from another data source.

Configuring the tile

Configure the following in the tile editor:

1. SQL Analytics: Enable the toggle.
2. dataset1:
   1. Data Source: Select Azure DevOps.
   2. Data Stream: Select Build Runs.
   3. Objects: Select the projects you want data from.
   4. Parameters > Result: Select Succeeded to return only successful builds.

   5. Timeframe: Select the timeframe you want to track. Note that after adding a monitor or configuring a KPI the Use dashboard timeframe option is disabled.

3. Click Add dataset to add another data source.
4. dataset2:
   1. Data Source: Select Jira Service Management.
   2. Data Stream: Select JQL Query.
   3. Parameters > JQL query: Enter a query to return incidents. For example:
      Copy

      TYPE = 'Bug' and Priority = 'Incident'

   4. Select the same timeframe you selected for dataset1.
   5. SQL > Query: Enter a query such as the following to combine your results to calculate the change failure rate:
      
      Copy

      SELECT
        builds.successful_builds,
        bugs.incident_bugs,
        ROUND(bugs.incident_bugs * 100.0 / NULLIF(builds.successful_builds, 0), 1) AS change_failure_rate
      FROM
        (SELECT COUNT(*) AS successful_builds FROM dataset1) AS builds,
        (SELECT COUNT(*) AS incident_bugs FROM dataset2) AS bugs;

   6. SQL > Columns: Click Edit next to the Change Failure Rate column and configure the following:
      1. Type: select Percent.
      2. Decimal Places: Enter 0.
      3. Click Save.
5. Visualization: Select Gauge.
   1. Mapping > Value: Select Change Failure Rate.
   2. Range > Min: Select Fixed and enter 0.
   3. Range > Max: Select Fixed and enter 100.
6. Click Save.

Adding a monitor

With a healthy baseline change failure rate of around 5-10%, we trigger a warning when failure begins to exceed expected levels, and an error when it reaches a point that indicates elevated deployment risk.

Configure the monitor

Configure the following in the tile editor:

1. Monitoring: Enable the Monitoring toggle.
2. Type: Select Threshold.
3. Value: Select top.
4. Column: Select Change Failure Rate.
5. Conditions:
   1. Error: Enable the toggle, then configure as Greater than, and supply an appropriate value. For our example, we’ll enter 20.
   2. Warning: Enable the toggle, then configure as Greater than, and supply an appropriate value. For our example, we’ll enter 12.
6. Click Save.

Publishing a KPI

To give even more value to this dashboard (that's right, it gets even better!), we can promote key operational metrics to first-class KPIs.

Once published, these KPIs can be selected as a data stream when configuring a tile, making it both easier to track on-call performance over time and to reuse this data across your other dashboards.

Once you've finished, don't forget to finish configuring your Risk Alerts tile!

Add the KPI type

Configure the following on the Settings > KPI > KPI Types page:

1. Click Add KPI type. The Add KPI type window opens.
2. Name: Enter Change failure rate.
3. Click Save.

Configure the tile KPI

Configure the following in the tile editor for each of the KPI types you configured.

1. KPI: Enable the toggle.
2. Type: Select Change failure rate.
3. Click Save.

Was this article helpful?