Unlock enhanced CI/CD insights in GitLab for pipeline visibility

As organizations and platform engineers increasingly embrace DevOps and CI/CD pipelines, features such as GitLab Pipelines have emerged as a powerful tool for automating development and delivery processes. But as with many other automation tools, it’s difficult to maintain the performance and visibility of these pipelines. That’s where Dynatrace, and its Software Development Lifecycle (SDLC) Events come into play to provide a full view into your CI/CD pipelines. This blog post shows how GitLab Webhooks and Dynatrace SDLC Events can be used to monitor pipelines and merge requests. But first, let’s explore some of the advantages of monitoring your CI/CD pipelines using Dynatrace.

Maximizing CI/CD efficiency

1. Enhanced observability and release validation

Dynatrace already excels at delivering a comprehensive end-to-end view of your systems and user journeys. Extending this visibility into your CI/CD pipelines offers even greater value. By using Dynatrace, organizations can proactively monitor for potential issues or slowdowns in their delivery processes. Additionally, it enables validation of releases to ensure quality and reliability. This integration empowers teams to detect, confirm, and resolve problems faster by correlating new issues directly to updates deployed by a pipeline, streamlining troubleshooting and minimizing downtime.

2. Improving collaboration across your organization

By surfacing actionable insights and centralized monitoring data, Dynatrace fosters collaboration between development, operations, and business teams. Everyone involved in the software delivery lifecycle can work together more effectively with a single source of truth and a shared understanding of pipeline performance and health.

3. Optimizing resource utilization

Whether you’re using self-hosted or cloud-hosted runners, their resource demands can significantly impact your organization. Inefficient or resource-intensive runners can lead to increased costs and underutilized infrastructure. With Dynatrace, you can monitor runner resource usage, enabling organizations to monitor consumption patterns, identify inefficiencies, and optimize runner performance. This allows you to allocate resources more effectively, reducing costs while ensuring your CI/CD pipelines operate smoothly and efficiently.

Now, on to the implementation.

Implementation process

Disclaimer: This blog post demonstrates pipeline observability for GitLab. The detailed description of this best practice is maintained here. The approach is currently in Preview and will be soon available for Azure DevOps, GitHub, and ArgoCD. If you are interested in testing the preview and providing feedback, please follow this link.

Prerequisites

1. Install Dynatrace Configuration as Code via Monaco.

2. Create an OAuth client with the following permissions.

• Run apps: app-engine:apps:run
• View OpenPipeline configurations: openpipeline:configurations:read
• Edit OpenPipeline configurations: openpipeline:configurations:write
• Create and edit documents: document:documents:write
• View documents: document:documents:read

3. Store the retrieved client ID, secret, and token endpoint as an environment variable.

• export OAUTH_CLIENT_ID=''
• export OAUTH_CLIENT_SECRET=''
• export OAUTH_TOKEN_ENDPOINT='https://sso.dynatrace.com/sso/oauth2/token'

4. Clone the Dynatrace configuration as code sample repository, then go to the gitlab_pipeline_observability directory.

• git clone https://github.com/Dynatrace/dynatrace-configuration-as-code-samples.git
• cd dynatrace-configuration-as-code-samples/gitlab_pipeline_observability

5. Edit the manifest.yaml by exchanging the placeholder with your Dynatrace environment ID.

Configure Dynatrace using Monaco

With the prerequisites set up, we can now use `monaco` to upload the two dashboards and the configurations for our GitLab ingest endpoints in OpenPipeline. If you already have any configurations in your OpenPipeline besides the default settings, start at step 1 to avoid overwriting your configs. If you do not, then you can skip to step 3.

1. Download your OpenPipeline configuration

• monaco download -e --only-openpipeline

2. Merge the content of download__/project/openpipline/events.sdlc.json into the file pipeline_observability/openpipeline/events.sdlc.gitlab.json

3. Run Monaco deploy

• monaco deploy manifest.yaml

To confirm that it worked, you can go to the OpenPipeline app and check out the newly deployed configurations under Events > Software Deveolpment Lifecycle. There should be a new custom ingest source, three dynamic routes, and three pipelines as shown in the images below.

Configure GitLab to send webhook events to Dynatrace

Now that Dynatrace is ready to transfer GitLab webhooks into SDLC events, we need to setup a webhook that allows GitLab to send the data to Dynatrace. To do that we will first need an Access Token from Dynatrace. You can navigate in Dynatrace to Access Tokens, then generate a new token using the following scopes:

• openpipeline.events_sdlc.custom
• openpipeline.events_sdlc

Save that token as we will need to use it in the next steps to configure the webhook.

1. In GitLab, select your group or project and go to Settings > Webhooks

2. Click Add new webhook, use the following settings:

• URL: Please exchange the placeholders with your Dynatrace environment ID and access token, respectively. https://.live.dynatrace.com/platform/ingest/custom/events.sdlc/gitlab
• Select the following triggers:
  • Merge Request Events
  • Job Events
  • Pipeline Events

Add custom header to your webhook with the name Authorization and value Api-Token .

• Click on Add webhook

And just like that, you are all set. Whenever a merge request is created, closed, assigned, etc., or whenever a job is run, completed, queued, etc., the events will be pushed automatically to Dynatrace. And because of the OpenPipelines configured in the step prior, those events will be processed and transferred into SDLC events, which are then stored in Grail.

To test out, have your developers create some merge requests or run some jobs. Then open the GitLab Pipeline Pulse and the GitLab Merge Requests Dashboard to see your information visualized. It will look something like this:

Next steps: Unlocking the potential of GitLab Data in Dynatrace

And with that, we now have our GitLab repository and pipelines monitored by Dynatrace. Visualizations such as the ones above are only just the beginning. The Dashboards are fully customizable with more information that the images show in this blog, but it’s not the end goal. With SDLC events, you can directly influence and enhance your DevOps and DevSecOps processes with Dynatrace Workflows or directly improve your flaky and failed tests resolution with code-level debug snapshot from the Live Debugger, for instance.

Below is an example of a workflow that triggers from our merged merge requests, using the Site Reliability Guardian to validate performance and vulnerabilities, and finally to either create a Jira ticket on failure, or auto merge the changes to the next stage if they pass.

The examples provided were designed to be straightforward and easy to implement, offering a foundation for integrating GitLab data into Dynatrace. However, these are just the beginning. Now that your data is within Dynatrace, the possibilities for analysis and visualization are virtually limitless.

Here are some next steps to consider:

• Visualizing job success trends
• Identifying pipeline performance bottlenecks
• Correlating runner activity with job execution times

Discover the possibilities

With the flexibility of DQL and the powerful visualizations from Dynatrace, you can create customized dashboards and reports that deliver actionable insights tailored to your needs. The only limit is your imagination—experiment with new queries, uncover hidden trends, and continually refine your monitoring setup to maximize the value of your GitLab data. If you are interested in testing and providing feedback to improve upon the preview, go to this community post and share you experiences and any suggestions you may have. And if you want to read more about Dynatrace and DevSecOps, give this blog post a quick read.