A command-line tool (and Lambda) for collecting Buildkite agent metrics
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
Lachlan Donald 8188dca2af
Merge pull request #87 from philwo/stackdriver-orgs
2 years ago
.buildkite Use a specific buildkite/github-release image 2 years ago
backend Use org-specific metric names 2 years ago
collector Support multiple queue params 2 years ago
lambda Merge branch 'master' into support-multiple-queues 2 years ago
vendor Update dependencies 3 years ago
version Bump version to v5.1.0 2 years ago
.gitignore Move to github.com/aws/aws-lambda-go/lambda 3 years ago
CHANGELOG.md Bump version to v5.1.0 2 years ago
Dockerfile feat: multi-stage build 2 years ago
LICENSE.md Initial migration from buildkite-cloudwatch-metrics-publisher 6 years ago
README.md Fix typo in README.md. 2 years ago
go.mod Add New Relic backend 2 years ago
go.sum Add New Relic backend 2 years ago
main.go Merge branch 'master' into support-multiple-queues 2 years ago

README.md

Buildkite Agent Metrics

A command-line tool for collecting Buildkite agent metrics, focusing on enabling auto-scaling. Currently AWS Cloudwatch, StatsD, Prometheus, Stackdriver and New Relic are supported.

Build status

Note: Formerly known as buildkite-metrics, but now buildkite-agent-metrics to reflect the focus of the tool.

Installing

Either download the latest binary from Github Releases or install with:

go get github.com/buildkite/buildkite-agent-metrics

Running

Several running modes are supported. All of them require an Agent Registration Token, found on the Buildkite Agents page.

Running as a Daemon

The simplest deployment is to run as a long-running daemon that collects metrics across all queues in an organization.

buildkite-agent-metrics -token abc123 -interval 30s

Restrict it to a single queue with -queue if you're scaling a single cluster of agents:

buildkite-agent-metrics -token abc123 -interval 30s -queue my-queue

Restrict it to a multiple queues with -queue:

buildkite-agent-metrics -token abc123 -interval 30s -queue my-queue1 -queue my-queue2

Running as an AWS Lambda

An AWS Lambda bundle is created and published as part of the build process. The lambda will require the cloudwatch:PutMetricData IAM permission.

It's entrypoint is handler, it requires a go1.x environment and respects the following env vars:

  • BUILDKITE_AGENT_TOKEN
  • BUILDKITE_BACKEND
  • BUILDKITE_QUEUE
  • BUILDKITE_QUIET
  • BUILDKITE_CLOUDWATCH_DIMENSIONS
aws lambda create-function \
  --function-name buildkite-agent-metrics \
  --memory 128 \
  --role arn:aws:iam::account-id:role/execution_role \
  --runtime go1.x \
  --zip-file fileb://handler.zip \
  --handler handler

Running as a Container

You can build a docker image for the buildkite-agent-metrics following:

docker build -t buildkite-agent-metrics .

This will create a local docker image named as buildkite-agent-metrics that you can tag and push to your own registry.

You can use the command-line arguments in a docker execution in the same way as described before:

docker run --rm buildkite-agent-metrics -token abc123 -interval 30s -queue my-queue

Backends

By default metrics will be submitted to CloudWatch but the backend can be switched to StatsD or Prometheus using the command-line argument -backend statsd or -backend prometheus respectively.

The Cloudwatch backend supports the following arguments:

  • -cloudwatch-dimensions: A optional custom dimension in the form of Key=Value, Key=Value

The StatsD backend supports the following arguments:

  • -statsd-host HOST: The StatsD host and port (defaults to 127.0.0.1:8125).
  • -statsd-tags: Some StatsD servers like the agent provided by Datadog support tags. If specified, metrics will be tagged by queue otherwise metrics will include the queue name in the metric. Only enable this option if you know your StatsD server supports tags.

The Prometheus backend supports the following arguments

  • -prometheus-addr: The local address to listen on (defaults to :8080).
  • -prometheus-path: The path under prometheus-addr to expose metrics on (defaults to /metrics).

The Stackdriver backend supports the following arguments

  • -stackdriver-projectid: The Google Cloud Platform project to report metrics for.

Upgrading from v2 to v3

  1. The -org argument is no longer needed
  2. The -token argument is now an Agent Registration Token — the same used in the Buildkite Agent configuration file, and found on the Buildkite Agents page.
  3. Build and pipeline metrics have been removed, focusing on agents and jobs by queue for auto–scaling. If you have a compelling reason to gather build or pipeline metrics please continue to use the previous version or open an issue with details.

Development

This tool is built with Go 1.11+ and Go Modules.

You can build and run the binary tool locally with golang installed:

export GO111MODULE=on
go run *.go -token [buildkite agent registration token]

Currently this will publish metrics to Cloudwatch under the custom metric prefix of Buildkite, using AWS credentials from your environment. The machine will require the cloudwatch:PutMetricData IAM permission.

Metrics

The following metrics are gathered when no specific queue is supplied:

Buildkite > (Org) > RunningJobsCount
Buildkite > (Org) > ScheduledJobsCount
Buildkite > (Org) > UnfinishedJobsCount
Buildkite > (Org) > IdleAgentsCount
Buildkite > (Org) > BusyAgentsCount
Buildkite > (Org) > BusyAgentPercentage
Buildkite > (Org) > TotalAgentsCount

Buildkite > (Org, Queue) > RunningJobsCount
Buildkite > (Org, Queue) > ScheduledJobsCount
Buildkite > (Org, Queue) > UnfinishedJobsCount
Buildkite > (Org, Queue) > IdleAgentsCount
Buildkite > (Org, Queue) > BusyAgentsCount
Buildkite > (Org, Queue) > BusyAgentPercentage
Buildkite > (Org, Queue) > TotalAgentsCount

When a queue is specified, only that queue's metrics are published.

License

See LICENSE.md (MIT)