Authenticating to GitLab Without Hitting Rate Limits

Posted: 2026-10-10

GitLab.com is tightening its rate limits. Free and unauthenticated limits change on 19 October 2026, and the Premium and Ultimate limits follow in January 2027. Unauthenticated traffic drops to 60 requests an hour per IP address.

During the 7 October preview/testing window at work lots of our Terraform jobs started failing with 429s while pulling modules from other GitLab projects.

But isn’t git exempt?

The rate limits docs state that Git over HTTPS requests shouldn’t count toward the 60 per hour unauthenticated limit, and that git has its own, much larger per-IP limit. However, we hit the limits anyway, so I don’t know if I trust that.

Git asks anonymously first

One gotcha we found was that, even with credentials configured, git doesn’t send them on the first request! It asks anonymously, waits for GitLab to answer 401 Unauthorized, and only then retries with credentials. Every module source is a fresh connection, so every module costs one anonymous request on top of the real one.

trace-auth.sh makes this visible:

#!/usr/bin/env bash
# Show which requests git sends to GitLab, and whether each one carries an
# Authorization header. Point it at a private repo: a public one answers the
# anonymous request with 200 and git never needs to authenticate at all.
#
#   ./trace-auth.sh https://gitlab.com/group/private-repo.git
#   ./trace-auth.sh https://gitlab.com/group/private-repo.git -c http.proactiveAuth=basic
set -euo pipefail

repo_url="$1"
git_options=("${@:2}")

GIT_TERMINAL_PROMPT=0 GIT_TRACE_CURL=1 GIT_TRACE_CURL_NO_DATA=1 \
  git "${git_options[@]}" ls-remote "$repo_url" 2>&1 >/dev/null \
  | grep -E 'Send header: (GET|POST|Authorization)|Recv header: HTTP/[0-9.]+ [0-9]{3}' \
  | sed -E 's/^[0-9:.]+ http\.c:[0-9]+ +//'

$ ./trace-auth.sh https://gitlab.com/group/private-repo.git
=> Send header: GET /group/private-repo.git/info/refs?service=git-upload-pack HTTP/1.1
<= Recv header: HTTP/1.1 401 Unauthorized
=> Send header: GET /group/private-repo.git/info/refs?service=git-upload-pack HTTP/1.1
=> Send header: Authorization: Basic <redacted>
<= Recv header: HTTP/1.1 200 OK

$ ./trace-auth.sh https://gitlab.com/group/private-repo.git -c http.proactiveAuth=basic
=> Send header: GET /group/private-repo.git/info/refs?service=git-upload-pack HTTP/1.1
=> Send header: Authorization: Basic <redacted>
<= Recv header: HTTP/1.1 200 OK

A common fix you’ll find for auth in CI is rewriting URLs to https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/ with insteadOf.

---
# The usual CI recipe: rewrite every GitLab URL to one with the job token in
# it. Module sources are often written as SSH (git@ or ssh://), and the job has
# no SSH key, so those forms are rewritten to authenticated HTTPS as well.
#
# This authenticates, but git still sends the anonymous request first. Adding
# `git config --global http.proactiveAuth basic` (git >= 2.46) makes it send
# the URL's credentials up front instead.
terraform-plan:
  image:
    name: hashicorp/terraform:1.13
    entrypoint: [""]
  before_script:
    - authenticated_url="https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/"
    - git config --global --add "url.${authenticated_url}.insteadOf" "https://${CI_SERVER_HOST}/"
    - git config --global --add "url.${authenticated_url}.insteadOf" "git@${CI_SERVER_HOST}:"
    - git config --global --add "url.${authenticated_url}.insteadOf" "ssh://git@${CI_SERVER_HOST}/"
  script:
    - terraform init
    - terraform plan

This sets auth, but doesn’t stop git from doing the unauthenticated try first.

So you can hit (or at least consume a share of) the rate limit first, and not get to do the authenticated request.

This happens because git doesn’t know whether a URL needs auth, or which kind, until the server says so with a 401, so by default it asks first.

Solutions

Ok, so how can you fix it?

Credential helper and proactive auth

Git 2.46 added http.proactiveAuth, which sends credentials on the first request, with a configured credential helper that uses $CI_JOB_TOKEN at runtime:

---
# Needs git >= 2.46 for http.proactiveAuth. The hashicorp/terraform image ships
# a recent enough git; check yours with `git --version`.
terraform-plan:
  image:
    name: hashicorp/terraform:1.13
    entrypoint: [""]
  before_script:
    # Without proactiveAuth git sends every new connection anonymously first and
    # only offers credentials after GitLab answers 401, so each module fetch
    # costs an extra unauthenticated request against the per-IP limit.
    - git config --global http.proactiveAuth basic
    # The helper is single-quoted, so $CI_JOB_TOKEN is expanded each time git
    # asks for credentials rather than being written into ~/.gitconfig.
    # Answering only `get` stops git trying to store or erase through it.
    - >-
      git config --global "credential.https://${CI_SERVER_HOST}.helper"
      '!f() { test "$1" = get && echo username=gitlab-ci-token && echo "password=${CI_JOB_TOKEN}"; }; f'
  script:
    - terraform init
    - terraform plan

Terraform runs git for git:: sources, so the module block itself needs no token:

module "network" {
  # Terraform hands git:: sources to the git binary, so every git config
  # setting in the job (global config file or GIT_CONFIG_* variables) applies.
  source = "git::https://gitlab.com/example-group/terraform-modules/network.git?ref=v1.4.0"
}

(Note: Each module project must allow the calling project under Settings > CI/CD > Job token permissions, or the fetch fails with a 404 that looks like a typo.)

Older git: send the header yourself

On git older than 2.46, http.proactiveAuth is silently ignored and so we’re back to the anonymous request. Debian bookworm ships 2.39, and hashicorp/terraform:1.9.8 ships 2.45, for example. http.extraHeader attaches a header to every request, so you can build the Basic auth header yourself:

---
# For images whose git predates http.proactiveAuth (2.46), e.g. Debian
# bookworm's 2.39. An extraHeader is sent on every request to the matching
# URL, so the first request is already authenticated.
fetch-modules:
  image: python:3.12-slim-bookworm
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends git
    # The header value lives in ~/.gitconfig for the rest of the job. That is
    # acceptable on a throwaway CI container; don't do it on a shared machine.
    - >-
      git config --global "http.https://${CI_SERVER_HOST}/.extraHeader"
      "Authorization: Basic $(printf 'gitlab-ci-token:%s' "$CI_JOB_TOKEN" | base64 -w0)"
  script:
    - git clone "https://${CI_SERVER_HOST}/example-group/terraform-modules/network.git"

Docker-in-Docker: config through env vars

If the job starts Terraform in its own container (for example hashicorp/terraform via docker:dind), then we need to configure git inside the container, not on the host. To run the commands like we have above, we’d need to either modify the image, or override its entrypoint to run the git commands first.

To work around this, we can configure git via environment variables instead!

With git 2.31+, it reads config from GIT_CONFIG_COUNT, GIT_CONFIG_KEY_<n> and GIT_CONFIG_VALUE_<n>, so the settings can be set via --env flags on the docker run command, without modifying the image at all:

---
# When the job runs Terraform in a container of its own, git config from the
# job's before_script doesn't reach it. GIT_CONFIG_COUNT/KEY/VALUE (git >= 2.31)
# carry the same settings in through --env, with nothing mounted or baked in.
terraform-plan:
  image: docker:29
  services:
    - docker:29-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    # The helper is the same one-liner as in the non-dind job. Single quotes
    # leave $CI_JOB_TOKEN for git to expand inside the container, so
    # `--env CI_JOB_TOKEN` (no value, copied from the job) passes the token
    # through. Writing the real value into the helper instead also works. The
    # token then sits in the docker CLI's arguments rather than only its
    # environment, which matters little on a single-use CI container.
    - >-
      docker run --rm
      --volume "$PWD:/workspace" --workdir /workspace
      --env CI_JOB_TOKEN
      --env GIT_CONFIG_COUNT=2
      --env GIT_CONFIG_KEY_0=http.proactiveAuth
      --env GIT_CONFIG_VALUE_0=basic
      --env "GIT_CONFIG_KEY_1=credential.https://${CI_SERVER_HOST}.helper"
      --env 'GIT_CONFIG_VALUE_1=!f() { test "$1" = get &&
      echo username=gitlab-ci-token && echo "password=${CI_JOB_TOKEN}"; }; f'
      hashicorp/terraform:1.13 init

If the container’s git is too old for proactiveAuth, send the header the same way. The header is a fixed string, so the job builds it before starting the container:

---
# The extraHeader fallback for a container whose git predates
# http.proactiveAuth. hashicorp/terraform:1.9.8 ships git 2.45, one release
# short, and silently ignores the setting.
terraform-plan:
  image: docker:29
  services:
    - docker:29-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    # Unlike the credential helper, the header is a fixed string, so it has to
    # be built here in the job. Exporting it and passing `--env NAME` with no
    # value keeps the token out of the docker CLI's arguments. The job's own
    # git ignores the variable, because GIT_CONFIG_COUNT is only set for the
    # container.
    - >-
      export GIT_CONFIG_VALUE_0="Authorization: Basic
      $(printf 'gitlab-ci-token:%s' "$CI_JOB_TOKEN" | base64 -w0)"
    - >-
      docker run --rm
      --volume "$PWD:/workspace" --workdir /workspace
      --env GIT_CONFIG_COUNT=1
      --env "GIT_CONFIG_KEY_0=http.https://${CI_SERVER_HOST}/.extraHeader"
      --env GIT_CONFIG_VALUE_0
      hashicorp/terraform:1.9.8 init

Option: move modules to the registry

Rather than the git workarounds we mentioned so far, a more proper fix could be to use GitLab’s Terraform module registry to house the modules instead.

Modules published to GitLab’s Terraform module registry are fetched by Terraform over the API with its own credentials, TF_TOKEN_<host>, plus you get to use version constraints instead of ?ref= pins.

This needs changes on both sides (producer and consumer):

The module project publishes a package on each release tag and consumers need read access to the project.
Every consumer changes its source from a git:: URL to gitlab.com/<namespace>/<name>/<system> with a version.
A published module that itself calls git:: sources still goes through git for those, so migrate the leaves first.

Publishing, in the module’s project:

---
# Lives in the module's own project. GitLab's Terraform-Module.gitlab-ci.yml
# template does the same with more options; the plain version shows what it
# sends. Module names must be unique within the top-level group.
publish-module:
  image: curlimages/curl:latest
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  variables:
    TERRAFORM_MODULE_NAME: network
    TERRAFORM_MODULE_SYSTEM: aws
  script:
    # The registry wants a bare semver, so strip the tag's leading "v".
    - TERRAFORM_MODULE_VERSION="${CI_COMMIT_TAG#v}"
    - tar -czf /tmp/module.tgz --exclude=./.git .
    - >-
      curl --fail-with-body --location
      --header "JOB-TOKEN: ${CI_JOB_TOKEN}"
      --upload-file /tmp/module.tgz
      "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_MODULE_SYSTEM}/${TERRAFORM_MODULE_VERSION}/file"

Consuming:

---
# Modules published to GitLab's Terraform module registry are fetched by
# Terraform itself over the API, not by git, so none of the git settings above
# are involved. Terraform reads registry credentials from TF_TOKEN_<host>, with
# dots in the hostname written as underscores.
#
#   module "network" {
#     source  = "gitlab.com/example-group/network/aws"
#     version = "~> 1.4"
#   }
terraform-plan:
  image:
    name: hashicorp/terraform:1.13
    entrypoint: [""]
  variables:
    TF_TOKEN_gitlab_com: $CI_JOB_TOKEN
  script:
    - terraform init
    - terraform plan

Other things that help

Cache .terraform/modules between jobs so most pipelines don’t fetch at all. terraform init only downloads modules whose source has changed, so the cache key doesn’t need to track them.

---
# Combine with whichever auth setup above applies; the cache only cuts how
# often it's needed.
#
# `terraform init` records each installed module's source in
# .terraform/modules/modules.json and skips any whose source is unchanged, so a
# stale cache is safe: a bumped ?ref= is fetched again and the rest are reused.
# The key therefore doesn't need to track module sources. The exception is a
# ref that moves, like a branch name, which stays at the cached commit until
# `terraform init -upgrade`.
terraform-plan:
  image:
    name: hashicorp/terraform:1.13
    entrypoint: [""]
  cache:
    key: terraform-modules-$CI_COMMIT_REF_SLUG
    fallback_keys:
      - terraform-modules-$CI_DEFAULT_BRANCH
    paths:
      - .terraform/modules
  script:
    - terraform init
    - terraform plan