# devstats-docker-images

**Repository Path**: cncf/devstats-docker-images

## Basic Information

- **Project Name**: devstats-docker-images
- **Description**: 📈DevStats docker images: minimal (hourly cron job sync), full (provisioning/bootstraping), Grafana (UI endpoint)
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-07-25
- **Last Updated**: 2026-10-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# devstats-docker-images

DevStats docker images: minimal (hourly cron job sync), full (provisioning/bootstraping), Grafana (UI endpoint), Patroni (database), tests.


# Create and test images

Create and remove docker images:

- To create DevStats docker container images and publish them, use: `DOCKER_USER=... ./images/build_images.sh`.
- To drop local DevStats docker container images use: `DOCKER_USER=... ./images/remove_images.sh`.
- You can add various flags to skip specific images like `SKIP_FULL=1`, `SKIP_MIN=1`, `SKIP_TEST=1`, `SKIP_PROD=1` see `images/build_images.sh`.
- You can skip publishing to docker hub via `SKIP_PUSH=1`.
- `IMAGE_TAG=xyz` builds and pushes every image as `name:xyz` instead of the default tag (`latest`) - e.g. `IMAGE_TAG=try SKIP_PROD=1 SKIP_GRAFANA=1 SKIP_PATRONI=1 SKIP_STATIC_NOBINS=1 SKIP_TESTS=1 DOCKER_USER=... ./images/build_images.sh` to try new images in the test namespace (`kubectl -n devstats-test set image cronjob/devstats-riff devstats-riff=DOCKER_USER/devstats-minimal-test:go127`) without touching the tags the production CronJobs pull.

Image contents (Go images, `images/Dockerfile.*`):

- Binaries are built with `golang:1.27` (statically linked, stripped) and installed under `/usr/bin`; the full images also have `/etc/gha2db` (metrics, SQLs, data files) and `/go/src/devstats` (the `devstats` data repository + `devstats-docker-images` scripts, `patch.sh`-ed for the test/prod hostname, with symlinks to the binaries).
- No source code is shipped: after the build the full images remove everything that came with the `devstatscode` clone (Go/Rust sources, `.git`, `go.mod`/`go.sum`, developer scripts) unless the `devstats`/`devstats-docker-images` tarballs provide a file at the same path (`cron/`, `devel/`, `git/` scripts, `projects.yaml`, ...), so `/go/src/devstats` is exactly the same file set in the Go and `-rust` images.
- The full images use the official `postgres:18-alpine` image (psql/pg_dump/pg_restore matching the server version, `docker-entrypoint.sh`) minus server-only leftovers nothing in DevStats uses (no PostgreSQL server ever runs from these images): the LLVM JIT (`libLLVM` ~170 MB, `llvmjit.so`, bitcode), C headers and static libraries. Because layers are additive the trimmed root filesystem is flattened into a fresh image (`FROM scratch` + `COPY --from=pgbase / /`) and the base image's runtime configuration (`ENV`, `VOLUME`, `ENTRYPOINT`, `STOPSIGNAL`, `EXPOSE`, `CMD`) is re-declared in the Dockerfile - keep it in sync with `docker image inspect postgres:18-alpine` when bumping the base image. Result: `devstats-test`/`devstats-prod` ~600 MB instead of ~900 MB with identical behaviour.

Shortcuts:

- Build only API images: `./example/build_api.sh`.

Rust images (DevStats binaries from the Rust port in `devstatscode/rust` instead of the Go ones):

- `RUST=1 DOCKER_USER=... SKIP_PATRONI=1 ./images/build_images.sh` builds every image that contains at least one DevStats binary under an alternate name with the `-rust` suffix: `devstats-test-rust`, `devstats-prod-rust`, `devstats-minimal-test-rust`, `devstats-minimal-prod-rust`, `devstats-tests-rust`, `devstats-static-test-rust`, `devstats-static-prod-rust`, `devstats-reports-rust`, `devstats-api-prod-rust`, `devstats-api-test-rust`. The regular (Go) images are not touched and images without DevStats binaries (grafana, patroni, static-cdf/graphql/default, backups-page) are skipped in this mode. All `SKIP_*` flags work as usual.
- The binaries are compiled inside Docker (`images/Dockerfile.rust-bins`, `rust:alpine` builder, static Linux musl executables with the same names/CLI/env as the Go ones) (a local intermediate image `DOCKER_USER/devstats-rust-bins`, never pushed, exported to `./rust-bins/`) and shipped via `grafana-bins.tar`/`api-bins.tar` (same Dockerfiles as the Go images) and `devstats-bins.tar` (`images/Dockerfile.{full,minimal}.{prod,test}.rust`, generated from the Go Dockerfiles by `images/rust_dockerfile.sh` so project lists stay single-sourced; `images/Makefile.{full,minimal}.rust` lay them out). The `-rust` images have the same runtime content as the Go ones (same base images and packages, `/etc/gha2db`, identical `/go/src/devstats` file set, binaries under the same names, no source code in either) but are smaller: static Rust binaries are less than half the size of the Go ones. Docker with BuildKit (Docker 23+) is required; the cargo registry and build directory are BuildKit cache mounts, so rebuilds are incremental.
- `devstats-tests-rust` runs `rust/test.sh` (rustfmt, clippy, unit tests and the Go⇄Rust compatibility tests against a local PostgreSQL 18): `docker run -ti DOCKER_USER/devstats-tests-rust`.
- `RUST=1 DOCKER_USER=... ./images/remove_images.sh` removes the `-rust` images.
- `DOCKER_USER=... ./images/build_rust_bins.sh [dir]` only builds the static Rust binaries (into `./rust-bins/` by default) - `build_images.sh` uses it in `RUST=1` mode and `devstats/devel/create_grafana_shared_data.sh` takes the `replacer`, `sqlitedb` and `runq` shipped in the shared Grafana data from it, so grafana pods run the same binaries as the images.
- Since 2026-09-12 the `-rust` images are the ones deployed (all CronJobs and the API in `devstats-test` and `devstats-prod`, `devstats-helm` defaults); the Go images are still built and pushed under their original names as the rollback path.
- The `full`/`minimal` Dockerfiles (Go and Rust) set `git config --system maintenance.autoDetach false` and `gc.autoDetach false`: git 2.46+ otherwise forks a detached `git maintenance run --auto` after every `fetch`/`pull`/`clone`, and because the `devstats` binary is PID 1 in the sync pods and does not reap orphaned processes, every such run stayed behind as a `[git]` zombie until the pod ended (1-2 per repository per run, harmless but noisy: `pids.max` is in the hundreds of thousands). With the setting the auto maintenance still runs, but in the foreground.


# Testing images

Using kubernetes:

- To test sync DevStats image (`devstats-minimal-test`, `devstats-minimal-prod` containers): `DOCKER_USER=... ./images/test_image_kubernetes.sh devstats-minimal-test`.
- To test provisioning DevStats image (`devstats-test`, `devstats-prod` containers): `DOCKER_USER=... ./images/test_image_kubernetes.sh devstats-prod`.
- To test Grafana DevStats image (`devstats-grafana` container): `DOCKER_USER=... ./images/test_image_kubernetes.sh devstats-grafana`.
- To execute test coverage: `./tests/test_from_docker.sh`.
- Making pushes to GitHb triggers automatic Travis CI builds.

Using docker:

- Replace `./images/test_image_kubernetes.sh` with `./images/test_image_docker.sh`.
- To execute test coverage: `./tests/test_from_k8s.sh`.


# Reports

- To run `devstats-reports` image using docker: `DOCKER_USER=... ./images/devstats_reports_docker.sh`.

# Backups

`devstats-helm/backups.sh` backs up every selected project sequentially (artificial-event archive, then full dump), followed by the shared `affiliations` database if present. Dump age and randomness no longer select projects; `NOAGE` is unnecessary. Existing schedules, `ONLY` selection and replica endpoint remain unchanged.

By default it waits indefinitely for provisioning, project syncs (even those older than nine hours), and shared affiliations imports. `BACKUP_WAIT_STEP` defaults to 60 seconds; a positive `BACKUP_WAIT_MAX` opts into a wait limit. A failed activity check refuses that backup, and failed or skipped backups make the run exit nonzero after processing the remaining databases. Stale flags therefore require investigation rather than automatic expiry. These checks do not lock out new syncs for the duration of a dump.

Full dumps use fast compression (`-Z1`) and replace the previous `.dump` only after successful completion; allow disk space for one additional in-progress dump. Backup connections disable parallel query workers and use 16MB `work_mem`; the existing `FASTXZ=1` chart setting keeps artificial-archive compression light. Backup-only Helm defaults in `cncf/devstats-helm` limit the client pod to 1 CPU/2Gi (request 100m/256Mi); these are not server-side CPU/I/O limits.

Isolated orchestration tests (no real database or `/root` writes): `python3 -m unittest discover -s tests -p test_backups.py`.


# Adding new projects

See `cncf/devstats-helm`:`ADDING_NEW_PROJECTS.md` or `NEW_PROJECT.md` (this is only for updating docker images and adding to bare kubernetes deployment).