# pipeloom

**Repository Path**: Adnois/pipeloom

## Basic Information

- **Project Name**: pipeloom
- **Description**: 自托管 DevOps 平台：流水线编排 · 代码扫描 · 制品晋级 · 部署发布 · 服务器/容器运维。
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 37
- **Forks**: 15
- **Created**: 2026-09-24
- **Last Updated**: 2026-11-01

## Categories & Tags

**Categories**: DevOps

**Tags**: None

## README

<div align="center">

# Pipeloom

**A self-hosted DevOps platform: pipeline orchestration · code scanning · artifact promotion · deployment · server & container ops.**
Kotlin / Spring Boot control plane, Vue 3 console, MySQL as the single source of state — plus a tiny Go probe for the hosts you manage.

![Kotlin](https://img.shields.io/badge/Kotlin-2.3-A97BFF?logo=kotlin&logoColor=white)
![Spring Boot](https://img.shields.io/badge/Spring%20Boot-4.1-6DB33F?logo=springboot&logoColor=white)
![Vue](https://img.shields.io/badge/Vue-3.5-42B883?logo=vue.js&logoColor=white)
![MySQL](https://img.shields.io/badge/MySQL-8.0-4479A1?logo=mysql&logoColor=white)
![License: Private](https://img.shields.io/badge/license-Private-lightgrey)

English | [简体中文](README.zh-CN.md)

</div>

---

## Try it live

- Entry point: <https://robust-personalized-metres-unix.trycloudflare.com>
- Account: `dadmin` / `dadmin123`

The demo runs behind a Cloudflare Quick Tunnel: **the URL changes whenever the tunnel restarts**, so treat this line as the source of truth and update it in place rather than forwarding an old link. The account is a public demo account — anyone with the link can sign in — so don't put real code, real credentials or production secrets on it.

## Why Pipeloom

A typical internal delivery stack is an assembly: a CI server, a SonarQube box, Ansible playbooks, a Portainer tab, a spreadsheet of "who approved this release", and someone's cron script for alerts. Every hand-off between those tools loses context, and nobody can answer "what is running in prod right now, and who approved it".

Pipeloom puts the whole loop in one control plane: the pipeline that pulls the code, the scanners that grade it, the artifact that promotion approval gates, the deploy that ships it, and the ops console you land in when it breaks. The node catalogue, the scan metrics and the credential vault are shared across all of it — a quality gate reads the very numbers the scan node produced, a deploy refuses to start until the artifact is approved for that environment, and every write ends up in one tamper-evident audit trail.

| | Pipeloom | Jenkins | GitLab CI + SonarQube + Ansible | In-house scripts + cron |
|---|:---:|:---:|:---:|:---:|
| Visual pipeline canvas (Stage→Job DAG) | ✅ built-in | plugin | `.gitlab-ci` only | ❌ |
| Code scanning + red-line gates | ✅ 14 engines (built-in rules included) | plugins | separate SonarQube | ❌ |
| Artifact promotion approval (per environment) | ✅ enforced server-side | ❌ | ❌ | ❌ |
| Multi-target deploy (SSH / Compose / Kubernetes) | ✅ built-in | plugin | Ansible | hand-written |
| Server & container ops + web terminal | ✅ built-in | ❌ | Portainer | ❌ |
| Agentless SSH **and** an outbound-only probe | ✅ | ❌ | Ansible only | ❌ |
| Plugin marketplace (hot-swappable nodes, no restart) | ✅ | ✅ | ❌ | ❌ |
| Credential vault (AES-256-GCM, ciphertext only) | ✅ | plugin | CI variables | env files |
| Append-only audit (DB triggers block UPDATE/DELETE) | ✅ | plugin | ❌ | ❌ |
| RBAC (SUPER_ADMIN / TENANT_ADMIN / OPERATOR / DEVELOPER / VIEWER) + scoped API tokens | ✅ | plugin | ✅ | ❌ |
| Multi-tenant isolation (ownership · visibility · grant list) | ✅ enforced server-side | ❌ | partial (groups) | ❌ |
| AI workstations (design · code · test · repo wiki · self-heal tickets) | ✅ optional, fully degradable | plugin | ❌ | ❌ |
| Ops footprint | 1 jar + 1 static site + MySQL | JVM + plugins | 3+ services | whatever you kept |

## Console tour

Screenshots are included in this repo, in the order you'd use them:

- **Overview** — pipeline / server / probe counts, credentials, 24h runs with success rate, trends, recent activity.
![alt text](docs/screenshots/overview.png)
- **Pipelines** — the two-level canvas (stages left→right, jobs stacked inside a stage), the node drawer with dynamic fields, YAML import, per-version config history with rollback.
![alt text](docs/screenshots/pipeline.png)
- **Runs** — stage timeline, per-step status, live job logs (server flushes step logs every 2s, console polls at 2s), reports, artifacts, approval actions, rerun / cancel / force-kill; the "run diagnosis" tab turns *why is it slow / where is it stuck / is it dead* into verdicts, and a failed step shows the real cause with its exception chain unwrapped instead of a bare `ExecutionException`.
![alt text](docs/screenshots/run-detail.png)
- **Repos** — register Git repositories (URL, credential, default branch) and mark them public / private; open one to browse sources by branch and remote ref. The credential comes from the vault, and public repos can leave it empty.
![alt text](docs/screenshots/projects.png)
- **Build agents & probes** — two kinds of build agent: container agents spawn a one-shot container inside the target Docker daemon, host agents run over SSH with a resident toolchain; both bind to pipelines via `agentId` / `agentLabels`. The probe page handles enrolment of the outbound Go binary — kind (`deploy` / `build`), tags, live heartbeat — and the token is shown only on create and on reset.
![alt text](docs/screenshots/build-agents.png)
![alt text](docs/screenshots/agents.png)
- **Credential vault** — six credential kinds in one place; plaintext is neither stored nor returned in lists, revealing it is a separate high-privilege action that lands in the audit trail, and expiry warns 7 days ahead.
![alt text](docs/screenshots/credential-vault.png)
- **Servers & Containers** — SSH and Kubernetes targets, aggregate container view, images / volumes / networks / stacks, `system df` + prune, live stats, web terminals.
![alt text](docs/screenshots/server-and-container.png)
- **Code scanning & quality** — tool matrix filtered by project language, scan rule inventory, quality metrics, red-line rules, promotion approvals; a red line can be a single threshold or a confidence expression folding several metrics into one score.
![alt text](docs/screenshots/artifact-report.png)
![alt text](docs/screenshots/quality-redline.png)
- **Promotion queue** — deploys stopped by a "requires promotion" gate wait here: pending / approved / rejected tabs, how long they have waited, how many times they were blocked, and each row expands into its evidence (change scope and scan verdicts assembled on demand). What gets approved is the **artifact**, not a run.
![alt text](docs/screenshots/promotion-queue.png)
- **AI task ledger** — one requirement's journey to shipping: a delivery funnel (filed → shipped → PR merged, three different denominators), token spend per model, and the three ticket kinds (requirement / self-heal / review); "trigger fix" is a human click, AI never rewrites code on its own.
![alt text](docs/screenshots/ai-tasks.png)
- **Repo wikis** — repo understanding persisted per commit: overall summary, module map, build & test commands (only those that really exist in manifest files — otherwise it says "unrecognised" instead of guessing); an unchanged commit costs zero tokens.
![alt text](docs/screenshots/repo-wikis.png)
- **Enterprise norms** — discipline written for the model: `guidance` goes into the prompt, `denyPatterns` are machine-enforced to refuse shipping; each norm carries a priority and a sharing scope (all tenants / listed tenants / one tenant).
![alt text](docs/screenshots/ai-norms.png)
- **Marketplace** — plugin packages, versions, per-user installs, comments, the built-in plugin development guide, Kotlin/Shell templates and the `plg` scaffold.
![alt text](docs/screenshots/plugin-marketplace.png)
- **Tenants** — the tenant ledger: members, pipelines, this month's usage against its cap, status (healthy / near budget / over budget / disabled), plus per-tenant execution quotas (concurrent runs, daily triggers, build-agent slots, shared build-agent minutes), a "claim ownerless pipelines" flow and a usage export.
![alt text](docs/screenshots/tenants.png)
- **Message centre** — failure, approval and ticket events are stored once per recipient with independent read state; filter by unread, and the header bell only ever shows your own count.
![alt text](docs/screenshots/messages.png)
- **Settings** — users & roles, notification channels, API tokens, audit log, AI provider, account.
![alt text](docs/screenshots/setting.png)

## Feature overview

- **🔐 Security foundation** — argon2id password hashing, RSA-OAEP encrypted password transport at login, CSRF token per session, login lockout (5 attempts / 15 min by default), session cookies with an opt-in `Secure` flag, Bearer API tokens (`pipeline:trigger` scope), RBAC with five roles enforced on the server (`SUPER_ADMIN` for platform governance, `TENANT_ADMIN` for one tenant, plus `OPERATOR` / `DEVELOPER` / `VIEWER`): reads are open to any signed-in user, **every write defaults to super admin / tenant admin / operator / developer** (a bare `@RequireAuth` still keeps a VIEWER out), and the two gates are distinct: `@PlatformOnly` admits only a `SUPER_ADMIN` who belongs to **no** tenant and guards the platform itself (tenant ledger and memberships, API token issuance, AI providers, marketplace registry sync and catalogue writes, reverse-proxy routes, schedule refresh, cross-tenant user operations); `@AdminOnly` is the business tier below it (super admin + tenant admin: revealing credential plaintext, red lines, notification channels, `docker system prune`). Login side: TOTP (with recovery codes) and WebAuthn as second factors, a session list you can prune other sessions from (with device and region from the offline `ip2region` DB), e-mail verification for unrecognised sources plus anomaly alerts — user-level switches where alerts default on and verification defaults off, because "no mail → cannot log in" has to be a deliberate choice rather than an upgrade side effect. Tokens whose scope already matched the path keep working. `AuthContractTest` + `AuthInterceptorRoleTest` + `PipelineVisibilityTest` pin the policy down (a new anonymous endpoint must be registered with a reason, and someone else's private pipeline has to vanish from lists, detail views and run history alike), and the UI mirrors it per role.
- **🗝 Credential vault** — git token / git HTTP / SSH key / SSH password / registry / kubeconfig, encrypted with AES-256-GCM under `PIPELOOM_VAULT_KEY` (base64, 32 bytes) and stored as `nonce ‖ ciphertext ‖ tag`. Plaintext never lands in the DB and never comes back in a GET. Without the key, vault-backed features answer `503 vault_unconfigured` instead of half-working. Credential expiry is scanned hourly, with a 7-day warning.
- **🧩 Pipeline orchestration** — Stage→Job two-level DAG, `needs` dependencies, topological levels where stages in the same level run concurrently (jobs inside a stage run in declaration order unless `job.needs` reorders them, and an upstream that failed or was skipped makes its dependents skip), stage-level and job-level `allowFailure`, stage/job `when` conditions (branch patterns, trigger events, expressions), per-job retries on every node type with exponential backoff (gate nodes — `approval` / `quality_gate` / `concurrency_group` — are never retried), per-job timeouts enforced as a shared budget across retries (see Known boundaries), stage-level approval gates and the standalone `approval` node, per-stage container `image` plus `cachePaths` / `cacheKey` dependency caching, per-stage isolated working directories via `isolatedWorkspace`, build machine binding by `agentId` or `agentLabels`, matrix builds with `matrix` / `matrixExclude` / `matrixInclude` / `matrixFailFast`.
- **🚦 Triggers** — manual, cron schedule (5-field expressions are normalised to Spring's 6-field form), Git webhooks from GitHub / GitLab / Gitea with push · tag · PR/MR · release event parsing, a generic webhook endpoint, pipeline chaining (`call_pipeline`, self-call rejected, outputs propagated back), REST trigger with a Bearer token. Inbound webhooks are HMAC-verified (`X-Pipeloom-Signature` + timestamp, `X-Hub-Signature-256`, or `X-Gitlab-Token`), deduplicated by delivery id for 5 minutes, IP-rate-limited after repeated signature failures, and auto-disabled after 10. Terminal runs are written back as a commit status on GitHub / GitLab / Gitea (best-effort — a failure there never changes the run result), so the PR shows ✓/✗ without opening the platform. Write-back needs a webhook-triggered run (a commit to mark) plus a `git_source` node carrying `repoUrl` and a **git_token** credential — public repos clone anonymously but cannot be written to without a token — and every reason for not writing back is logged as its own named outcome.
- **🏗 Build, artifacts, promotion** — `git_source` clones into the workspace (or into the stage's container) with credentials embedded from the vault, and captures commit, author, file count and per-file diffs; JGit backs repo browsing and remote ref listing. `build_image` / `push_image` tag `pipeloom-<runId>:<commitShort>` and push to a private registry; `archive_artifact` keeps the real bytes of the build output as a downloadable artifact; per-run workspaces are cleaned up when the run reaches a terminal state. Deploy nodes that move artifacts (`deploy_ssh`, `file_distribute`) can require **promotion approval per (artifact, environment)**: approval is on the artifact, not the run, and a missing or incomplete gate **fails closed** while registering a pending request the approver can act on from the run page.
- **🔍 Code scanning & quality red-lines** — the `code_scan` node orchestrates 14 registered engines. The built-in engine (24 source rules across security / defects / style / complexity, plus 17 secret rules) is pure JVM and always available; gitleaks · SpotBugs · Checkstyle · PMD · detekt · ktlint · ESLint · Pylint · cpplint · clang · gometalinter · StyleCop · OcCheck each resolve through a `PATH → Docker image → skip` chain, and the skip is reported instead of silently producing nothing. `semgrep_scan` and `tca_scan` are separate nodes precisely because they need an external runtime (a Semgrep environment, or a reachable TCA server). Output is SARIF plus aggregated issue counts; scan results become run variables (`CODE_SCAN_TOTAL`, `CODE_SCAN_ERRORS`, `CODE_COMPLEXITY_MAX`, `CODE_DUPLICATION_RATIO`, `CODE_LOC_CODE`, …) that the `quality_gate` node evaluates against your red-line rules — fail the stage, or block for approval; a rule can also carry a confidence expression that folds several metrics into one score instead of a single threshold. Reports, per-file drill-down and a JSON export live on the run.
- **⏱ Three things scans used to hide** — ① **where the time went**: per-engine elapsed time and the total (`engineTimings`) are written into the report and shown in the run's source breakdown, so "which tool was slow" is answerable instead of collapsing into one opaque duration. ② **truncation**: the built-in engine scans at most 2000 files per run; it used to stop silently at 2000, producing a report indistinguishable from a full-repo pass. It now sorts by path, takes the first 2000, and says so in the step log ("N files in total, only the first 2000 were scanned, X remain unscanned — this is not 'the repo is clean'"), with the in-process Kotlin path and the shell path selecting the same files. ③ **change attribution**: the report gains a `changeSet` (baseline commit → HEAD, changed files, files with issues, whether it was capped), every issue is tagged `changed`, and `summary.changed*` plus five **new** metrics (`CODE_SCAN_CHANGED_FILES/TOTAL/ERRORS/WARNINGS/INFOS`) count only findings inside the change set. Crucially it does **not** shrink the scan: red lines, trends, LOC and the full-repo verdict are unchanged — attribution is a second dimension layered on top, so a full-repo `<= 0` red line cannot turn green without a fix. A first run, a non-git workspace or an unreachable baseline yields an explicit reason rather than "0 new issues"; `incremental=false` on the node turns attribution off (that is not "incremental scanning", and the log says so).
- **🧪 Dependency and licence scanning** — `dependency_scan` sits beside `code_scan`: it scans dependency lockfiles rather than the sources, dispatching `npm audit` / `pip-audit` / `govulncheck` / OWASP dependency-check / `cargo audit` by ecosystem, plus Go · Gradle · npm · pip licence scanners. Findings are normalised into a CVE list (report type `dependency-scan`) and `DEP_SCAN_TOTAL/CRITICAL/HIGH/MEDIUM/LOW` become run variables for the gate; the `PATH → Docker → skip` chain is the same as `code_scan`.
- **🧹 Scan noise reduction (AI optional)** — 200 findings are first clustered locally into 5–10 groups (by rule + file extension + message prefix, pure Java, no model call), then one representative per cluster is sent to the model to ask "is this cluster a false positive?" — N calls for N clusters, not N calls for N findings. The model only ranks and merges, it **never deletes** findings, and the raw list stays expandable. A cluster judged false-positive can be turned into a suppression rule (`tenant + rule + file pattern + message prefix`, idempotent and audited) in one click; suppressed findings stop appearing in reports, and platform-level suppressions apply to every tenant.
- **🚀 Deployment** — `deploy_ssh` (load image, run container, ports + run args), `deploy_compose`, `file_distribute`, `server_script` for command-style ops, `health_check` with HTTP or command probes, retries and fatal/non-fatal semantics. Kubernetes targets support `image_update` (rolling · recreate · blue-green), `manifest_apply` (server-side apply with an explicit `fieldManager`, cluster-scoped kinds ordered first) and `helm_chart` (`helm upgrade --install` with values).
- **🖥 Server & container ops** — SSH, Kubernetes and local targets plus probe-backed hosts; aggregated container list across hosts, create / inspect / prune, image · volume · network · stack management, `system df` and prune, live stats, service (systemd) start/stop/restart/status behind a command whitelist, host and container web terminals, Kubernetes pod exec, streaming log tails.
- **🛰 Probe (agent)** — a single static `CGO_ENABLED=0` Go binary that dials **out** to `GET /ws/agents/connect?token=…` and never listens on a port. It reports OS · arch · CPU cores · memory · hostname on each heartbeat, executes `exec` / `upload` / `download` tasks with streaming output, and is classified `deploy` or `build` with tags and label-based scheduling. `SshExecutor` routes to the probe when a host is bound to one, and fails loudly when the probe is offline. The server serves the matching binary at `GET /api/agents/binary/{os}/{arch}`.
- **📣 Notifications** — WeCom · DingTalk · Feishu · Slack · Microsoft Teams · generic webhook · email, routed by explicit event subscription (`succeeded`, `failed`, `gate_pending`, `canceled`, `custom`, `concurrency_rejected`, `gate_rejected` — success is opt-in on purpose, failure and approval are opt-out), template rendering with run variables, a per-delivery ledger, one-click **redelivery that re-renders with the current channel config**, and a test-send button. There is also a `notify` node inside the pipeline.
- **📥 In-app inbox** — the same events reach an in-site message centre, and it does **not** depend on any channel being configured: a fresh instance with zero webhooks still notifies people. Fan-out is decided by event semantics rather than channel subscription — the pipeline owner and the run's trigger actor always get a copy, failures additionally reach the tenant admin (scoped to that pipeline's own tenant, not a platform-wide blast), and approval-type events (`gate_pending` / `gate_rejected` / `ticket_open`) reach tenant admins and operators because those are exactly the roles allowed to approve. Every recipient gets **their own row with their own read state** (Alice reading a gate request must not clear Bob's badge), names that are not live enabled accounts (API tokens, deleted owners, Git commit authors) are dropped, one event is capped at 200 copies, a pipeline can name extra recipients (`notifyRecipients`) which are filtered the same way, and the header bell plus `/messages` page render it. Inbox rows are deleted together with their run by the retention sweeper.
- **🧩 Plugin marketplace & SDK** — plugins are `task.json` + a Kotlin/Java fat JAR (or a shell handler script). Installing registers a new node type in `JobHandlerRegistry`, or a new rule in `ScanRuleRegistry`, **without a restart**; uninstalling retires the type so old pipelines fail with an explanation instead of silently succeeding. Installations are per user, versions are checksum-verified and path-traversal-safe, reserved node types are refused, and plugins reach the platform through a one-time run-scoped API token that can only fetch credentials the pipeline itself declares. Five templates ship (four Kotlin plus one shell) together with an in-app development guide and a `plg` scaffold whose local checks share the server's rule implementation.
- **🤖 AI software factory (optional, fully degradable)** — any OpenAI-compatible endpoint (`PIPELOOM_AI_BASE_URL` / `MODEL` / `API_KEY`, so DeepSeek · Moonshot · vLLM work too; the key lives in the vault). Four first-class nodes: `ai_design` (a plan and task breakdown grounded in the **checked-out** repo), `ai_coding` (requirement → edit → guardrail → push and open a PR/MR), `ai_test` (three machine checks defuse "write a test that is always green": the existing suite must be green before it starts, it may only add files, and green must come from re-running the suite itself) and `ai_wiki` (repo wiki cached per commit — the same `source_commit` means zero tokens). They are real `JobHandler`s, so they inherit build-machine / container routing, the job time budget, retries and audit; on an instance without AI they fail loudly instead of succeeding as a no-op. `ai_coding` also runs a **machine-judged** secret red line before shipping: a hit stops it at "no commit, no push, no PR" — no exception makes it acceptable to write credentials into repo history.
- **🔁 Tickets: self-heal and review loops (filing is automatic, starting work is a decision)** — `ai_tasks` holds three kinds in one ledger: human-filed requirements, self-heal tickets opened for failed runs (failure attribution + log tail + repro context) and review tickets opened when a PR asks for changes (they reuse the original branch, so a push updates the existing PR). Both are produced by pollers reading the fact tables rather than by hooks on the terminal hot path, and a closed or merged PR retires its untriggered review ticket. **Tickets are filed automatically; code is not changed automatically** — a human clicks "trigger fix" before `ai_coding` starts, pipelines without `selfHeal` never call the API, and a run triggered by `ai_task` never files another ticket (no self-amplification). A policy layer (`ai_norms`, enterprise norms) feeds `guidance` into the prompt and lets `denyPatterns` be machine-rejected at ship time, with every change audited.
- **🧠 The rest of the AI surface** — pipeline generation from a repo, run diagnosis, fix suggestions, and the ops assistant (natural language → shell, command explanation, compose generation, container diagnosis). Unconfigured or unreachable means `available=false` and the UI degrades quietly — the CI/CD path never depends on AI.
- **🏷 Multi-tenancy and resource grants** — a tenant ledger with memberships (super admin / tenant admin / operator / developer / viewer), per-tenant execution quotas (concurrent runs, daily runs, probe slots on one build agent, shared build-agent minutes per day), and platform assets that can be **granted to named tenants** instead of the old all-or-nothing switch (`share_mode = all | listed` plus a grant list carrying grantor, expiry and revocation history, across servers / build agents / probes / credentials / repos / custom nodes / enterprise norms). Three layers decide: **ownership** (`tenant_id`) → **visibility** (`public` / `private`, where private means owner + super admin + the tenant admin of the same tenant) → **grant list**. What the console hides, the runtime also refuses to reference (a run is judged by the `tenant_id` snapshot on its own row, and fails closed when it cannot tell).
- **🧹 Housekeeping & platform** — MySQL 8 + Flyway with MyBatis-Plus, run status changes and cron fires persisted as an event stream (`pipeline_run_events` / `cron_trigger_log`), run retention sweeper (default 90 days), two-layer crash recovery (`StaleRunRecoveryService` finalizes orphaned `running` / `queued` / `canceling` runs on boot and records the last event as the reason, and a 2-minute watchdog terminalizes any run whose step heartbeat stopped for 3 hours), gate timeout enforcement, expired slot cleanup, i18n console (简体中文 / English), `/api/health`, Actuator `health,info,prometheus,metrics`, and Swagger UI at `/swagger-ui.html`.
- **🩺 Run diagnosis & business metrics** — the "运行诊断" tab on a run answers *why is it slow / where is it stuck / is it dead*: queue time separated from execution time, heartbeat silence as a liveness verdict, exhausted per-job time budget, who the gate is waiting for, the first failed step with its last log line and whether `allowFailure` tolerates it, the bottleneck step's share of total time, and platform interrupt codes translated into what they mean. Every threshold lives in one server-side place (`RunDiagnosisService`); the UI only renders. `/actuator/prometheus` now carries pipeline business metrics next to the framework's own JVM/HTTP series: terminal counts by status and trigger, queue and duration distributions, retries, budget exhaustion, `when`/`needs` skips, gate approvals and rejections, matrix fan-out sizes, Git status outcomes, promotion blocks, and an active-runs gauge. Labels are always finite sets from code — never `pipelineId`, `runId`, usernames or step names, because those belong in the per-run diagnosis endpoint.
- **⚖️ Concurrency control** — pipeline-level mutual exclusion groups (`deploy-prod`, `deploy-${branch}` …) that reject a conflicting trigger with a `409` naming the occupying run, and stage-level resource slots leased by the `concurrency_group` node with expiry sweeping.

> Security is non-negotiable: secrets are ciphertext at rest, commands are whitelisted or arrayified, webhooks are signed, sessions and tokens are scoped, and audit rows cannot be updated or deleted.

## Install / deploy

Requirements: **JDK 25 for the server toolchain** (Gradle downloads it if absent; the plugin SDK targets 17 so plugins stay portable), **MySQL 8.0+**, **Node.js 22+** for the console, and Docker only where you actually use it — container-isolated builds, image deploys and container ops.

### ① Docker Compose (recommended for self-hosting)

```bash
git clone https://gitee.com/Adnois/pipeloom.git && cd pipeloom
cp .env.example .env            # set DB / admin / vault secrets before the first up
docker compose up -d --build    # mysql + server + web
open http://localhost:8092      # login: admin / $PIPELOOM_ADMIN_PASSWORD
```

The `web` service is nginx: static console on port `8092`, `/api/` proxied to `server:8888`, `/ws/` proxied with an upgrade header and a 24h read timeout. Data lives in the `mysql-data`, `artifacts` and `plugins` named volumes. The server image runs as UID `10001` (non-root), aligned with the Kubernetes manifests.

### ② Kubernetes

```bash
kubectl apply -f k8s/namespace.yaml
# edit k8s/secret.yaml first — DB password, initial admin password, vault key
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/mysql.yaml
kubectl apply -f k8s/server.yaml   # PVCs: 20Gi artifacts, 5Gi plugins
kubectl apply -f k8s/web.yaml
kubectl apply -f k8s/ingress.yaml  # set your host, enable the TLS block
```

`k8s/server.yaml` pins `SERVER_PORT=8888` on purpose: Kubernetes injects `SERVER_PORT=tcp://…` for the Service named `server`, which Spring Boot's relaxed binding would otherwise read as `server.port` and fail to start. Liveness and readiness probe `/api/health`. Keep `replicas: 1` — cron scheduling and in-flight runs are driven from the process (see [Boundaries](#known-boundaries-honest-list)).

### ③ From source (development)

```bash
# 1. database
mysql -u root -p -e "CREATE DATABASE pipeloom CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"

# 2. backend — http://localhost:8888, Flyway migrates on boot
cd server && ./gradlew bootRun

# 3. console — http://localhost:5173, proxies /api and /ws to :8888
cd web && npm ci && npm run dev
```

Secrets come from environment variables, or from `server/src/main/resources/application-local.yml` (copy `application-local.yml.example`, it is git-ignored). Produce artifacts with `./gradlew bootJar` (→ `server/target/pipeloom-server.jar`) and `npm run build` (→ `web/dist`).

The probe is a separate Go module:

```bash
cd agent && make                 # pipeloom-agent-linux-amd64 / -linux-arm64 / -darwin-amd64 / -darwin-arm64
# Console → Agents → New agent → copy the connect command
./pipeloom-agent-darwin-arm64 -server=http://localhost:8888 -token=<token> -v
```

### Configuration (environment variables)

`.env` really only needs three values — the DB password, the initial admin password and the vault key — plus `PIPELOOM_API_URL` once you serve it on a real host. Everything else has a usable default.

**Core**

| Variable | Description | Default |
|---|---|---|
| `PIPELOOM_ADMIN_PASSWORD` | Initial admin password, **used only on first launch** (change it later in account settings). Minimum 8 chars. No default — the project's own rule for password properties is fail fast rather than shipping an instance nobody can log into | none (required on first launch) |
| `PIPELOOM_ADMIN_USERNAME` | Admin username created on first launch | `admin` |
| `PIPELOOM_VAULT_KEY` | Base64-encoded 32-byte vault master key. Unset ⇒ credentials, terminals, K8s exec, AI config and all AI features are unavailable (`503 vault_unconfigured`) | none |
| `PIPELOOM_API_URL` | Externally reachable base URL — used in notification links and webhook `curl` examples | `http://localhost:8888` |
| `PIPELOOM_AUTH_COOKIE_SECURE` | Set `true` once you serve over HTTPS | `false` |
| `SERVER_PORT` | HTTP port | `8888` |

**Database**

| Variable | Description | Default |
|---|---|---|
| `PIPELOOM_DB_HOST` | MySQL host | `localhost` |
| `PIPELOOM_DB_USERNAME` | MySQL user | `root` |
| `PIPELOOM_DB_PASSWORD` | MySQL password. Compose reads it from `.env`, K8s from the Secret — always set it explicitly; note `application.yml` still carries a local-development fallback that production must never rely on | env var (`.env` for Compose) |

**Runs, plugins & retention**

| Variable | Description | Default |
|---|---|---|
| `PIPELOOM_RETENTION_RUNS_MAX_AGE_DAYS` | Days a terminal run is kept before the sweeper removes it with its logs, artifacts, reports, notification rows and in-app copies | `90` |
| `PIPELOOM_PLUGIN_STORAGE` | Directory for uploaded plugin packages | `./data/plugins` |
| `PIPELOOM_PLUGIN_EXEC_TIMEOUT_SECONDS` | Default plugin execution ceiling (a node may override it, clamped 1s–24h) | `3600` |
| `pipeloom.agent-binary-dir` | Where the server looks for probe binaries to serve | `./agent` |
| `pipeloom.server.local-bootstrap.enabled` | Auto-register the host it runs on as a `local` target | `true` |
| `pipeloom.security.ws.allowed-origins` | WebSocket origin allow-list (CSWSH guard) — add your console origin when you deploy | `localhost` / `127.0.0.1` patterns |

**AI (optional)**

| Variable | Description | Default |
|---|---|---|
| `PIPELOOM_AI_BASE_URL` | OpenAI-compatible base URL | `https://api.openai.com/v1` |
| `PIPELOOM_AI_API_KEY` | API key; empty ⇒ AI disabled (startup is never blocked) | empty |
| `PIPELOOM_AI_MODEL` | Model name | `gpt-4o-mini` |
| `PIPELOOM_AI_TIMEOUT_SECONDS` | Request timeout | `120` |

## Pipeline reference

Node types available on the canvas, and who executes them:

| Category | Node | Executor |
|---|---|---|
| Source | `trigger` | engine (fixed first stage: events, cron, concurrency) |
| Source | `git_source` | engine (`git clone` + commit/diff capture); JGit for repo browsing |
| Build | `build_image`, `push_image`, `archive_artifact` | engine (Docker / registry credentials from the vault) |
| Build | `script`, `custom` | engine (container or local shell, default image `alpine`, `commands` interpolated) |
| Build | `test` | **platform-reserved placeholder** — no-op unless a plugin provides it (“no-op” badge on the step) |
| Deploy | `deploy_ssh`, `deploy_compose`, `file_distribute`, `server_script` | built-in handlers (SSH or probe transport) |
| Deploy | `deploy_k8s` | built-in handler (fabric8: image rollout / manifests / Helm) |
| Quality | `code_scan`, `semgrep_scan`, `tca_scan`, `dependency_scan`, `quality_gate`, `health_check` | built-in handlers |
| AI | `ai_design`, `ai_coding`, `ai_test`, `ai_wiki` | built-in handlers (fail loudly when AI is unconfigured) |
| Flow | `approval`, `concurrency_group`, `call_pipeline` | built-in handlers |
| Notify | `notify`, `webhook` | built-in handler (`webhook` calls an external HTTP service, sync or async callback) |

`test` is deliberately kept as a reserved type: it exists so older pipelines do not turn red overnight, but it is a **no-op** (beyond archiving whatever `artifactPath` names) unless a plugin provides it. The canvas and the save-time validator say so instead of letting you discover it mid-run — and the run page badges a succeeded placeholder step as 「空操作」 (no-op), so a green check never claims work that did not happen. `script` and `custom` are not placeholders: they really run `commands` inside a container (default `alpine`, overridable per stage or job) or in the local shell.

Built-in variables available to commands, env values and node config (`${VAR}` and `${VAR:-default}`): `RUN_ID`, `PIPELINE_ID`, `PIPELINE_NAME`, `BRANCH`, `COMMIT_SHA`, `COMMIT_SHORT`, `TRIGGER_TYPE`, `TRIGGER_ACTOR`, `STAGE_NAME`, `JOB_NAME`, `WORKSPACE`, plus anything a node publishes through run variables (artifact names, scan metrics, matrix axes, sub-pipeline outputs).

## Plugin marketplace in short

```json
{
  "name": "hello-world",
  "description": "Hello World 示例插件",
  "nodeType": "hello_world",
  "version": "1.0.0",
  "category": "custom",
  "execution": { "language": "java", "packagePath": "build/libs/hello-world-1.0.0.jar" },
  "fields": [{ "key": "message", "label": "消息内容", "kind": "text" }],
  "outputs": [{ "name": "RESULT", "desc": "输出结果" }],
  "defaultConfig": { "message": "Hello, Pipeloom!" }
}
```

```bash
gradle jar
zip hello-plugin.zip task.json build/libs/hello-world-1.0.0.jar
# Console → Plugin marketplace → Upload package (drag & drop works) → Install
```

A plugin implements `PluginTask<P : PluginParam>` from `server/plugin-sdk`; the platform injects config as `PLG_*` environment variables plus `$PLG_INPUT_FILE`, and reads back `$PLG_OUTPUTS_FILE` (variables), `$PLG_REPORT_FILE` (structured report) and `$PLG_API_URL` + `$PLG_API_TOKEN` (a one-time, run-scoped token that is revoked when the run reaches a terminal state). Templates: `tpl-kotlin-basic`, `tpl-kotlin-deploy`, `tpl-kotlin-notify`, `tpl-kotlin-scan`.

A plugin is just a `java -jar` process driven by `PLG_*` env vars, so it can be developed and iterated on **without the platform**: point `PLG_INPUT_FILE` at a JSON file, read results from `PLG_OUTPUTS_FILE` / `PLG_REPORT_FILE`, and use `PLG_CREDENTIALS_FILE` to fake credentials (honoured only while `PLG_API_URL` is unset, so a pipeline variable can never redirect a plugin to an arbitrary file on the build host). Exit codes match the platform's verdict: 0 success, 1 failure, 2 error. Each template README ships the exact command, and every template now carries a runnable unit test suite (`gradle test`) that drives `execute()` through `ctx.result` assertions.

## Tech stack

- **Backend** — Kotlin 2.3 · Spring Boot 4.1 (web, websocket, mail, flyway, actuator) · MyBatis-Plus 3.5 · MySQL 8 + Flyway · BouncyCastle · sshj (SSH) · fabric8 `kubernetes-client` · Eclipse JGit · Jackson 3 (`tools.jackson`) · springdoc OpenAPI · Micrometer/Prometheus · ArchUnit for layering guards · ktlint + detekt in the build.
- **Console** — Vue 3.5 `<script setup>` · TypeScript · Vite 8 (Rolldown) · naive-ui · vue-router + Pinia · vue-i18n (zh-CN / en) · Monaco + CodeMirror · xterm.js for terminals · Tabler icons.
- **Probe** — Go 1.23 + `gorilla/websocket`, `CGO_ENABLED=0`, four `GOOS/GOARCH` targets from one `Makefile`.
- **Runtime** — JDK 25 toolchain (virtual threads for the engine), nginx for the static console, eclipse-temurin for the server image, MySQL 8 for state.

## Architecture

```
pipeloom/
├── server/                         # Kotlin / Spring Boot control plane (~560 files)
│   ├── src/main/kotlin/com/pipeloom/server/
│   │   ├── controller/             # 49 REST controllers: auth · account · project · pipeline
│   │   │                           #   server · credential · notification · marketplace · agent · ai
│   │   │                           #   audit · dashboard · proxy · system
│   │   ├── service/                # domain services, one package per aggregate
│   │   │   ├── pipeline/          # spec, runs, DAG engine, artifacts, promotion, quality, webhooks
│   │   │   │   └── handler/        # JobHandler SPI + ScannerRegistry + ScanRuleRegistry
│   │   │   ├── marketplace/        # plugin lifecycle, runtime API, custom nodes
│   │   │   ├── server/             # SSH targets, containers, KubeClientService, ops
│   │   │   ├── notification/       # channels, senders, delivery ledger, templates
│   │   │   ├── credential/         # VaultCrypto (AES-256-GCM) + vault CRUD
│   │   │   ├── agent/              # probe registry + connection manager
│   │   │   └── auth/ · account/ · audit/ · ai/ · dashboard/ · project/
│   │   ├── components/             # SshExecutor · AgentExecutor · CommandWhitelist · CSRF
│   │   ├── websocket/              # SSH terminal · container terminal · kube exec · log tail · probe
│   │   ├── schedule/               # retention · gate timeout · slot sweep · stale-run recovery · expiry
│   │   ├── interceptor/ · aspect/  # @RequireAuth · @Audit · unified error envelope
│   │   └── runner/                 # admin bootstrap · local server bootstrap
│   ├── src/main/resources/
│   │   ├── application.yml
│   │   ├── db/migration/           # V0002 … V0167
│   │   ├── builtin-plugins/        # code-scan orchestrator + per-engine scripts
│   │   └── plugin-templates/       # five plugin templates (4 Kotlin + 1 shell)
│   └── plugin-sdk/                 # Kotlin/Java 17: PluginTask / PluginContext / PluginParam / PluginResult
├── web/                            # Vue 3 console (40 views, 66 components, 34 API modules)
├── agent/                          # Go probe: outbound WebSocket, exec/upload/download
├── k8s/                            # namespace · secret · mysql · server · web · ingress
├── docker-compose.yml              # mysql + server + web
└── .github/workflows/ci.yml        # compile · ktlint · detekt · test · vue-tsc · build
```

The dependency rule is `controller → service → mapper/entity`, enforced by ArchUnit tests; HTTP concerns never leak into services, and services never know what a form looks like. The engine (`PipelineRunService` + `RunDispatcher` + `JobHandlerRegistry`) is the only place that decides what a node does — node types are dispatched through the registry, so a marketplace plugin is executed by exactly the same path as a built-in handler.

## Development

```bash
cd server && ./gradlew compileKotlin      # build
cd server && ./gradlew test               # unit + ArchUnit layer tests
cd server && ./gradlew ktlintCheck detekt # lint (CI runs both)
cd web    && npm run typecheck            # vue-tsc --noEmit
cd web    && npm run build                # production console
cd agent  && make                          # probe binaries
```

Migrations live in `server/src/main/resources/db/migration/` and are named `V<NNNN>__<description>.sql`; Flyway runs with `baseline-on-migrate` and `out-of-order`, so add the next number rather than editing history. Local config belongs in `server/src/main/resources/application-local.yml` (git-ignored), not in `application.yml`. Note that Gradle output lands in `server/target/` rather than `server/build/`, and the current `.gitignore` has one merged line (`server/target` + `logs/`), so `server/target/` is not actually ignored — don't commit it.

## Known boundaries (honest list)

- **Single instance.** Cron scheduling and in-flight run state are process-local; run one server replica. Crashes are recovered explicitly: `StaleRunRecoveryService` marks orphaned `running` / `queued` / `canceling` runs as failed with an `interrupt_reason` on boot, a 2-minute watchdog does the same for any step whose heartbeat is older than 3 hours, while `waiting` runs (approval gates) are preserved.
- **`matrixMaxParallel` is not honoured, and the reason is structural rather than "not scheduled yet".** Combinations run in declaration order, and both the save-time `matrix_parallel_blocked` warning and the run log quote the same reason code from `ParallelSafety`. The blocker is the **working directory**: `_WORKSPACE_DIR` is a run-scoped variable pinned by whichever stage creates it first, and every later stage shares that path — a container build agent shares it too, since it mounts `-v $workspace:$workspace`. So "split the combinations into sibling stages" is not a workaround: it trades an explicit limitation for silent artifact clobbering. The engine names those cases at save time with `parallel_stages_share_workspace` (same needs level, same execution target, both read/write the workspace). Half of that has landed: **`stage.isolatedWorkspace` gives a stage its own working directory** (nested under the same per-run base, reclaimed per execution target, and re-reclaimed after a restart via a run-variable ledger), so sibling stages can now run concurrently without clobbering each other. What is still missing is *combo-grained* isolation plus gate-wait state that counts per pending target, so matrix combinations still run sequentially. `matrixFailFast` is honoured.
- **Jobs inside a stage run sequentially.** Concurrency exists at stage level (same topological level) and across runs.
- **`job.timeoutMinutes` is a per-job budget, enforced at call granularity.** One deadline is created when the job starts and is shared by all retries; every command or remote call may only claim the time that is left, and once the budget is spent nothing new is started (a `call_pipeline` node stops waiting and cancels its child run). It cannot cut an already-blocked call to the second — that is bounded by the call's own timeout. Gate nodes (`approval` / `quality_gate` / `concurrency_group`) are exempt: their waits are engine control flow and are governed by `gateTimeoutMinutes` instead.
- **File transfers are bounded, but by a fixed default.** `SshExecutor.uploadFile` / `downloadToFile` now take a `timeoutSeconds` (default 30 minutes) and force-disconnect the SSH connection when the deadline passes, so a stalled transfer fails the job with a named reason instead of blocking forever. `deploy_ssh`'s image upload claims the job's remaining budget (`ctx.budget`); other call sites use the default ceiling rather than the job deadline.
- **Retries are not idempotent replays.** `retryCount` now applies to every node type (it previously only worked for handler-backed nodes), but retrying `call_pipeline` really does start another child pipeline run — a new run id, new notifications, and a repeated deployment if the child pipeline deploys. The validator flags this at save time.
- **The inbox is bounded by what the run record actually knows.** Recipients are resolved from `pipelines.owner_username` and `pipeline_runs.trigger_actor`, so a pipeline with no owner that was triggered by an API token reaches only the role group its event names — that is correct behaviour, not a bug, but it means "I got nothing" is often "nobody owned this". Names that are not live enabled accounts are dropped rather than queued into an unreadable black hole. One event writes at most 200 copies and logs a named warning when it truncates. Inbox rows are deleted with their run by the retention sweeper: there is no separate inbox retention window, so a message older than the retention period is gone for everyone.
- **Resource-level isolation has two layers: owner and tenant.** `pipelines` carries `owner_username` / `visibility`: `public` (the default) is visible to **members of the same tenant**, `private` only to its owner and administrators (super admin / the tenant admin of the same tenant), and a private pipeline's **runs disappear from global run history too**. Anything you may not see answers `404`, never `403`, so its existence is not disclosed. **A tenant-constrained member sees only pipelines whose `tenant_id` equals their tenant (a column added by V0117); a row is also always visible to its own owner, so a membership reshuffle can never eat someone's workflow**. A NULL `tenant_id` is a **platform-level asset** — not in any tenant, hidden from tenant members. V0117 backfills historical leftovers into `default-tenant`; after that a row's tenant comes from its creator, and only pipelines created by an account outside every tenant stay NULL. That column means the *opposite* of the same-named column on `servers` / `credentials` / `projects`, where NULL is a deliberate platform-public resource. The console no longer advertises "resources owned by no tenant stay visible" as if it were the product rule. Only super admins see those rows, badged *No tenant owner*. The fix is a human one, and the API now exposes it: `ownerUsername` is an independent update field, super-admin-only, whose target must be an existing enabled account — blank values and `api-token:` subjects are refused, and a tenant admin can only hand pipelines to members of that same tenant (cross-tenant assignment is reserved for platform admins who belong to no tenant). `pipelines` has no creator column, so **ownership cannot be backfilled automatically**. Changing the owner rewrites `tenant_id` from the new owner's tenant — there is deliberately no separate "set tenant" knob, because two knobs always drift into one fake one. Changing a member's tenant does **not** retroactively move the pipelines in their name: a silent org reshuffle that re-homes other tenants' workflows is worse than a stale ownership. API tokens stay outside the visibility model — otherwise a CI token could not see the very pipeline it was issued to trigger; scopes remain their boundary. `servers` / `build_agents` / `credentials` / `projects` / `custom_nodes` work differently: V0116 gave them a `tenant_id` where `NULL` is a deliberate *platform public* resource (credentials, projects and shared quality rules still resolve as "this tenant ∪ platform public"), which is not the same rule as strict pipeline ownership, and the UI wording is split accordingly. Within one tenant two OPERATORs can still see and edit each other's pipelines unless they are `private` — that half of the model is unchanged.
- **Agents and promotion approvals now resolve by tenant too; a column nobody reads is not isolation.** `build_agents` gained `tenant_id` back in V0116 and the probe registry `agents` got it in V0118, but no code consulted either one, so "already isolated" was only a statement about the schema. Listing, single reads, creation, renaming, deletion, heartbeats and token resets all check the tenant now. The `token` in a probe row is a connection credential, so it is returned exactly once — on create and on regenerate — and never in list or detail payloads. **Creating a build agent that references another tenant's server or credential is refused outright** (an agent pointing at someone else's docker host is someone else's machine running my builds; filtering the agent list while letting references through leaves only a UI layer of isolation). The error text merges "does not exist" with "is not yours", so enumerating ids gives no feedback. Execution paths deliberately stay unfiltered — fetching an agent by id, the stale-heartbeat sweep, a probe reporting its own status — because conditioning those on tenant makes background jobs fail in the middle of a run; the boundary is console authorization, and execution inherits the authorization of that run.
- **A promotion approval is a gate action, not a read.** Approving and rejecting were previously blocked only by **role** (OPERATOR+), and nothing asked whose promotion it was — so any OPERATOR on the platform could push another tenant's artifact to production. Approve, reject, the review queue, the review card, per-run listings and manual requests now share one decision (reading and writing are not two different gates), and an unresolvable row answers with the same 404. The `pendingTotal` badge shares the queue's predicate, otherwise "12 awaiting review" hides nine rows belonging to someone else — a number leaks more easily than a list, because it carries no permission check. Attribution is **copied onto the promotion row itself** (V0119 `pipeline_id` / `tenant_id`) rather than resolved through `run_id`: `pipeline_runs` is periodically purged by retention, and a tenant boundary that depends on a table that gets deleted fails in both directions — a tenant's own legitimate approvals become unreadable once the run expires (strict hiding degrading into silent data loss), while "cannot resolve, so allow" turns cross-tenant approval into "just wait for their record to expire". Legacy rows whose run is long gone and which carry no attribution stay visible only to super admins and API tokens, who are the ones able to clean them up.
- **Plugin versions can be pinned per pipeline, and a pin actually holds.** A new job-level `pluginVersion` (deliberately *not* inside `config`: the whole `job.config` is written to `PLG_INPUT_FILE` for the plugin, so platform routing metadata in there would leak to plugins and could collide with a field a plugin declares itself). When set, the step executes from that row of `marketplace_plugin_versions` — script text, execution language, package path, SHA-256 — **regardless of who has which version installed, handler registration order, or who triggered the run**. When empty, nothing changes at all: not even a database query. One entry point (`PluginVersionPinning.resolve`) serves both save-time validation and runtime resolution, so "why the editor rejected it" and "why the run failed" are literally the same sentence.
- **If the pin cannot be honoured, the step fails on the spot — it never falls back to latest.** No such version row, package file absent on this machine (a DB restored without `data/plugins` is a real scenario), file no longer matches the recorded SHA-256, a pre-snapshot half-record, or the same version belonging to two plugins: each is refused with an actionable reason. Falling back to latest would trade "a breaking update" for "a random version", which is harder to diagnose and inverts the promise. Version strings are validated *before* being interpolated into a storage path (so `".."`, `"1..0"`, `".5"` are rejected), and the resolver re-checks that the returned row really is the requested version.
- **Uninstall and disable stay authoritative.** A pin chooses which bytes to run; it does not resurrect a retired node type, and the step log says so explicitly, so nobody assumes a pin lets them work around a delisting.
- **A fact that has to be admitted first: without a pin there is no "current version".** `installed_plugins` is unique on `(user_id, node_type)` (V0091 moved installs per user) while `JobHandlerRegistry` keeps exactly one handler per nodeType, last registration wins. So if A installed 1.2.0 and B installed 2.0.0, which one their pipelines actually run depends on the order `selectList` returns at startup. This change deliberately does **not** swap that semantics (resolving by triggerer would make the same spec behave differently depending on who clicked run — one ambiguity traded for another); instead it makes the ambiguity visible: the step log states "not pinned, this machine has N install records (1.2.0, 2.0.0); the registered one is what executed". Pin a version on the node when you need determinism.
- **When a version is pinned, required-field validation reads that version's declared fields.** Otherwise the form validates against latest while the engine honours pinned: rename a field between versions and you simultaneously get false save-time rejections and runtime "missing parameter" failures.
- **Plugin-package rules have exactly one implementation, so the local tool and the server cannot disagree.** The rules live in `plugin-sdk`'s `PluginPackageContract` (entry count, decompressed size, the 256KB `handler.sh` cap, `schemaVersion`, required task.json fields, nodeType format and reserved types, whether `execution.language` matches what the package actually contains, whether `packagePath` resolves). `PluginPackageController` and `MarketplaceService` call it; so does `PluginDoctor` (`plg validate` / `gradle pluginCheck`). **Why this is emphasized**: developers used to learn the rules only by uploading and being rejected, and copying the rules into a local tool produces "validate passed locally, platform still rejects" — worse than having no local check, because it trains people to distrust error messages. The hardest test here builds a package with `PluginDoctor` and feeds those exact bytes to the server's `parseZip`, asserting both reach the same conclusion.
- **Packaging is one step inside the template, not a version number hand-copied twice.** All four Kotlin templates ship `gradle buildPlugin` (produces `build/distributions/<name>-<version>.zip`, aligns the jar's in-package entry name with `execution.packagePath`, prints SHA-256) and `gradle pluginCheck`. The old `zip X.zip task.json build/libs/x-1.0.0.jar` wrote the version twice; bumping it in one place only produced "packagePath not found". SHA-256 is not decoration: the platform computes it once at upload, then re-computes and compares on every install/download (`MarketplaceService.verifyPackageIntegrity`, mismatch is rejected), so entry order and timestamps are pinned — repackaging the same sources yields the same checksum, and **re-zipping the file in the middle will break the comparison**.
- **`plg` is distributed by the platform instead of being released separately.** `GET /api/marketplace/templates/sdk/plg` serves the script, `GET .../sdk/script` serves the `plg-sdk.sh` that shell plugins source — the same classpath resource the platform injects into the running plugin, so local and platform execute the same implementation. The CLI does three things only: fetch a template and rewrite `name`/`nodeType` from the project name (printing what it changed rather than editing files silently), delegate package/validate to `PluginDoctor`, and delegate Java builds to `gradle buildPlugin`. **The rules are deliberately not reimplemented in the CLI**; that is what lets it follow the server instead of drifting into a second source of truth.
- **One row per nodeType in the catalog (behavior change — the impact first).** Package upload and web publish used to insert a brand-new row every time (`UUID.randomUUID()`), and `marketplace_plugins.node_type` had only a plain index, no unique constraint, so the same plugin could occupy many rows. That is not cosmetic debt: `JobHandlerRegistry` keeps exactly one handler per nodeType, so the extra rows were never equal at runtime, and anything built on "the local row" (sync, version diffing) was referring to something undefined. Both publish paths now merge by nodeType: a higher version rewrites the existing row and appends a version snapshot; the same version with identical content is an idempotent skip that writes nothing; the same version with different bytes, an older version, an incomparable version, or several local rows are all refused with a full-sentence reason. Existing duplicates are not auto-merged (choosing a row is an operator decision), but sync and upload both stop first and list the exact row ids. This also closes a real hole: **package upload never wrote `marketplace_plugin_versions`**, so uploaded plugins had no version history and could not be pinned at all. Both entry points now record snapshots, and `PluginVersionPinning` can fall back to the catalog row for the version it currently holds, so existing plugins can be pinned without re-uploading.
- **Syncing plugin catalogs between instances: preview, then confirm — catalog only, never install.** Configure `pipeloom.plugin.registry-sync.registries[]` (`name` / `url` / `token-env`). `POST /api/marketplace/sync/preview` is read-only and reports a disposition per entry (writable / identical-skip / conflict / needs attention); `POST /api/marketplace/sync/apply` is what writes to the local catalog. Deliberate choices: (1) the confirmation step must show the content hash that will be written, and `apply` carries it back for verification — otherwise "confirm" confirms nothing but a click; (2) `apply` re-reads the upstream catalog and re-decides, so content that changed between preview and click is refused; (3) downloaded packages are re-hashed locally and compared against the value upstream *claims* — the remote metadata's hash is never trusted, and a mismatch is recorded as `conflict` and refused; (4) pulled content goes through `PluginPackageIngest`, the same entry point as a manual upload with the same package-structure and reserved-type rules — there is no second "write metadata straight into the DB" path (for script plugins the upstream has no package for, a deterministic package is rebuilt from the upstream metadata and goes through the same door, so sync duplicates no packaging rules); (5) **no scheduled sync**: A→B→A loops are harmless by construction ("pull only" plus identical-hash idempotent skip), plus a configured upstream equal to this instance's own `api-url` is refused outright.
- **Sync writes executable content, so the boundary defaults to refusal.** Plain `http` is rejected unless `allow-insecure-http`; loopback and link-local addresses are rejected by default (the cloud metadata endpoint `169.254.169.254` lives in link-local, and sync requests carry a token); a URL containing `user:token@` is rejected outright — credentials reach the process only through the environment variable named by `token-env`, never the database, logs, or the UI. The log table stores `scheme://host[:port]` and **keeps both the hash upstream declared and the hash this machine computed**: with a single column you cannot see when the two started to diverge, which is the one thing this feature exists to catch. Actions are `synced / skipped / conflict / rejected`, pinned by a database CHECK, because "upstream mutated a released version" and "this package was never valid" must be queryable apart. Sync is a super-admin action and each run is audited.
- **To be plain about it: a matching catalog is not a matching run.** Sync guarantees only that this instance's plugin catalog contains that version; it does not guarantee any pipeline executes it — installation is per user (`installed_plugins` is keyed by `(user_id, node_type)`) and the executed version is decided by `pluginVersion` on the node. That sentence is printed at the top of the drawer, and the button says "sync the selected items" rather than "sync everything": unselected entries are never written just because upstream has them. One side effect that must be disclosed: each package pull increments the upstream's `downloads` counter, so a sync looks like someone downloading on that instance.
- **Shell plugins finally have a template.** The platform has always supported shell execution (`parseZip` accepts `handler.sh`, `PluginJobHandler.buildCommand()` runs the script text itself, inputs arrive as `PLG_<FIELD_UPPERCASE>` env vars), yet all four templates were Kotlin, leaving a twenty-line-script author with no starting point. `tpl-shell-basic` adds one (`task.json` + `handler.sh` + `dev-input.json` + README). The README pins a trap unique to shell: every `${name}` in the script body is substituted by **pipeline variables** first (`VariableService.resolveText`), so plugin fields must be read via `plg_get_input <field>` or `$PLG_<FIELD>` — `${field}` reads a same-named pipeline variable, not your plugin config.

- **Visibility and tenant conditions must be a single `IN (SELECT …)` subquery.** Splitting them into two `apply` calls leaves the tenant layer referencing the subquery's internal alias `p` at the top level (`WHERE p.tenant_id = ?`), and the outer `pipelines` / `pipeline_runs` has no such alias — MySQL errors with Unknown column. Only *tenant-constrained members* ever reach that SQL, so the isolation feature would break the exact users it protects the moment it ships. `PipelineVisibilityTest` now asserts every `p.` prefix stays inside the subquery.

- **`stage.isolatedWorkspace` cuts file passing between stages.** By default every stage shares one run-level checkout, and "clone here, build later" relies on that; an isolated stage sees none of the files an upstream stage left behind (`needs` orders execution, it does not copy anything), which the save-time `isolated_workspace_hides_upstream` warning calls out. Pass things between stages as artifacts, or give the stage its own `git_source`.

- **Stage caching (`cachePaths` / `cacheKey`) is namespaced per matrix combination and finally bounded.** `CacheGovernance` derives the namespace once for restore and save; the old duplicated `take(64)` let two keys sharing a 64-character prefix silently collide. Matrix stages restore and save *at the combo boundary* — saving used to happen after matrix variables were restored, so a `${os}` in the key no longer resolved and the save landed in a different directory than the restore, which made "configure the cache correctly" a guarantee of a permanent miss. Failed combos are never saved, so a broken dependency tree cannot come back next run labelled as a cache hit. Restore reports hit / miss honestly instead of always logging "restored", and counts reach `/actuator/prometheus` as `pipeloom.cache.restore{result}`, `pipeloom.cache.save{result}` and `pipeloom.cache.evict{reason}`. Limits come from `pipeloom.cache.*` (root, `ttl-days` default 14, `max-bytes` default 10 GiB); the namespace written this run is never evicted. `cachePaths` resolve relative to the workspace — an absolute path is relocated inside it and flagged at save time (`cache_path_relocated_into_workspace`), and a `cacheKey` with no `cachePaths` caches nothing (`cache_key_without_paths`). Sharing across runs still requires the **same execution target**: the same path on two build machines is two directories.

- **Reverse proxy / auto-HTTPS and DNS-provider onboarding are stubs.** The console renders the panels, the endpoints return empty or placeholder objects, and no certificate is issued. The panel now opens with an explicit *not-implemented* banner so placeholder routes and certificate states cannot be mistaken for real configuration. Do not plan around them yet.
- **`helm_chart` deploys need `helm` on the control node**, and blue-green under Helm is a values-level concern (the node logs that it only records the strategy).
- **`stage.post` (post-stage steps) and `stage.services` (sidecar services) are still not executed — but they no longer pass silently.** The engine has no branch that runs them, so saving a pipeline that declares them is now rejected (`stage_post_not_executed` / `stage_services_not_executed`), and a run that reaches them via API/Webhook fails the stage with `SpecFieldNotSupportedException` instead of pretending. `cachePaths` / `cacheKey`, `allowFailure`, `when`, `retryCount` and `gateTimeoutMinutes` are all live.
- **Remote (delegated) scanning covers fewer engines than local scanning.**
- **Scan metrics such as Semgrep detail counts are produced on the local execution path**, and a gate reads a missing metric as "not found" instead of zero.
- **`test` is a no-op placeholder until a plugin provides it** (see the node table above) — `script` / `custom` are not: they really run `commands`. This is the easiest boundary to trip over.
- **Change attribution is not incremental scanning, and that line is deliberate.** Apart from the new `CODE_SCAN_CHANGED_*` metrics, red lines, trends, LOC and the full-repo verdict are untouched: attribution is a second dimension layered on top. Shrinking the scan set to a diff still lacks hard prerequisites — `semgrep` (1.176) has no `--diff`, SpotBugs analyses bytecode rather than a file list, and "no secrets anywhere in the repo" is itself a whole-repo invariant, so a diff-scoped gitleaks run would let a `CODE_SCAN_GITLEAKS <= 0` red line turn green without a fix. If that ever lands it must be a **new** `*_CHANGED` gate, never a silent swap of the existing full-repo lines. Attribution also needs git history back to the baseline commit of the **previous successful run** on the same branch: a first run, a non-git workspace or an unreachable baseline (shallow clone, after a force push) has no baseline, and then the report states the reason instead of "0 new issues".
- **The built-in engine caps a single scan at 2000 files.** Above that it sorts by path, takes the first 2000, and names both the total and the number left unscanned in the step log; that warning exists so "scanned part of it" cannot masquerade as "the repo is clean". Walking and sorting the whole tree is the price of knowing the real total — skip it and you would not even know the cap was hit.
- **"Visible" and "usable" are two different questions; both must be asked.** List filtering stops enumeration but not id guessing, so the grant list (`share_mode = listed` + `resource_grants`, with expiry and revocation) applies to **lists and references alike**, and the runtime re-checks against the `tenant_id` snapshot on the run row, failing closed when it cannot tell. The `private` boundary is owner + super admin + the tenant admin of the same tenant; a tenant-owned asset cannot be flipped to `listed`, because that would open a bypass around "created in a tenant means isolated".
- **"Administrator" is two roles now.** `TENANT_ADMIN` manages its own tenant's members and resources (`@AdminOnly`: credentials, red lines, notification channels, plugin enable/disable); platform surfaces (users, tenant ledger, API token issuance, AI providers, marketplace registry sync, reverse proxy) are `@PlatformOnly`, i.e. only a `SUPER_ADMIN` who belongs to **no** tenant. What a role means is decided by which tenant it belongs to, not by its name — so "promote a tenant admin to a platform admin" is not a single field change.
- **Probe binaries are served from `pipeloom.agent-binary-dir`** — `make` in `agent/`, or drop the files there; a missing binary is a 404, not a fallback.

## Versioning

Starting from **v1.0.0**, pipeloom follows [Semantic Versioning](https://semver.org/):

| Bump | When | Example |
|---|---|---|
| MAJOR | Breaking change (API incompatible, irreversible migration) | v1.0.0 → v2.0.0 |
| MINOR | New feature (backward compatible) | v1.0.0 → v1.1.0 |
| PATCH | Bug fix (backward compatible) | v1.0.0 → v1.0.1 |

**Stable / development tracks** — the MINOR segment determines the channel:

- **Even MINOR** = stable release (v1.**0**.x, v1.**2**.x) — safe for production; only receives PATCH bug fixes.
- **Odd MINOR** = development release (v1.**1**.x, v1.**3**.x) — preview / testing; new features land here first.

Git tags use the `vMAJOR.MINOR.PATCH` format. CHANGELOG is auto-generated from [Conventional Commits](https://conventionalcommits.org/) at release time. See [docs/release-checklist.md](docs/release-checklist.md) for the full release procedure.

## Project status

Internal, actively developed: version `1.0.0` (SemVer), database schema at Flyway, last commits daily. It is used to drive real pipelines, scans, deploys and daily ops; treat `main` as a moving target and pin the commit you deploy. The API surface is documented live at `/swagger-ui.html` once the server is up.

## Contributing

Branch off `main`, keep CI green (`ktlint`, `detekt`, `test`, `vue-tsc`), and land a Flyway migration for every schema change rather than editing an existing one. Conventional-style commit prefixes are the convention here (`feat:` · `fix:` · `refactor:` · `style:` · `test:`), with the scope in parentheses when it helps (`fix(pipeline): …`). Be careful with secrets: `application-local.yml`, `data/`, `logs/` and `*.db*` are ignored, but **`.env` is currently tracked in this repository**, so never put a production password or vault key in it — generate deployment secrets out-of-band and keep `.env` for local placeholders only.

## License

Private — internal use only.

---

<div align="center">
<sub>Pipeloom — CI, quality gates, deployment and ops in one control plane.</sub>
</div>