diff --git a/README.md b/README.md index 190da84dc2d68928f99e0ff24bcc79df8904814e..2ec47e0016aa13a09ebaee326293706447af8990 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ http://127.0.0.1:3030 ## 步骤4:接入业务系统 -首先，在管理控制台 `系统设置` 页面中，将业务系统和对话应用所在的服务器 IP 地址（对话应用和业务系统需要部署在同一台服务器中）设置到 `对话应用/业务应用服务器`。因为修改后该配置后，系统会同步调整 IP 白名单，需重启容器后生效。 +首先，在管理控制台 `系统设置` 页面中，将业务系统和对话应用所在的服务器 IP 地址或域名（对话应用和业务系统需要部署在同一台服务器中）设置到 `对话应用/业务应用服务器`。服务器地址保存到系统数据库后会立即用于业务调用；IP 地址会同步到 IP 白名单，白名单变更需重启容器后生效；域名按外网地址处理，不加入白名单。 然后，在“业务系统”菜单中添加接入的业务系统。活字格系统通常需要填写： @@ -112,24 +112,25 @@ MQTT/MinIO 密码可在 `系统设置` 页面中 `基础设置` 选项卡获取 ## 性能测试 -- 测试版本：20260701.01 -- 测试环境：2C4G -- LLM：本地模拟，收到请求后等待 5 秒再返回，非阻塞 +- 测试日期：2026-08-06 +- 测试环境：2C4G，100 个数字员工 +- LLM：本地模拟，在 4-6 秒之间随机等待后返回，理论均值和中位数均为 5 秒，容量 128 - 测试规格： - - 数字员工 x 100 - 用户 x 1000，每个数字员工关联 10 个用户 - 数字员工模板 x 100，每个工作区模板 100 个文件 - 技能 x 100 - 系统 x 100 -- 测试计划： - - 预热：1 分钟 - - 稳态：10 分钟 - -| 并发 | 结果 | 稳态成功/请求 | 失败 | 吞吐 req/s | P50 | P95 | P99 | CPU 峰值 | 内存峰值 | -| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| 20 | 通过 | 986/986 | 0 | 1.64 | 11.857s | 17.119s | 21.945s | 202.39% | 1.66GiB | -| 50 | 通过 | 1000/1000 | 0 | 1.67 | 28.185s | 40.100s | 58.307s | 199.70% | 2.05GiB | -| 100 | 通过 | 998/998 | 0 | 1.66 | 56.599s | 74.609s | 93.801s | 180.19% | 2.13GiB | +- 测试计划：每档预热 30 秒，稳态压测 15 分钟；并发档位为 5、10、30、100 +- 测试结论：10 并发下，2C4G 的配置可以正常使用；更高并发量不会导致错误，但体验下滑严重。 + +| 并发 | 结果 | 稳态成功/请求 | 失败 | 吞吐 req/s | Avg | P50 | P90 | P99 | CPU 峰值 | 内存峰值 | +| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 5 | 通过 | 707/707 | 0 | 0.79 | 6.370s | 6.344s | 7.487s | 8.467s | 198.26% | 2.33GiB | +| 10 | 通过 | 1213/1213 | 0 | 1.35 | 7.411s | 7.280s | 8.758s | 10.435s | 203.01% | 2.57GiB | +| 30 | 通过 | 1109/1109 | 0 | 1.23 | 24.347s | 23.691s | 27.510s | 46.764s | 201.26% | 3.23GiB | +| 100 | 通过 | 1110/1110 | 0 | 1.23 | 79.367s | 78.747s | 90.629s | 120.325s | 199.54% | 3.99GiB | + +详细测试报告见 [DAEDALUS_AGENT-100a-2C4G-stress-report.md](./doc/DAEDALUS_AGENT-100a-2C4G-stress-report.md)。 ## 本地开发文件 diff --git a/doc/DAEDALUS_AGENT-100a-2C4G-stress-report.md b/doc/DAEDALUS_AGENT-100a-2C4G-stress-report.md new file mode 100644 index 0000000000000000000000000000000000000000..d15a8373394bdfb5978e663bf0b88768dd556446 --- /dev/null +++ b/doc/DAEDALUS_AGENT-100a-2C4G-stress-report.md @@ -0,0 +1,60 @@ +# DAEDALUS_AGENT 容器压力测试报告 + +**100 Agent 并发矩阵正式测试结果** + +| 项目 | 内容 | +| --- | --- | +| 测试日期 | 2026-08-06 | +| 测试环境 | 2 CPU / 4 GiB，100 个数字员工 | +| 容器 | `daedalus-agent-stress-100a-reuse-mock` | +| 镜像 | `daedalus-agent-x64:local` (`sha256:ff3d7e3f058376ad29b8cc67b8660803281e3ea3e753b44552f72d84b2cbd1f4`) | +| 负载策略 | 5 / 10 / 30 / 100 并发；每档预热 30 秒、稳态压测 15 分钟 | +| 模拟模型 | Mock OpenAI，4000-6000ms 均匀随机延迟，理论均值和中位数均为 5000ms，容量 128 | +| Gateway 并发 | `maxConcurrent = 128` | + +> **结论**：四档并发压测全部通过，稳态成功率均为 100%，失败请求为 0，Mock LLM 容量拒绝为 0。 + +## 1. 测试范围与方法 + +本次测试复用同一个 100 Agent 容器，使用固定随机种子选择 Agent，依次执行 5、10、30、100 并发。每档在 30 秒预热后持续施加 900 秒稳态负载，模型端使用容量为 128 的本地 Mock LLM，单次模型等待时间从 4000-6000ms 均匀随机采样。 + +通过条件为压测脚本正常退出、稳态请求无失败且 Mock LLM 未发生容量拒绝。容器资源限制保持为 2 CPU / 4 GiB。 + +## 2. 压测结果 + +| 并发 | 结果 | 稳态成功/请求 | 失败 | 吞吐 req/s | Avg | P50 | P90 | P99 | +| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 5 | 通过 | 707/707 | 0 | 0.79 | 6.370s | 6.344s | 7.487s | 8.467s | +| 10 | 通过 | 1213/1213 | 0 | 1.35 | 7.411s | 7.280s | 8.758s | 10.435s | +| 30 | 通过 | 1109/1109 | 0 | 1.23 | 24.347s | 23.691s | 27.510s | 46.764s | +| 100 | 通过 | 1110/1110 | 0 | 1.23 | 79.367s | 78.747s | 90.629s | 120.325s | + +响应时间为从压测客户端发起 MQTT 请求到收到完整回复的端到端耗时，包含网关、Agent、模型调用和消息链路开销。 + +## 3. 资源与容量观察 + +| 并发 | CPU 平均 | CPU 峰值 | 内存平均 | 内存峰值 | 内存峰值占限额 | Mock 实际峰值 | 容量拒绝 | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 5 | 71.82% | 198.26% | 1.67 GiB | 2.33 GiB | 58.15% | 5 | 0 | +| 10 | 127.09% | 203.01% | 1.92 GiB | 2.57 GiB | 64.18% | 10 | 0 | +| 30 | 134.17% | 201.26% | 2.37 GiB | 3.23 GiB | 80.68% | 30 | 0 | +| 100 | 128.55% | 199.54% | 2.76 GiB | 3.99 GiB | 99.68% | 100 | 0 | + +Docker CPU 百分比按多核累计，约 200% 表示 2 个 CPU 核心接近满载。100 并发时内存峰值为 3.99 GiB，占 4 GiB 限制的 99.68%；本次未发生 OOM，但可用内存余量非常有限。 + +## 4. 运行验证 + +| 验证项 | 结果 | +| --- | --- | +| 压测矩阵 | 5 / 10 / 30 / 100，四档均通过 | +| 时间策略 | 每档预热 30 秒，稳态 900 秒 | +| Mock 延迟 | 所有档位健康检查返回 4000-6000ms，中心值 5000ms，抖动 1000ms | +| Mock 容量 | 容量配置 128；四档实际峰值分别为 5、10、30、100；容量拒绝均为 0 | +| Gateway 配置 | `openclaw.json`、`env.json`、`runtime-env.json` 均为 128 | +| 容器状态 | 复用原容器，Docker `RestartCount = 0` | +| 进程状态 | 压测结束后 Mock LLM 与压测进程均已退出 | +| 结果字段 | 原始报告记录 Avg、P50、P90、P99 | + +## 5. 结论 + +本次 100 Agent 并发矩阵压力测试整体通过。系统在 5、10、30、100 并发下均保持 100% 稳态成功率，随机 Mock LLM 未触发容量限制。100 并发场景下 CPU 接近满载，内存达到 99.68%，后续容量规划应重点增加内存余量并关注 CPU 饱和。 diff --git a/docker/BUILD.md b/docker/BUILD.md index 4fca08e468edfe629d00cc193e01f8157b25b920..b82253507c408fbae28e310e3cf4301d88197a84 100644 --- a/docker/BUILD.md +++ b/docker/BUILD.md @@ -92,12 +92,12 @@ Docker 容器测试放在根目录 `test`，目录按职责拆分为 `common/` ```powershell npm --prefix test run test:integration:fast npm --prefix test run test:integration:full -npm --prefix test run test:stress:20a10c +npm --prefix test run test:stress npm --prefix test run test:stress:50a20c npm --prefix test run test:stress:100a50c ``` -集成与压测 suite 全部使用 mock LLM。mock LLM 收到 chat completion 请求后默认等待 5 秒再返回；响应时间在各 suite 配置的 `mockLlm.delayMs`、`mockLlm.chunkDelayMs` 中配置，也可以用 `MOCK_OPENAI_DELAY_MS`、`MOCK_OPENAI_CHUNK_DELAY_MS` 覆盖。suite 运行时状态默认写入 `test/.runtime/`。 +集成与压测 suite 全部使用 mock LLM。mock LLM 收到 chat completion 请求后，默认在 4-6 秒之间随机等待再返回，理论均值和中位数均为 5 秒；响应中心值在各 suite 配置的 `mockLlm.delayMs` 中配置，随机半径可用 `MOCK_OPENAI_DELAY_JITTER_MS` 覆盖，`mockLlm.chunkDelayMs` 和 `MOCK_OPENAI_CHUNK_DELAY_MS` 仍用于流式分片间隔。suite 运行时状态默认写入 `test/.runtime/`。 底层调试脚本仍保留独立配置和独立运行状态文件，适合单独排查 case 或准备阶段。 diff --git a/docker/README.md b/docker/README.md index 360c0b590d67706d48c474c017d9c8c7bbe3ada6..b1d14023fd38bfb5baf53ff5e48e806aedf48393 100644 --- a/docker/README.md +++ b/docker/README.md @@ -47,8 +47,7 @@ docker start daedalus-agent ```json { "DA_MANAGEMENT_CONSOLE_PORT": 3030, - "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 16, - "DA_HUOZIGE_SERVER_IP": "" + "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 128 } ``` @@ -56,13 +55,7 @@ docker start daedalus-agent 管理控制台的系统状态采用缓存刷新机制：启动后默认延迟 60 秒首次采集，之后每 60 秒刷新一次，覆盖 MQTT、S3、内核、插件加载和调用服务等状态；可通过 `DA_SYSTEM_STATUS_INITIAL_DELAY_MS`、`DA_SYSTEM_STATUS_REFRESH_INTERVAL_MS`、`DA_SYSTEM_STATUS_TIMEOUT_MS` 调整。 -如需允许 OpenClaw/agent 访问指定局域网地址，可追加： - -```json -{ - "DA_INTERNAL_NETWORK_ALLOWLIST": "192.168.1.10,10.20.0.0/16" -} -``` +OpenClaw/agent 的局域网白名单通过管理控制台“系统设置 -> 基础设置”维护并保存在 SQLite；修改后重启容器生效。紧急运维时也可通过容器环境变量 `DA_INTERNAL_NETWORK_ALLOWLIST` 临时覆盖数据库值。 ## 运行数据 @@ -120,7 +113,7 @@ daedalus-agent-mqtt-channel-chat-{yyyyMMdd}.log 顶层脚本守护 MQTT、MinIO、OpenClaw supervisor 和 apps supervisor。apps supervisor 内部以 `daedalus-svc` 守护 management web 和 `daedalus-agent-apps-invoke-cli serve`；OpenClaw supervisor 内部守护 OpenClaw gateway。 -对话应用/业务应用所在的活字格服务器地址通过管理控制台“系统设置 -> 基础设置”统一配置。修改后管理控制台会同步维护 `DA_INTERNAL_NETWORK_ALLOWLIST`：移除旧地址并加入新地址；白名单变更需重启容器后生效。OpenClaw/agent 侧仍只执行 CLI client，并连接固定内部 socket `/var/daedalus-agent/run/app-invoke.sock`；真实业务系统访问由 `daedalus-svc` 运行的 invoke service 完成。 +对话应用/业务应用所在的活字格服务器地址和内部网络白名单都由管理控制台保存到 SQLite，并在一次事务内更新；修改后的地址会立即用于业务调用，白名单在重启容器后生效。IP 地址会自动加入白名单，域名按外网地址处理，不加入白名单。旧版本 `env.json` 中的 `DA_HUOZIGE_SERVER_IP` 和 `DA_INTERNAL_NETWORK_ALLOWLIST` 仅作为一次性迁移入口，迁移后会从文件中删除。OpenClaw/agent 侧仍只执行 CLI client，并连接固定内部 socket `/var/daedalus-agent/run/app-invoke.sock`；真实业务系统访问由 `daedalus-svc` 运行的 invoke service 完成。 ## 验收与压测 @@ -129,12 +122,12 @@ daedalus-agent-mqtt-channel-chat-{yyyyMMdd}.log ```powershell npm --prefix test run test:integration:fast npm --prefix test run test:integration:full -npm --prefix test run test:stress:20a10c +npm --prefix test run test:stress npm --prefix test run test:stress:50a20c npm --prefix test run test:stress:100a50c ``` -集成与压测 suite 全部使用 mock LLM。mock LLM 收到 chat completion 请求后默认等待 5 秒再返回；响应时间可在各 suite 配置的 `mockLlm.delayMs`、`mockLlm.chunkDelayMs` 中配置，也可以用 `MOCK_OPENAI_DELAY_MS`、`MOCK_OPENAI_CHUNK_DELAY_MS` 覆盖。 +集成与压测 suite 全部使用 mock LLM。mock LLM 收到 chat completion 请求后，默认在 4-6 秒之间随机等待再返回，理论均值和中位数均为 5 秒；响应中心值可在各 suite 配置的 `mockLlm.delayMs` 中配置，随机半径可用 `MOCK_OPENAI_DELAY_JITTER_MS` 覆盖，分片间隔可用 `mockLlm.chunkDelayMs` 或 `MOCK_OPENAI_CHUNK_DELAY_MS` 配置。 底层脚本仍保留独立配置和独立运行状态文件。首次运行前可分别复制本地配置文件： diff --git a/docker/assets/config.json b/docker/assets/config.json index 3a6c0d35edf156fb198cd147f8d1f29d7ddd128a..c074978e940e86a608cbe370a029286f302fe049 100644 --- a/docker/assets/config.json +++ b/docker/assets/config.json @@ -4,6 +4,7 @@ "workspace": "/var/daedalus-agent/.openclaw/workspaces/default", "skipBootstrap": true, "heartbeat": { + "every": "0m", "isolatedSession": true, "lightContext": true } diff --git a/docker/assets/scripts/kernel-startup.sh b/docker/assets/scripts/kernel-startup.sh index ca880419740aa2ffae31c7e68ec599d21486b3cc..b69f6f8a092840a6521a580e37021f3fc77d3561 100644 --- a/docker/assets/scripts/kernel-startup.sh +++ b/docker/assets/scripts/kernel-startup.sh @@ -1963,14 +1963,14 @@ ensure_runtime_env_defaults() { temp_env="${RUNTIME_ENV_JSON}.tmp.$$" jq ' (if has("DA_INTERNAL_NETWORK_ALLOWLIST") then . else . + { "DA_INTERNAL_NETWORK_ALLOWLIST": "" } end) - | (if has("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT") then . else . + { "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": "16" } end) + | (if has("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT") then . else . + { "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": "128" } end) ' "$RUNTIME_ENV_JSON" > "$temp_env" mv "$temp_env" "$RUNTIME_ENV_JSON" ensure_service_file_mode "$RUNTIME_ENV_JSON" 0640 } normalize_openclaw_gateway_max_concurrent() { - local value="${DA_OPENCLAW_GATEWAY_MAX_CONCURRENT:-16}" + local value="${DA_OPENCLAW_GATEWAY_MAX_CONCURRENT:-128}" if [[ ! "$value" =~ ^[1-9][0-9]*$ ]]; then startup_step_error "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT must be a positive integer" @@ -2207,6 +2207,7 @@ patch_openclaw_config() { .agents.defaults.workspace = $workspace | .agents.defaults.maxConcurrent = $gatewayMaxConcurrent | .agents.defaults.skipBootstrap = true + | .agents.defaults.heartbeat.every = "0m" | .agents.defaults.heartbeat.isolatedSession = true | .agents.defaults.heartbeat.lightContext = true | .gateway.mode = "local" diff --git a/docker/assets/scripts/startup.sh b/docker/assets/scripts/startup.sh index 237ea9ae6ad3dbf85d7c4398ab4bb4c5bccfcae1..0ca4abf73d45612f49199948ec3c33bbdbdbdf66 100644 --- a/docker/assets/scripts/startup.sh +++ b/docker/assets/scripts/startup.sh @@ -300,14 +300,13 @@ reject_unsupported_user_env() { ensure_user_env_defaults() { local temp_env - if jq -e 'has("DA_INTERNAL_NETWORK_ALLOWLIST") and has("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT")' "${USER_ENV_JSON}" >/dev/null; then + if jq -e 'has("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT")' "${USER_ENV_JSON}" >/dev/null; then return fi temp_env="${USER_ENV_JSON}.tmp.$$" jq ' - (if has("DA_INTERNAL_NETWORK_ALLOWLIST") then . else . + { "DA_INTERNAL_NETWORK_ALLOWLIST": "" } end) - | (if has("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT") then . else . + { "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 16 } end) + (if has("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT") then . else . + { "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 128 } end) ' "${USER_ENV_JSON}" > "${temp_env}" mv "${temp_env}" "${USER_ENV_JSON}" ensure_service_file "${USER_ENV_JSON}" 0640 @@ -668,11 +667,11 @@ ensure_default_startup_hook() { } generate_runtime_files() { - node - "${USER_ENV_JSON}" "${USER_CONFIG_JSON}" "${CREDENTIALS_JSON}" "${GENERATED_DIR}" "${DA_ROOT}" <<'NODE' + node - "${USER_ENV_JSON}" "${USER_CONFIG_JSON}" "${CREDENTIALS_JSON}" "${GENERATED_DIR}" "${DA_ROOT}" "${DA_WEB_DATA_DIR}/management-console.db" <<'NODE' const fs = require("node:fs"); const path = require("node:path"); -const [userEnvPath, userConfigPath, credentialsPath, generatedDir, productRoot] = process.argv.slice(2); +const [userEnvPath, userConfigPath, credentialsPath, generatedDir, productRoot, databasePath] = process.argv.slice(2); function readJson(filePath) { return JSON.parse(fs.readFileSync(filePath, "utf8").replace(/^\uFEFF/, "")); } @@ -681,6 +680,35 @@ const userEnv = readJson(userEnvPath); const credentials = readJson(credentialsPath); const config = readJson(userConfigPath); +function readRuntimeSetting(name) { + if (!fs.existsSync(databasePath)) { + return undefined; + } + let database; + try { + const { DatabaseSync } = require("node:sqlite"); + database = new DatabaseSync(databasePath, { readOnly: true }); + const table = database.prepare(` + SELECT 1 + FROM sqlite_master + WHERE type = 'table' AND name = 'runtime_settings' + `).get(); + if (!table) { + return undefined; + } + return database.prepare(` + SELECT setting_value + FROM runtime_settings + WHERE setting_key = ? + `).get(name)?.setting_value; + } catch (error) { + process.stderr.write(`[daedalus-agent] failed to read runtime setting ${name}: ${error.message}\n`); + return undefined; + } finally { + database?.close(); + } +} + function value(name, fallback = "") { const current = userEnv[name]; if (current === undefined || current === null || current === "") { @@ -742,8 +770,10 @@ const shared = { DA_ONTOLOGY_DIR: path.posix.join(productRoot, "ontology"), DA_TMP_DIR: path.posix.join(productRoot, ".openclaw/tmp"), DA_WEB_DATA_DIR: path.posix.join(productRoot, "management/data"), - DA_INTERNAL_NETWORK_ALLOWLIST: valueWithDefault("DA_INTERNAL_NETWORK_ALLOWLIST", ""), - DA_OPENCLAW_GATEWAY_MAX_CONCURRENT: positiveIntegerWithDefault("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT", 16), + DA_INTERNAL_NETWORK_ALLOWLIST: process.env.DA_INTERNAL_NETWORK_ALLOWLIST + ?? readRuntimeSetting("internal_network_allowlist") + ?? valueWithDefault("DA_INTERNAL_NETWORK_ALLOWLIST", ""), + DA_OPENCLAW_GATEWAY_MAX_CONCURRENT: positiveIntegerWithDefault("DA_OPENCLAW_GATEWAY_MAX_CONCURRENT", 128), DA_MANAGEMENT_WEBSITE_TIMEOUT: String(value("DA_MANAGEMENT_WEBSITE_TIMEOUT", "180")), DA_SYSTEM_STATUS_INITIAL_DELAY_MS: positiveIntegerWithDefault("DA_SYSTEM_STATUS_INITIAL_DELAY_MS", 60000), DA_SYSTEM_STATUS_REFRESH_INTERVAL_MS: positiveIntegerWithDefault("DA_SYSTEM_STATUS_REFRESH_INTERVAL_MS", 60000), @@ -757,6 +787,7 @@ config.agents.defaults = config.agents.defaults || {}; config.agents.defaults.workspace = path.posix.join(productRoot, ".openclaw/workspaces/default"); config.agents.defaults.heartbeat = { ...(config.agents.defaults.heartbeat || {}), + every: "0m", isolatedSession: true, lightContext: true }; diff --git a/docker/docs/environment-variables.md b/docker/docs/environment-variables.md index 2c33195a4c9c933631a347256b792de733c383a7..021b03b2dc2548e68b066057efb2e3337da51c81 100644 --- a/docker/docs/environment-variables.md +++ b/docker/docs/environment-variables.md @@ -7,9 +7,7 @@ DAEDALUS_AGENT Docker 支持通过 `/var/daedalus-agent/env.json` 配置少量 | 变量 | 默认值 | 说明 | | --- | --- | --- | | `DA_MANAGEMENT_CONSOLE_PORT` | `3030` | management web 监听端口。建议宿主机映射整个 `3030-3050` 区间。 | -| `DA_OPENCLAW_GATEWAY_MAX_CONCURRENT` | `16` | OpenClaw gateway 执行 lane 并发数，启动时写入 `openclaw.json` 的 `agents.defaults.maxConcurrent`。 | -| `DA_HUOZIGE_SERVER_IP` | 空字符串 | 对话应用/业务应用所在的活字格服务器 IP 地址或主机名。通过管理控制台修改时会同步维护 `DA_INTERNAL_NETWORK_ALLOWLIST`。 | -| `DA_INTERNAL_NETWORK_ALLOWLIST` | 空字符串 | 允许 `daedalus-agent` 访问的局域网地址或 CIDR，支持逗号、空格、分号或 JSON 数组字符串。通过基础设置修改活字格服务器地址时会自动移除旧地址并加入新地址。 | +| `DA_OPENCLAW_GATEWAY_MAX_CONCURRENT` | `128` | OpenClaw gateway 执行 lane 并发数，启动时写入 `openclaw.json` 的 `agents.defaults.maxConcurrent`。 | | `DA_MANAGEMENT_WEBSITE_TIMEOUT` | `180` | management web 访问内部管理动作的超时时间，单位秒。 | | `DA_SYSTEM_STATUS_INITIAL_DELAY_MS` | `60000` | 管理控制台启动后首次采集系统状态的延迟，单位毫秒；覆盖 MQTT、S3、内核、插件加载和调用服务等状态。 | | `DA_SYSTEM_STATUS_REFRESH_INTERVAL_MS` | `60000` | 首次采集完成后系统状态缓存的刷新间隔，单位毫秒。 | @@ -20,21 +18,22 @@ DAEDALUS_AGENT Docker 支持通过 `/var/daedalus-agent/env.json` 配置少量 ```json { "DA_MANAGEMENT_CONSOLE_PORT": 3030, - "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 16, - "DA_HUOZIGE_SERVER_IP": "", - "DA_INTERNAL_NETWORK_ALLOWLIST": "", + "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 128, "DA_SYSTEM_STATUS_INITIAL_DELAY_MS": 60000, "DA_SYSTEM_STATUS_REFRESH_INTERVAL_MS": 60000, "DA_SYSTEM_STATUS_TIMEOUT_MS": 5000 } ``` +对话应用/业务应用所在的活字格服务器地址和内部网络白名单保存在 management web 的 SQLite 数据库中，应通过管理控制台“系统设置 -> 基础设置”修改。二者在同一事务内提交；IP 地址会自动加入白名单，域名按外网地址处理，不加入白名单。旧版本 `env.json` 中的 `DA_HUOZIGE_SERVER_IP` 和 `DA_INTERNAL_NETWORK_ALLOWLIST` 仅作为升级迁移入口，迁移后会从文件中删除。 + ## 容器启动调试变量 这些变量通过 `docker run -e` 或编排系统注入，用于运维调试，不建议放入 `/var/daedalus-agent/env.json`。 | 变量 | 默认值 | 说明 | | --- | --- | --- | +| `DA_INTERNAL_NETWORK_ALLOWLIST` | 数据库配置 | 临时覆盖 SQLite 中的内部网络白名单，支持逗号、空格、分号或 JSON 数组字符串；仅建议应急运维使用。 | | `DA_SERVICE_RESTART_DELAY_SECONDS` | `2` | 顶层 MQTT、MinIO、OpenClaw supervisor、apps supervisor 异常退出后的重启间隔。 | | `DA_APPS_RESTART_DELAY_SECONDS` | `2` | apps supervisor 内部重启 management web 和 invoke service 的间隔。 | | `OPENCLAW_RESTART_DELAY_SECONDS` | `2` | OpenClaw gateway 重启间隔。 | diff --git a/docker/docs/startup-steps.md b/docker/docs/startup-steps.md index e84c10f2a32ca69bc04e0fa82cc5cba5c368e608..842b6f6a7573c7133044fcacdff90b61e9d64d1a 100644 --- a/docker/docs/startup-steps.md +++ b/docker/docs/startup-steps.md @@ -9,7 +9,7 @@ | 1/14 | `require_input_files` | 检查 `/var/daedalus-agent/env.json` 和 `config.json`；缺失时播种 sample 文件并退出，存在时校验为 JSON object，并拒绝用户覆盖托管变量。 | | 2/14 | `ensure_directories` | 创建统一运行目录：`.openclaw`、`logs`、`generated`、`management/data`、`ontology`、`workspace-templates`、`minio`、`mqtt/aedes`、`startup-state`、`scripts`、`run`，并规范用户/组/权限。 | | 3/14 | `ensure_credentials` | 生成或复用 MQTT/MinIO 随机密码，写入 `credentials.json`。 | -| 4/14 | `generate_runtime_files` | 合并用户环境、凭据和配置，生成 `generated/runtime-env.json` 与 `generated/runtime-config.json`。 | +| 4/14 | `generate_runtime_files` | 合并容器环境、SQLite 运行设置、用户环境、凭据和配置，生成 `generated/runtime-env.json` 与 `generated/runtime-config.json`。 | | 5/14 | `sync_optional_user_files` | 同步可选 `agent_workspace` 到 `workspace-templates/default`。 | | 6/14 | `ensure_default_startup_hook` | 确保 `/var/daedalus-agent/run_after_startup.sh` 存在；缺失时写入默认脚本。 | | 7/14 | `start_mqtt` | 启动内置 MQTT broker 并等待 1883 可用。 | diff --git a/docker/sample/env.json b/docker/sample/env.json index 2472b6a99eb5c9378d25d6f489908b987b6e3338..cd49e333d6ff0cab5d0c775071da507c3fd4f537 100644 --- a/docker/sample/env.json +++ b/docker/sample/env.json @@ -1,6 +1,4 @@ { "DA_MANAGEMENT_CONSOLE_PORT": 3030, - "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 100, - "DA_HUOZIGE_SERVER_IP": "", - "DA_INTERNAL_NETWORK_ALLOWLIST": "127.0.0.1,::1" + "DA_OPENCLAW_GATEWAY_MAX_CONCURRENT": 128 } diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent-template/service.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent-template/service.ts index 488004c54ecdd950cb9ed47608511e81a415bdf0..ddb9880441f099338d6f6b307e6b143d7c536f09 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent-template/service.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent-template/service.ts @@ -1,5 +1,6 @@ import type { Stats } from "node:fs"; -import { lstat, mkdir, readdir, rm, stat } from "node:fs/promises"; +import { randomUUID } from "node:crypto"; +import { access, lstat, mkdir, readdir, rename, rm, stat } from "node:fs/promises"; import path from "node:path"; import { ManagedStateStore } from "../agent/state-store.js"; @@ -74,26 +75,55 @@ export class TemplateManager { throw new Error("agent.template.create requires objectKey"); } + const templatesRoot = this.getTemplatesRoot(); + await mkdir(templatesRoot, { recursive: true }); const targetDir = this.getTemplateDir(params.templateName); - if (params.replace) { - await rm(targetDir, { recursive: true, force: true }); - } else { + const operationId = `${process.pid}-${randomUUID()}`; + const stagingDir = path.join(templatesRoot, `.${params.templateName}.${operationId}.staging`); + const backupDir = path.join(templatesRoot, `.${params.templateName}.${operationId}.backup`); + let backupCreated = false; + + try { + const zipPayload = await this.getObjectStore().getObject(params.bucket, params.objectKey); + await extractZipToDirectory(zipPayload, stagingDir, "workspace templates root"); + const agentsFile = await stat(path.join(stagingDir, "AGENTS.md")).catch(() => null); + if (!agentsFile?.isFile()) { + throw new Error(`Agent template must contain top-level AGENTS.md: ${params.templateName}`); + } + await this.normalizeTemplateTree(stagingDir); + + const targetExists = await this.pathExists(targetDir); + if (targetExists && !params.replace) { + throw new Error(`Template already exists: ${params.templateName}`); + } + if (targetExists) { + await rename(targetDir, backupDir); + backupCreated = true; + } try { - const info = await stat(targetDir); - if (info.isDirectory()) { - throw new Error(`Template already exists: ${params.templateName}`); - } - } catch (error: unknown) { - if ((error as NodeJS.ErrnoException).code !== "ENOENT") { - throw error; + await rename(stagingDir, targetDir); + } catch (error) { + if (backupCreated) { + await rename(backupDir, targetDir).catch(() => undefined); + backupCreated = false; } + throw error; + } + if (backupCreated) { + await rm(backupDir, { recursive: true, force: true }); + backupCreated = false; + } + return await this.readTemplateRecord(params.templateName); + } finally { + await rm(stagingDir, { recursive: true, force: true }).catch(() => undefined); + if (backupCreated && !(await this.pathExists(targetDir))) { + await rename(backupDir, targetDir).catch(() => undefined); + backupCreated = false; + } + if (!backupCreated) { + await rm(backupDir, { recursive: true, force: true }).catch(() => undefined); } } - - const zipPayload = await this.getObjectStore().getObject(params.bucket, params.objectKey); - await extractZipToDirectory(zipPayload, targetDir, "workspace templates root"); - await this.normalizeTemplateTree(targetDir); - return await this.readTemplateRecord(params.templateName); } async deleteTemplate(params: AgentTemplateDeleteParams): Promise<{ templateName: string; path: string }> { @@ -176,6 +206,18 @@ export class TemplateManager { } } + private async pathExists(targetPath: string): Promise { + try { + await access(targetPath); + return true; + } catch (error: unknown) { + if (isMissingPathError(error)) { + return false; + } + throw error; + } + } + private getObjectStore(): S3ObjectStore { if (!this.objectStore) { this.objectStore = createS3ObjectStore(this.env); diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/lifecycle-service.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/lifecycle-service.ts index d8b1e842579fe4866e507b7e39b26e1d7f05e38a..339a211bcd3c97d0c209af1b370c2bd743cd110d 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/lifecycle-service.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/lifecycle-service.ts @@ -1,4 +1,4 @@ -import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { mkdir, readFile } from "node:fs/promises"; import path from "node:path"; import { CliRunner } from "../../tools/cli-runner.js"; @@ -7,6 +7,7 @@ import { toChinaISOString } from "../../china-time.js"; import type { AgentNameSetParams, AgentModelSetParams, + AgentUpdateParams, EnvironmentConfig, ManagedAgentRecord } from "../../types.js"; @@ -19,6 +20,7 @@ import { type ResolvedAgentCreateOptions } from "./support.js"; import { ensureSharedMode } from "../shared-permissions.js"; +import { atomicWriteFile } from "../../tools/atomic-file.js"; const USAGE_QUERY_LIMIT = 100000; const GATEWAY_CALL_TIMEOUT_MS = "30000"; @@ -169,6 +171,54 @@ export class AgentLifecycleService { return (await this.findAgentConfigIndex(agentId)) >= 0; } + async deriveManagedAgents(fallbackRecords: ManagedAgentRecord[]): Promise { + const config = await this.readOpenClawConfig(); + const agents = Array.isArray(config.agents?.list) ? config.agents.list : []; + const bindings = Array.isArray(config.bindings) ? config.bindings : []; + const accounts = config.channels?.[MQTT_CHANNEL_ID]?.accounts ?? {}; + const fallbackById = new Map(fallbackRecords.map((record) => [record.id, record])); + const now = toChinaISOString(); + const hasUnsupportedMetadata = agents.some((agent: Record) => ( + Object.prototype.hasOwnProperty.call(agent, "daedalusAgent") + )); + + const derived = agents.flatMap((agent: Record) => { + const binding = bindings.find((item: Record) => ( + item?.type === "route" + && item?.agentId === agent.id + && item?.match?.channel === MQTT_CHANNEL_ID + )); + const accountId = firstText(binding?.match?.accountId, agent.daedalusAgent?.accountId); + const account = accountId ? accounts[accountId] : undefined; + if (!agent.id || !accountId || !account) { + return []; + } + const fallback = fallbackById.get(agent.id); + return [{ + id: agent.id, + accountId, + workspace: firstText(fallback?.workspace, agent.workspace), + inboundTopic: firstText(fallback?.inboundTopic, account.topics?.inbound, agent.daedalusAgent?.inboundTopic), + outboundTopic: firstText(fallback?.outboundTopic, account.topics?.outbound, agent.daedalusAgent?.outboundTopic), + enabled: account.enabled !== false, + templateName: firstText(fallback?.templateName, agent.daedalusAgent?.templateName, "default"), + createdAt: fallback?.createdAt, + updatedAt: fallback?.updatedAt || now + }]; + }); + if (hasUnsupportedMetadata) { + for (const agent of agents) { + delete agent.daedalusAgent; + } + await this.writeOpenClawConfig(config); + } + const derivedIds = new Set(derived.map((record: ManagedAgentRecord) => record.id)); + return [ + ...derived, + ...fallbackRecords.filter((record) => !derivedIds.has(record.id)) + ].sort((left: ManagedAgentRecord, right: ManagedAgentRecord) => left.id.localeCompare(right.id)); + } + async ensureManagedTopologyBase(): Promise { const config = await this.readOpenClawConfig(); config.plugins ??= {}; @@ -455,7 +505,53 @@ export class AgentLifecycleService { const serialized = `${JSON.stringify(config, null, 2)}\n`; await mkdir(path.dirname(targetPath), { recursive: true, mode: SHARED_DIRECTORY_MODE }); - await writeFile(targetPath, serialized, "utf8"); + await atomicWriteFile(targetPath, serialized, {}, SHARED_FILE_MODE); await ensureSharedMode(targetPath, SHARED_FILE_MODE); } + + async snapshotOpenClawConfig(): Promise> { + return await this.readOpenClawConfig(); + } + + async restoreOpenClawConfig(config: Record): Promise { + await this.writeOpenClawConfig(config); + } + + async updateManagedTopology(record: ManagedAgentRecord, params: AgentUpdateParams): Promise { + const config = await this.readOpenClawConfig(); + const agents = Array.isArray(config.agents?.list) ? config.agents.list : []; + const agent = agents.find((entry: { id?: string }) => entry?.id === record.id); + if (!agent) { + throw new Error(`Agent config not found: ${record.id}`); + } + + if (params.name !== undefined) { + const name = firstText(params.name); + if (!name) { + throw new Error("Agent name is required"); + } + agent.name = name; + } + if (params.model !== undefined) { + agent.model = normalizeModelConfig(params.model); + } + if (params.enabled !== undefined) { + this.upsertManagedAccountInConfig( + config, + record.accountId, + record.inboundTopic, + record.outboundTopic, + params.enabled + ); + this.upsertManagedBindingInConfig(config, record.id, record.accountId); + } + + await this.writeOpenClawConfig(config); + return { + agentId: record.id, + name: agent.name, + model: agent.model, + enabled: params.enabled ?? record.enabled + }; + } } diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/service.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/service.ts index e6b7459d6cff8c99b47a5546f724f69211b1c2d0..776e0beb75cee9f9c7f56dbf4446db91b7c77c4b 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/service.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/service.ts @@ -7,6 +7,7 @@ import type { AgentGetParams, AgentNameSetParams, AgentModelSetParams, + AgentUpdateParams, AgentMutationParams, AgentWorkspaceExportParams, AgentWorkspaceExportResult, @@ -73,7 +74,7 @@ export class OpenClawManager { } async listAgents(): Promise { - return await this.lifecycleService.listAgents(await this.stateStore.listAgents()); + return await this.lifecycleService.listAgents(await this.reconcileManagedState()); } async getAgent(params: AgentGetParams): Promise { @@ -89,7 +90,7 @@ export class OpenClawManager { async createAgent(params: AgentCreateParams): Promise { const resolved = resolveAgentCreateOptions(this.env, params); - const existing = await this.stateStore.getAgent(resolved.agentId); + const existing = (await this.reconcileManagedState()).find((record) => record.id === resolved.agentId); if (existing) { throw new Error(`Managed agent already exists: ${resolved.agentId}`); } @@ -98,10 +99,6 @@ export class OpenClawManager { throw new Error(`OpenClaw agent already exists: ${resolved.agentId}`); } - await this.lifecycleService.ensureManagedTopologyBase(); - await this.workspaceService.ensureWorkspaceSeeded(resolved.workspace, resolved.templateName, { overwrite: true }); - await this.lifecycleService.createManagedTopology(resolved); - const record: ManagedAgentRecord = { id: resolved.agentId, accountId: resolved.accountId, @@ -114,54 +111,83 @@ export class OpenClawManager { updatedAt: toChinaISOString() }; - await this.stateStore.upsertAgent(record); - await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange( - params.restart, - `agent.create:${resolved.agentId}` - ); - - return record; + const configSnapshot = await this.lifecycleService.snapshotOpenClawConfig(); + try { + await this.lifecycleService.ensureManagedTopologyBase(); + await this.workspaceService.ensureWorkspaceSeeded(resolved.workspace, resolved.templateName, { overwrite: true }); + await this.lifecycleService.createManagedTopology(resolved); + await this.stateStore.upsertAgent(record); + await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange( + params.restart, + `agent.create:${resolved.agentId}` + ); + return record; + } catch (error) { + await this.lifecycleService.restoreOpenClawConfig(configSnapshot).catch(() => undefined); + await this.stateStore.removeAgent(record.id).catch(() => undefined); + throw error; + } } async enableAgent(params: AgentMutationParams): Promise { const record = await this.requireManagedAgent(params.agentId); - await this.lifecycleService.enableManagedTopology(record); - const updated: ManagedAgentRecord = { ...record, enabled: true, updatedAt: toChinaISOString() }; - await this.stateStore.upsertAgent(updated); - await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange(params.restart, `agent.enable:${params.agentId}`); - return updated; + const configSnapshot = await this.lifecycleService.snapshotOpenClawConfig(); + try { + await this.lifecycleService.enableManagedTopology(record); + await this.stateStore.upsertAgent(updated); + await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange(params.restart, `agent.enable:${params.agentId}`); + return updated; + } catch (error) { + await this.lifecycleService.restoreOpenClawConfig(configSnapshot).catch(() => undefined); + await this.stateStore.upsertAgent(record).catch(() => undefined); + throw error; + } } async disableAgent(params: AgentMutationParams): Promise { const record = await this.requireManagedAgent(params.agentId); - await this.lifecycleService.disableManagedTopology(record); - const updated: ManagedAgentRecord = { ...record, enabled: false, updatedAt: toChinaISOString() }; - await this.stateStore.upsertAgent(updated); - await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange(params.restart, `agent.disable:${params.agentId}`); - return updated; + const configSnapshot = await this.lifecycleService.snapshotOpenClawConfig(); + try { + await this.lifecycleService.disableManagedTopology(record); + await this.stateStore.upsertAgent(updated); + await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange(params.restart, `agent.disable:${params.agentId}`); + return updated; + } catch (error) { + await this.lifecycleService.restoreOpenClawConfig(configSnapshot).catch(() => undefined); + await this.stateStore.upsertAgent(record).catch(() => undefined); + throw error; + } } async deleteAgent(params: AgentMutationParams): Promise<{ agentId: string; accountId: string }> { const record = await this.requireManagedAgent(params.agentId); - await this.lifecycleService.deleteManagedTopology(record); - await this.workspaceService.moveWorkspaceToRecycleBin(record.workspace, record.id); - await this.stateStore.removeAgent(record.id); - await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange(params.restart, `agent.delete:${params.agentId}`); - - return { - agentId: record.id, - accountId: record.accountId - }; + const configSnapshot = await this.lifecycleService.snapshotOpenClawConfig(); + let recycledWorkspacePath: string | undefined; + try { + await this.lifecycleService.deleteManagedTopology(record); + recycledWorkspacePath = (await this.workspaceService.moveWorkspaceToRecycleBin(record.workspace, record.id)).path; + await this.stateStore.removeAgent(record.id); + await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange(params.restart, `agent.delete:${params.agentId}`); + return { + agentId: record.id, + accountId: record.accountId + }; + } catch (error) { + await this.lifecycleService.restoreOpenClawConfig(configSnapshot).catch(() => undefined); + await this.workspaceService.restoreWorkspaceFromRecycleBin(recycledWorkspacePath, record.workspace).catch(() => undefined); + await this.stateStore.upsertAgent(record).catch(() => undefined); + throw error; + } } async queryAllUsage(params: { agentIds?: string[] } = {}): Promise { @@ -180,6 +206,41 @@ export class OpenClawManager { return result; } + async updateAgent(params: AgentUpdateParams): Promise { + const record = await this.requireManagedAgent(params.agentId); + const configSnapshot = await this.lifecycleService.snapshotOpenClawConfig(); + const previousDocs = params.docsContent === undefined + ? undefined + : await this.workspaceService.readAgentDocs(record.workspace); + const updatedRecord: ManagedAgentRecord = { + ...record, + enabled: params.enabled ?? record.enabled, + updatedAt: toChinaISOString() + }; + + try { + const result = await this.lifecycleService.updateManagedTopology(record, params); + if (params.docsContent !== undefined) { + await this.workspaceService.updateAgentDocs(record.workspace, params.docsContent); + } + if (params.enabled !== undefined && params.enabled !== record.enabled) { + await this.stateStore.upsertAgent(updatedRecord); + } + await this.operationsService.maybeRefreshGatewayAfterManagedTopologyChange( + params.restart, + `agent.update:${params.agentId}` + ); + return result; + } catch (error) { + await this.lifecycleService.restoreOpenClawConfig(configSnapshot).catch(() => undefined); + if (params.docsContent !== undefined) { + await this.workspaceService.restoreAgentDocs(record.workspace, previousDocs).catch(() => undefined); + } + await this.stateStore.upsertAgent(record).catch(() => undefined); + throw error; + } + } + async listLlmProviders(): Promise { return await this.llmModelConfigService.listProviders(); } @@ -280,7 +341,7 @@ export class OpenClawManager { } private async requireManagedAgent(agentId: string): Promise { - const record = await this.stateStore.getAgent(agentId); + const record = (await this.reconcileManagedState()).find((item) => item.id === agentId); if (!record) { throw new Error(`Managed agent not found: ${agentId}`); } @@ -288,6 +349,15 @@ export class OpenClawManager { return record; } + private async reconcileManagedState(): Promise { + const stored = await this.stateStore.listAgents(); + const derived = await this.lifecycleService.deriveManagedAgents(stored); + if (JSON.stringify(derived) !== JSON.stringify(stored)) { + await this.stateStore.write({ version: 1, agents: derived }).catch(() => undefined); + } + return derived; + } + private getObjectStore(): S3ObjectStore { if (!this.objectStore) { this.objectStore = createS3ObjectStore(this.env); diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/state-store.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/state-store.ts index 3817477fb3543f43ed3c108dadb68be1ee169454..5f5efe90426042d5c32a83a59bbd66239785284d 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/state-store.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/state-store.ts @@ -1,7 +1,7 @@ -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import path from "node:path"; +import { readFile } from "node:fs/promises"; import type { ManagedAgentRecord, ManagedState } from "../../types.js"; +import { atomicWriteFile } from "../../tools/atomic-file.js"; const DEFAULT_STATE: ManagedState = { version: 1, @@ -30,8 +30,7 @@ export class ManagedStateStore { } async write(state: ManagedState): Promise { - await mkdir(path.dirname(this.filePath), { recursive: true }); - await writeFile(this.filePath, `${JSON.stringify(state, null, 2)}\n`, "utf8"); + await atomicWriteFile(this.filePath, `${JSON.stringify(state, null, 2)}\n`); } async listAgents(): Promise { diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/update.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/update.ts new file mode 100644 index 0000000000000000000000000000000000000000..812e73518716ea22b6cb8b654724ce2d78a89ae0 --- /dev/null +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/update.ts @@ -0,0 +1,6 @@ +import type { AgentUpdateParams } from "../../types.js"; +import { readCapabilityParams, type CapabilityHandler } from "../context.js"; + +export const agentUpdate: CapabilityHandler = async (request, context) => { + return await context.agentManager.updateAgent(readCapabilityParams(request)); +}; diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/workspace-service.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/workspace-service.ts index 5ded28e25325a443f4ac4a478963f42a0c0012eb..7eb58dc6ccf2bd9ec5e0ad3f2d7186570d715eee 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/workspace-service.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/agent/workspace-service.ts @@ -1,5 +1,5 @@ import { constants as fsConstants } from "node:fs"; -import { copyFile, lstat, mkdir, readFile, readdir, rename, stat, writeFile } from "node:fs/promises"; +import { copyFile, lstat, mkdir, readFile, readdir, rename, rm, stat } from "node:fs/promises"; import path from "node:path"; import AdmZip from "adm-zip"; @@ -12,6 +12,7 @@ import { SHARED_FILE_MODE } from "./support.js"; import { ensureSharedMode, ensureSharedPathPermissions } from "../shared-permissions.js"; +import { atomicWriteFile } from "../../tools/atomic-file.js"; const WORKSPACE_TEMPLATE_METADATA_FILES = new Set([ ".daedalus-agent-builtin.fingerprint" @@ -44,11 +45,30 @@ export class AgentWorkspaceService { await mkdir(workspace, { recursive: true, mode: SHARED_DIRECTORY_MODE }); await ensureSharedPathPermissions(workspace, SHARED_DIRECTORY_MODE, workspaceGroupId); const normalized = content.endsWith("\n") ? content : `${content}\n`; - await writeFile(targetPath, normalized, "utf8"); + await atomicWriteFile(targetPath, normalized, {}, SHARED_AGENT_DOCS_FILE_MODE); await ensureSharedPathPermissions(targetPath, SHARED_AGENT_DOCS_FILE_MODE, workspaceGroupId); return { path: targetPath }; } + async readAgentDocs(workspace: string): Promise { + try { + return await readFile(path.join(workspace, "AGENTS.md"), "utf8"); + } catch (error: unknown) { + if ((error as NodeJS.ErrnoException)?.code === "ENOENT") { + return undefined; + } + throw error; + } + } + + async restoreAgentDocs(workspace: string, content: string | undefined): Promise { + if (content === undefined) { + await rm(path.join(workspace, "AGENTS.md"), { force: true }); + return; + } + await this.updateAgentDocs(workspace, content); + } + async buildWorkspaceExportArchive(workspace: string): Promise<{ buffer: Buffer; entries: string[]; @@ -100,6 +120,22 @@ export class AgentWorkspaceService { return { path: targetPath }; } + async restoreWorkspaceFromRecycleBin(recyclePath: string | undefined, workspace: string): Promise { + if (!recyclePath) { + return; + } + try { + await stat(recyclePath); + } catch (error: unknown) { + if ((error as NodeJS.ErrnoException)?.code === "ENOENT") { + return; + } + throw error; + } + await mkdir(path.dirname(workspace), { recursive: true, mode: SHARED_DIRECTORY_MODE }); + await rename(recyclePath, workspace); + } + private async copyDirectoryContents(sourceDir: string, targetDir: string, options: { overwrite?: boolean } = {}): Promise { const workspaceGroupId = await this.resolveWorkspaceGroupId(targetDir); await this.copyDirectoryContentsWithGroup(sourceDir, targetDir, workspaceGroupId, options); diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/apps/service.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/apps/service.ts index 937bc7721728e350be2548393aa1376414ea3b81..6f8be12db9885665273db3ae00ff77911afc347d 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/apps/service.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/apps/service.ts @@ -1,4 +1,5 @@ -import { mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { rmSync } from "node:fs"; +import { stat } from "node:fs/promises"; import path from "node:path"; import type { @@ -14,51 +15,20 @@ import type { import { toChinaISOString } from "../../china-time.js"; import { OntologyManager } from "../ontology/service.js"; -interface SystemsState { - version: 1; - items: BusinessSystemRecord[]; -} - -interface SystemLogsState { - version: 1; - items: BusinessSystemLogRecord[]; -} - -const DEFAULT_SYSTEMS_STATE: SystemsState = { - version: 1, - items: [] -}; - -const DEFAULT_SYSTEM_LOGS_STATE: SystemLogsState = { - version: 1, - items: [] -}; - const SUPPORTED_SYSTEM_PROVIDERS = new Set(["hzg", "phx"]); +const OBSOLETE_GENERATED_STATE_FILES = ["business-systems.json", "system-logs.json"]; -function slugifySystemId(value: string): string { - return value - .trim() - .toLowerCase() - .replace(/[^a-z0-9_-]+/g, "-") - .replace(/^-+|-+$/g, ""); -} - -function validateSystemId(systemId: string): void { - if (!/^[A-Za-z0-9_-]+$/.test(systemId)) { - throw new Error(`Invalid system id: ${systemId}`); +function removeObsoleteGeneratedStateFiles(stateFile: string): void { + const stateRoot = path.dirname(stateFile); + for (const fileName of OBSOLETE_GENERATED_STATE_FILES) { + rmSync(path.join(stateRoot, fileName), { force: true }); } } -function normalizeSystemStatus(status?: string): "active" | "disabled" { - return status === "disabled" ? "disabled" : "active"; -} - function normalizeRequiredText(value: string | undefined, errorMessage: string): string { if (!value?.trim()) { throw new Error(errorMessage); } - return value.trim(); } @@ -72,61 +42,63 @@ function normalizeSystemProvider(value: string | undefined, action: string): Bus if (!SUPPORTED_SYSTEM_PROVIDERS.has(normalized as BusinessSystemProvider)) { throw new Error(`${action} supports provider hzg or phx only`); } - return normalized as BusinessSystemProvider; } -function normalizeOptionalSystemProvider(value: string | undefined, fallback: BusinessSystemProvider): BusinessSystemProvider { - return value === undefined ? fallback : normalizeSystemProvider(value, "business system update"); +function normalizeSystemId(value: string | undefined, fallback?: string): string { + const systemId = normalizeOptionalText(value) || normalizeOptionalText(fallback); + if (!systemId || !/^[A-Za-z0-9_-]+$/.test(systemId)) { + throw new Error(`Invalid system id: ${systemId || ""}`); + } + return systemId; } -function normalizeUrl(value: string | undefined, errorMessage: string): string { - const normalized = normalizeRequiredText(value, errorMessage); - try { - return new URL(normalized).toString(); - } catch (error: unknown) { - const message = error instanceof Error ? error.message : String(error); - throw new Error(`${errorMessage}: ${message}`); - } +function operationRecord(params: { + systemId: string; + name?: string; + provider?: string; + ontologyName?: string; + ontologyPackage?: string; + description?: string; + status?: "active" | "disabled"; + bucket?: string; + objectKey?: string; + lastImportedAt?: string; + operationAt?: string; +}): BusinessSystemRecord { + const now = params.operationAt || toChinaISOString(); + return { + id: params.systemId, + name: normalizeOptionalText(params.name) || params.systemId, + provider: normalizeSystemProvider(params.provider || "hzg", "apps operation"), + ontologyName: normalizeOptionalText(params.ontologyName) || params.systemId, + ontologyPackage: normalizeOptionalText(params.ontologyPackage) || normalizeOptionalText(params.objectKey), + description: normalizeOptionalText(params.description), + status: params.status === "disabled" ? "disabled" : "active", + bucket: normalizeOptionalText(params.bucket), + objectKey: normalizeOptionalText(params.objectKey), + createdAt: now, + updatedAt: now, + lastImportedAt: params.lastImportedAt + }; } export class BusinessSystemManager { constructor( - private readonly env: EnvironmentConfig, + env: EnvironmentConfig, private readonly ontologyManager: OntologyManager - ) {} + ) { + removeObsoleteGeneratedStateFiles(env.stateFile); + } async listSystems(): Promise<{ items: BusinessSystemRecord[] }> { - const state = await this.readSystemsState(); - return { - items: [...state.items].sort((left, right) => left.id.localeCompare(right.id)) - }; + return { items: [] }; } async createSystem(params: BusinessSystemCreateParams): Promise { const name = normalizeRequiredText(params.name, "apps.create requires name"); + const systemId = normalizeSystemId(params.systemId, name); const provider = normalizeSystemProvider(params.provider, "apps.create"); - const applicationEndpoint = normalizeUrl( - params.applicationEndpoint, - "apps.create requires applicationEndpoint" - ); - const sessionServiceEndpoint = normalizeUrl( - params.sessionServiceEndpoint, - "apps.create requires sessionServiceEndpoint" - ); - - const systemId = params.systemId?.trim() || slugifySystemId(name); - if (!systemId) { - throw new Error("apps.create requires systemId or name that can be normalized into an id"); - } - validateSystemId(systemId); - - const state = await this.readSystemsState(); - if (state.items.some((item) => item.id === systemId)) { - throw new Error(`Business system already exists: ${systemId}`); - } - - const now = toChinaISOString(); const ontologyName = normalizeOptionalText(params.ontologyName) || systemId; const bucket = normalizeOptionalText(params.bucket); const objectKey = normalizeOptionalText(params.objectKey); @@ -134,7 +106,7 @@ export class BusinessSystemManager { throw new Error("apps.create requires bucket and objectKey together"); } - let lastImportedAt: string | undefined; + let operationAt: string | undefined; if (bucket && objectKey) { const ontology = await this.ontologyManager.createOntology({ name: ontologyName, @@ -143,137 +115,60 @@ export class BusinessSystemManager { replace: params.replaceOntology ?? true }); await this.assertBusinessOntologyIndex(ontology.path); - lastImportedAt = now; + operationAt = toChinaISOString(); } - const record: BusinessSystemRecord = { - id: systemId, + return operationRecord({ + ...params, + systemId, name, provider, ontologyName, - ontologyPackage: normalizeOptionalText(params.ontologyPackage) || objectKey, - description: normalizeOptionalText(params.description), - applicationEndpoint, - sessionServiceEndpoint, - status: normalizeSystemStatus(params.status), bucket, objectKey, - createdAt: now, - updatedAt: now, - lastImportedAt - }; - - state.items.push(record); - await this.writeSystemsState(state); - return record; + lastImportedAt: operationAt, + operationAt + }); } async updateSystem(params: BusinessSystemUpdateParams): Promise { - validateSystemId(params.systemId); - const state = await this.readSystemsState(); - const index = state.items.findIndex((item) => item.id === params.systemId); - if (index < 0) { - throw new Error(`Business system not found: ${params.systemId}`); - } - - const current = state.items[index]!; - const next: BusinessSystemRecord = { - ...current, - name: normalizeOptionalText(params.name) || current.name, - provider: normalizeOptionalSystemProvider(params.provider, current.provider), - ontologyName: normalizeOptionalText(params.ontologyName) || current.ontologyName, - ontologyPackage: normalizeOptionalText(params.ontologyPackage) || current.ontologyPackage, - description: normalizeOptionalText(params.description) || current.description, - applicationEndpoint: params.applicationEndpoint === undefined - ? current.applicationEndpoint - : normalizeUrl(params.applicationEndpoint, "business system update requires applicationEndpoint"), - sessionServiceEndpoint: params.sessionServiceEndpoint === undefined - ? current.sessionServiceEndpoint - : normalizeUrl(params.sessionServiceEndpoint, "business system update requires sessionServiceEndpoint"), - status: normalizeSystemStatus(params.status ?? current.status), - bucket: normalizeOptionalText(params.bucket) || current.bucket, - objectKey: normalizeOptionalText(params.objectKey) || current.objectKey, - updatedAt: toChinaISOString() - }; - - if ((params.bucket !== undefined || params.objectKey !== undefined) && !(normalizeOptionalText(params.bucket) && normalizeOptionalText(params.objectKey))) { - throw new Error("business system update requires bucket and objectKey together"); - } - - if ((params.bucket?.trim() && params.objectKey?.trim()) || params.replaceOntology) { - if (!(next.bucket && next.objectKey)) { - throw new Error("business system update requires bucket and objectKey when replacing ontology"); - } - const ontology = await this.ontologyManager.createOntology({ - name: next.ontologyName, - bucket: next.bucket, - objectKey: next.objectKey, - replace: params.replaceOntology ?? true - }); - await this.assertBusinessOntologyIndex(ontology.path); - next.lastImportedAt = next.updatedAt; - } - - state.items[index] = next; - await this.writeSystemsState(state); - return next; + return await this.createSystem({ + ...params, + name: normalizeOptionalText(params.name) || params.systemId, + provider: params.provider || "hzg" + }); } async enableSystem(params: BusinessSystemMutationParams): Promise { - return await this.updateSystem({ - systemId: params.systemId, - status: "active" - }); + const systemId = normalizeSystemId(params.systemId); + return operationRecord({ systemId, status: "active" }); } async disableSystem(params: BusinessSystemMutationParams): Promise { - return await this.updateSystem({ - systemId: params.systemId, - status: "disabled" - }); + const systemId = normalizeSystemId(params.systemId); + return operationRecord({ systemId, status: "disabled" }); } async deleteSystem(params: BusinessSystemMutationParams): Promise<{ systemId: string; ontologyName: string }> { - validateSystemId(params.systemId); - const state = await this.readSystemsState(); - const current = state.items.find((item) => item.id === params.systemId); - if (!current) { - throw new Error(`Business system not found: ${params.systemId}`); - } - - await this.ontologyManager.deleteOntology({ name: current.ontologyName }); - await this.writeSystemsState({ - version: 1, - items: state.items.filter((item) => item.id !== params.systemId) - }); - + const systemId = normalizeSystemId(params.systemId); + await this.ontologyManager.deleteOntology({ name: systemId }); return { - systemId: current.id, - ontologyName: current.ontologyName + systemId, + ontologyName: systemId }; } async recordLog(params: BusinessSystemLogRecordParams): Promise { - validateSystemId(params.systemId); - const systems = await this.readSystemsState(); - const system = systems.items.find((item) => item.id === params.systemId); - if (!system) { - throw new Error(`Business system not found: ${params.systemId}`); - } + const systemId = normalizeSystemId(params.systemId); const provider = normalizeSystemProvider(params.provider, "apps.log.record"); - if (provider !== system.provider) { - throw new Error(`apps.log.record provider mismatch: expected ${system.provider}, received ${provider}`); - } const applicationName = normalizeRequiredText(params.applicationName, "apps.log.record requires applicationName"); const commandName = normalizeRequiredText(params.commandName, "apps.log.record requires commandName"); if (!Number.isFinite(params.responseTimeMs)) { throw new Error("apps.log.record requires responseTimeMs"); } - - const logs = await this.readLogsState(); - const record: BusinessSystemLogRecord = { - id: `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`, - systemId: system.id, + return { + id: `${systemId}-${Date.now()}`, + systemId, agentId: normalizeOptionalText(params.agentId), conversationId: normalizeOptionalText(params.conversationId), provider, @@ -286,22 +181,6 @@ export class BusinessSystemManager { error: params.error, createdAt: params.createdAt ?? toChinaISOString() }; - - logs.items.push(record); - await this.writeLogsState(logs); - return record; - } - - private getStateRoot(): string { - return path.dirname(this.env.stateFile); - } - - private getSystemsStatePath(): string { - return path.join(this.getStateRoot(), "business-systems.json"); - } - - private getLogsStatePath(): string { - return path.join(this.getStateRoot(), "system-logs.json"); } private async assertBusinessOntologyIndex(ontologyPath: string): Promise { @@ -318,46 +197,4 @@ export class BusinessSystemManager { throw error; } } - - private async readSystemsState(): Promise { - try { - const content = await readFile(this.getSystemsStatePath(), "utf8"); - const parsed = JSON.parse(content) as Partial; - return { - version: 1, - items: Array.isArray(parsed.items) ? parsed.items : [] - }; - } catch (error: unknown) { - if ((error as NodeJS.ErrnoException)?.code === "ENOENT") { - return DEFAULT_SYSTEMS_STATE; - } - throw error; - } - } - - private async writeSystemsState(state: SystemsState): Promise { - await mkdir(this.getStateRoot(), { recursive: true }); - await writeFile(this.getSystemsStatePath(), `${JSON.stringify(state, null, 2)}\n`, "utf8"); - } - - private async readLogsState(): Promise { - try { - const content = await readFile(this.getLogsStatePath(), "utf8"); - const parsed = JSON.parse(content) as Partial; - return { - version: 1, - items: Array.isArray(parsed.items) ? parsed.items : [] - }; - } catch (error: unknown) { - if ((error as NodeJS.ErrnoException)?.code === "ENOENT") { - return DEFAULT_SYSTEM_LOGS_STATE; - } - throw error; - } - } - - private async writeLogsState(state: SystemLogsState): Promise { - await mkdir(this.getStateRoot(), { recursive: true }); - await writeFile(this.getLogsStatePath(), `${JSON.stringify(state, null, 2)}\n`, "utf8"); - } } diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/catalog/snapshot.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/catalog/snapshot.ts index 7274bda3ce1f9c54b91bdc04adaff0b0cc3ba728..40e90c6bb8c6ad0700f293ad96fdff7aaa09b87f 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/catalog/snapshot.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/catalog/snapshot.ts @@ -111,33 +111,6 @@ function mapSkills(payload: unknown) { })); } -function endpointFrom(value: unknown) { - const raw = firstText(value); - if (!raw) { - return null; - } - - try { - const url = new URL(raw); - return { - scheme: url.protocol.replace(/:$/, ""), - host: url.hostname, - port: url.port ? Number(url.port) : (url.protocol === "https:" ? 443 : 80) - }; - } catch { - return null; - } -} - -function mapSystems(payload: unknown) { - return listItems(payload).map((system) => ({ - id: firstText(system.id), - provider: firstText(system.provider), - status: firstText(system.status, "active"), - endpoint: endpointFrom(system.applicationEndpoint) - })); -} - function mapOntologies(payload: unknown) { return listItems(payload).map((ontology) => ({ id: firstText(ontology.name), @@ -151,14 +124,12 @@ export const catalogSnapshot: CapabilityHandler = async (_request, context) => { templates, agents, skills, - systems, ontologies ] = await Promise.all([ context.agentManager.listLlmProviders(), context.templateManager.listTemplates(), context.agentManager.listAgents(), context.agentManager.listGlobalSkills(), - context.businessSystemManager.listSystems(), context.ontologyManager.listOntologies() ]); @@ -167,7 +138,7 @@ export const catalogSnapshot: CapabilityHandler = async (_request, context) => { agents: mapAgents(agents), templates: mapTemplates(templates), skills: mapSkills(skills), - systems: mapSystems(systems), + systems: [], ontologies: mapOntologies(ontologies) }; diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/context.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/context.ts index 2894e34ab59663b3c5a8fa1ad0ffaa20ef93aa14..2e387defcfb8f7b3debcc4d72258523555f80465 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/context.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/context.ts @@ -5,6 +5,7 @@ import type { AgentNameSetParams, AgentModelSetParams, AgentMutationParams, + AgentUpdateParams, AgentWorkspaceExportParams, AgentWorkspaceExportResult, AgentTemplateCreateParams, @@ -44,6 +45,7 @@ export interface AgentCapacityService { setAgentModel(params: AgentModelSetParams): Promise; setAgentName(params: AgentNameSetParams): Promise; updateAgentDocs(params: AgentDocsUpdateParams): Promise<{ path: string }>; + updateAgent(params: AgentUpdateParams): Promise; exportAgentWorkspace(params: AgentWorkspaceExportParams): Promise; listLlmProviders(): Promise; createLlmProvider(params: LlmProviderCreateParams): Promise; diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/ontology/service.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/ontology/service.ts index a5379186b48bdbe27eb19f9258e046438ddf7d07..3f54013190e2f37d9c302b683a1ac27e757b8994 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/ontology/service.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/ontology/service.ts @@ -1,4 +1,5 @@ -import { mkdir, rm } from "node:fs/promises"; +import { randomUUID } from "node:crypto"; +import { access, mkdir, rename, rm, stat } from "node:fs/promises"; import path from "node:path"; import { createS3ObjectStore, extractZipToDirectory, type S3ObjectStore } from "../../tools/s3-archive.js"; @@ -53,15 +54,52 @@ export class OntologyManager { throw new Error("ontology.create requires objectKey"); } + await mkdir(this.env.ontologyRoot, { recursive: true }); const targetDir = path.join(this.env.ontologyRoot, params.name); - if (params.replace) { - await rm(targetDir, { recursive: true, force: true }); - } + const operationId = `${process.pid}-${randomUUID()}`; + const stagingDir = path.join(this.env.ontologyRoot, `.${params.name}.${operationId}.staging`); + const backupDir = path.join(this.env.ontologyRoot, `.${params.name}.${operationId}.backup`); + let backupCreated = false; - const zipPayload = await this.getObjectStore().getObject(params.bucket, params.objectKey); - await extractZipToDirectory(zipPayload, targetDir, "ontology root"); + try { + const zipPayload = await this.getObjectStore().getObject(params.bucket, params.objectKey); + await extractZipToDirectory(zipPayload, stagingDir, "ontology root"); + await this.assertTopLevelIndex(stagingDir); - return { path: targetDir }; + const targetExists = await this.pathExists(targetDir); + if (targetExists && !params.replace) { + throw new Error(`Ontology already exists: ${params.name}`); + } + if (targetExists) { + await rename(targetDir, backupDir); + backupCreated = true; + } + + try { + await rename(stagingDir, targetDir); + } catch (error) { + if (backupCreated) { + await rename(backupDir, targetDir).catch(() => undefined); + backupCreated = false; + } + throw error; + } + + if (backupCreated) { + await rm(backupDir, { recursive: true, force: true }); + backupCreated = false; + } + return { path: targetDir }; + } finally { + await rm(stagingDir, { recursive: true, force: true }).catch(() => undefined); + if (backupCreated && !(await this.pathExists(targetDir))) { + await rename(backupDir, targetDir).catch(() => undefined); + backupCreated = false; + } + if (!backupCreated) { + await rm(backupDir, { recursive: true, force: true }).catch(() => undefined); + } + } } async deleteOntology(params: OntologyDeleteParams): Promise<{ path: string }> { @@ -78,4 +116,31 @@ export class OntologyManager { return this.objectStore; } + + private async assertTopLevelIndex(ontologyPath: string): Promise { + const indexPath = path.join(ontologyPath, "index.md"); + try { + const info = await stat(indexPath); + if (!info.isFile()) { + throw new Error(`Business system ontology must contain top-level index.md: ${ontologyPath}`); + } + } catch (error: unknown) { + if ((error as NodeJS.ErrnoException)?.code === "ENOENT") { + throw new Error(`Business system ontology must contain top-level index.md: ${ontologyPath}`); + } + throw error; + } + } + + private async pathExists(targetPath: string): Promise { + try { + await access(targetPath); + return true; + } catch (error: unknown) { + if ((error as NodeJS.ErrnoException)?.code === "ENOENT") { + return false; + } + throw error; + } + } } diff --git a/packages/daedalus-agent-management-web/server/management-src/capabilities/registry.ts b/packages/daedalus-agent-management-web/server/management-src/capabilities/registry.ts index 3f2b87911ea45c7845b94572df05f42f1e497be8..70f9001a414268a5b287dfcb5567773c6925bd0e 100644 --- a/packages/daedalus-agent-management-web/server/management-src/capabilities/registry.ts +++ b/packages/daedalus-agent-management-web/server/management-src/capabilities/registry.ts @@ -7,6 +7,7 @@ import { agentGet } from "./agent/get.js"; import { agentModelSet } from "./agent/model-set.js"; import { agentNameSet } from "./agent/name-set.js"; import { agentUsageList } from "./agent/usage-list.js"; +import { agentUpdate } from "./agent/update.js"; import { agentWorkspaceExport } from "./agent/workspace-export.js"; import { agentTemplateCreate } from "./agent-template/create.js"; import { agentTemplateDelete } from "./agent-template/delete.js"; @@ -47,6 +48,7 @@ export const capabilityHandlers: Record = { "agent.model.set": agentModelSet, "agent.name.set": agentNameSet, "agent.docs.update": agentDocsUpdate, + "agent.update": agentUpdate, "agent.workspace.export": agentWorkspaceExport, "agent.template.create": agentTemplateCreate, "agent.template.delete": agentTemplateDelete, diff --git a/packages/daedalus-agent-management-web/server/management-src/tools/atomic-file.ts b/packages/daedalus-agent-management-web/server/management-src/tools/atomic-file.ts new file mode 100644 index 0000000000000000000000000000000000000000..d258fac9cbcbbb8fd4719d57b3e2296f6ae04b96 --- /dev/null +++ b/packages/daedalus-agent-management-web/server/management-src/tools/atomic-file.ts @@ -0,0 +1,51 @@ +import { randomUUID } from "node:crypto"; +import { + mkdir, + open, + rename, + rm, + type FileHandle +} from "node:fs/promises"; +import path from "node:path"; + +interface AtomicFileOperations { + mkdir: typeof mkdir; + open: typeof open; + rename: typeof rename; + rm: typeof rm; +} + +const DEFAULT_OPERATIONS: AtomicFileOperations = { + mkdir, + open, + rename, + rm +}; + +export async function atomicWriteFile( + targetPath: string, + content: string | Buffer, + operationOverrides: Partial = {}, + mode = 0o640 +): Promise { + const operations = { ...DEFAULT_OPERATIONS, ...operationOverrides }; + const targetDir = path.dirname(targetPath); + const tempPath = path.join(targetDir, `.${path.basename(targetPath)}.${process.pid}.${randomUUID()}.tmp`); + let handle: FileHandle | undefined; + + await operations.mkdir(targetDir, { recursive: true }); + try { + handle = await operations.open(tempPath, "wx", mode); + await handle.writeFile(content); + await handle.sync(); + await handle.close(); + handle = undefined; + await operations.rename(tempPath, targetPath); + } catch (error) { + if (handle) { + await handle.close().catch(() => undefined); + } + await operations.rm(tempPath, { force: true }).catch(() => undefined); + throw error; + } +} diff --git a/packages/daedalus-agent-management-web/server/management-src/types.ts b/packages/daedalus-agent-management-web/server/management-src/types.ts index 490bd9fc7ac1fad7397408188bfa762bc52c2ab2..01d34a28a2a2f8a24e2ace18ea8354212c747d1c 100644 --- a/packages/daedalus-agent-management-web/server/management-src/types.ts +++ b/packages/daedalus-agent-management-web/server/management-src/types.ts @@ -137,6 +137,18 @@ export interface AgentNameSetParams { restart?: boolean; } +export interface AgentUpdateParams { + agentId: string; + name?: string; + model?: string | { + primary: string; + fallbacks?: string[]; + }; + docsContent?: string; + enabled?: boolean; + restart?: boolean; +} + export interface LlmProviderCreateParams { providerId: string; baseUrl: string; diff --git a/packages/daedalus-agent-management-web/server/management-test/atomic-file.test.ts b/packages/daedalus-agent-management-web/server/management-test/atomic-file.test.ts new file mode 100644 index 0000000000000000000000000000000000000000..907ac87660fe17eb73ad6b2d02d367e75e4418f2 --- /dev/null +++ b/packages/daedalus-agent-management-web/server/management-test/atomic-file.test.ts @@ -0,0 +1,23 @@ +import assert from "node:assert/strict"; +import { mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { atomicWriteFile } from "../management-src/tools/atomic-file.js"; + +test("atomicWriteFile preserves the previous file when the final rename fails", async () => { + const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-atomic-file-")); + const targetPath = path.join(root, "state.json"); + await writeFile(targetPath, "{\"version\":1}\n", "utf8"); + + await assert.rejects( + atomicWriteFile(targetPath, "{\"version\":2}\n", { + async rename() { + throw new Error("injected rename failure"); + } + }), + /injected rename failure/ + ); + + assert.equal(await readFile(targetPath, "utf8"), "{\"version\":1}\n"); +}); diff --git a/packages/daedalus-agent-management-web/server/management-test/business-system-manager.test.ts b/packages/daedalus-agent-management-web/server/management-test/business-system-manager.test.ts index 279ee7bfd567b61189733c5f4102fd714bdc8793..48a0917aead83f8e5e703cf463409bb434971077 100644 --- a/packages/daedalus-agent-management-web/server/management-test/business-system-manager.test.ts +++ b/packages/daedalus-agent-management-web/server/management-test/business-system-manager.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { access, mkdir, mkdtemp, writeFile } from "node:fs/promises"; import path from "node:path"; import { tmpdir } from "node:os"; @@ -42,13 +42,44 @@ function createEnv(overrides: Partial = {}): EnvironmentConfi }; } -test("BusinessSystemManager creates systems with validated provider, endpoints, and ontology index", async () => { +async function assertFileDoesNotExist(filePath: string): Promise { + await assert.rejects(access(filePath), (error: NodeJS.ErrnoException) => error.code === "ENOENT"); +} + +test("BusinessSystemManager removes obsolete generated JSON state files", async () => { + const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-system-legacy-state-")); + const stateRoot = path.join(root, "state"); + const managedAgentsPath = path.join(stateRoot, "managed-agents.json"); + await mkdir(stateRoot, { recursive: true }); + await Promise.all([ + writeFile(path.join(stateRoot, "business-systems.json"), "{}\n", "utf8"), + writeFile(path.join(stateRoot, "system-logs.json"), "{}\n", "utf8"), + writeFile(managedAgentsPath, "{}\n", "utf8") + ]); + + new BusinessSystemManager( + createEnv({ stateFile: managedAgentsPath }), + { + async createOntology() { + return { path: "" }; + }, + async deleteOntology() { + return { path: "" }; + } + } as never + ); + + await assertFileDoesNotExist(path.join(stateRoot, "business-systems.json")); + await assertFileDoesNotExist(path.join(stateRoot, "system-logs.json")); + await access(managedAgentsPath); +}); + +test("BusinessSystemManager imports ontology without persisting systems or endpoints", async () => { const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-system-")); + const stateRoot = path.join(root, "state"); let imported: { name: string; bucket: string; objectKey: string; replace?: boolean } | undefined; const manager = new BusinessSystemManager( - createEnv({ - stateFile: path.join(root, "state", "managed-agents.json") - }), + createEnv({ stateFile: path.join(stateRoot, "managed-agents.json") }), { async createOntology(params) { imported = params; @@ -67,8 +98,8 @@ test("BusinessSystemManager creates systems with validated provider, endpoints, systemId: "crm", name: "CRM", provider: "HZG", - applicationEndpoint: "http://172.12.2.23:8081/wms", - sessionServiceEndpoint: "http://172.12.2.23:8080/chat", + applicationEndpoint: "http://legacy.example.com:8081/wms", + sessionServiceEndpoint: "http://legacy.example.com:8080/chat", bucket: "ontology-artifacts", objectKey: "crm/crm.zip" }); @@ -80,25 +111,16 @@ test("BusinessSystemManager creates systems with validated provider, endpoints, replace: true }); assert.equal(created.provider, "hzg"); - assert.equal(created.applicationEndpoint, "http://172.12.2.23:8081/wms"); - assert.equal(created.sessionServiceEndpoint, "http://172.12.2.23:8080/chat"); - - const listed = await manager.listSystems(); - assert.equal(listed.items.length, 1); - assert.equal(listed.items[0]?.id, "crm"); + assert.equal(created.applicationEndpoint, undefined); + assert.equal(created.sessionServiceEndpoint, undefined); + assert.deepEqual((await manager.listSystems()).items, []); + await assertFileDoesNotExist(path.join(stateRoot, "business-systems.json")); }); test("BusinessSystemManager rejects imported ontologies without top-level index.md", async () => { const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-system-bad-ontology-")); - await mkdir(path.join(root, "state"), { recursive: true }); - await writeFile(path.join(root, "state", "business-systems.json"), `${JSON.stringify({ - version: 1, - items: [] - }, null, 2)}\n`, "utf8"); const manager = new BusinessSystemManager( - createEnv({ - stateFile: path.join(root, "state", "managed-agents.json") - }), + createEnv({ stateFile: path.join(root, "state", "managed-agents.json") }), { async createOntology(params) { const ontologyPath = path.join(root, "ontology", params.name); @@ -112,42 +134,20 @@ test("BusinessSystemManager rejects imported ontologies without top-level index. } as never ); - await assert.rejects(async () => { - await manager.createSystem({ - systemId: "crm", - name: "CRM", - provider: "hzg", - applicationEndpoint: "http://172.12.2.23:8081/wms", - sessionServiceEndpoint: "http://172.12.2.23:8080/chat", - bucket: "ontology-artifacts", - objectKey: "crm/crm.zip" - }); - }, /top-level index\.md/); + await assert.rejects(manager.createSystem({ + systemId: "crm", + name: "CRM", + provider: "hzg", + bucket: "ontology-artifacts", + objectKey: "crm/crm.zip" + }), /top-level index\.md/); }); -test("BusinessSystemManager records log errors and keeps provider aligned with the system", async () => { +test("BusinessSystemManager returns compatible log results without persisting log JSON", async () => { const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-system-logs-")); const stateRoot = path.join(root, "state"); - await mkdir(stateRoot, { recursive: true }); - await writeFile(path.join(stateRoot, "business-systems.json"), `${JSON.stringify({ - version: 1, - items: [{ - id: "crm", - name: "CRM", - provider: "hzg", - ontologyName: "crm", - applicationEndpoint: "http://172.12.2.23:8081/wms", - sessionServiceEndpoint: "http://172.12.2.23:8080/chat", - status: "active", - createdAt: "2026-05-21T00:00:00.000Z", - updatedAt: "2026-05-21T00:00:00.000Z" - }] - }, null, 2)}\n`, "utf8"); - const manager = new BusinessSystemManager( - createEnv({ - stateFile: path.join(stateRoot, "managed-agents.json") - }), + createEnv({ stateFile: path.join(stateRoot, "managed-agents.json") }), { async createOntology() { return { path: "" }; @@ -165,63 +165,28 @@ test("BusinessSystemManager records log errors and keeps provider aligned with t commandName: "searchCustomer", responseTimeMs: 128, success: false, - error: { - code: "upstream_error" - } - }); - - assert.deepEqual(recorded.error, { - code: "upstream_error" + error: { code: "upstream_error" } }); - await assert.rejects(async () => { - await manager.recordLog({ - systemId: "crm", - provider: "phx", - applicationName: "crm", - commandName: "searchCustomer", - responseTimeMs: 100, - success: true - }); - }, /provider mismatch/); - - const persisted = JSON.parse(await readFile(path.join(stateRoot, "system-logs.json"), "utf8")) as { - items: Array<{ error?: unknown }>; - }; - assert.deepEqual(persisted.items[0]?.error, { - code: "upstream_error" - }); + assert.deepEqual(recorded.error, { code: "upstream_error" }); + await assert.rejects(manager.recordLog({ + systemId: "crm", + provider: "unsupported" as "hzg", + applicationName: "crm", + commandName: "searchCustomer", + responseTimeMs: 100, + success: true + }), /supports provider hzg or phx only/); + await assertFileDoesNotExist(path.join(stateRoot, "system-logs.json")); }); -test("BusinessSystemManager updates, toggles, and deletes according to the capability contract", async () => { +test("BusinessSystemManager keeps mutation actions as stateless compatibility adapters", async () => { const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-system-lifecycle-")); const stateRoot = path.join(root, "state"); let deletedOntologyName: string | undefined; let imported: { name: string; bucket: string; objectKey: string; replace?: boolean } | undefined; - - await mkdir(stateRoot, { recursive: true }); - await writeFile(path.join(stateRoot, "business-systems.json"), `${JSON.stringify({ - version: 1, - items: [{ - id: "crm", - name: "CRM", - provider: "hzg", - ontologyName: "crm", - ontologyPackage: "crm/crm.zip", - description: "old", - applicationEndpoint: "http://172.12.2.23:8081/wms", - sessionServiceEndpoint: "http://172.12.2.23:8080/chat", - status: "active", - bucket: "old-bucket", - objectKey: "old/object.zip", - createdAt: "2026-05-21T00:00:00.000Z", - updatedAt: "2026-05-21T00:00:00.000Z" - }] - }, null, 2)}\n`, "utf8"); const manager = new BusinessSystemManager( - createEnv({ - stateFile: path.join(stateRoot, "managed-agents.json") - }), + createEnv({ stateFile: path.join(stateRoot, "managed-agents.json") }), { async createOntology(params) { imported = params; @@ -255,15 +220,10 @@ test("BusinessSystemManager updates, toggles, and deletes according to the capab replace: true }); assert.equal(updated.description, "updated"); - assert.equal(updated.bucket, "new-bucket"); - assert.equal(updated.objectKey, "crm/crm.zip"); assert.equal(updated.lastImportedAt, updated.updatedAt); assert.equal(disabled.status, "disabled"); assert.equal(enabled.status, "active"); - assert.deepEqual(deleted, { - systemId: "crm", - ontologyName: "crm" - }); + assert.deepEqual(deleted, { systemId: "crm", ontologyName: "crm" }); assert.equal(deletedOntologyName, "crm"); - assert.deepEqual((await manager.listSystems()).items, []); + await assertFileDoesNotExist(path.join(stateRoot, "business-systems.json")); }); diff --git a/packages/daedalus-agent-management-web/server/management-test/capabilities-registry.test.ts b/packages/daedalus-agent-management-web/server/management-test/capabilities-registry.test.ts index 792a3c392470d1a41b9ca0b9f4efff4e6dfcd72e..cf41a765ea9775ad0800901c2c3ae00a52899691 100644 --- a/packages/daedalus-agent-management-web/server/management-test/capabilities-registry.test.ts +++ b/packages/daedalus-agent-management-web/server/management-test/capabilities-registry.test.ts @@ -20,6 +20,7 @@ const expectedActions = [ "agent.model.set", "agent.name.set", "agent.docs.update", + "agent.update", "agent.workspace.export", "agent.template.create", "agent.template.delete", @@ -72,6 +73,7 @@ function createContext(calls: Array<{ method: string; args: unknown[] }>): Capab setAgentModel: respond("agentManager.setAgentModel"), setAgentName: respond("agentManager.setAgentName"), updateAgentDocs: respond("agentManager.updateAgentDocs") as never, + updateAgent: respond("agentManager.updateAgent"), exportAgentWorkspace: respond("agentManager.exportAgentWorkspace") as never, listLlmProviders: respond("agentManager.listLlmProviders"), createLlmProvider: respond("agentManager.createLlmProvider"), @@ -216,6 +218,12 @@ test("capability handlers delegate to the expected service methods", async () => expectedMethod: "agentManager.updateAgentDocs", expectedArgs: [{ agentId: "lowcode", content: "# AGENTS" }] }, + { + action: "agent.update", + params: { agentId: "lowcode", name: "Lowcode Agent Updated", enabled: false }, + expectedMethod: "agentManager.updateAgent", + expectedArgs: [{ agentId: "lowcode", name: "Lowcode Agent Updated", enabled: false }] + }, { action: "agent.workspace.export", params: { agentId: "lowcode", bucket: "admin-out", objectKey: "agent-workspace/lowcode.zip" }, @@ -374,7 +382,6 @@ test("capability handlers delegate to the expected service methods", async () => "agentManager.listAgents", "agentManager.listGlobalSkills", "agentManager.listLlmProviders", - "businessSystemManager.listSystems", "ontologyManager.listOntologies", "templateManager.listTemplates" ].sort(), item.action); @@ -506,16 +513,9 @@ test("catalog.snapshot emits compact catalog payloads", async () => { isBuiltIn: true }] }) as never; - context.businessSystemManager.listSystems = async () => ({ - items: [{ - id: "crm", - name: "CRM", - provider: "phx", - status: "active", - applicationEndpoint: "https://crm.example.com:8443", - builtIn: true - }] - }) as never; + context.businessSystemManager.listSystems = async () => { + throw new Error("catalog.snapshot must not read kernel business-system state"); + }; context.ontologyManager.listOntologies = async () => ({ items: [{ name: "crm", @@ -565,16 +565,7 @@ test("catalog.snapshot emits compact catalog payloads", async () => { id: "planner", builtIn: true }); - assert.deepEqual(result.catalogs?.systems?.items?.[0], { - id: "crm", - provider: "phx", - status: "active", - endpoint: { - scheme: "https", - host: "crm.example.com", - port: 8443 - } - }); + assert.deepEqual(result.catalogs?.systems?.items, []); assert.deepEqual(result.catalogs?.ontologies?.items?.[0], { id: "crm", builtIn: true diff --git a/packages/daedalus-agent-management-web/server/management-test/ontology-manager.test.ts b/packages/daedalus-agent-management-web/server/management-test/ontology-manager.test.ts index 388f152978ae01825f08df23f723ab1fd1b68878..0463b13ddba60cfe9314b90e7e2ede14c94a6d1d 100644 --- a/packages/daedalus-agent-management-web/server/management-test/ontology-manager.test.ts +++ b/packages/daedalus-agent-management-web/server/management-test/ontology-manager.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { access, mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; import path from "node:path"; import { tmpdir } from "node:os"; @@ -46,6 +46,7 @@ test("OntologyManager creates, lists, and deletes ontologies", async () => { const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-ontology-")); const AdmZip = (await import("adm-zip")).default; const zip = new AdmZip(); + zip.addFile("index.md", Buffer.from("# CRM", "utf8")); zip.addFile("schema.json", Buffer.from("{\"name\":\"crm\"}", "utf8")); const manager = new OntologyManager( createEnv({ @@ -90,6 +91,63 @@ test("OntologyManager creates, lists, and deletes ontologies", async () => { ]); }); +test("OntologyManager preserves the current ontology when replacement download fails", async () => { + const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-ontology-download-failure-")); + const ontologyRoot = path.join(root, "ontology"); + const currentRoot = path.join(ontologyRoot, "crm"); + await mkdir(currentRoot, { recursive: true }); + await writeFile(path.join(currentRoot, "index.md"), "# Current\n", "utf8"); + + const manager = new OntologyManager(createEnv({ ontologyRoot }), { + objectStore: { + async getObject(): Promise { + throw new Error("injected download failure"); + } + } + }); + + await assert.rejects( + manager.createOntology({ + name: "crm", + bucket: "ontology-bucket", + objectKey: "crm/crm.zip", + replace: true + }), + /injected download failure/ + ); + assert.equal(await readFile(path.join(currentRoot, "index.md"), "utf8"), "# Current\n"); +}); + +test("OntologyManager rejects an invalid replacement before switching directories", async () => { + const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-ontology-validation-failure-")); + const ontologyRoot = path.join(root, "ontology"); + const currentRoot = path.join(ontologyRoot, "crm"); + await mkdir(currentRoot, { recursive: true }); + await writeFile(path.join(currentRoot, "index.md"), "# Current\n", "utf8"); + const invalidZip = new (await import("adm-zip")).default(); + invalidZip.addFile("schema.json", Buffer.from("{}", "utf8")); + + const manager = new OntologyManager(createEnv({ ontologyRoot }), { + objectStore: { + async getObject(): Promise { + return invalidZip.toBuffer(); + } + } + }); + + await assert.rejects( + manager.createOntology({ + name: "crm", + bucket: "ontology-bucket", + objectKey: "crm/crm.zip", + replace: true + }), + /top-level index\.md/ + ); + assert.equal(await readFile(path.join(currentRoot, "index.md"), "utf8"), "# Current\n"); + await assert.rejects(access(path.join(ontologyRoot, "crm.staging"))); +}); + test("OntologyManager marks built-in ontologies from manifest", async () => { const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-ontology-built-in-")); await mkdir(path.join(root, "ontology", "crm"), { recursive: true }); diff --git a/packages/daedalus-agent-management-web/server/management-test/openclaw-manager.test.ts b/packages/daedalus-agent-management-web/server/management-test/openclaw-manager.test.ts index 25480e3c57a827b5b5fa79b9ce23fe009d911232..86ce9fb33a3aaa23ac5a09c353d78a455b1d4964 100644 --- a/packages/daedalus-agent-management-web/server/management-test/openclaw-manager.test.ts +++ b/packages/daedalus-agent-management-web/server/management-test/openclaw-manager.test.ts @@ -683,6 +683,63 @@ test("listAgents includes managed agents missing from openclaw config", async () }); }); +test("listAgents migrates unsupported managed metadata out of openclaw agent config", async () => { + const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-agent-metadata-migration-")); + const configPath = path.join(root, ".openclaw", "openclaw.json"); + const stateStore = new ManagedStateStore(path.join(root, "managed-state.json")); + await writeOpenClawConfig(configPath, { + agents: { + list: [{ + id: "lowcode", + name: "Lowcode Agent", + workspace: "/var/platform_data/openclaw/workspaces/lowcode", + daedalusAgent: { + accountId: "lowcode", + templateName: "sales-template", + inboundTopic: "legacy/inbound", + outboundTopic: "legacy/outbound" + } + }] + }, + channels: { + "daedalus-agent-mqtt-channel": { + accounts: { + lowcode: { + enabled: true, + topics: { + inbound: "daedalus-agent/agent/lowcode/inbound", + outbound: "daedalus-agent/agent/lowcode/outbound" + } + } + } + } + }, + bindings: [{ + type: "route", + agentId: "lowcode", + match: { + channel: "daedalus-agent-mqtt-channel", + accountId: "lowcode" + } + }] + }); + + const manager = new OpenClawManager( + new MockCliRunner() as never, + new MockCliRunner() as never, + createEnv({ openclawConfigPath: configPath }), + stateStore + ); + + const result = await manager.listAgents() as { items: Array> }; + const config = await readOpenClawConfig(configPath); + const record = await stateStore.getAgent("lowcode"); + + assert.equal(result.items[0]?.templateName, "sales-template"); + assert.equal(record?.templateName, "sales-template"); + assert.equal("daedalusAgent" in config.agents.list[0], false); +}); + test("createAgent issues the expected openclaw commands", async () => { const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-agent-create-")); const configPath = path.join(root, ".openclaw", "openclaw.json"); diff --git a/packages/daedalus-agent-management-web/server/management-test/template-manager.test.ts b/packages/daedalus-agent-management-web/server/management-test/template-manager.test.ts index 73ad61abc74c3995aa3c3a8fcfc0ae8bd759c9e0..ec1e402f00072d691fe726daf5f627ef475f24f4 100644 --- a/packages/daedalus-agent-management-web/server/management-test/template-manager.test.ts +++ b/packages/daedalus-agent-management-web/server/management-test/template-manager.test.ts @@ -144,3 +144,31 @@ test("TemplateManager marks built-in templates from manifest", async () => { { templateName: "default", "is-built-in": true } ]); }); + +test("TemplateManager preserves the current template when replacement download fails", async () => { + const root = await mkdtemp(path.join(tmpdir(), "oc-mgmt-template-failure-")); + const templatesRoot = path.join(root, "workspace-templates"); + const templateRoot = path.join(templatesRoot, "sales"); + await mkdir(templateRoot, { recursive: true }); + await writeFile(path.join(templateRoot, "AGENTS.md"), "# Current\n", "utf8"); + + const manager = new TemplateManager( + createEnv({ workspaceTemplatesRoot: templatesRoot }), + new ManagedStateStore(path.join(root, "managed-state.json")), + { + objectStore: { + async getObject(): Promise { + throw new Error("injected template download failure"); + } + } + } + ); + + await assert.rejects(manager.createTemplate({ + templateName: "sales", + bucket: "template-bucket", + objectKey: "workspace/sales.zip", + replace: true + }), /injected template download failure/); + assert.equal(await readFile(path.join(templateRoot, "AGENTS.md"), "utf8"), "# Current\n"); +}); diff --git a/packages/daedalus-agent-management-web/server/src/api/routes/index.js b/packages/daedalus-agent-management-web/server/src/api/routes/index.js index cbda22dba04ffda91e353e14880e5ba489355cc1..3bed25f8da8b345b06b39220f6dcf766be06b88e 100644 --- a/packages/daedalus-agent-management-web/server/src/api/routes/index.js +++ b/packages/daedalus-agent-management-web/server/src/api/routes/index.js @@ -928,7 +928,7 @@ export function createRoutes(services) { services, req, "保存 IP 白名单", - "更新 env.json 中的 DA_INTERNAL_NETWORK_ALLOWLIST / DA_HUOZIGE_SERVER_IP" + "原子更新数据库中的活字格服务器地址及内部网络白名单" ); res.json(result); })); diff --git a/packages/daedalus-agent-management-web/server/src/app.js b/packages/daedalus-agent-management-web/server/src/app.js index 1cdddc8d0da60310fb7ac450095760e5520bb699..833bdba58928b1f42571c775f6cff675ce4d2850 100644 --- a/packages/daedalus-agent-management-web/server/src/app.js +++ b/packages/daedalus-agent-management-web/server/src/app.js @@ -35,7 +35,7 @@ export function createServerApp() { const updateNotifier = new UpdateNotifier({ authService }); const hzgProviderClient = new HzgProviderClient(); const auditLogService = new AuditLogService({ db }); - const runtimeEnvConfigService = new RuntimeEnvConfigService({ env }); + const runtimeEnvConfigService = new RuntimeEnvConfigService({ db, env }); const runtimeComponentHealthService = new RuntimeComponentHealthService({ env }); const containerService = new ContainerService({ env }); const catalogSyncService = new CatalogSyncService({ db, managementClient, auditLogService, env }); diff --git a/packages/daedalus-agent-management-web/server/src/db/index.js b/packages/daedalus-agent-management-web/server/src/db/index.js index 25fd6106cb8d82d79d002023bf89cac5675047e1..ac2e0a42620e72f86f97b8eaee1202582a2accc1 100644 --- a/packages/daedalus-agent-management-web/server/src/db/index.js +++ b/packages/daedalus-agent-management-web/server/src/db/index.js @@ -8,10 +8,10 @@ import { loadEnv } from "../config/env.js"; import { hashPassword } from "../utils/security.js"; const env = loadEnv(); -const SCHEMA_VERSION = 11; +const SCHEMA_VERSION = 12; const DEFAULT_ACCESS_TOKEN = "sk-1234567890qwertyuiop"; -const DEFAULT_PORTAL_NAME = "智能体管理与运营平台"; -const DEFAULT_BRAND_SUBTITLE = "Your company's name here"; +const DEFAULT_PORTAL_NAME = "智能体集群管理控制台"; +const DEFAULT_BRAND_SUBTITLE = "Agents Management Console"; const DEFAULT_THEME_COLOR = "#a5522d"; const DEFAULT_ADMIN_USERNAME = "admin"; const DEFAULT_ADMIN_DISPLAY_NAME = "管理员"; @@ -494,6 +494,91 @@ function migrateV10ToV11() { ); } +function createRuntimeSettingsTable() { + return ` + CREATE TABLE IF NOT EXISTS runtime_settings ( + setting_key TEXT PRIMARY KEY, + setting_value TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + `; +} + +function readLegacyHuozigeServerHost() { + try { + if (!fs.existsSync(env.userEnvJsonPath)) { + return ""; + } + const payload = JSON.parse(fs.readFileSync(env.userEnvJsonPath, "utf8").replace(/^\uFEFF/, "")); + return String(payload?.DA_HUOZIGE_SERVER_IP || "").trim(); + } catch { + return ""; + } +} + +function migrateV11ToV12() { + db.exec(createRuntimeSettingsTable()); + + const legacyHost = readLegacyHuozigeServerHost(); + const storedHost = hasTableColumn("business_systems", "host") + ? String(db.prepare(` + SELECT host + FROM business_systems + WHERE trim(host) <> '' + ORDER BY updated_at DESC, id DESC + LIMIT 1 + `).get()?.host || "").trim() + : ""; + const serverHost = legacyHost || storedHost; + if (serverHost) { + const now = toChinaISOString(); + db.prepare(` + INSERT INTO runtime_settings ( + setting_key, setting_value, created_at, updated_at + ) VALUES ('huozige_server_host', ?, ?, ?) + ON CONFLICT(setting_key) DO NOTHING + `).run(serverHost, now, now); + } + + if (!hasTableColumn("business_systems", "host")) { + return; + } + + db.exec(` + CREATE TABLE business_systems_v12 ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + provider TEXT NOT NULL CHECK (provider IN ('hzg', 'phx')), + application_name TEXT NOT NULL UNIQUE, + description TEXT NOT NULL, + ontology_artifact_id INTEGER, + scheme TEXT NOT NULL CHECK (scheme IN ('http', 'https')), + port INTEGER NOT NULL, + status TEXT NOT NULL CHECK (status IN ('active', 'disabled')), + last_connectivity_test_status TEXT NOT NULL DEFAULT 'unknown', + last_connectivity_test_result_json TEXT NOT NULL DEFAULT '{}', + is_builtin INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + FOREIGN KEY (ontology_artifact_id) REFERENCES artifacts(id) ON DELETE SET NULL + ); + + INSERT INTO business_systems_v12 ( + id, provider, application_name, description, ontology_artifact_id, + scheme, port, status, last_connectivity_test_status, + last_connectivity_test_result_json, is_builtin, created_at, updated_at + ) + SELECT + id, provider, application_name, description, ontology_artifact_id, + scheme, port, status, last_connectivity_test_status, + last_connectivity_test_result_json, is_builtin, created_at, updated_at + FROM business_systems; + + DROP TABLE business_systems; + ALTER TABLE business_systems_v12 RENAME TO business_systems; + `); +} + function addAgentCreatorColumn() { if (!hasTable("agents")) { return; @@ -539,6 +624,8 @@ function createSchema() { updated_at TEXT NOT NULL ); + ${createRuntimeSettingsTable()} + ${createComponentVersionsTable()} ${createStateTable("usage_refresh_state")} @@ -688,7 +775,6 @@ function createSchema() { description TEXT NOT NULL, ontology_artifact_id INTEGER, scheme TEXT NOT NULL CHECK (scheme IN ('http', 'https')), - host TEXT NOT NULL, port INTEGER NOT NULL, status TEXT NOT NULL CHECK (status IN ('active', 'disabled')), last_connectivity_test_status TEXT NOT NULL DEFAULT 'unknown', @@ -856,225 +942,25 @@ function ensureCoreRows() { function migrate() { const version = currentSchemaVersion(); - - if (version === 1 && SCHEMA_VERSION >= 2) { - db.exec("BEGIN"); - try { - migrateV1ToV2(); - if (SCHEMA_VERSION >= 3) { - migrateV2ToV3(); - } - if (SCHEMA_VERSION >= 4) { - migrateV3ToV4(); - } - if (SCHEMA_VERSION >= 5) { - migrateV4ToV5(); - } - if (SCHEMA_VERSION >= 6) { - migrateV5ToV6(); - } - if (SCHEMA_VERSION >= 7) { - migrateV6ToV7(); - } - if (SCHEMA_VERSION >= 8) { - migrateV7ToV8(); - } - if (SCHEMA_VERSION >= 9) { - migrateV8ToV9(); - } - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 2 && SCHEMA_VERSION >= 3) { - db.exec("BEGIN"); - try { - migrateV2ToV3(); - if (SCHEMA_VERSION >= 4) { - migrateV3ToV4(); - } - if (SCHEMA_VERSION >= 5) { - migrateV4ToV5(); - } - if (SCHEMA_VERSION >= 6) { - migrateV5ToV6(); - } - if (SCHEMA_VERSION >= 7) { - migrateV6ToV7(); - } - if (SCHEMA_VERSION >= 8) { - migrateV7ToV8(); - } - if (SCHEMA_VERSION >= 9) { - migrateV8ToV9(); - } - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 3 && SCHEMA_VERSION >= 4) { - db.exec("BEGIN"); - try { - migrateV3ToV4(); - if (SCHEMA_VERSION >= 5) { - migrateV4ToV5(); - } - if (SCHEMA_VERSION >= 6) { - migrateV5ToV6(); - } - if (SCHEMA_VERSION >= 7) { - migrateV6ToV7(); - } - if (SCHEMA_VERSION >= 8) { - migrateV7ToV8(); - } - if (SCHEMA_VERSION >= 9) { - migrateV8ToV9(); - } - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 4 && SCHEMA_VERSION >= 5) { - db.exec("BEGIN"); - try { - migrateV4ToV5(); - if (SCHEMA_VERSION >= 6) { - migrateV5ToV6(); - } - if (SCHEMA_VERSION >= 7) { - migrateV6ToV7(); - } - if (SCHEMA_VERSION >= 8) { - migrateV7ToV8(); - } - if (SCHEMA_VERSION >= 9) { - migrateV8ToV9(); - } - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 5 && SCHEMA_VERSION >= 6) { - db.exec("BEGIN"); - try { - migrateV5ToV6(); - if (SCHEMA_VERSION >= 7) { - migrateV6ToV7(); - } - if (SCHEMA_VERSION >= 8) { - migrateV7ToV8(); - } - if (SCHEMA_VERSION >= 9) { - migrateV8ToV9(); - } - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 6 && SCHEMA_VERSION >= 7) { - db.exec("BEGIN"); - try { - migrateV6ToV7(); - if (SCHEMA_VERSION >= 8) { - migrateV7ToV8(); - } - if (SCHEMA_VERSION >= 9) { - migrateV8ToV9(); - } - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 7 && SCHEMA_VERSION >= 8) { - db.exec("BEGIN"); - try { - migrateV7ToV8(); - if (SCHEMA_VERSION >= 9) { - migrateV8ToV9(); - } - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 8 && SCHEMA_VERSION >= 9) { - db.exec("BEGIN"); - try { - migrateV8ToV9(); - if (SCHEMA_VERSION >= 10) { - migrateV9ToV10(); - } - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); - } - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } - } else if (version === 9 && SCHEMA_VERSION >= 10) { + const migrations = [ + migrateV1ToV2, + migrateV2ToV3, + migrateV3ToV4, + migrateV4ToV5, + migrateV5ToV6, + migrateV6ToV7, + migrateV7ToV8, + migrateV8ToV9, + migrateV9ToV10, + migrateV10ToV11, + migrateV11ToV12 + ]; + + if (version >= 1 && version < SCHEMA_VERSION) { db.exec("BEGIN"); try { - migrateV9ToV10(); - if (SCHEMA_VERSION >= 11) { - migrateV10ToV11(); + for (let current = version; current < SCHEMA_VERSION; current += 1) { + migrations[current - 1](); } setSchemaVersion(SCHEMA_VERSION); db.exec("COMMIT"); @@ -1082,16 +968,6 @@ function migrate() { db.exec("ROLLBACK"); throw error; } - } else if (version === 10 && SCHEMA_VERSION >= 11) { - db.exec("BEGIN"); - try { - migrateV10ToV11(); - setSchemaVersion(SCHEMA_VERSION); - db.exec("COMMIT"); - } catch (error) { - db.exec("ROLLBACK"); - throw error; - } } else if (version !== SCHEMA_VERSION) { db.exec("PRAGMA foreign_keys = OFF;"); db.exec("BEGIN"); @@ -1115,6 +991,7 @@ function migrate() { if (!hasTableColumn("system_invocation_logs", "request_size_bytes")) { migrateV8ToV9(); } + migrateV11ToV12(); ensureCoreRows(); } diff --git a/packages/daedalus-agent-management-web/server/src/infra/s3/object-storage.js b/packages/daedalus-agent-management-web/server/src/infra/s3/object-storage.js index 76511e6b3c0c89582c563bb4007ea5c619e18a7a..c0dee32d4399ff3f3584cae829c3866c068aff0c 100644 --- a/packages/daedalus-agent-management-web/server/src/infra/s3/object-storage.js +++ b/packages/daedalus-agent-management-web/server/src/infra/s3/object-storage.js @@ -1,4 +1,4 @@ -import { GetObjectCommand, PutObjectCommand, S3Client } from "@aws-sdk/client-s3"; +import { DeleteObjectCommand, GetObjectCommand, PutObjectCommand, S3Client } from "@aws-sdk/client-s3"; import { randomUUID } from "node:crypto"; import { Readable } from "node:stream"; import { chinaDateKey } from "../../utils/china-time.js"; @@ -95,4 +95,14 @@ export class ObjectStorage { return await bodyToBuffer(response.Body); } + + async deleteObject(bucket, objectKey) { + if (!bucket || !objectKey) { + return; + } + await this.getClient().send(new DeleteObjectCommand({ + Bucket: bucket, + Key: objectKey + })); + } } diff --git a/packages/daedalus-agent-management-web/server/src/services/agent-service.js b/packages/daedalus-agent-management-web/server/src/services/agent-service.js index b8b4718596505e9a500e23167df6ee6a5edfebde..6abbf2543efd4797fed0ff9581c1b525bf862416 100644 --- a/packages/daedalus-agent-management-web/server/src/services/agent-service.js +++ b/packages/daedalus-agent-management-web/server/src/services/agent-service.js @@ -886,62 +886,71 @@ export class AgentService { const nextAgentName = firstText(payload.agent_name, payload.name, current.agent_name, current.slug); const runtimeAction = runtimeActionForStatusChange(current.status, nextStatus); - if (runtimeAction) { - await this.managementClient.call(runtimeAction, { - agentId: current.slug, - restart: payload.restart !== false - }); - } - - if (payload.docs_content !== undefined && payload.docs_content !== current.docs_content) { - await this.managementClient.call("agent.docs.update", { - agentId: current.slug, - content: payload.docs_content - }); + const docsChanged = payload.docs_content !== undefined && payload.docs_content !== current.docs_content; + const nameChanged = nextAgentName !== current.agent_name; + const modelChanged = llmSelection.changed || fallbackLlmSelection.changed; + const remoteParams = { + agentId: current.slug, + restart: payload.restart !== false, + ...(runtimeAction ? { enabled: nextStatus !== "disabled" } : {}), + ...(docsChanged ? { docsContent: payload.docs_content } : {}), + ...(nameChanged ? { name: nextAgentName } : {}), + ...(modelChanged ? { model: modelConfig } : {}) + }; + const remoteChanged = runtimeAction || docsChanged || nameChanged || modelChanged; + let remoteApplied = false; + if (remoteChanged) { + await this.managementClient.call("agent.update", remoteParams); + remoteApplied = true; } - if (nextAgentName !== current.agent_name) { - await this.managementClient.call("agent.name.set", { - agentId: current.slug, - name: nextAgentName, - restart: payload.restart !== false - }); - } + try { + runInTransaction(this.db, () => { + this.db.prepare(` + UPDATE agents + SET agent_name = ?, description = ?, docs_content = ?, status = ?, tags_json = ?, + daily_limit = ?, llm_provider_id = ?, llm_model_id = ?, llm_model_ref = ?, + fallback_llm_provider_id = ?, fallback_llm_model_id = ?, fallback_llm_model_ref = ?, + updated_at = ? + WHERE id = ? + `).run( + nextAgentName, + payload.description ?? current.description, + payload.docs_content ?? current.docs_content, + nextStatus, + jsonStringify(payload.tags ?? jsonParse(current.tags_json)), + Number(payload.daily_limit ?? current.daily_limit), + llmSelection.providerId, + llmSelection.modelId, + llmSelection.modelRef, + fallbackLlmSelection.providerId, + fallbackLlmSelection.modelId, + fallbackLlmSelection.modelRef, + toChinaISOString(), + agentId + ); - if (llmSelection.changed || fallbackLlmSelection.changed) { - await this.managementClient.call("agent.model.set", { - agentId: current.slug, - model: modelConfig, - restart: payload.restart !== false + if (hasOwnValue(payload, "permission_usernames") || hasOwnValue(payload, "users")) { + this.replacePermissions(agentId, resolvePermissionUsernames(payload)); + } }); - } - - this.db.prepare(` - UPDATE agents - SET agent_name = ?, description = ?, docs_content = ?, status = ?, tags_json = ?, - daily_limit = ?, llm_provider_id = ?, llm_model_id = ?, llm_model_ref = ?, - fallback_llm_provider_id = ?, fallback_llm_model_id = ?, fallback_llm_model_ref = ?, - updated_at = ? - WHERE id = ? - `).run( - nextAgentName, - payload.description ?? current.description, - payload.docs_content ?? current.docs_content, - nextStatus, - jsonStringify(payload.tags ?? jsonParse(current.tags_json)), - Number(payload.daily_limit ?? current.daily_limit), - llmSelection.providerId, - llmSelection.modelId, - llmSelection.modelRef, - fallbackLlmSelection.providerId, - fallbackLlmSelection.modelId, - fallbackLlmSelection.modelRef, - toChinaISOString(), - agentId - ); - - if (hasOwnValue(payload, "permission_usernames") || hasOwnValue(payload, "users")) { - this.replacePermissions(agentId, resolvePermissionUsernames(payload)); + } catch (error) { + if (remoteApplied) { + await this.managementClient.call("agent.update", { + agentId: current.slug, + restart: payload.restart !== false, + ...(runtimeAction ? { enabled: current.status !== "disabled" } : {}), + ...(docsChanged ? { docsContent: current.docs_content } : {}), + ...(nameChanged ? { name: current.agent_name } : {}), + ...(modelChanged ? { + model: buildAgentModelConfig( + { modelRef: current.llm_model_ref }, + { modelRef: current.fallback_llm_model_ref } + ) + } : {}) + }).catch(() => undefined); + } + throw error; } return (await this.listAgents()).find((item) => item.id === agentId); diff --git a/packages/daedalus-agent-management-web/server/src/services/catalog-sync-service.js b/packages/daedalus-agent-management-web/server/src/services/catalog-sync-service.js index 42f37383334994994a23c5a5b0b524d3e10c387f..76df81e628662a31c4ab92be7e990f51c7bc4819 100644 --- a/packages/daedalus-agent-management-web/server/src/services/catalog-sync-service.js +++ b/packages/daedalus-agent-management-web/server/src/services/catalog-sync-service.js @@ -244,17 +244,6 @@ function builtInOntologyNames(items) { .filter(Boolean)); } -function normalizeSystemEndpoint(item, existing = null) { - const endpoint = item?.endpoint && typeof item.endpoint === "object" ? item.endpoint : {}; - return { - scheme: firstText(endpoint.scheme, existing?.scheme, "http"), - host: firstText(endpoint.host, existing?.host, item?.host, "localhost"), - port: Number.isInteger(Number(endpoint.port)) - ? Number(endpoint.port) - : (Number.isInteger(Number(existing?.port)) ? Number(existing.port) : 80) - }; -} - export class CatalogSyncService { constructor({ db, managementClient, auditLogService = null, env = null }) { this.db = db; @@ -391,8 +380,8 @@ export class CatalogSyncService { const templateItems = readCatalogItems(remote, "templates"); const agentItems = readCatalogItems(remote, "agents"); const skillItems = readCatalogItems(remote, "skills"); - const systemItems = readCatalogItems(remote, "systems"); const ontologyItems = readCatalogItems(remote, "ontologies"); + const systemCount = Number(this.db.prepare("SELECT COUNT(*) AS count FROM business_systems").get()?.count || 0); const summaries = { llm_sync_state: { models: countLargeLanguageModels(llmItems) @@ -407,7 +396,7 @@ export class CatalogSyncService { skills: skillItems.length }, system_sync_state: { - systems: systemItems.length + systems: systemCount } }; @@ -416,7 +405,7 @@ export class CatalogSyncService { this.syncTemplates(templateItems, now); this.syncAgents(agentItems, new Map(), now); this.syncSkills(skillItems, now); - this.syncSystems(systemItems, now, builtInOntologyNames(ontologyItems)); + this.syncSystemBuiltInFlags(builtInOntologyNames(ontologyItems), now); const finishedAt = toChinaISOString(); for (const item of CATALOG_SYNC_STATES) { @@ -913,46 +902,16 @@ export class CatalogSyncService { } } - syncSystems(items, now = toChinaISOString(), builtInOntologies = new Set()) { - const existingRows = this.db.prepare("SELECT * FROM business_systems").all(); - const existingMap = new Map(existingRows.map((row) => [row.application_name, row])); - const seen = new Set(); + syncSystemBuiltInFlags(builtInOntologies = new Set(), now = toChinaISOString()) { const update = this.db.prepare(` UPDATE business_systems - SET provider = ?, scheme = ?, host = ?, port = ?, status = ?, is_builtin = ?, updated_at = ? + SET is_builtin = ?, updated_at = ? WHERE id = ? `); - const remove = this.db.prepare("DELETE FROM business_systems WHERE id = ?"); - - for (const item of items) { - const applicationName = firstText(item?.id, item?.name).toLowerCase(); - if (!applicationName) { - throw new Error("catalogs.systems.items[].id is required"); - } - - seen.add(applicationName); - const existing = existingMap.get(applicationName); - if (!existing) { - continue; - } - - const endpoint = normalizeSystemEndpoint(item, existing); - const isBuiltin = builtInOntologies.has(applicationName) ? 1 : 0; - update.run( - firstText(item?.provider, existing.provider), - endpoint.scheme, - endpoint.host, - endpoint.port, - firstText(item?.status, existing.status) === "disabled" ? "disabled" : "active", - isBuiltin, - now, - existing.id - ); - } - - for (const row of existingRows) { - if (!seen.has(row.application_name)) { - remove.run(row.id); + for (const row of this.db.prepare("SELECT id, application_name, is_builtin FROM business_systems").all()) { + const isBuiltin = builtInOntologies.has(row.application_name) ? 1 : 0; + if (Number(row.is_builtin) !== isBuiltin) { + update.run(isBuiltin, now, row.id); } } } diff --git a/packages/daedalus-agent-management-web/server/src/services/large-language-model-service.js b/packages/daedalus-agent-management-web/server/src/services/large-language-model-service.js index c5069a990f35fb227dec68f2382ba14d91b02284..17ec8d427494fba693d40d98c58fec316a2cca64 100644 --- a/packages/daedalus-agent-management-web/server/src/services/large-language-model-service.js +++ b/packages/daedalus-agent-management-web/server/src/services/large-language-model-service.js @@ -275,6 +275,53 @@ export class LargeLanguageModelService { return this.getModel(model.provider_id, model.model_id); } + async refreshProviderProjection(providerId) { + const snapshot = await this.managementClient.call("catalog.snapshot", {}); + const providers = snapshot?.catalogs?.llmProviders?.items; + if (!Array.isArray(providers)) { + throw new Error("catalog.snapshot did not return llmProviders"); + } + const provider = providers.find((item) => firstText(item?.id) === providerId); + if (!provider) { + this.db.prepare("DELETE FROM llm_models WHERE provider_id = ?").run(providerId); + this.db.prepare("DELETE FROM llm_providers WHERE provider_id = ?").run(providerId); + return null; + } + + this.upsertProvider({ + providerId, + baseUrl: provider.baseUrl, + apiKeyMask: provider.secret?.mask, + apiKeyFingerprint: provider.secret?.fingerprint, + api: firstText(provider.api, "openaiCompletions") === "openaiCompletions" + ? "openai-completions" + : provider.api + }); + const seenModels = new Set(); + for (const model of Array.isArray(provider.models) ? provider.models : []) { + const modelId = firstText(model?.id); + if (!modelId) { + continue; + } + const modelRef = firstText(model.ref, buildModelRef(providerId, modelId)); + seenModels.add(modelRef); + this.upsertModel({ + providerId, + modelId, + modelRef, + name: firstText(model.name, modelId), + contextTokens: model.limits?.contextTokens ?? null, + maxTokens: model.limits?.maxOutputTokens ?? null + }); + } + for (const local of this.db.prepare("SELECT model_ref FROM llm_models WHERE provider_id = ?").all(providerId)) { + if (!seenModels.has(local.model_ref)) { + this.db.prepare("DELETE FROM llm_models WHERE model_ref = ?").run(local.model_ref); + } + } + return this.getProvider(providerId); + } + async createProvider(payload) { const providerId = normalizeProviderId(payload.provider_id ?? payload.providerId); const baseUrl = normalizeBaseUrl(payload.base_url ?? payload.baseUrl); @@ -285,12 +332,21 @@ export class LargeLanguageModelService { apiKey, restart: false }); - return this.upsertProvider(remote, { - providerId, - baseUrl, - apiKeyMask: maskApiKey(apiKey), - apiKeyFingerprint: fingerprintApiKey(apiKey) - }); + try { + return this.upsertProvider(remote, { + providerId, + baseUrl, + apiKeyMask: maskApiKey(apiKey), + apiKeyFingerprint: fingerprintApiKey(apiKey) + }); + } catch (error) { + try { + return await this.refreshProviderProjection(providerId); + } catch { + await this.managementClient.call("llm.provider.delete", { providerId, restart: false }).catch(() => undefined); + throw error; + } + } } async updateProvider(providerId, payload) { @@ -308,12 +364,20 @@ export class LargeLanguageModelService { } const remote = await this.managementClient.call("llm.provider.update", remotePayload); - return this.upsertProvider(remote, { - providerId: normalizedProviderId, - baseUrl: remotePayload.baseUrl, - apiKeyMask: remotePayload.apiKey ? maskApiKey(remotePayload.apiKey) : undefined, - apiKeyFingerprint: remotePayload.apiKey ? fingerprintApiKey(remotePayload.apiKey) : undefined - }); + try { + return this.upsertProvider(remote, { + providerId: normalizedProviderId, + baseUrl: remotePayload.baseUrl, + apiKeyMask: remotePayload.apiKey ? maskApiKey(remotePayload.apiKey) : undefined, + apiKeyFingerprint: remotePayload.apiKey ? fingerprintApiKey(remotePayload.apiKey) : undefined + }); + } catch (error) { + try { + return await this.refreshProviderProjection(normalizedProviderId); + } catch { + throw error; + } + } } async deleteProvider(providerId) { @@ -330,7 +394,15 @@ export class LargeLanguageModelService { providerId: normalizedProviderId, restart: false }); - this.db.prepare("DELETE FROM llm_providers WHERE provider_id = ?").run(normalizedProviderId); + try { + this.db.prepare("DELETE FROM llm_providers WHERE provider_id = ?").run(normalizedProviderId); + } catch (error) { + try { + await this.refreshProviderProjection(normalizedProviderId); + } catch { + throw error; + } + } return { ok: true, provider_id: normalizedProviderId }; } @@ -414,14 +486,24 @@ export class LargeLanguageModelService { maxTokens, restart: false }); - return this.upsertModel(remote, { - providerId, - modelId, - modelRef: buildModelRef(providerId, modelId), - name: firstText(payload.name, modelId), - contextTokens, - maxTokens - }); + try { + return this.upsertModel(remote, { + providerId, + modelId, + modelRef: buildModelRef(providerId, modelId), + name: firstText(payload.name, modelId), + contextTokens, + maxTokens + }); + } catch (error) { + try { + await this.refreshProviderProjection(providerId); + return this.getModel(providerId, modelId); + } catch { + await this.managementClient.call("llm.model.delete", { providerId, modelId, restart: false }).catch(() => undefined); + throw error; + } + } } async updateModel(providerId, modelId, payload) { @@ -442,14 +524,23 @@ export class LargeLanguageModelService { } const remote = await this.managementClient.call("llm.model.update", remotePayload); - return this.upsertModel(remote, { - providerId: normalizedProviderId, - modelId: normalizedModelId, - modelRef: buildModelRef(normalizedProviderId, normalizedModelId), - name: payload.name, - contextTokens: payload.context_tokens ?? payload.contextTokens, - maxTokens: payload.max_tokens ?? payload.maxTokens - }); + try { + return this.upsertModel(remote, { + providerId: normalizedProviderId, + modelId: normalizedModelId, + modelRef: buildModelRef(normalizedProviderId, normalizedModelId), + name: payload.name, + contextTokens: payload.context_tokens ?? payload.contextTokens, + maxTokens: payload.max_tokens ?? payload.maxTokens + }); + } catch (error) { + try { + await this.refreshProviderProjection(normalizedProviderId); + return this.getModel(normalizedProviderId, normalizedModelId); + } catch { + throw error; + } + } } async deleteModel(providerId, modelId) { @@ -465,7 +556,15 @@ export class LargeLanguageModelService { modelId: normalizedModelId, restart: false }); - this.db.prepare("DELETE FROM llm_models WHERE model_ref = ?").run(model.model_ref); + try { + this.db.prepare("DELETE FROM llm_models WHERE model_ref = ?").run(model.model_ref); + } catch (error) { + try { + await this.refreshProviderProjection(normalizedProviderId); + } catch { + throw error; + } + } return { ok: true, model_ref: model.model_ref }; } } diff --git a/packages/daedalus-agent-management-web/server/src/services/portal-service.js b/packages/daedalus-agent-management-web/server/src/services/portal-service.js index 865ec8325cb47d5bb79d671cb3040bfec97c1298..77bfba41db91d403917b24698071e5d7cb28af0d 100644 --- a/packages/daedalus-agent-management-web/server/src/services/portal-service.js +++ b/packages/daedalus-agent-management-web/server/src/services/portal-service.js @@ -725,24 +725,66 @@ export class PortalService { async persistArtifact({ kind, file, createdBy }) { const upload = await this.objectStorage.uploadAdminArtifact({ kind, file }); - const result = this.db.prepare(` - INSERT INTO artifacts ( - kind, bucket, object_key, original_name, mime_type, byte_size, created_by, created_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - `).run( - kind, - upload.bucket, - upload.objectKey, - file.originalname, - file.mimetype || "application/octet-stream", - file.size, - createdBy, - toChinaISOString() - ); - return { - id: result.lastInsertRowid, - ...upload - }; + try { + const result = this.db.prepare(` + INSERT INTO artifacts ( + kind, bucket, object_key, original_name, mime_type, byte_size, created_by, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + `).run( + kind, + upload.bucket, + upload.objectKey, + file.originalname, + file.mimetype || "application/octet-stream", + file.size, + createdBy, + toChinaISOString() + ); + return { + id: result.lastInsertRowid, + ...upload + }; + } catch (error) { + await this.objectStorage.deleteObject?.(upload.bucket, upload.objectKey).catch(() => undefined); + throw error; + } + } + + getArtifactStorageInfo(artifactId) { + if (!artifactId) { + return null; + } + return this.db.prepare(` + SELECT id, bucket, object_key + FROM artifacts + WHERE id = ? + `).get(artifactId) || null; + } + + async discardArtifact(artifact) { + if (!artifact) { + return; + } + if (artifact.id) { + try { + const references = this.db.prepare(` + SELECT + (SELECT COUNT(*) FROM business_systems WHERE ontology_artifact_id = ?) + + (SELECT COUNT(*) FROM agent_templates WHERE artifact_id = ?) + + (SELECT COUNT(*) FROM skills WHERE artifact_id = ?) AS count + `).get(artifact.id, artifact.id, artifact.id); + if (Number(references?.count || 0) > 0) { + return; + } + this.db.prepare("DELETE FROM artifacts WHERE id = ?").run(artifact.id); + } catch { + return; + } + } + const objectKey = artifact.object_key || artifact.objectKey; + if (artifact.bucket && objectKey && this.objectStorage.deleteObject) { + await this.objectStorage.deleteObject(artifact.bucket, objectKey).catch(() => undefined); + } } listTemplates() { @@ -832,39 +874,63 @@ export class PortalService { const defaultSelection = this.resolveTemplateDefaultLlm(defaultLlm); ensureZipContains(file.buffer, "AGENTS.md"); + const existing = this.db.prepare("SELECT * FROM agent_templates WHERE template_name = ?").get(templateName); + const previousArtifact = this.getArtifactStorageInfo(existing?.artifact_id); const artifact = await this.persistArtifact({ kind: "template", file, createdBy }); - const remoteResult = await this.managementClient.call("agent.template.create", { - templateName, - bucket: artifact.bucket, - objectKey: artifact.objectKey, - replace: true - }); - const now = toChinaISOString(); - this.db.prepare(` - INSERT INTO agent_templates ( - template_name, artifact_id, remote_status, remote_result_json, - default_llm_provider_id, default_llm_model_id, default_llm_model_ref, - created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(template_name) DO UPDATE SET - artifact_id = excluded.artifact_id, - remote_status = excluded.remote_status, - remote_result_json = excluded.remote_result_json, - default_llm_provider_id = excluded.default_llm_provider_id, - default_llm_model_id = excluded.default_llm_model_id, - default_llm_model_ref = excluded.default_llm_model_ref, - updated_at = excluded.updated_at - `).run( - templateName, - artifact.id, - "ready", - JSON.stringify(remoteResult ?? {}), - defaultSelection.providerId, - defaultSelection.modelId, - defaultSelection.modelRef, - now, - now - ); + let remoteApplied = false; + try { + const remoteResult = await this.managementClient.call("agent.template.create", { + templateName, + bucket: artifact.bucket, + objectKey: artifact.objectKey, + replace: true + }); + remoteApplied = true; + const now = toChinaISOString(); + this.db.prepare(` + INSERT INTO agent_templates ( + template_name, artifact_id, remote_status, remote_result_json, + default_llm_provider_id, default_llm_model_id, default_llm_model_ref, + created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(template_name) DO UPDATE SET + artifact_id = excluded.artifact_id, + remote_status = excluded.remote_status, + remote_result_json = excluded.remote_result_json, + default_llm_provider_id = excluded.default_llm_provider_id, + default_llm_model_id = excluded.default_llm_model_id, + default_llm_model_ref = excluded.default_llm_model_ref, + updated_at = excluded.updated_at + `).run( + templateName, + artifact.id, + "ready", + JSON.stringify(remoteResult ?? {}), + defaultSelection.providerId, + defaultSelection.modelId, + defaultSelection.modelRef, + now, + now + ); + } catch (error) { + if (remoteApplied) { + if (previousArtifact) { + await this.managementClient.call("agent.template.create", { + templateName, + bucket: previousArtifact.bucket, + objectKey: previousArtifact.object_key, + replace: true + }).catch(() => undefined); + } else { + await this.managementClient.call("agent.template.delete", { templateName }).catch(() => undefined); + } + } + await this.discardArtifact(artifact); + throw error; + } + if (previousArtifact?.id !== artifact.id) { + await this.discardArtifact(previousArtifact); + } return this.listTemplates().find((item) => item.template_name === templateName); } @@ -908,8 +974,22 @@ export class PortalService { throw conflict("该模板仍有数字员工使用，无法删除"); } + const artifact = this.getArtifactStorageInfo(existing.artifact_id); await this.managementClient.call("agent.template.delete", { templateName }); - this.db.prepare("DELETE FROM agent_templates WHERE template_name = ?").run(templateName); + try { + this.db.prepare("DELETE FROM agent_templates WHERE template_name = ?").run(templateName); + } catch (error) { + if (artifact) { + await this.managementClient.call("agent.template.create", { + templateName, + bucket: artifact.bucket, + objectKey: artifact.object_key, + replace: true + }).catch(() => undefined); + } + throw error; + } + await this.discardArtifact(artifact); return { ok: true }; } @@ -947,31 +1027,39 @@ export class PortalService { const normalizedArchive = normalizeSkillArchive(file); const slug = normalizedArchive.metadata.slug; const artifact = await this.persistArtifact({ kind: "skill", file: normalizedArchive.file, createdBy }); - - await this.managementClient.call("skills.global.install.local", { - slug, - bucket: artifact.bucket, - objectKey: artifact.objectKey, - force: false - }); - const remoteStatus = "installed"; - - const now = toChinaISOString(); - const result = this.db.prepare(` - INSERT INTO skills ( - slug, description, artifact_id, remote_status, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?) - `).run( - slug, - "", - artifact.id, - remoteStatus, - now, - now - ); - - const skillId = result.lastInsertRowid; - return this.listSkills().find((item) => item.id === skillId); + let remoteApplied = false; + try { + await this.managementClient.call("skills.global.install.local", { + slug, + bucket: artifact.bucket, + objectKey: artifact.objectKey, + force: false + }); + remoteApplied = true; + + const now = toChinaISOString(); + const result = this.db.prepare(` + INSERT INTO skills ( + slug, description, artifact_id, remote_status, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?) + `).run( + slug, + "", + artifact.id, + "installed", + now, + now + ); + + const skillId = result.lastInsertRowid; + return this.listSkills().find((item) => item.id === skillId); + } catch (error) { + if (remoteApplied) { + await this.managementClient.call("skills.global.delete", { slug }).catch(() => undefined); + } + await this.discardArtifact(artifact); + throw error; + } } async updateSkill(skillId, { payload, file, createdBy }) { @@ -984,6 +1072,7 @@ export class PortalService { if (normalizedArchive.metadata.slug !== slug) { throw badRequest(`技能 zip 中的 name 归一化后必须与当前技能 ${slug} 一致`); } + const previousArtifact = this.getArtifactStorageInfo(current.artifact_id); const artifact = await this.persistArtifact({ kind: "skill", file: normalizedArchive.file, createdBy }); const next = { @@ -993,25 +1082,40 @@ export class PortalService { updated_at: toChinaISOString() }; - await this.managementClient.call("skills.global.install.local", { - slug, - bucket: artifact.bucket, - objectKey: artifact.objectKey, - force: true - }); - const remoteStatus = "installed"; - - this.db.prepare(` - UPDATE skills - SET description = ?, artifact_id = ?, remote_status = ?, updated_at = ? - WHERE id = ? - `).run( - "", - next.artifact_id, - remoteStatus, - next.updated_at, - skillId - ); + let remoteApplied = false; + try { + await this.managementClient.call("skills.global.install.local", { + slug, + bucket: artifact.bucket, + objectKey: artifact.objectKey, + force: true + }); + remoteApplied = true; + + this.db.prepare(` + UPDATE skills + SET description = ?, artifact_id = ?, remote_status = ?, updated_at = ? + WHERE id = ? + `).run( + "", + next.artifact_id, + "installed", + next.updated_at, + skillId + ); + } catch (error) { + if (remoteApplied && previousArtifact) { + await this.managementClient.call("skills.global.install.local", { + slug, + bucket: previousArtifact.bucket, + objectKey: previousArtifact.object_key, + force: true + }).catch(() => undefined); + } + await this.discardArtifact(artifact); + throw error; + } + await this.discardArtifact(previousArtifact); return this.listSkills().find((item) => item.id === skillId); } @@ -1035,11 +1139,25 @@ export class PortalService { throw conflict("内置技能不允许删除"); } + const artifact = this.getArtifactStorageInfo(current.artifact_id); if (current.remote_status === "installed") { await this.managementClient.call("skills.global.delete", { slug: current.slug }); } - this.db.prepare("DELETE FROM skills WHERE id = ?").run(skillId); + try { + this.db.prepare("DELETE FROM skills WHERE id = ?").run(skillId); + } catch (error) { + if (current.remote_status === "installed" && artifact) { + await this.managementClient.call("skills.global.install.local", { + slug: current.slug, + bucket: artifact.bucket, + objectKey: artifact.object_key, + force: true + }).catch(() => undefined); + } + throw error; + } + await this.discardArtifact(artifact); return { ok: true }; } diff --git a/packages/daedalus-agent-management-web/server/src/services/runtime-env-config-service.js b/packages/daedalus-agent-management-web/server/src/services/runtime-env-config-service.js index c3a029213dff5733e51bc8511f831ac8f310ae40..29b195e59eb9110e977182b9ba70f3ccfea65ea8 100644 --- a/packages/daedalus-agent-management-web/server/src/services/runtime-env-config-service.js +++ b/packages/daedalus-agent-management-web/server/src/services/runtime-env-config-service.js @@ -1,10 +1,14 @@ import fs from "node:fs"; +import { isIP } from "node:net"; import path from "node:path"; import { badRequest } from "../utils/errors.js"; +import { toChinaISOString } from "../utils/china-time.js"; export const AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY = "DA_INTERNAL_NETWORK_ALLOWLIST"; export const HUOZIGE_SERVER_IP_KEY = "DA_HUOZIGE_SERVER_IP"; +const HUOZIGE_SERVER_HOST_SETTING_KEY = "huozige_server_host"; +const INTERNAL_NETWORK_ALLOWLIST_SETTING_KEY = "internal_network_allowlist"; function firstText(...values) { for (const value of values) { @@ -27,7 +31,7 @@ function normalizeAllowlistValue(value) { return String(value).trim(); } -function normalizeServerIpValue(value) { +function normalizeServerHostValue(value) { const normalized = String(value || "").trim(); if ( normalized.includes("/") @@ -80,13 +84,17 @@ function uniqueAllowlistItems(items) { return result; } -function updateAllowlistServerIp(allowlistValue, previousIp, nextIp) { - const previous = normalizeServerIpValue(previousIp); - const next = normalizeServerIpValue(nextIp); +function updateAllowlistServerHost(allowlistValue, previousHost, nextHost) { + const previous = normalizeServerHostValue(previousHost); + const next = normalizeServerHostValue(nextHost); + const excludedItems = new Set([previous]); + if (next && !isIP(next)) { + excludedItems.add(next); + } const retainedItems = parseAllowlistItems(allowlistValue) - .filter((item) => item !== previous); + .filter((item) => !excludedItems.has(item)); - if (next) { + if (isIP(next)) { retainedItems.push(next); } @@ -130,16 +138,101 @@ function writeJsonObject(filePath, payload) { } export class RuntimeEnvConfigService { - constructor({ env, envPath, credentialsPath } = {}) { + constructor({ db, env, envPath, credentialsPath } = {}) { + this.db = db; this.envPath = envPath || env?.userEnvJsonPath || path.resolve("env.json"); this.credentialsPath = credentialsPath || env?.credentialsJsonPath || path.resolve("credentials.json"); + this.migrateLegacyRuntimeSettings(); + } + + getRuntimeSetting(key) { + return this.db?.prepare(` + SELECT setting_value + FROM runtime_settings + WHERE setting_key = ? + `).get(key)?.setting_value; + } + + upsertRuntimeSetting(key, value, now = toChinaISOString()) { + this.db.prepare(` + INSERT INTO runtime_settings ( + setting_key, setting_value, created_at, updated_at + ) VALUES (?, ?, ?, ?) + ON CONFLICT(setting_key) DO UPDATE SET + setting_value = excluded.setting_value, + updated_at = excluded.updated_at + `).run(key, value, now, now); + } + + getHuozigeServerHost() { + return normalizeServerHostValue(this.getRuntimeSetting(HUOZIGE_SERVER_HOST_SETTING_KEY)); + } + + getInternalNetworkAllowlist() { + return normalizeAllowlistValue(this.getRuntimeSetting(INTERNAL_NETWORK_ALLOWLIST_SETTING_KEY)); + } + + setHuozigeServerHost(value) { + if (!this.db) { + throw new Error("Runtime settings database is not configured"); + } + const normalized = normalizeServerHostValue(value); + this.upsertRuntimeSetting(HUOZIGE_SERVER_HOST_SETTING_KEY, normalized); + return normalized; + } + + migrateLegacyRuntimeSettings() { + if (!this.db || !fs.existsSync(this.envPath)) { + return; + } + const current = readJsonObject(this.envPath); + const hasLegacyHost = Object.prototype.hasOwnProperty.call(current, HUOZIGE_SERVER_IP_KEY); + const legacyHost = hasLegacyHost + ? normalizeServerHostValue(current[HUOZIGE_SERVER_IP_KEY]) + : ""; + const hasLegacyAllowlist = Object.prototype.hasOwnProperty.call(current, AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY); + const storedHost = this.getRuntimeSetting(HUOZIGE_SERVER_HOST_SETTING_KEY); + const storedAllowlist = this.getRuntimeSetting(INTERNAL_NETWORK_ALLOWLIST_SETTING_KEY); + const configuredHost = normalizeServerHostValue(storedHost === undefined ? legacyHost : storedHost); + const previousAllowlist = normalizeAllowlistValue( + storedAllowlist === undefined ? current[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY] : storedAllowlist + ); + const domainHosts = new Set([legacyHost, configuredHost].filter((host) => host && !isIP(host))); + let nextAllowlist = uniqueAllowlistItems( + parseAllowlistItems(previousAllowlist).filter((item) => !domainHosts.has(item)) + ).join(","); + nextAllowlist = updateAllowlistServerHost(nextAllowlist, "", configuredHost); + + this.db.exec("BEGIN IMMEDIATE"); + try { + const now = toChinaISOString(); + if (storedHost === undefined && configuredHost) { + this.upsertRuntimeSetting(HUOZIGE_SERVER_HOST_SETTING_KEY, configuredHost, now); + } + if (storedAllowlist === undefined) { + this.upsertRuntimeSetting(INTERNAL_NETWORK_ALLOWLIST_SETTING_KEY, nextAllowlist, now); + } + this.db.exec("COMMIT"); + } catch (error) { + this.db.exec("ROLLBACK"); + throw error; + } + + if (hasLegacyHost) { + delete current[HUOZIGE_SERVER_IP_KEY]; + } + if (hasLegacyAllowlist) { + delete current[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY]; + } + if (hasLegacyHost || hasLegacyAllowlist) { + writeJsonObject(this.envPath, current); + } } getNetworkAllowlistConfig() { - const payload = readJsonObject(this.envPath, "env.json"); return { - agent_internal_network_allowlist: normalizeAllowlistValue(payload[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY]), - huozige_server_ip: normalizeServerIpValue(payload[HUOZIGE_SERVER_IP_KEY]) + agent_internal_network_allowlist: this.getInternalNetworkAllowlist(), + huozige_server_ip: this.getHuozigeServerHost() }; } @@ -154,36 +247,47 @@ export class RuntimeEnvConfigService { } updateNetworkAllowlistConfig(payload) { - const current = readJsonObject(this.envPath); - const previousServerIp = normalizeServerIpValue(current[HUOZIGE_SERVER_IP_KEY]); - const hasServerIpPayload = Object.prototype.hasOwnProperty.call(payload || {}, "huozige_server_ip"); + if (!this.db) { + throw new Error("Runtime settings database is not configured"); + } + const previousAllowlist = this.getInternalNetworkAllowlist(); + const previousServerHost = this.getHuozigeServerHost(); + const hasServerHostPayload = Object.prototype.hasOwnProperty.call(payload || {}, "huozige_server_ip"); + let nextAllowlist = previousAllowlist; + let nextServerHost = previousServerHost; if (Object.prototype.hasOwnProperty.call(payload || {}, "agent_internal_network_allowlist")) { - current[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY] = normalizeAllowlistValue( + nextAllowlist = normalizeAllowlistValue( payload?.agent_internal_network_allowlist ); } - if (hasServerIpPayload) { - const nextServerIp = normalizeServerIpValue(payload?.huozige_server_ip); - current[HUOZIGE_SERVER_IP_KEY] = nextServerIp; - current[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY] = updateAllowlistServerIp( - current[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY], - previousServerIp, - nextServerIp + if (hasServerHostPayload) { + nextServerHost = normalizeServerHostValue(payload?.huozige_server_ip); + nextAllowlist = updateAllowlistServerHost( + nextAllowlist, + previousServerHost, + nextServerHost ); - } else { - const currentServerIp = normalizeServerIpValue(current[HUOZIGE_SERVER_IP_KEY]); - if (currentServerIp) { - current[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY] = updateAllowlistServerIp( - current[AGENT_INTERNAL_NETWORK_ALLOWLIST_KEY], - "", - currentServerIp - ); - } + } else if (nextServerHost) { + nextAllowlist = updateAllowlistServerHost(nextAllowlist, nextServerHost, nextServerHost); } - writeJsonObject(this.envPath, current); - return this.getNetworkAllowlistConfig(); + this.db.exec("BEGIN IMMEDIATE"); + try { + const now = toChinaISOString(); + if (hasServerHostPayload) { + this.upsertRuntimeSetting(HUOZIGE_SERVER_HOST_SETTING_KEY, nextServerHost, now); + } + this.upsertRuntimeSetting(INTERNAL_NETWORK_ALLOWLIST_SETTING_KEY, nextAllowlist, now); + this.db.exec("COMMIT"); + } catch (error) { + this.db.exec("ROLLBACK"); + throw error; + } + return { + ...this.getNetworkAllowlistConfig(), + network_allowlist_changed: nextAllowlist !== previousAllowlist + }; } } diff --git a/packages/daedalus-agent-management-web/server/src/services/system-service.js b/packages/daedalus-agent-management-web/server/src/services/system-service.js index da78f59eea21dd5a22702ae258f7b7b5351548c6..66a46b1ee0924fb317524dd229ed2882c80e580f 100644 --- a/packages/daedalus-agent-management-web/server/src/services/system-service.js +++ b/packages/daedalus-agent-management-web/server/src/services/system-service.js @@ -80,7 +80,7 @@ function decorateSystemRow(row, configuredHost = "") { return null; } - const host = normalizeText(configuredHost) || row.host; + const host = normalizeText(configuredHost); const baseUrl = buildBaseUrl(row.scheme, host, row.port, row.application_name); return { ...row, @@ -204,10 +204,8 @@ export class SystemService { getConfiguredHuozigeServerIp() { return normalizeText( - this.runtimeEnvConfigService - ?.getNetworkAllowlistConfig - ?.() - ?.huozige_server_ip + this.runtimeEnvConfigService?.getHuozigeServerHost?.() + || this.runtimeEnvConfigService?.getNetworkAllowlistConfig?.()?.huozige_server_ip ); } @@ -220,7 +218,7 @@ export class SystemService { } resolveSystemHost(row) { - const host = this.getConfiguredHuozigeServerIp() || normalizeText(row?.host); + const host = this.getConfiguredHuozigeServerIp(); if (!host) { throw badRequest("请先在基础设置中配置对话应用/业务应用服务器"); } @@ -311,13 +309,13 @@ export class SystemService { normalizeCreatePayload(payload) { assertHostNotProvided(payload); + this.requireConfiguredHuozigeServerIp(); const applicationName = normalizeApplicationName(payload.application_name); return { provider: FIXED_BUSINESS_SYSTEM_PROVIDER, application_name: applicationName, description: normalizeText(payload.description), scheme: normalizeScheme(payload.scheme), - host: this.requireConfiguredHuozigeServerIp(), port: normalizePort(payload.port), status: normalizeText(payload.status) === "disabled" ? "disabled" : "active" }; @@ -325,6 +323,7 @@ export class SystemService { normalizeUpdatePayload(payload, current) { assertHostNotProvided(payload); + this.requireConfiguredHuozigeServerIp(); const applicationName = hasOwn(payload, "application_name") ? normalizeApplicationName(payload.application_name) : current.application_name; @@ -333,7 +332,6 @@ export class SystemService { application_name: applicationName, description: hasOwn(payload, "description") ? normalizeText(payload.description) : current.description, scheme: hasOwn(payload, "scheme") ? normalizeScheme(payload.scheme) : current.scheme, - host: this.resolveSystemHost(current), port: hasOwn(payload, "port") ? normalizePort(payload.port) : current.port, status: hasOwn(payload, "status") ? (normalizeText(payload.status) === "disabled" ? "disabled" : "active") @@ -347,12 +345,53 @@ export class SystemService { } return this.db.prepare(` - SELECT bucket, object_key + SELECT id, bucket, object_key FROM artifacts WHERE id = ? `).get(artifactId) || null; } + buildRemoteSystemParams(system, artifact) { + const objectKey = artifact?.object_key || artifact?.objectKey; + return { + systemId: system.application_name, + name: system.application_name, + provider: system.provider || FIXED_BUSINESS_SYSTEM_PROVIDER, + ontologyName: system.application_name, + description: system.description || "", + status: system.status || "active", + bucket: artifact?.bucket, + objectKey, + replaceOntology: Boolean(artifact?.bucket && objectKey) + }; + } + + async discardArtifact(artifact) { + if (!artifact) { + return; + } + if (artifact.id) { + try { + const references = this.db.prepare(` + SELECT + (SELECT COUNT(*) FROM business_systems WHERE ontology_artifact_id = ?) + + (SELECT COUNT(*) FROM agent_templates WHERE artifact_id = ?) + + (SELECT COUNT(*) FROM skills WHERE artifact_id = ?) AS count + `).get(artifact.id, artifact.id, artifact.id); + if (Number(references?.count || 0) > 0) { + return; + } + this.db.prepare("DELETE FROM artifacts WHERE id = ?").run(artifact.id); + } catch { + return; + } + } + const objectKey = artifact.object_key || artifact.objectKey; + if (artifact.bucket && objectKey && this.objectStorage.deleteObject) { + await this.objectStorage.deleteObject(artifact.bucket, objectKey).catch(() => undefined); + } + } + async deleteRemoteOntologiesIfPresent(applicationNames, { ignoreDeleteErrors = false } = {}) { const normalizedNames = Array.from(new Set( applicationNames @@ -390,55 +429,45 @@ export class SystemService { const artifact = await this.persistOntologyArtifact({ file, createdBy }); const artifactId = artifact.id; - const bucket = artifact.bucket; - const objectKey = artifact.objectKey; - - const { deletedApplicationNames } = await this.deleteRemoteOntologiesIfPresent([ - normalized.application_name - ]); - - const baseUrl = this.buildSystemBaseUrl(normalized); - await this.managementClient.call("apps.create", { - systemId: normalized.application_name, - name: normalized.application_name, - provider: normalized.provider, - ontologyName: normalized.application_name, - description: normalized.description, - applicationEndpoint: baseUrl, - sessionServiceEndpoint: baseUrl, - status: normalized.status, - bucket, - objectKey, - replaceOntology: Boolean(bucket && objectKey) - }); - const now = toChinaISOString(); - const result = this.db.prepare(` - INSERT INTO business_systems ( - provider, application_name, description, ontology_artifact_id, - scheme, host, port, status, last_connectivity_test_status, last_connectivity_test_result_json, - created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `).run( - normalized.provider, - normalized.application_name, - normalized.description, - artifactId, - normalized.scheme, - normalized.host, - normalized.port, - normalized.status, - "unknown", - JSON.stringify({}), - now, - now - ); + let remoteCreated = false; + try { + await this.managementClient.call("apps.create", this.buildRemoteSystemParams(normalized, artifact)); + remoteCreated = true; + + const now = toChinaISOString(); + const result = this.db.prepare(` + INSERT INTO business_systems ( + provider, application_name, description, ontology_artifact_id, + scheme, port, status, last_connectivity_test_status, last_connectivity_test_result_json, + created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `).run( + normalized.provider, + normalized.application_name, + normalized.description, + artifactId, + normalized.scheme, + normalized.port, + normalized.status, + "unknown", + JSON.stringify({}), + now, + now + ); - return { - system: this.getSystemById(result.lastInsertRowid), - kernelOntologyDeleted: deletedApplicationNames.length > 0, - deletedKernelOntologyNames: deletedApplicationNames - }; + return { + system: this.getSystemById(result.lastInsertRowid), + kernelOntologyDeleted: false, + deletedKernelOntologyNames: [] + }; + } catch (error) { + if (remoteCreated) { + await this.deleteRemoteOntologiesIfPresent([normalized.application_name], { ignoreDeleteErrors: true }); + } + await this.discardArtifact(artifact); + throw error; + } } async persistOntologyArtifact({ file, createdBy }) { @@ -452,25 +481,30 @@ export class SystemService { } const artifact = await this.objectStorage.uploadAdminArtifact({ kind: "ontology", file }); - const result = this.db.prepare(` - INSERT INTO artifacts ( - kind, bucket, object_key, original_name, mime_type, byte_size, created_by, created_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - `).run( - "ontology", - artifact.bucket, - artifact.objectKey, - file.originalname, - file.mimetype || "application/octet-stream", - file.size, - createdBy, - toChinaISOString() - ); + try { + const result = this.db.prepare(` + INSERT INTO artifacts ( + kind, bucket, object_key, original_name, mime_type, byte_size, created_by, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + `).run( + "ontology", + artifact.bucket, + artifact.objectKey, + file.originalname, + file.mimetype || "application/octet-stream", + file.size, + createdBy, + toChinaISOString() + ); - return { - id: result.lastInsertRowid, - ...artifact - }; + return { + id: result.lastInsertRowid, + ...artifact + }; + } catch (error) { + await this.objectStorage.deleteObject?.(artifact.bucket, artifact.objectKey).catch(() => undefined); + throw error; + } } async updateSystem(id, { payload, file, createdBy }) { @@ -490,45 +524,54 @@ export class SystemService { } const existingArtifact = artifact || this.getArtifactStorageInfo(artifactId); - - const baseUrl = this.buildSystemBaseUrl(normalized); + const previousArtifact = this.getArtifactStorageInfo(current.ontology_artifact_id); const previousSystemId = current.application_name; - const { deletedApplicationNames } = await this.deleteRemoteOntologiesIfPresent([ - previousSystemId, - normalized.application_name - ], { ignoreDeleteErrors: true }); - - await this.managementClient.call("apps.create", { - systemId: normalized.application_name, - name: normalized.application_name, - provider: normalized.provider, - ontologyName: normalized.application_name, - description: normalized.description, - applicationEndpoint: baseUrl, - sessionServiceEndpoint: baseUrl, - status: normalized.status, - bucket: existingArtifact?.bucket, - objectKey: existingArtifact?.object_key || existingArtifact?.objectKey, - replaceOntology: Boolean(existingArtifact?.bucket && (existingArtifact?.object_key || existingArtifact?.objectKey)) - }); + let remoteUpdated = false; - this.db.prepare(` - UPDATE business_systems - SET provider = ?, application_name = ?, description = ?, ontology_artifact_id = ?, - scheme = ?, host = ?, port = ?, status = ?, updated_at = ? - WHERE id = ? - `).run( - normalized.provider, - normalized.application_name, - normalized.description, - artifactId, - normalized.scheme, - normalized.host, - normalized.port, - normalized.status, - toChinaISOString(), - id - ); + try { + await this.managementClient.call("apps.create", this.buildRemoteSystemParams(normalized, existingArtifact)); + remoteUpdated = true; + + this.db.prepare(` + UPDATE business_systems + SET provider = ?, application_name = ?, description = ?, ontology_artifact_id = ?, + scheme = ?, port = ?, status = ?, updated_at = ? + WHERE id = ? + `).run( + normalized.provider, + normalized.application_name, + normalized.description, + artifactId, + normalized.scheme, + normalized.port, + normalized.status, + toChinaISOString(), + id + ); + } catch (error) { + if (remoteUpdated) { + await this.managementClient.call( + "apps.create", + this.buildRemoteSystemParams(current, previousArtifact) + ).catch(() => undefined); + if (previousSystemId !== normalized.application_name) { + await this.deleteRemoteOntologiesIfPresent([normalized.application_name], { ignoreDeleteErrors: true }); + } + } + if (artifact) { + await this.discardArtifact(artifact); + } + throw error; + } + + const deletedApplicationNames = []; + if (previousSystemId !== normalized.application_name) { + const deleted = await this.deleteRemoteOntologiesIfPresent([previousSystemId], { ignoreDeleteErrors: true }); + deletedApplicationNames.push(...deleted.deletedApplicationNames); + } + if (artifact && previousArtifact?.id !== artifact.id) { + await this.discardArtifact(previousArtifact); + } return { system: this.getSystemById(id), @@ -544,9 +587,6 @@ export class SystemService { } const normalizedStatus = normalizeText(status) === "disabled" ? "disabled" : "active"; - await this.managementClient.call(normalizedStatus === "active" ? "apps.enable" : "apps.disable", { - systemId: current.application_name - }); this.db.prepare(` UPDATE business_systems SET status = ?, updated_at = ? @@ -566,8 +606,10 @@ export class SystemService { throw conflict("内置 ontology 不允许删除"); } + const artifact = this.getArtifactStorageInfo(current.ontology_artifact_id); await this.managementClient.call("apps.delete", { systemId: current.application_name }); this.db.prepare("DELETE FROM business_systems WHERE id = ?").run(id); + await this.discardArtifact(artifact); return { ok: true }; } @@ -642,7 +684,7 @@ export class SystemService { return filePath; } - async persistInvocationArtifact({ traceId, direction, applicationName, summary, createdAt }) { + async prepareInvocationArtifact({ traceId, direction, applicationName, summary, createdAt }) { let storage = null; let redactionStatus = "metadata_only"; try { @@ -656,50 +698,79 @@ export class SystemService { storage = null; } + return { + traceId, + direction, + summary, + storage, + redactionStatus, + createdAt + }; + } + + insertInvocationArtifact(prepared) { const result = this.db.prepare(` INSERT INTO system_invocation_artifacts ( trace_id, direction, content_type, byte_size, sha256, storage_bucket, storage_object_key, preview_json, redaction_status, created_at ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `).run( - traceId, - direction, + prepared.traceId, + prepared.direction, "application/json", - summary.byteSize, - summary.sha256, - storage?.bucket || null, - storage?.objectKey || null, - JSON.stringify(summary.preview ?? {}), - redactionStatus, - createdAt + prepared.summary.byteSize, + prepared.summary.sha256, + prepared.storage?.bucket || null, + prepared.storage?.objectKey || null, + JSON.stringify(prepared.summary.preview ?? {}), + prepared.redactionStatus, + prepared.createdAt ); return { id: result.lastInsertRowid, - direction, + direction: prepared.direction, content_type: "application/json", - byte_size: summary.byteSize, - sha256: summary.sha256, - storage_bucket: storage?.bucket || null, - storage_object_key: storage?.objectKey || null, - preview: summary.preview, - redaction_status: redactionStatus + byte_size: prepared.summary.byteSize, + sha256: prepared.summary.sha256, + storage_bucket: prepared.storage?.bucket || null, + storage_object_key: prepared.storage?.objectKey || null, + preview: prepared.summary.preview, + redaction_status: prepared.redactionStatus }; } + async removeInvocationArtifactFile(prepared) { + if (!prepared?.storage?.objectKey) { + return; + } + const filePath = this.resolveInvocationArtifactFilePath(prepared.storage.objectKey); + await fs.rm(filePath, { force: true }).catch(() => undefined); + } + + async persistInvocationArtifact(params) { + const prepared = await this.prepareInvocationArtifact(params); + try { + return this.insertInvocationArtifact(prepared); + } catch (error) { + await this.removeInvocationArtifactFile(prepared); + throw error; + } + } + async recordInvocation(log) { const createdAt = log.created_at || toChinaISOString(); const traceId = log.trace_id; const requestSummary = summarizePayload(log.request_payload ?? {}); const responseSummary = summarizePayload(log.response_payload ?? {}); - const requestArtifact = await this.persistInvocationArtifact({ + const requestPrepared = await this.prepareInvocationArtifact({ traceId, direction: "request", applicationName: log.application_name, summary: requestSummary, createdAt }); - const responseArtifact = await this.persistInvocationArtifact({ + const responsePrepared = await this.prepareInvocationArtifact({ traceId, direction: "response", applicationName: log.application_name, @@ -707,48 +778,56 @@ export class SystemService { createdAt }); - this.db.prepare(` - INSERT INTO system_invocation_logs ( - trace_id, agent_slug, session_id, application_name, command_name, method, - request_size_bytes, response_size_bytes, request_sha256, response_sha256, - request_preview_json, response_preview_json, request_artifact_id, response_artifact_id, - response_time_ms, success, error_message, created_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `).run( - traceId, - log.agent_slug || null, - log.session_id || null, - log.application_name, - log.command_name, - log.method || "", - requestSummary.byteSize, - responseSummary.byteSize, - requestSummary.sha256, - responseSummary.sha256, - JSON.stringify(requestSummary.preview ?? {}), - JSON.stringify(responseSummary.preview ?? {}), - requestArtifact.id, - responseArtifact.id, - log.response_time_ms, - log.success ? 1 : 0, - log.error_message || null, - createdAt - ); - + const supportsTransactions = typeof this.db.exec === "function"; try { - await this.managementClient.call("apps.log.record", { - systemId: log.application_name, - agentId: log.agent_slug, - conversationId: log.session_id, - provider: FIXED_BUSINESS_SYSTEM_PROVIDER, - applicationName: log.application_name, - commandName: log.command_name, - inputParams: requestSummary.preview, - responseTimeMs: log.response_time_ms, - success: log.success, - responseContent: responseSummary.preview - }); - } catch {} + if (supportsTransactions) { + this.db.exec("BEGIN IMMEDIATE"); + } + const requestArtifact = this.insertInvocationArtifact(requestPrepared); + const responseArtifact = this.insertInvocationArtifact(responsePrepared); + this.db.prepare(` + INSERT INTO system_invocation_logs ( + trace_id, agent_slug, session_id, application_name, command_name, method, + request_size_bytes, response_size_bytes, request_sha256, response_sha256, + request_preview_json, response_preview_json, request_artifact_id, response_artifact_id, + response_time_ms, success, error_message, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `).run( + traceId, + log.agent_slug || null, + log.session_id || null, + log.application_name, + log.command_name, + log.method || "", + requestSummary.byteSize, + responseSummary.byteSize, + requestSummary.sha256, + responseSummary.sha256, + JSON.stringify(requestSummary.preview ?? {}), + JSON.stringify(responseSummary.preview ?? {}), + requestArtifact.id, + responseArtifact.id, + log.response_time_ms, + log.success ? 1 : 0, + log.error_message || null, + createdAt + ); + if (supportsTransactions) { + this.db.exec("COMMIT"); + } + } catch (error) { + if (supportsTransactions) { + try { + this.db.exec("ROLLBACK"); + } catch {} + } + await Promise.all([ + this.removeInvocationArtifactFile(requestPrepared), + this.removeInvocationArtifactFile(responsePrepared) + ]); + throw error; + } + } listLogs({ applicationName = "", agentSlug = "" } = {}) { diff --git a/packages/daedalus-agent-management-web/server/test/agent-service-create.test.js b/packages/daedalus-agent-management-web/server/test/agent-service-create.test.js index 880b270c167eb21d5861ad5f4493b32bb6162954..7c671c5a8210b1b2aadd62d3b6abd8f563a79e39 100644 --- a/packages/daedalus-agent-management-web/server/test/agent-service-create.test.js +++ b/packages/daedalus-agent-management-web/server/test/agent-service-create.test.js @@ -23,7 +23,7 @@ const FALLBACK_LLM_SELECTION = { }; const FALLBACK_MODEL_REF = "backup-openai/qwen-fallback"; -function createDb({ failAgentInsert = false, templates = ["default"] } = {}) { +function createDb({ failAgentInsert = false, failAgentUpdate = false, templates = ["default"] } = {}) { const insertedRows = []; const permissions = []; const llmModels = new Map([ @@ -221,6 +221,9 @@ function createDb({ failAgentInsert = false, templates = ["default"] } = {}) { if (sql.includes("UPDATE agents")) { return { run: (...params) => { + if (failAgentUpdate) { + throw new Error("injected agent update failure"); + } if (!insertedRows[0]) { insertedRows.push({ id: 1, @@ -959,7 +962,7 @@ it("updates remote and local llm model during agent update", async () => { expect(updated.llm_model_ref).toBe(ALT_MODEL_REF); expect(actionCalls).toEqual([{ - action: "agent.model.set", + action: "agent.update", params: { agentId: "agent-a", model: { @@ -970,6 +973,42 @@ it("updates remote and local llm model during agent update", async () => { }]); }); +it("restores the remote agent when the local update fails", async () => { + const actionCalls = []; + const service = new AgentService({ + db: createDb({ failAgentUpdate: true }), + managementClient: { + async call(action, params) { + actionCalls.push({ action, params }); + return { ok: true }; + } + }, + objectStorage: {} + }); + + await expectAsync(service.updateAgent(1, { + agent_name: "Agent A Updated" + })).toBeRejectedWithError(/injected agent update failure/); + expect(actionCalls).toEqual([ + { + action: "agent.update", + params: { + agentId: "agent-a", + name: "Agent A Updated", + restart: true + } + }, + { + action: "agent.update", + params: { + agentId: "agent-a", + name: "Agent A", + restart: true + } + } + ]); +}); + it("updates fallback llm model without changing the primary model", async () => { const actionCalls = []; const service = new AgentService({ @@ -990,7 +1029,7 @@ it("updates fallback llm model without changing the primary model", async () => expect(updated.llm_model_ref).toBe(DEFAULT_MODEL_REF); expect(updated.fallback_llm_model_ref).toBe(FALLBACK_MODEL_REF); expect(actionCalls).toEqual([{ - action: "agent.model.set", + action: "agent.update", params: { agentId: "agent-a", model: { @@ -1047,7 +1086,7 @@ it("clears fallback llm model during agent update", async () => { expect(updated.fallback_llm_model_id).toBeNull(); expect(updated.fallback_llm_model_ref).toBeNull(); expect(actionCalls).toEqual([{ - action: "agent.model.set", + action: "agent.update", params: { agentId: "agent-a", model: { @@ -1112,7 +1151,7 @@ it("keeps agent-user bindings when update payload omits assignment fields", asyn { id: 11, username: "zhangsan" } ]); expect(actionCalls).toEqual([{ - action: "agent.name.set", + action: "agent.update", params: { agentId: "agent-a", name: "Agent A Updated", diff --git a/packages/daedalus-agent-management-web/server/test/catalog-sync-service.test.js b/packages/daedalus-agent-management-web/server/test/catalog-sync-service.test.js index 83ef56eb26a7e7f7f0c5dd6b77a9e1f838d14b94..779583c01ed2fc89aea0bfafbea3644ad6e37038 100644 --- a/packages/daedalus-agent-management-web/server/test/catalog-sync-service.test.js +++ b/packages/daedalus-agent-management-web/server/test/catalog-sync-service.test.js @@ -286,7 +286,6 @@ function createDb() { description TEXT NOT NULL, ontology_artifact_id INTEGER, scheme TEXT NOT NULL, - host TEXT NOT NULL, port INTEGER NOT NULL, status TEXT NOT NULL, last_connectivity_test_status TEXT NOT NULL, @@ -380,16 +379,15 @@ it("syncs startup catalog snapshot with one management action runtime call", asy db.prepare(` INSERT INTO business_systems ( provider, application_name, description, ontology_artifact_id, - scheme, host, port, status, last_connectivity_test_status, last_connectivity_test_result_json, + scheme, port, status, last_connectivity_test_status, last_connectivity_test_result_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `).run( "hzg", "crm", "keep-local", null, "http", - "crm.example.com", 8080, "active", "unknown", @@ -400,16 +398,15 @@ it("syncs startup catalog snapshot with one management action runtime call", asy db.prepare(` INSERT INTO business_systems ( provider, application_name, description, ontology_artifact_id, - scheme, host, port, status, last_connectivity_test_status, last_connectivity_test_result_json, + scheme, port, status, last_connectivity_test_status, last_connectivity_test_result_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `).run( "hzg", "legacy", "remove-local", null, "http", - "legacy.example.com", 8080, "active", "unknown", @@ -509,7 +506,7 @@ it("syncs startup catalog snapshot with one management action runtime call", asy templates: 1, agents: 1, skills: 1, - systems: 1 + systems: 2 }); expect(calls).toEqual(["catalog.snapshot"]); @@ -545,21 +542,21 @@ it("syncs startup catalog snapshot with one management action runtime call", asy jasmine.objectContaining({ slug: "global-skill", description: "" }) ]); expect(db.prepare("SELECT application_name, is_builtin FROM business_systems ORDER BY application_name").all()).toEqual([ - jasmine.objectContaining({ application_name: "crm", is_builtin: 1 }) + jasmine.objectContaining({ application_name: "crm", is_builtin: 1 }), + jasmine.objectContaining({ application_name: "legacy", is_builtin: 0 }) ]); - expect(db.prepare("SELECT provider, scheme, host, port, status FROM business_systems WHERE application_name = ?").get("crm")).toEqual({ - provider: "phx", - scheme: "https", - host: "crm-kernel.example.com", - port: 8443, - status: "disabled" + expect(db.prepare("SELECT provider, scheme, port, status FROM business_systems WHERE application_name = ?").get("crm")).toEqual({ + provider: "hzg", + scheme: "http", + port: 8080, + status: "active" }); expect(service.getLlmSyncStatus().summary).toEqual({ models: 2 }); expect(service.getTemplateSyncStatus().summary).toEqual({ templates: 1 }); expect(service.getAgentSyncStatus().summary).toEqual({ agents: 1 }); expect(service.getSkillSyncStatus().summary).toEqual({ skills: 1 }); - expect(service.getSystemSyncStatus().summary).toEqual({ systems: 1 }); + expect(service.getSystemSyncStatus().summary).toEqual({ systems: 2 }); expect(service.getUsageRefreshStatus().status).toBe("idle"); expect(service.getUsageRefreshStatus().summary).toEqual({}); }); diff --git a/packages/daedalus-agent-management-web/server/test/db-reset-migration.test.js b/packages/daedalus-agent-management-web/server/test/db-reset-migration.test.js index 664ad7b3d4d3d543458331c9ef18edd842042570..a0c22a7b3f6bdf1f0e2daf9d77ecf7f74b238934 100644 --- a/packages/daedalus-agent-management-web/server/test/db-reset-migration.test.js +++ b/packages/daedalus-agent-management-web/server/test/db-reset-migration.test.js @@ -38,6 +38,14 @@ function restoreDataDir(previousDataDir) { } } +function restoreEnvironmentValue(name, previousValue) { + if (previousValue === undefined) { + delete process.env[name]; + } else { + process.env[name] = previousValue; + } +} + it("resets drifted legacy schemas to the latest database structure", async () => { const previousDataDir = process.env.DA_WEB_DATA_DIR; const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "daedalus-agent-web-db-reset-test-")); @@ -162,7 +170,7 @@ it("resets drifted legacy schemas to the latest database structure", async () => const settingsColumns = db.prepare("PRAGMA table_info(settings)").all().map((row) => row.name); - expect(schemaVersion?.value).toBe("11"); + expect(schemaVersion?.value).toBe("12"); expect(settingsColumns).toContain("logo_mime_type"); expect(settingsColumns).toContain("logo_data"); expect(agentUsersTable?.name).toBe("agent_users"); @@ -190,6 +198,7 @@ it("resets drifted legacy schemas to the latest database structure", async () => expect(skillColumns).not.toContain("distribution_scope"); expect(skillColumns).toContain("is_builtin"); expect(systemColumns).toContain("scheme"); + expect(systemColumns).not.toContain("host"); expect(systemColumns).not.toContain("schema"); expect(systemColumns).toContain("last_connectivity_test_status"); expect(systemColumns).toContain("last_connectivity_test_result_json"); @@ -334,7 +343,7 @@ it("migrates v1 agents to template relationships without dropping data", async ( ORDER BY slug `).all(); - expect(schemaVersion?.value).toBe("11"); + expect(schemaVersion?.value).toBe("12"); expect(agentColumns).toContain("template_name"); expect(agentColumns).toContain("llm_model_ref"); expect(agentColumns).toContain("fallback_llm_model_ref"); @@ -456,7 +465,7 @@ it("migrates v3 templates to default llm columns and removes template descriptio const templateColumns = db.prepare("PRAGMA table_info(agent_templates)").all().map((row) => row.name); const template = db.prepare("SELECT template_name, default_llm_model_ref FROM agent_templates WHERE template_name = ?").get("default"); - expect(schemaVersion?.value).toBe("11"); + expect(schemaVersion?.value).toBe("12"); expect(templateColumns).toContain("default_llm_provider_id"); expect(templateColumns).toContain("default_llm_model_id"); expect(templateColumns).toContain("default_llm_model_ref"); @@ -540,10 +549,10 @@ it("migrates v4 settings through uploaded logo columns without dropping data", a const schemaVersion = db.prepare("SELECT value FROM schema_meta WHERE key = 'schema_version'").get(); const settings = db.prepare("SELECT portal_name, brand_subtitle, theme_color, logo_mime_type, logo_data FROM settings WHERE id = 1").get(); - expect(schemaVersion?.value).toBe("11"); + expect(schemaVersion?.value).toBe("12"); expect(settings).toEqual({ - portal_name: "daedalus-agent", - brand_subtitle: "智能体管理与运营平台", + portal_name: "智能体集群管理控制台", + brand_subtitle: "Agents Management Console", theme_color: "#a5522d", logo_mime_type: "", logo_data: null @@ -681,7 +690,7 @@ it("migrates v5 templates by dropping template description without dropping data WHERE template_name = ? `).get("default"); - expect(schemaVersion?.value).toBe("11"); + expect(schemaVersion?.value).toBe("12"); expect(templateColumns).not.toContain("description"); expect(template).toEqual({ id: 1, @@ -789,10 +798,10 @@ it("migrates v10 users and portal settings to the daedalus-agent identity", asyn `).get(); const user = db.prepare("SELECT id, role, username, display_name, password_hash FROM users WHERE id = 1").get(); - expect(schemaVersion?.value).toBe("11"); + expect(schemaVersion?.value).toBe("12"); expect(settings).toEqual({ - portal_name: "daedalus-agent", - brand_subtitle: "智能体管理与运营平台", + portal_name: "智能体集群管理控制台", + brand_subtitle: "Agents Management Console", theme_color: "#a5522d", logo_mime_type: "", logo_data: null @@ -807,3 +816,90 @@ it("migrates v10 users and portal settings to the daedalus-agent identity", asyn restoreDataDir(previousDataDir); }); + +it("migrates the configured huozige host from v11 into runtime settings", async () => { + const previousDataDir = process.env.DA_WEB_DATA_DIR; + const previousUserEnvJson = process.env.DA_USER_ENV_JSON; + const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "daedalus-agent-web-db-v11-v12-test-")); + const envPath = path.join(dataDir, "env.json"); + const dbFile = path.join(dataDir, "management-console.db"); + + try { + process.env.DA_WEB_DATA_DIR = dataDir; + process.env.DA_USER_ENV_JSON = envPath; + + const seedModuleUrl = new URL(`../src/db/index.js?case=${Date.now()}-v11-seed`, import.meta.url); + const { db: seedDb } = await import(seedModuleUrl.href); + seedDb.close(); + + const setupDb = new DatabaseSync(dbFile); + setupDb.exec(` + DROP TABLE runtime_settings; + ALTER TABLE business_systems ADD COLUMN host TEXT NOT NULL DEFAULT ''; + UPDATE schema_meta SET value = '11' WHERE key = 'schema_version'; + `); + setupDb.prepare(` + INSERT INTO business_systems ( + provider, application_name, description, ontology_artifact_id, + scheme, port, status, last_connectivity_test_status, + last_connectivity_test_result_json, is_builtin, created_at, updated_at, host + ) VALUES (?, ?, ?, NULL, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `).run( + "hzg", + "crm", + "CRM", + "http", + 8080, + "active", + "unknown", + "{}", + 0, + "2026-08-01T00:00:00.000Z", + "2026-08-01T00:00:00.000Z", + "stale.example.com" + ); + setupDb.close(); + fs.writeFileSync(envPath, JSON.stringify({ + DA_HUOZIGE_SERVER_IP: "configured.example.com", + DA_INTERNAL_NETWORK_ALLOWLIST: "configured.example.com" + }, null, 2)); + + const moduleUrl = new URL(`../src/db/index.js?case=${Date.now()}-v11-v12`, import.meta.url); + const { db } = await import(moduleUrl.href); + const schemaVersion = db.prepare("SELECT value FROM schema_meta WHERE key = 'schema_version'").get(); + const systemColumns = db.prepare("PRAGMA table_info(business_systems)").all().map((row) => row.name); + const runtimeHost = db.prepare(` + SELECT setting_value + FROM runtime_settings + WHERE setting_key = 'huozige_server_host' + `).get(); + const system = db.prepare(` + SELECT application_name, scheme, port, status + FROM business_systems + WHERE application_name = 'crm' + `).get(); + + expect(schemaVersion?.value).toBe("12"); + expect(systemColumns).not.toContain("host"); + expect(runtimeHost?.setting_value).toBe("configured.example.com"); + expect(system).toEqual({ + application_name: "crm", + scheme: "http", + port: 8080, + status: "active" + }); + db.close(); + + const restartModuleUrl = new URL(`../src/db/index.js?case=${Date.now()}-v12-restart`, import.meta.url); + const { db: restartedDb } = await import(restartModuleUrl.href); + expect(restartedDb.prepare(` + SELECT setting_value + FROM runtime_settings + WHERE setting_key = 'huozige_server_host' + `).get()?.setting_value).toBe("configured.example.com"); + restartedDb.close(); + } finally { + restoreDataDir(previousDataDir); + restoreEnvironmentValue("DA_USER_ENV_JSON", previousUserEnvJson); + } +}); diff --git a/packages/daedalus-agent-management-web/server/test/external-service.test.js b/packages/daedalus-agent-management-web/server/test/external-service.test.js index 8d097b3e2e898e2f1b34c567c9f59720ebf28e0c..f7f5ad1d443c0790b6070415daf839b4e33f15d7 100644 --- a/packages/daedalus-agent-management-web/server/test/external-service.test.js +++ b/packages/daedalus-agent-management-web/server/test/external-service.test.js @@ -51,7 +51,6 @@ function createDb() { description TEXT NOT NULL DEFAULT '', ontology_artifact_id INTEGER, scheme TEXT NOT NULL, - host TEXT NOT NULL, port INTEGER NOT NULL, status TEXT NOT NULL, last_connectivity_test_status TEXT NOT NULL DEFAULT 'unknown', @@ -71,8 +70,8 @@ function createDb() { ); INSERT INTO agent_permissions (agent_id, agent_user_id) VALUES (1, 1); INSERT INTO business_systems ( - id, provider, application_name, scheme, host, port, status - ) VALUES (1, 'hzg', 'crm-app', 'https', 'example.com', 443, 'active'); + id, provider, application_name, scheme, port, status + ) VALUES (1, 'hzg', 'crm-app', 'https', 443, 'active'); `); return db; diff --git a/packages/daedalus-agent-management-web/server/test/portal-service-alignment.test.js b/packages/daedalus-agent-management-web/server/test/portal-service-alignment.test.js index 4d845303cd42b58364474c5a3a4ad69a81cf3279..6d657e4b86ec6490c16c927e56b1bb0c9a1d6ccc 100644 --- a/packages/daedalus-agent-management-web/server/test/portal-service-alignment.test.js +++ b/packages/daedalus-agent-management-web/server/test/portal-service-alignment.test.js @@ -399,8 +399,8 @@ it("updates portal settings without touching uploaded logo data", () => { const calls = []; let settings = { id: 1, - portal_name: "daedalus-agent", - brand_subtitle: "智能体管理与运营平台", + portal_name: "智能体集群管理控制台", + brand_subtitle: "Agents Management Console", theme_color: "#a5522d", logo_mime_type: "image/png", has_logo: true, @@ -447,8 +447,8 @@ it("updates and clears uploaded portal logo", () => { const calls = []; let settings = { id: 1, - portal_name: "daedalus-agent", - brand_subtitle: "智能体管理与运营平台", + portal_name: "智能体集群管理控制台", + brand_subtitle: "Agents Management Console", theme_color: "#a5522d", logo_mime_type: "", has_logo: false, @@ -753,6 +753,43 @@ it("rejects updating a global skill with a zip for a different slug before persi expect(actionCalls.length).toBe(0); }); +it("deletes an uploaded object when the artifact row cannot be persisted", async () => { + const deleted = []; + const service = new PortalService({ + db: { + prepare() { + return { + run() { + throw new Error("injected artifact insert failure"); + } + }; + } + }, + objectStorage: { + async uploadAdminArtifact() { + return { bucket: "admin-in", objectKey: "skill/new.zip" }; + }, + async deleteObject(bucket, objectKey) { + deleted.push({ bucket, objectKey }); + } + }, + managementClient: {}, + authService: {} + }); + + await expectAsync(service.persistArtifact({ + kind: "skill", + file: { + buffer: Buffer.from("zip"), + originalname: "new.zip", + mimetype: "application/zip", + size: 3 + }, + createdBy: 1 + })).toBeRejectedWithError(/injected artifact insert failure/); + expect(deleted).toEqual([{ bucket: "admin-in", objectKey: "skill/new.zip" }]); +}); + it("deletes installed global skill through skills.global.delete using skill slug only", async () => { const actionCalls = []; const db = { diff --git a/packages/daedalus-agent-management-web/server/test/runtime-env-config-service.test.js b/packages/daedalus-agent-management-web/server/test/runtime-env-config-service.test.js index fffe0973bdfbe5ae9ea7b8517a1377b17491e61a..ea0c9fea8672e8d84b76e62d1c1b5e31f6fbf04c 100644 --- a/packages/daedalus-agent-management-web/server/test/runtime-env-config-service.test.js +++ b/packages/daedalus-agent-management-web/server/test/runtime-env-config-service.test.js @@ -1,6 +1,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; +import { DatabaseSync } from "node:sqlite"; import { ContainerService } from "../src/services/container-service.js"; import { RuntimeEnvConfigService } from "../src/services/runtime-env-config-service.js"; @@ -9,7 +10,20 @@ function createTempDir() { return fs.mkdtempSync(path.join(os.tmpdir(), "daedalus-agent-env-config-")); } -it("reads the agent internal network allowlist from env.json", () => { +function createRuntimeSettingsDb() { + const db = new DatabaseSync(":memory:"); + db.exec(` + CREATE TABLE runtime_settings ( + setting_key TEXT PRIMARY KEY, + setting_value TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + `); + return db; +} + +it("migrates the agent internal network allowlist from env.json to SQLite", () => { const tempDir = createTempDir(); const envPath = path.join(tempDir, "env.json"); fs.writeFileSync(envPath, JSON.stringify({ @@ -18,28 +32,31 @@ it("reads the agent internal network allowlist from env.json", () => { }, null, 2)); try { - const service = new RuntimeEnvConfigService({ envPath }); + const service = new RuntimeEnvConfigService({ db: createRuntimeSettingsDb(), envPath }); expect(service.getNetworkAllowlistConfig()).toEqual({ agent_internal_network_allowlist: "192.168.1.10,10.20.0.0/16", huozige_server_ip: "" }); + expect(JSON.parse(fs.readFileSync(envPath, "utf8"))).toEqual({ + DA_MANAGEMENT_CONSOLE_PORT: 3030 + }); } finally { fs.rmSync(tempDir, { recursive: true, force: true }); } }); -it("updates the agent internal network allowlist without dropping other env.json values", () => { +it("updates the agent internal network allowlist in SQLite without rewriting env.json", () => { const tempDir = createTempDir(); const envPath = path.join(tempDir, "env.json"); fs.writeFileSync(envPath, JSON.stringify({ DA_MANAGEMENT_CONSOLE_PORT: 3030, - DA_OPENCLAW_GATEWAY_MAX_CONCURRENT: 16, + DA_OPENCLAW_GATEWAY_MAX_CONCURRENT: 128, DA_INTERNAL_NETWORK_ALLOWLIST: "" }, null, 2)); try { - const service = new RuntimeEnvConfigService({ envPath }); + const service = new RuntimeEnvConfigService({ db: createRuntimeSettingsDb(), envPath }); const result = service.updateNetworkAllowlistConfig({ agent_internal_network_allowlist: " 192.168.1.10 ; 10.20.0.0/16 " }); @@ -47,19 +64,19 @@ it("updates the agent internal network allowlist without dropping other env.json expect(result).toEqual({ agent_internal_network_allowlist: "192.168.1.10 ; 10.20.0.0/16", - huozige_server_ip: "" + huozige_server_ip: "", + network_allowlist_changed: true }); expect(saved).toEqual({ DA_MANAGEMENT_CONSOLE_PORT: 3030, - DA_OPENCLAW_GATEWAY_MAX_CONCURRENT: 16, - DA_INTERNAL_NETWORK_ALLOWLIST: "192.168.1.10 ; 10.20.0.0/16" + DA_OPENCLAW_GATEWAY_MAX_CONCURRENT: 128 }); } finally { fs.rmSync(tempDir, { recursive: true, force: true }); } }); -it("updates the huozige server ip and replaces it in the allowlist", () => { +it("migrates the huozige server ip to SQLite and removes the legacy env.json value", () => { const tempDir = createTempDir(); const envPath = path.join(tempDir, "env.json"); fs.writeFileSync(envPath, JSON.stringify({ @@ -69,20 +86,160 @@ it("updates the huozige server ip and replaces it in the allowlist", () => { }, null, 2)); try { - const service = new RuntimeEnvConfigService({ envPath }); + const db = createRuntimeSettingsDb(); + const service = new RuntimeEnvConfigService({ db, envPath }); + expect(service.getNetworkAllowlistConfig().huozige_server_ip).toBe("192.168.1.10"); + const result = service.updateNetworkAllowlistConfig({ huozige_server_ip: "192.168.1.11" }); const saved = JSON.parse(fs.readFileSync(envPath, "utf8")); + const persisted = db.prepare(` + SELECT setting_value + FROM runtime_settings + WHERE setting_key = 'huozige_server_host' + `).get(); expect(result).toEqual({ agent_internal_network_allowlist: "10.0.0.2,10.20.0.0/16,192.168.1.11", - huozige_server_ip: "192.168.1.11" + huozige_server_ip: "192.168.1.11", + network_allowlist_changed: true }); expect(saved).toEqual({ - DA_MANAGEMENT_CONSOLE_PORT: 3030, - DA_HUOZIGE_SERVER_IP: "192.168.1.11", - DA_INTERNAL_NETWORK_ALLOWLIST: "10.0.0.2,10.20.0.0/16,192.168.1.11" + DA_MANAGEMENT_CONSOLE_PORT: 3030 + }); + expect(persisted.setting_value).toBe("192.168.1.11"); + } finally { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +}); + +it("does not add a huozige domain to the internal network allowlist", () => { + const tempDir = createTempDir(); + const envPath = path.join(tempDir, "env.json"); + fs.writeFileSync(envPath, JSON.stringify({ + DA_INTERNAL_NETWORK_ALLOWLIST: "10.20.0.0/16" + }, null, 2)); + + try { + const db = createRuntimeSettingsDb(); + const service = new RuntimeEnvConfigService({ db, envPath }); + const domainResult = service.updateNetworkAllowlistConfig({ + huozige_server_ip: "apps.example.com" + }); + + expect(domainResult).toEqual({ + agent_internal_network_allowlist: "10.20.0.0/16", + huozige_server_ip: "apps.example.com", + network_allowlist_changed: false + }); + expect(JSON.parse(fs.readFileSync(envPath, "utf8"))).toEqual({ + }); + expect(service.getNetworkAllowlistConfig()).toEqual({ + agent_internal_network_allowlist: "10.20.0.0/16", + huozige_server_ip: "apps.example.com" + }); + + const nextDomainResult = service.updateNetworkAllowlistConfig({ + huozige_server_ip: "service.example.com" + }); + expect(nextDomainResult).toEqual({ + agent_internal_network_allowlist: "10.20.0.0/16", + huozige_server_ip: "service.example.com", + network_allowlist_changed: false + }); + + const ipResult = service.updateNetworkAllowlistConfig({ + huozige_server_ip: "192.168.1.20" + }); + expect(ipResult).toEqual({ + agent_internal_network_allowlist: "10.20.0.0/16,192.168.1.20", + huozige_server_ip: "192.168.1.20", + network_allowlist_changed: true + }); + + const domainAfterIpResult = service.updateNetworkAllowlistConfig({ + huozige_server_ip: "apps.example.com" + }); + expect(domainAfterIpResult).toEqual({ + agent_internal_network_allowlist: "10.20.0.0/16", + huozige_server_ip: "apps.example.com", + network_allowlist_changed: true + }); + } finally { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +}); + +it("removes a migrated huozige domain from the internal network allowlist", () => { + const tempDir = createTempDir(); + const envPath = path.join(tempDir, "env.json"); + fs.writeFileSync(envPath, JSON.stringify({ + DA_HUOZIGE_SERVER_IP: "apps.example.com", + DA_INTERNAL_NETWORK_ALLOWLIST: "10.20.0.0/16,apps.example.com" + }, null, 2)); + + try { + const service = new RuntimeEnvConfigService({ db: createRuntimeSettingsDb(), envPath }); + + expect(service.getHuozigeServerHost()).toBe("apps.example.com"); + expect(JSON.parse(fs.readFileSync(envPath, "utf8"))).toEqual({ + }); + } finally { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +}); + +it("rolls back the server host when the allowlist update fails", () => { + const tempDir = createTempDir(); + const envPath = path.join(tempDir, "env.json"); + fs.writeFileSync(envPath, "{}\n"); + + try { + const db = createRuntimeSettingsDb(); + const service = new RuntimeEnvConfigService({ db, envPath }); + service.updateNetworkAllowlistConfig({ + huozige_server_ip: "192.168.1.10", + agent_internal_network_allowlist: "10.20.0.0/16" + }); + db.exec(` + CREATE TRIGGER fail_allowlist_update + BEFORE UPDATE ON runtime_settings + WHEN NEW.setting_key = 'internal_network_allowlist' + BEGIN + SELECT RAISE(ABORT, 'injected allowlist failure'); + END; + `); + + expect(() => service.updateNetworkAllowlistConfig({ + huozige_server_ip: "192.168.1.11" + })).toThrowError(/injected allowlist failure/); + expect(service.getNetworkAllowlistConfig()).toEqual({ + agent_internal_network_allowlist: "10.20.0.0/16,192.168.1.10", + huozige_server_ip: "192.168.1.10" + }); + } finally { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +}); + +it("removes the selected domain when it already exists in the allowlist", () => { + const tempDir = createTempDir(); + const envPath = path.join(tempDir, "env.json"); + fs.writeFileSync(envPath, JSON.stringify({ + DA_INTERNAL_NETWORK_ALLOWLIST: "10.20.0.0/16,apps.example.com" + }, null, 2)); + + try { + const service = new RuntimeEnvConfigService({ db: createRuntimeSettingsDb(), envPath }); + const result = service.updateNetworkAllowlistConfig({ + huozige_server_ip: "apps.example.com" + }); + + expect(result).toEqual({ + agent_internal_network_allowlist: "10.20.0.0/16", + huozige_server_ip: "apps.example.com", + network_allowlist_changed: true }); } finally { fs.rmSync(tempDir, { recursive: true, force: true }); diff --git a/packages/daedalus-agent-management-web/server/test/runtime-settings-route.test.js b/packages/daedalus-agent-management-web/server/test/runtime-settings-route.test.js index df53c018df6162fc9368bc161ed4970088811635..4c2cdd5f0c76a1d94adea234e3f928f6fac3a99a 100644 --- a/packages/daedalus-agent-management-web/server/test/runtime-settings-route.test.js +++ b/packages/daedalus-agent-management-web/server/test/runtime-settings-route.test.js @@ -176,7 +176,7 @@ it("updates the env allowlist config and writes an audit log", async () => { userId: 1, username: "daedalus-agent", action: "保存 IP 白名单", - detail: "更新 env.json 中的 DA_INTERNAL_NETWORK_ALLOWLIST / DA_HUOZIGE_SERVER_IP" + detail: "原子更新数据库中的活字格服务器地址及内部网络白名单" }]); } finally { await close(server); diff --git a/packages/daedalus-agent-management-web/server/test/system-service-alignment.test.js b/packages/daedalus-agent-management-web/server/test/system-service-alignment.test.js index afba27ca556de13c40a5e2da2db03b1b8f546ac1..dae7b87772ebf83519cd6b49b0214d49bf6bc3c5 100644 --- a/packages/daedalus-agent-management-web/server/test/system-service-alignment.test.js +++ b/packages/daedalus-agent-management-web/server/test/system-service-alignment.test.js @@ -1,6 +1,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; +import { DatabaseSync } from "node:sqlite"; const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "daedalus-agent-web-system-test-")); process.env.DA_WEB_DATA_DIR = dataDir; @@ -40,6 +41,62 @@ function createDb({ currentSystem = null, createdSystem = null } = {}) { }; } +it("requires the global huozige server host when normalizing systems", () => { + const service = new SystemService({ + db: createDb(), + objectStorage: {}, + managementClient: {}, + runtimeEnvConfigService: { + getHuozigeServerHost() { + return ""; + } + } + }); + + expect(() => service.normalizeCreatePayload({ + application_name: "demo-app", + scheme: "https", + port: 443, + status: "active" + })).toThrowError(/请先在基础设置中配置/); +}); + +it("uses a changed global host without updating the business system row", () => { + const system = { + id: 1, + provider: "hzg", + application_name: "demo-app", + scheme: "https", + port: 443, + status: "active" + }; + let configuredHost = "old.example.com"; + const service = new SystemService({ + db: { + prepare() { + return { + get() { + return system; + } + }; + } + }, + objectStorage: {}, + managementClient: {}, + runtimeEnvConfigService: { + getHuozigeServerHost() { + return configuredHost; + } + } + }); + + expect(service.getInvocationBaseUrlByApplicationName("demo-app").baseUrl) + .toBe("https://old.example.com:443/demo-app"); + configuredHost = "new.example.com"; + expect(service.getInvocationBaseUrlByApplicationName("demo-app").baseUrl) + .toBe("https://new.example.com:443/demo-app"); +}); + it("maps website fields to management action runtime apps.create params", async () => { const createdSystem = { id: 1, @@ -48,7 +105,6 @@ it("maps website fields to management action runtime apps.create params", async description: "desc", ontology_artifact_id: 9, scheme: "https", - host: "example.com", port: 443, status: "active", last_connectivity_test_status: "unknown", @@ -124,29 +180,23 @@ it("maps website fields to management action runtime apps.create params", async expect(result.system.application_name).toBe("demo-app"); expect(result.system.base_url).toBe("https://example.com:443/demo-app"); - expect(result.kernelOntologyDeleted).toBe(true); - expect(result.deletedKernelOntologyNames).toEqual(["demo-app"]); - expect(actionCalls.length).toBe(2); + expect(result.kernelOntologyDeleted).toBe(false); + expect(result.deletedKernelOntologyNames).toEqual([]); + expect(actionCalls.length).toBe(1); expect(actionCalls[0]).toEqual({ - action: "apps.delete", + action: "apps.create", params: { - systemId: "demo-app" + systemId: "demo-app", + name: "demo-app", + provider: "hzg", + ontologyName: "demo-app", + description: "desc", + status: "active", + bucket: "admin-in", + objectKey: "ontology/demo.zip", + replaceOntology: true } }); - expect(actionCalls[1].action).toBe("apps.create"); - expect(actionCalls[1].params).toEqual({ - systemId: "demo-app", - name: "demo-app", - provider: "hzg", - ontologyName: "demo-app", - description: "desc", - applicationEndpoint: "https://example.com:443/demo-app", - sessionServiceEndpoint: "https://example.com:443/demo-app", - status: "active", - bucket: "admin-in", - objectKey: "ontology/demo.zip", - replaceOntology: true - }); }); it("recreates remote system with the configured huozige server host", async () => { @@ -157,7 +207,6 @@ it("recreates remote system with the configured huozige server host", async () = description: "old desc", ontology_artifact_id: 7, scheme: "https", - host: "old.example.com", port: 443, status: "disabled" }; @@ -165,7 +214,6 @@ it("recreates remote system with the configured huozige server host", async () = ...currentSystem, description: "new desc", scheme: "http", - host: "configured.example.com", port: 8443, status: "active" }; @@ -201,29 +249,23 @@ it("recreates remote system with the configured huozige server host", async () = }); expect(result.system.application_name).toBe("demo-app"); - expect(result.kernelOntologyDeleted).toBe(true); - expect(result.deletedKernelOntologyNames).toEqual(["demo-app"]); - expect(actionCalls.length).toBe(2); + expect(result.kernelOntologyDeleted).toBe(false); + expect(result.deletedKernelOntologyNames).toEqual([]); + expect(actionCalls.length).toBe(1); expect(actionCalls[0]).toEqual({ - action: "apps.delete", + action: "apps.create", params: { - systemId: "demo-app" + systemId: "demo-app", + name: "demo-app", + provider: "hzg", + ontologyName: "demo-app", + description: "new desc", + status: "active", + bucket: "admin-in", + objectKey: "ontology/existing.zip", + replaceOntology: true } }); - expect(actionCalls[1].action).toBe("apps.create"); - expect(actionCalls[1].params).toEqual({ - systemId: "demo-app", - name: "demo-app", - provider: "hzg", - ontologyName: "demo-app", - description: "new desc", - applicationEndpoint: "http://configured.example.com:8443/demo-app", - sessionServiceEndpoint: "http://configured.example.com:8443/demo-app", - status: "active", - bucket: "admin-in", - objectKey: "ontology/existing.zip", - replaceOntology: true - }); }); it("rejects deleting built-in ontology systems", async () => { @@ -234,7 +276,6 @@ it("rejects deleting built-in ontology systems", async () => { description: "desc", ontology_artifact_id: 7, scheme: "https", - host: "example.com", port: 443, status: "active", is_builtin: 1 @@ -262,6 +303,87 @@ it("rejects deleting built-in ontology systems", async () => { expect(actionCalls).toEqual([]); }); +it("restores the previous remote ontology when the local system update fails", async () => { + const currentSystem = { + id: 1, + provider: "hzg", + application_name: "demo-app", + description: "old desc", + ontology_artifact_id: 7, + scheme: "https", + port: 443, + status: "active", + is_builtin: 0 + }; + const db = createDb({ currentSystem }); + const originalPrepare = db.prepare.bind(db); + db.prepare = (sql) => { + const statement = originalPrepare(sql); + if (sql.includes("UPDATE business_systems")) { + statement.run = () => { + throw new Error("injected system update failure"); + }; + } + return statement; + }; + const actionCalls = []; + const service = new SystemService({ + db, + objectStorage: {}, + managementClient: { + async call(action, params) { + actionCalls.push({ action, params }); + return {}; + } + }, + runtimeEnvConfigService: { + getHuozigeServerHost: () => "configured.example.com" + } + }); + + await expectAsync(service.updateSystem(1, { + payload: { description: "new desc" }, + file: null, + createdBy: 1 + })).toBeRejectedWithError(/injected system update failure/); + expect(actionCalls.map((item) => item.action)).toEqual(["apps.create", "apps.create"]); + expect(actionCalls[0].params.description).toBe("new desc"); + expect(actionCalls[1].params.description).toBe("old desc"); +}); + +it("deletes an uploaded ontology object when artifact persistence fails", async () => { + const deleted = []; + const service = new SystemService({ + db: { + prepare() { + return { + run() { + throw new Error("injected artifact insert failure"); + } + }; + } + }, + objectStorage: { + async uploadAdminArtifact() { + return { bucket: "admin-in", objectKey: "ontology/new.zip" }; + }, + async deleteObject(bucket, objectKey) { + deleted.push({ bucket, objectKey }); + } + }, + managementClient: {} + }); + const zip = new (await import("adm-zip")).default(); + zip.addFile("index.md", Buffer.from("# index", "utf8")); + const buffer = zip.toBuffer(); + + await expectAsync(service.persistOntologyArtifact({ + file: { buffer, originalname: "new.zip", mimetype: "application/zip", size: buffer.length }, + createdBy: 1 + })).toBeRejectedWithError(/injected artifact insert failure/); + expect(deleted).toEqual([{ bucket: "admin-in", objectKey: "ontology/new.zip" }]); +}); + it("stores invocation payload files under DAEDALUS_AGENT data logs invoke date folders", async () => { const invokeDataDir = fs.mkdtempSync(path.join(os.tmpdir(), "daedalus-agent-data-invoke-test-")); try { @@ -333,3 +455,74 @@ it("stores invocation payload files under DAEDALUS_AGENT data logs invoke date f fs.rmSync(invokeDataDir, { recursive: true, force: true }); } }); + +it("rolls back invocation rows and payload files when the log insert fails", async () => { + const invokeDataDir = fs.mkdtempSync(path.join(os.tmpdir(), "daedalus-agent-invoke-rollback-")); + const db = new DatabaseSync(":memory:"); + db.exec(` + CREATE TABLE system_invocation_artifacts ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + trace_id TEXT NOT NULL, + direction TEXT NOT NULL, + content_type TEXT NOT NULL, + byte_size INTEGER NOT NULL, + sha256 TEXT NOT NULL, + storage_bucket TEXT, + storage_object_key TEXT, + preview_json TEXT NOT NULL, + redaction_status TEXT NOT NULL, + created_at TEXT NOT NULL + ); + CREATE TABLE system_invocation_logs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + trace_id TEXT NOT NULL, + agent_slug TEXT, + session_id TEXT, + application_name TEXT NOT NULL, + command_name TEXT NOT NULL, + method TEXT NOT NULL, + request_size_bytes INTEGER NOT NULL, + response_size_bytes INTEGER NOT NULL, + request_sha256 TEXT NOT NULL, + response_sha256 TEXT NOT NULL, + request_preview_json TEXT NOT NULL, + response_preview_json TEXT NOT NULL, + request_artifact_id INTEGER, + response_artifact_id INTEGER, + response_time_ms INTEGER NOT NULL, + success INTEGER NOT NULL, + error_message TEXT, + created_at TEXT NOT NULL + ); + CREATE TRIGGER fail_invocation_log + BEFORE INSERT ON system_invocation_logs + BEGIN + SELECT RAISE(ABORT, 'injected invocation log failure'); + END; + `); + try { + const service = new SystemService({ + db, + objectStorage: {}, + managementClient: {}, + env: { productDataDir: invokeDataDir } + }); + await expectAsync(service.recordInvocation({ + trace_id: "trace-failure", + application_name: "crm-app", + command_name: "query", + request_payload: { input: "a" }, + response_payload: { output: "b" }, + response_time_ms: 10, + success: true, + created_at: "2026-07-06T05:04:03.002+08:00" + })).toBeRejectedWithError(/injected invocation log failure/); + + expect(db.prepare("SELECT COUNT(*) AS count FROM system_invocation_artifacts").get().count).toBe(0); + const artifactDir = path.join(invokeDataDir, "logs", "invoke", "20260706"); + expect(fs.readdirSync(artifactDir)).toEqual([]); + } finally { + db.close(); + fs.rmSync(invokeDataDir, { recursive: true, force: true }); + } +}); diff --git a/packages/daedalus-agent-management-web/src/brand.js b/packages/daedalus-agent-management-web/src/brand.js index 2a4e711bea87c253cceb3133db1179208dee1735..6eaf5fef26297ee2563b6ea8aa8654a7951f0473 100644 --- a/packages/daedalus-agent-management-web/src/brand.js +++ b/packages/daedalus-agent-management-web/src/brand.js @@ -1,5 +1,5 @@ -export const BRAND_NAME = "智能体管理与运营平台"; -export const BRAND_SUBTITLE = "Your company's name here"; +export const BRAND_NAME = "智能体集群管理控制台"; +export const BRAND_SUBTITLE = "Agents Management Console"; export const BRAND_SLOGAN = "说出你的困境，我来解决它"; export const BRAND_THEME_COLOR = "#a5522d"; diff --git a/packages/daedalus-agent-management-web/src/pages/SettingsPage.jsx b/packages/daedalus-agent-management-web/src/pages/SettingsPage.jsx index 6f1bbf5c201d76fa7a2e6267de9bb931724f96fe..c1f3f834f5f97a671ecc34ee2ade74613db95894 100644 --- a/packages/daedalus-agent-management-web/src/pages/SettingsPage.jsx +++ b/packages/daedalus-agent-management-web/src/pages/SettingsPage.jsx @@ -693,9 +693,11 @@ export function SettingsPage({ agent_internal_network_allowlist: savedRuntimeConfig.agent_internal_network_allowlist || "" }); notify?.success("对话应用/业务应用服务器已保存"); - setContainerRestartConfirmTitle("对话应用/业务应用服务器已保存"); - setContainerRestartConfirmContent("活字格服务器地址已保存到 env.json，并已同步调整 IP 白名单：旧地址已移除，新地址已加入。重启容器后生效。点击确定将立即重启整个容器；取消后可以继续使用当前页面。"); - setContainerRestartConfirmOpen(true); + if (savedRuntimeConfig.network_allowlist_changed) { + setContainerRestartConfirmTitle("对话应用/业务应用服务器已保存"); + setContainerRestartConfirmContent("活字格服务器地址已保存到系统数据库，并已同步调整 IP 白名单。新地址会立即用于业务调用；IP 白名单需重启容器后生效。点击确定将立即重启整个容器；取消后可以继续使用当前页面。"); + setContainerRestartConfirmOpen(true); + } } } catch (error) { notify?.error(`基础设置保存失败：${getErrorMessage(error, "保存失败")}`); @@ -981,7 +983,7 @@ export function SettingsPage({ { diff --git a/packages/daedalus-agent-management-web/test/integration/full.js b/packages/daedalus-agent-management-web/test/integration/full.js index 10ed1c32f6dc81f7c18fe5e22472778b7ee35445..d1111503fc7e01bc3b26e34e7b4bfb4653d015bd 100644 --- a/packages/daedalus-agent-management-web/test/integration/full.js +++ b/packages/daedalus-agent-management-web/test/integration/full.js @@ -370,6 +370,17 @@ async function main() { addAction(report, "settings.huozige-server.update", hzgServerUpdate.ok, hzgServerUpdate.status, hzgServerUpdate.body); assertOk(hzgServerUpdate, "settings.huozige-server.update"); + const internalSystemAfterHostUpdate = await session.get(`/api/internal/app-invoke/base-url?applicationName=${applicationName}`); + addAction( + report, + "internal.app-invoke.base-url-after-host-update", + internalSystemAfterHostUpdate.ok + && internalSystemAfterHostUpdate.body?.baseUrl === `https://service2.example.com:443/${applicationName}`, + internalSystemAfterHostUpdate.status, + internalSystemAfterHostUpdate.body + ); + assertOk(internalSystemAfterHostUpdate, "internal.app-invoke.base-url-after-host-update"); + const systemUpdate = await session.multipart("PUT", `/api/systems/${systemRow.id}`, { description: "integration system updated", status: "disabled" diff --git a/test/README.md b/test/README.md index 05eacde72a1aea287d647f8b308879170d6d31e1..5fc05a5253e4e2111d85acbecde35ee4a1ebf5a6 100644 --- a/test/README.md +++ b/test/README.md @@ -43,7 +43,7 @@ powershell -ExecutionPolicy Bypass -File .\test\integration\run-flow.windows.ps1 ## Mock LLM -Integration and stress suites use the mock LLM server by default. The mock waits 5 seconds after receiving a request, then returns the response. `chunkDelayMs` stays at `0` unless a test explicitly needs streamed chunk spacing. +Integration and stress suites use the mock LLM server by default. With the default `delayMs=5000`, the mock samples an integer delay uniformly from 4000ms through 6000ms (theoretical mean and median: 5000ms), then returns the response. Set `MOCK_OPENAI_DELAY_JITTER_MS=0` to restore a fixed delay. `chunkDelayMs` stays at `0` unless a test explicitly needs streamed chunk spacing. Use `common/mock-openai-completions-server.js` when testing providers configured as OpenAI Completions. It exposes both `/v1/completions` and `/v1/chat/completions`, extracts the requested stress-test marker, and returns only that marker after the configured delay: @@ -51,6 +51,7 @@ Use `common/mock-openai-completions-server.js` when testing providers configured $env:MOCK_OPENAI_HOST = "0.0.0.0" $env:MOCK_OPENAI_PORT = "18085" $env:MOCK_OPENAI_DELAY_MS = "5000" +$env:MOCK_OPENAI_DELAY_JITTER_MS = "1000" $env:MOCK_OPENAI_CHUNK_DELAY_MS = "0" node .\test\common\mock-openai-completions-server.js ``` @@ -66,14 +67,15 @@ When a containerized agent calls a mock server on the Windows host, configure th The stress suite runner starts `common/mock-openai-server.js`, prepares the local seed/report directory, syncs seed data into a Docker volume, starts the DAEDALUS_AGENT container, and then runs `case/stress/stress-flow.js`. ```powershell -npm --prefix test run test:stress:20a10c +npm --prefix test run test:stress npm --prefix test run test:stress:50a20c npm --prefix test run test:stress:100a50c ``` The suite names mean: -- `20a10c`: 20 agents, 10 concurrent requests/agents. +- `test:stress`: reuse 100 agents and run four mock-LLM load rounds at 5, 10, 30, and 100 concurrent requests/agents. +- `test:stress:20a10c`: quick single-round check with 20 agents and 10 concurrent requests/agents. - `50a20c`: 50 agents, 20 concurrent requests/agents. - `100a50c`: 100 agents, 50 concurrent requests/agents. @@ -85,7 +87,7 @@ $env:DA_STRESS_RUNTIME_STATE = "E:\CODE\daedalus-agent-oc\test\.runtime\suite\st node .\test\case\stress\stress-flow.js ``` -Mock LLM response time is configured in each suite config: +Mock LLM response time is configured in each suite config. `delayMs` is the center of the random delay range; the default jitter is 1000ms: ```json { @@ -100,6 +102,7 @@ Environment variables can override the config: ```powershell $env:MOCK_OPENAI_DELAY_MS = "5000" +$env:MOCK_OPENAI_DELAY_JITTER_MS = "1000" $env:MOCK_OPENAI_CHUNK_DELAY_MS = "25" $env:DA_STRESS_IMAGE_TAG = "daedalus-agent-root-only-perms:20260630-5" npm --prefix test run test:stress:50a20c diff --git a/test/case/branding/daedalus-brand-contract.test.js b/test/case/branding/daedalus-brand-contract.test.js index 6f4b89b3e7eb1ce1d87a82fc106142e3a163b7e2..20dedc02df1f434f53eb44df75236ce5a8fd81ea 100644 --- a/test/case/branding/daedalus-brand-contract.test.js +++ b/test/case/branding/daedalus-brand-contract.test.js @@ -75,8 +75,8 @@ test("exposes the daedalus-agent visual identity contract", async () => { const loginPageSource = fs.readFileSync(path.join(webRoot, "src", "pages", "LoginPage.jsx"), "utf8"); const faviconSource = fs.readFileSync(path.join(webRoot, "public", "daedalus-favicon.svg"), "utf8"); - assert.equal(brand.BRAND_NAME, "daedalus-agent"); - assert.equal(brand.BRAND_SUBTITLE, "智能体管理与运营平台"); + assert.equal(brand.BRAND_NAME, "智能体集群管理控制台"); + assert.equal(brand.BRAND_SUBTITLE, "Agents Management Console"); assert.equal(brand.BRAND_SLOGAN, "说出你的困境，我来解决它"); assert.equal(brand.BRAND_THEME_COLOR, "#a5522d"); assert.match(logoSource, /daedalus-mark/); diff --git a/test/case/integration/playwright-flow.js b/test/case/integration/playwright-flow.js index 724ca26b6b1ba00f25c77748a1e3852320ce541f..027c4c22b68c92cb8f322baac2ab4d0d173e70c3 100644 --- a/test/case/integration/playwright-flow.js +++ b/test/case/integration/playwright-flow.js @@ -1763,7 +1763,6 @@ async function agentPageWriteSenderIsolationMarker(page, baseUrl, marker, timeou ].join("\n"), timeoutMs, { - expectedText, maxReplyLength: 1200 } ); diff --git a/test/case/stress/100a-reuse-concurrency-matrix-contract.test.js b/test/case/stress/100a-reuse-concurrency-matrix-contract.test.js new file mode 100644 index 0000000000000000000000000000000000000000..3f7a28d0b5677a4dba2d99d6108c3b869f38b5d0 --- /dev/null +++ b/test/case/stress/100a-reuse-concurrency-matrix-contract.test.js @@ -0,0 +1,95 @@ +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; + +import { + DEFAULT_CONCURRENCY_MATRIX, + DEFAULT_DURATION_MS, + DEFAULT_MAX_CONCURRENT, + DEFAULT_WARMUP_MS, + buildConfig, + readMatrix +} from "../../suite/stress/run-100a-reuse-concurrency-matrix.js"; + +const testDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", ".."); +const packageJson = JSON.parse(fs.readFileSync(path.join(testDir, "package.json"), "utf8")); + +test("default stress command runs the 100-agent concurrency matrix", () => { + assert.equal( + packageJson.scripts["test:stress"], + "node ./suite/stress/run-100a-reuse-concurrency-matrix.js" + ); + assert.equal(packageJson.scripts["test:stress:100a-matrix"], packageJson.scripts["test:stress"]); +}); + +test("100-agent matrix defaults to 5, 10, 30, and 100 concurrent requests", () => { + assert.deepEqual([...DEFAULT_CONCURRENCY_MATRIX], [5, 10, 30, 100]); + assert.deepEqual(readMatrix("", DEFAULT_CONCURRENCY_MATRIX), DEFAULT_CONCURRENCY_MATRIX); + + for (const concurrency of DEFAULT_CONCURRENCY_MATRIX) { + const config = buildConfig(concurrency); + assert.equal(config.stress.counts.agents, 100); + assert.equal(config.stress.counts.concurrentRequests, concurrency); + assert.equal(config.stress.counts.concurrentAgents, concurrency); + } +}); + +test("all matrix rounds reuse the same resources and deterministic agent selection", () => { + const configs = DEFAULT_CONCURRENCY_MATRIX.map((concurrency) => buildConfig(concurrency)); + assert.deepEqual(new Set(configs.map((config) => config.stress.prefix)), new Set(["mock-100a-reuse"])); + assert.deepEqual(new Set(configs.map((config) => config.stress.randomSeed)), new Set(["mock-100a-reuse"])); + assert.ok(configs.every((config) => config.stress.skipResourceCreation)); + assert.ok(configs.every((config) => config.stress.skipSeededStartup)); + assert.deepEqual(readMatrix("5,10,30,100", []), [5, 10, 30, 100]); + assert.deepEqual(readMatrix("5,10,10,30,100,200", []), [5, 10, 30, 100]); +}); + +test("matrix runner always uses the local mock OpenAI server", () => { + for (const concurrency of DEFAULT_CONCURRENCY_MATRIX) { + const config = buildConfig(concurrency); + assert.equal(config.llm.key, "sk-mock-local"); + assert.equal(config.llm.model, "mock-openai"); + assert.equal(config.llm.modelName, "mock-openai"); + assert.match(config.llm.url, /^http:\/\/host\.docker\.internal:\d+\/v1$/); + assert.equal(config.mockLlm.delayMs, 5000); + assert.equal(config.mockLlm.delayJitterMs, 1000); + assert.equal(config.mockLlm.capacity, 128); + assert.equal(config.stress.gatewayMaxConcurrent, 128); + assert.equal(config.stress.load.warmupMs, 30000); + assert.equal(config.stress.load.durationMs, 900000); + } + + assert.equal(DEFAULT_MAX_CONCURRENT, 128); + assert.equal(DEFAULT_WARMUP_MS, 30000); + assert.equal(DEFAULT_DURATION_MS, 900000); +}); + +test("stress environment persists the configured gateway concurrency", () => { + const prepareSource = fs.readFileSync( + path.join(testDir, "prepare", "stress", "rebuild-stress-env.js"), + "utf8" + ); + + assert.match( + prepareSource, + /DA_OPENCLAW_GATEWAY_MAX_CONCURRENT:\s*env\.gatewayMaxConcurrent/ + ); +}); + +test("stress reports use Avg, P50, P90, and P99 response metrics", () => { + const stressFlowSource = fs.readFileSync( + path.join(testDir, "case", "stress", "stress-flow.js"), + "utf8" + ); + const matrixSource = fs.readFileSync( + path.join(testDir, "suite", "stress", "run-100a-reuse-concurrency-matrix.js"), + "utf8" + ); + + assert.match(stressFlowSource, /p90:\s*percentile\(responseTimes,\s*90\)/); + assert.doesNotMatch(stressFlowSource, /p95/i); + assert.match(matrixSource, /\| Avg \| P50 \| P90 \| P99 \|/); + assert.doesNotMatch(matrixSource, /p95/i); +}); diff --git a/test/case/stress/mock-delay.test.js b/test/case/stress/mock-delay.test.js new file mode 100644 index 0000000000000000000000000000000000000000..8896282d79ef93bea9cb8cdd0d3a5481f151fe4c --- /dev/null +++ b/test/case/stress/mock-delay.test.js @@ -0,0 +1,42 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + DEFAULT_DELAY_JITTER_MS, + DEFAULT_DELAY_MS, + resolveDelayRange, + sampleDelayMs +} from "../../common/mock-delay.js"; + +test("default mock delay is symmetric from 4 to 6 seconds", () => { + const range = resolveDelayRange(); + assert.deepEqual(range, { minMs: 4000, maxMs: 6000 }); + assert.equal(DEFAULT_DELAY_MS, 5000); + assert.equal(DEFAULT_DELAY_JITTER_MS, 1000); + assert.equal(sampleDelayMs(range, () => 0), 4000); + assert.equal(sampleDelayMs(range, () => 0.999999), 6000); +}); + +test("uniform integer samples have a 5 second mean and median", () => { + const range = resolveDelayRange(); + const samples = Array.from({ length: range.maxMs - range.minMs + 1 }, (_, index) => { + return sampleDelayMs( + range, + () => (index + 0.5) / (range.maxMs - range.minMs + 1) + ); + }); + const sorted = [...samples].sort((left, right) => left - right); + const mean = samples.reduce((sum, value) => sum + value, 0) / samples.length; + const median = sorted[Math.floor(sorted.length / 2)]; + + assert.equal(Math.min(...samples), 4000); + assert.equal(Math.max(...samples), 6000); + assert.equal(mean, 5000); + assert.equal(median, 5000); +}); + +test("zero jitter preserves the existing fixed-delay mode", () => { + const range = resolveDelayRange(5000, 0); + assert.deepEqual(range, { minMs: 5000, maxMs: 5000 }); + assert.equal(sampleDelayMs(range), 5000); +}); diff --git a/test/case/stress/stress-flow.js b/test/case/stress/stress-flow.js index 91e3c38f57f532075245b579c89db2eb581844a6..f9a222ff2d4e8427135ee762df8bf88d9f026b72 100644 --- a/test/case/stress/stress-flow.js +++ b/test/case/stress/stress-flow.js @@ -52,7 +52,7 @@ const DEFAULT_CREATION_CONCURRENCY = { }; const DEFAULT_LOAD_PROFILE = { warmupMs: 30000, - durationMs: 300000, + durationMs: 900000, progressEveryMs: 30000 }; const PROVIDER_ID_PREFIX = "provider"; @@ -766,7 +766,6 @@ async function createSystems(baseUrl, token, names, artifactRoot, stressConfig) application_name: systemName, description: `Generated stress system ${systemName}`, scheme: "http", - host: "127.0.0.1", port: 18080 + (index % 1000), status: "active" }, { @@ -1036,7 +1035,7 @@ function summarizeMqttCollector(collector, durationMs) { max: responseTimes.length ? Math.max(...responseTimes) : null, avg: average(responseTimes), p50: percentile(responseTimes, 50), - p95: percentile(responseTimes, 95), + p90: percentile(responseTimes, 90), p99: percentile(responseTimes, 99) }, sample: collector.sample, diff --git a/test/case/upgrade/openclaw-upgrade-contract.test.js b/test/case/upgrade/openclaw-upgrade-contract.test.js index dc644d081bb25ab72b6fbaddc65dcbaaff776402..ee259da8c06b5d068e0b85e9e9dd74aedc54e7ba 100644 --- a/test/case/upgrade/openclaw-upgrade-contract.test.js +++ b/test/case/upgrade/openclaw-upgrade-contract.test.js @@ -20,7 +20,7 @@ test("Docker image pins the supported OpenClaw and Node versions", async () => { assert.match(dockerfile, /^FROM npm-builder-base AS web-installer$/m); }); -test("default workspace skips idle heartbeat model calls", async () => { +test("Docker runtime disables idle heartbeat model calls", async () => { const dockerfile = await readRepoFile("docker/Dockerfile"); const config = JSON.parse(await readRepoFile("docker/assets/config.json")); const startup = await readRepoFile("docker/assets/scripts/startup.sh"); @@ -39,10 +39,12 @@ test("default workspace skips idle heartbeat model calls", async () => { ); assert.match(dockerfile, /for doc in AGENTS\.md USER\.md SOUL\.md HEARTBEAT\.md;/); assert.deepEqual(config.agents.defaults.heartbeat, { + every: "0m", isolatedSession: true, lightContext: true }); - assert.match(startup, /isolatedSession: true,\n lightContext: true/); + assert.match(startup, /every: "0m",\n isolatedSession: true,\n lightContext: true/); + assert.match(kernelStartup, /\.agents\.defaults\.heartbeat\.every = "0m"/); assert.match(kernelStartup, /\.agents\.defaults\.heartbeat\.isolatedSession = true/); assert.match(kernelStartup, /\.agents\.defaults\.heartbeat\.lightContext = true/); }); diff --git a/test/common/mock-delay.js b/test/common/mock-delay.js new file mode 100644 index 0000000000000000000000000000000000000000..a4629d6f782d37131e82e4972f7e3239d8c668b9 --- /dev/null +++ b/test/common/mock-delay.js @@ -0,0 +1,30 @@ +export const DEFAULT_DELAY_MS = 5000; +export const DEFAULT_DELAY_JITTER_MS = 1000; + +function normalizeNonNegativeInteger(value, fallback) { + const parsed = Number(value); + return Number.isFinite(parsed) && parsed >= 0 ? Math.round(parsed) : fallback; +} + +export function resolveDelayRange(centerMs = DEFAULT_DELAY_MS, jitterMs = DEFAULT_DELAY_JITTER_MS) { + const center = normalizeNonNegativeInteger(centerMs, DEFAULT_DELAY_MS); + const jitter = Math.min( + normalizeNonNegativeInteger(jitterMs, DEFAULT_DELAY_JITTER_MS), + center + ); + return { + minMs: center - jitter, + maxMs: center + jitter + }; +} + +export function sampleDelayMs(range, random = Math.random) { + const minMs = normalizeNonNegativeInteger(range?.minMs, 0); + const maxMs = Math.max(minMs, normalizeNonNegativeInteger(range?.maxMs, minMs)); + const bucketCount = maxMs - minMs + 1; + const randomValue = Number(random()); + const normalizedRandom = Number.isFinite(randomValue) + ? Math.min(Math.max(randomValue, 0), 1 - Number.EPSILON) + : 0.5; + return minMs + Math.floor(normalizedRandom * bucketCount); +} diff --git a/test/common/mock-openai-completions-server.js b/test/common/mock-openai-completions-server.js index 5c81edbfb3be84ef76f37f472c746ada81b8af96..ef25f176293cdbf6352bd0fb2b46a1402bc2a942 100644 --- a/test/common/mock-openai-completions-server.js +++ b/test/common/mock-openai-completions-server.js @@ -1,16 +1,26 @@ import http from "node:http"; import { randomUUID } from "node:crypto"; +import { + DEFAULT_DELAY_JITTER_MS, + DEFAULT_DELAY_MS, + resolveDelayRange, + sampleDelayMs +} from "./mock-delay.js"; function readPositiveInteger(value, fallback) { const parsed = Number(value); return Number.isFinite(parsed) && parsed >= 0 ? Math.round(parsed) : fallback; } -const DEFAULT_DELAY_MS = 5000; -const DEFAULT_CAPACITY = 1000; +const DEFAULT_CAPACITY = 128; const host = process.env.MOCK_OPENAI_HOST || "0.0.0.0"; const port = readPositiveInteger(process.env.MOCK_OPENAI_PORT, 18080); const delayMs = readPositiveInteger(process.env.MOCK_OPENAI_DELAY_MS, DEFAULT_DELAY_MS); +const delayJitterMs = readPositiveInteger( + process.env.MOCK_OPENAI_DELAY_JITTER_MS, + DEFAULT_DELAY_JITTER_MS +); +const delayRange = resolveDelayRange(delayMs, delayJitterMs); const chunkDelayMs = readPositiveInteger(process.env.MOCK_OPENAI_CHUNK_DELAY_MS, 0); const capacity = readPositiveInteger(process.env.MOCK_OPENAI_CAPACITY, DEFAULT_CAPACITY); const startedAt = new Date(); @@ -195,8 +205,9 @@ async function sendCompletion(request, response) { }); stats.recentMarkers = stats.recentMarkers.slice(-20); - if (delayMs > 0) { - await sleep(delayMs); + const sampledDelayMs = sampleDelayMs(delayRange); + if (sampledDelayMs > 0) { + await sleep(sampledDelayMs); } if (payload.stream === true) { @@ -249,8 +260,9 @@ async function sendChatCompletion(request, response) { }); stats.recentMarkers = stats.recentMarkers.slice(-20); - if (delayMs > 0) { - await sleep(delayMs); + const sampledDelayMs = sampleDelayMs(delayRange); + if (sampledDelayMs > 0) { + await sleep(sampledDelayMs); } if (payload.stream === true) { @@ -330,6 +342,9 @@ const server = http.createServer(async (request, response) => { ok: true, service: "mock-openai-completions-server", delayMs, + delayMinMs: delayRange.minMs, + delayMaxMs: delayRange.maxMs, + delayJitterMs, chunkDelayMs, capacity }); @@ -356,6 +371,9 @@ const server = http.createServer(async (request, response) => { startedAt: startedAt.toISOString(), uptimeMs: Date.now() - startedAt.getTime(), delayMs, + delayMinMs: delayRange.minMs, + delayMaxMs: delayRange.maxMs, + delayJitterMs, chunkDelayMs, capacity, ...stats @@ -369,6 +387,9 @@ const server = http.createServer(async (request, response) => { service: "mock-openai-completions-server", resetAt: new Date().toISOString(), delayMs, + delayMinMs: delayRange.minMs, + delayMaxMs: delayRange.maxMs, + delayJitterMs, chunkDelayMs, capacity }); @@ -397,5 +418,9 @@ server.keepAliveTimeout = 65000; server.headersTimeout = 66000; server.listen(port, host, () => { - console.log(`[mock-openai-completions] listening on http://${host}:${port}/v1 delayMs=${delayMs} chunkDelayMs=${chunkDelayMs} capacity=${capacity}`); + console.log( + `[mock-openai-completions] listening on http://${host}:${port}/v1 ` + + `delayMs=${delayMs} delayRangeMs=${delayRange.minMs}-${delayRange.maxMs} ` + + `chunkDelayMs=${chunkDelayMs} capacity=${capacity}` + ); }); diff --git a/test/common/mock-openai-server.js b/test/common/mock-openai-server.js index 7aa532730ff41f0b441cf990a18b074c025648e0..328c427a80a8c26b7d7787ef70e11253952840b6 100644 --- a/test/common/mock-openai-server.js +++ b/test/common/mock-openai-server.js @@ -1,16 +1,26 @@ import http from "node:http"; import { randomUUID } from "node:crypto"; +import { + DEFAULT_DELAY_JITTER_MS, + DEFAULT_DELAY_MS, + resolveDelayRange, + sampleDelayMs +} from "./mock-delay.js"; function readPositiveInteger(value, fallback) { const parsed = Number(value); return Number.isFinite(parsed) && parsed >= 0 ? Math.round(parsed) : fallback; } -const DEFAULT_DELAY_MS = 5000; -const DEFAULT_CAPACITY = 1000; +const DEFAULT_CAPACITY = 128; const host = process.env.MOCK_OPENAI_HOST || "0.0.0.0"; const port = readPositiveInteger(process.env.MOCK_OPENAI_PORT, 18080); const delayMs = readPositiveInteger(process.env.MOCK_OPENAI_DELAY_MS, DEFAULT_DELAY_MS); +const delayJitterMs = readPositiveInteger( + process.env.MOCK_OPENAI_DELAY_JITTER_MS, + DEFAULT_DELAY_JITTER_MS +); +const delayRange = resolveDelayRange(delayMs, delayJitterMs); const chunkDelayMs = readPositiveInteger(process.env.MOCK_OPENAI_CHUNK_DELAY_MS, 0); const capacity = readPositiveInteger(process.env.MOCK_OPENAI_CAPACITY, DEFAULT_CAPACITY); const startedAt = new Date(); @@ -146,8 +156,9 @@ async function sendChatCompletion(request, response) { const id = `chatcmpl-mock-${randomUUID()}`; const model = typeof payload.model === "string" ? payload.model : "mock-openai"; try { - if (delayMs > 0) { - await sleep(delayMs); + const sampledDelayMs = sampleDelayMs(delayRange); + if (sampledDelayMs > 0) { + await sleep(sampledDelayMs); } if (payload.stream === true) { @@ -247,6 +258,9 @@ const server = http.createServer(async (request, response) => { ok: true, service: "mock-openai-server", delayMs, + delayMinMs: delayRange.minMs, + delayMaxMs: delayRange.maxMs, + delayJitterMs, chunkDelayMs, capacity }); @@ -259,6 +273,9 @@ const server = http.createServer(async (request, response) => { startedAt: startedAt.toISOString(), uptimeMs: Date.now() - startedAt.getTime(), delayMs, + delayMinMs: delayRange.minMs, + delayMaxMs: delayRange.maxMs, + delayJitterMs, chunkDelayMs, capacity, ...stats @@ -272,6 +289,9 @@ const server = http.createServer(async (request, response) => { service: "mock-openai-server", resetAt: new Date().toISOString(), delayMs, + delayMinMs: delayRange.minMs, + delayMaxMs: delayRange.maxMs, + delayJitterMs, chunkDelayMs, capacity }); @@ -296,5 +316,9 @@ server.keepAliveTimeout = 65000; server.headersTimeout = 66000; server.listen(port, host, () => { - console.log(`[mock-openai] listening on http://${host}:${port}/v1 delayMs=${delayMs} chunkDelayMs=${chunkDelayMs} capacity=${capacity}`); + console.log( + `[mock-openai] listening on http://${host}:${port}/v1 ` + + `delayMs=${delayMs} delayRangeMs=${delayRange.minMs}-${delayRange.maxMs} ` + + `chunkDelayMs=${chunkDelayMs} capacity=${capacity}` + ); }); diff --git a/test/common/run-container-flow.core.sh b/test/common/run-container-flow.core.sh index e067b0391d0479752c9ca6870eed0f044196a4e3..efed1273bdec5e0c0dc2b840c7ab0ff2b8214467 100644 --- a/test/common/run-container-flow.core.sh +++ b/test/common/run-container-flow.core.sh @@ -192,6 +192,7 @@ const output = { SOURCE_RUNTIME_ENV_PATH: text(config.sourceRuntimeEnvPath, "/var/daedalus-agent/generated/runtime-env.json"), SOURCE_CONFIG_PATH: text(config.sourceConfigPath, "/opt/daedalus-agent-assets/config.json"), HUOZIGE_SERVER_IP: text(process.env.DA_HUOZIGE_SERVER_IP, text(config.huozigeServerIp, text(config.businessSystemHost))), + AGENT_NETWORK_ALLOWLIST: text(process.env.DA_INTERNAL_NETWORK_ALLOWLIST, text(process.env.MOCK_AGENT_NETWORK_ALLOWLIST, text(config.agentNetworkAllowlist))), CLAWHUB_FETCH_TIMEOUT_MS: positiveInteger(config.clawhubFetchTimeoutMs, 180000), STARTUP_WAIT_SECONDS: positiveInteger(config.startupWaitSeconds, 300) }; @@ -478,9 +479,9 @@ prepare_runtime() { docker cp "${SOURCE_CONTAINER}:${SOURCE_RUNTIME_ENV_PATH}" "${MOUNT_DIR}/env.json" docker cp "${SOURCE_CONTAINER}:${SOURCE_CONFIG_PATH}" "${MOUNT_DIR}/config.json" - node - "${MOUNT_DIR}/env.json" "${MOUNT_DIR}/credentials.json" "${HUOZIGE_SERVER_IP}" <<'NODE' + node - "${MOUNT_DIR}/env.json" "${MOUNT_DIR}/credentials.json" "${HUOZIGE_SERVER_IP}" "${AGENT_NETWORK_ALLOWLIST}" <<'NODE' const fs = require("node:fs"); -const [envTarget, credentialsTarget, huozigeServerIp] = process.argv.slice(2); +const [envTarget, credentialsTarget, huozigeServerIp, agentNetworkAllowlist] = process.argv.slice(2); function readJson(filePath) { return JSON.parse(fs.readFileSync(filePath, "utf8").replace(/^\uFEFF/, "")); @@ -512,7 +513,7 @@ const credentials = { const userEnv = { DA_MANAGEMENT_CONSOLE_PORT: 3030, - DA_INTERNAL_NETWORK_ALLOWLIST: sourceEnv.DA_INTERNAL_NETWORK_ALLOWLIST || "" + DA_INTERNAL_NETWORK_ALLOWLIST: agentNetworkAllowlist || sourceEnv.DA_INTERNAL_NETWORK_ALLOWLIST || "" }; if (huozigeServerIp) { userEnv.DA_HUOZIGE_SERVER_IP = huozigeServerIp; @@ -525,10 +526,10 @@ NODE return 0 fi - node - "${MOUNT_DIR}/env.json" "${REPO_ROOT}/docker/assets/config.json" "${HUOZIGE_SERVER_IP}" <<'NODE' + node - "${MOUNT_DIR}/env.json" "${REPO_ROOT}/docker/assets/config.json" "${HUOZIGE_SERVER_IP}" "${AGENT_NETWORK_ALLOWLIST}" <<'NODE' const fs = require("node:fs"); const path = require("node:path"); -const [envTarget, sampleConfig, huozigeServerIp] = process.argv.slice(2); +const [envTarget, sampleConfig, huozigeServerIp, agentNetworkAllowlist] = process.argv.slice(2); const payload = { DA_MANAGEMENT_CONSOLE_PORT: 3030 @@ -536,6 +537,9 @@ const payload = { if (huozigeServerIp) { payload.DA_HUOZIGE_SERVER_IP = huozigeServerIp; } +if (agentNetworkAllowlist) { + payload.DA_INTERNAL_NETWORK_ALLOWLIST = agentNetworkAllowlist; +} fs.writeFileSync(envTarget, `${JSON.stringify(payload, null, 2)}\n`); fs.copyFileSync(sampleConfig, path.join(path.dirname(envTarget), "config.json")); diff --git a/test/integration/config.integration.sample.json b/test/integration/config.integration.sample.json index f35c604041c0782993e93e72706fc3cdae75ed1d..a03bf118387e58c9f026397dd0dedf407726488c 100644 --- a/test/integration/config.integration.sample.json +++ b/test/integration/config.integration.sample.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18083/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": false, diff --git a/test/integration/run-flow.windows.ps1 b/test/integration/run-flow.windows.ps1 index 9179df25de27f5b8c1a3befc183340ce9eec2274..20a28470ee74f88e8e2d7214650a225af72ae2e3 100644 --- a/test/integration/run-flow.windows.ps1 +++ b/test/integration/run-flow.windows.ps1 @@ -582,7 +582,7 @@ function Initialize-MountDir { $script = @' const fs = require("node:fs"); -const [envTarget, credentialsTarget, huozigeServerIp] = process.argv.slice(2); +const [envTarget, credentialsTarget, huozigeServerIp, agentNetworkAllowlist] = process.argv.slice(2); function readJson(filePath) { return JSON.parse(fs.readFileSync(filePath, "utf8").replace(/^\uFEFF/, "")); @@ -614,7 +614,7 @@ const credentials = { const userEnv = { DA_MANAGEMENT_CONSOLE_PORT: 3030, - DA_INTERNAL_NETWORK_ALLOWLIST: sourceEnv.DA_INTERNAL_NETWORK_ALLOWLIST || "" + DA_INTERNAL_NETWORK_ALLOWLIST: agentNetworkAllowlist || sourceEnv.DA_INTERNAL_NETWORK_ALLOWLIST || "" }; if (huozigeServerIp) { userEnv.DA_HUOZIGE_SERVER_IP = huozigeServerIp; @@ -625,14 +625,19 @@ fs.writeFileSync(credentialsTarget, `${JSON.stringify(credentials, null, 2)}\n`, fs.chmodSync(credentialsTarget, 0o600); '@ - Invoke-NodeScript $script @((Join-Path $mountDir "env.json"), (Join-Path $mountDir "credentials.json"), $huozigeServerIp) + Invoke-NodeScript $script @( + (Join-Path $mountDir "env.json"), + (Join-Path $mountDir "credentials.json"), + $huozigeServerIp, + $agentNetworkAllowlist + ) return } $script = @' const fs = require("node:fs"); const path = require("node:path"); -const [envTarget, sampleConfig, huozigeServerIp] = process.argv.slice(2); +const [envTarget, sampleConfig, huozigeServerIp, agentNetworkAllowlist] = process.argv.slice(2); const payload = { DA_MANAGEMENT_CONSOLE_PORT: 3030 @@ -640,12 +645,20 @@ const payload = { if (huozigeServerIp) { payload.DA_HUOZIGE_SERVER_IP = huozigeServerIp; } +if (agentNetworkAllowlist) { + payload.DA_INTERNAL_NETWORK_ALLOWLIST = agentNetworkAllowlist; +} fs.writeFileSync(envTarget, `${JSON.stringify(payload, null, 2)}\n`); fs.copyFileSync(sampleConfig, path.join(path.dirname(envTarget), "config.json")); '@ - Invoke-NodeScript $script @((Join-Path $mountDir "env.json"), (Join-Path (Join-Path $repoRoot "docker\assets") "config.json"), $huozigeServerIp) + Invoke-NodeScript $script @( + (Join-Path $mountDir "env.json"), + (Join-Path (Join-Path $repoRoot "docker\assets") "config.json"), + $huozigeServerIp, + $agentNetworkAllowlist + ) } function Get-DockerDaedalusAgentMountArgs { @@ -898,6 +911,11 @@ $huozigeServerIp = Get-TextValue $env:DA_HUOZIGE_SERVER_IP ( Get-TextValue (Get-JsonProperty $config "businessSystemHost") ) ) +$agentNetworkAllowlist = Get-TextValue $env:DA_INTERNAL_NETWORK_ALLOWLIST ( + Get-TextValue $env:MOCK_AGENT_NETWORK_ALLOWLIST ( + Get-TextValue (Get-JsonProperty $config "agentNetworkAllowlist") + ) +) $clawhubFetchTimeoutMs = Get-PositiveInteger (Get-JsonProperty $config "clawhubFetchTimeoutMs") 180000 $startupWaitSeconds = Get-PositiveInteger (Get-JsonProperty $config "startupWaitSeconds") 300 $reservedHttpPortEnd = $webPort + 20 diff --git a/test/package.json b/test/package.json index 736d7fc49f7610bbb943b69d041192bd54488451..dc19c3fcd4f2924a88e218cd61df2aa99d8f3137 100644 --- a/test/package.json +++ b/test/package.json @@ -6,12 +6,15 @@ "test:brand": "node --test ./case/branding/daedalus-brand-contract.test.js", "test:upgrade-contract": "node --test ./case/upgrade/openclaw-upgrade-contract.test.js", "test:integration:contract": "node --test ./case/integration/file-output-content.test.js", + "test:mock-delay": "node --test ./case/stress/mock-delay.test.js", "case:integration": "node ./case/integration/playwright-flow.js", "case:stress": "node ./case/stress/stress-flow.js", "test:flow": "node ./suite/integration/run-integration-suite.js full", "test:integration:fast": "node ./suite/integration/run-integration-suite.js fast", "test:integration:full": "node ./suite/integration/run-integration-suite.js full", - "test:stress": "node ./suite/stress/run-mock-stress-suite.js 20a10c", + "test:stress": "node ./suite/stress/run-100a-reuse-concurrency-matrix.js", + "test:stress:contract": "node --test ./case/stress/100a-reuse-concurrency-matrix-contract.test.js", + "test:stress:100a-matrix": "node ./suite/stress/run-100a-reuse-concurrency-matrix.js", "test:stress:100a50c": "node ./suite/stress/run-mock-stress-suite.js 100a50c", "test:stress:50a20c": "node ./suite/stress/run-mock-stress-suite.js 50a20c", "test:stress:20a10c": "node ./suite/stress/run-mock-stress-suite.js 20a10c", diff --git a/test/prepare/stress/rebuild-stress-env.js b/test/prepare/stress/rebuild-stress-env.js index b7055daefed99c3f206cb0cce45f49d7910634a4..9972bce6b175b64b2f6a202f3ad3cc64bfcbbc45 100644 --- a/test/prepare/stress/rebuild-stress-env.js +++ b/test/prepare/stress/rebuild-stress-env.js @@ -671,7 +671,8 @@ function writeRuntimeFiles(env) { writeJson(path.join(env.mountDir, "env.json"), { DA_MANAGEMENT_CONSOLE_PORT: 3030, - DA_INTERNAL_NETWORK_ALLOWLIST: env.agentNetworkAllowlist + DA_INTERNAL_NETWORK_ALLOWLIST: env.agentNetworkAllowlist, + DA_OPENCLAW_GATEWAY_MAX_CONCURRENT: env.gatewayMaxConcurrent }, 0o640); writeJson(path.join(env.mountDir, "credentials.json"), credentials, 0o600); writeJson(path.join(env.mountDir, "generated", "runtime-env.json"), runtimeEnv, 0o640); @@ -735,9 +736,9 @@ async function createManagementDb(env, names, assignments) { const insertSystem = db.prepare(` INSERT INTO business_systems ( provider, application_name, description, ontology_artifact_id, - scheme, host, port, status, last_connectivity_test_status, + scheme, port, status, last_connectivity_test_status, last_connectivity_test_result_json, is_builtin, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, 'active', 'unknown', '{}', 0, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, 'active', 'unknown', '{}', 0, ?, ?) `); const insertAgent = db.prepare(` INSERT INTO agents ( @@ -755,6 +756,14 @@ async function createManagementDb(env, names, assignments) { db.exec("BEGIN"); try { + db.prepare(` + INSERT INTO runtime_settings ( + setting_key, setting_value, created_at, updated_at + ) VALUES ('huozige_server_host', '127.0.0.1', ?, ?) + ON CONFLICT(setting_key) DO UPDATE SET + setting_value = excluded.setting_value, + updated_at = excluded.updated_at + `).run(now, now); db.prepare(` INSERT INTO llm_providers ( provider_id, base_url, api_key_mask, api_key_fingerprint, api, @@ -834,7 +843,6 @@ async function createManagementDb(env, names, assignments) { `Generated stress test system ${systemName}.`, result.lastInsertRowid, "http", - "127.0.0.1", 18080, now, now @@ -947,29 +955,6 @@ function writeStressFiles(env, names, assignments, mqttPassword) { agents: managedAgents }); - const systems = names.systemNames.map((systemName, index) => ({ - id: systemName, - name: systemName, - provider: index % 2 === 0 ? "hzg" : "phx", - ontologyName: systemName, - ontologyPackage: `${systemName}.zip`, - description: `Generated stress test system ${systemName}.`, - applicationEndpoint: `http://127.0.0.1:18080/${systemName}`, - sessionServiceEndpoint: `http://127.0.0.1:18080/${systemName}`, - status: "active", - createdAt: toChinaISOString(), - updatedAt: toChinaISOString(), - lastImportedAt: toChinaISOString() - })); - writeJson(path.join(managementStateRoot, "business-systems.json"), { - version: 1, - items: systems - }); - writeJson(path.join(managementStateRoot, "system-logs.json"), { - version: 1, - items: [] - }); - const openClawConfig = buildOpenClawConfig({ names, model: env.model, diff --git a/test/stress/config.stress.sample.json b/test/stress/config.stress.sample.json index 586f3d4c1ccd2a8e6fcfb8e4de1118b79016ef8e..82bdeef6fb5171cc6d1112ddf1c02d665d371187 100644 --- a/test/stress/config.stress.sample.json +++ b/test/stress/config.stress.sample.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18080/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": false, @@ -83,7 +83,7 @@ }, "load": { "warmupMs": 30000, - "durationMs": 300000, + "durationMs": 900000, "progressEveryMs": 30000 } }, diff --git a/test/suite/integration/fast/config.integration.json b/test/suite/integration/fast/config.integration.json index 56ea2269ea707b68935cbc3d10e59e89b74671e2..42b676de7139f60729f8eddd99935489f19602f4 100644 --- a/test/suite/integration/fast/config.integration.json +++ b/test/suite/integration/fast/config.integration.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18083/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": false, diff --git a/test/suite/integration/full/config.integration.json b/test/suite/integration/full/config.integration.json index fc0d892ca48395b68e4408049e521b168ffb2426..e19ec1a33de1824a6c5a2a4a69c9e23f3eb9e191 100644 --- a/test/suite/integration/full/config.integration.json +++ b/test/suite/integration/full/config.integration.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18084/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": false, diff --git a/test/suite/integration/run-integration-suite.js b/test/suite/integration/run-integration-suite.js index 7f4fff83d96969e2b4ce4af268b7d2f6e3b09cab..104c6b2e9812feacf79787ee2ff7b983ac0c9007 100644 --- a/test/suite/integration/run-integration-suite.js +++ b/test/suite/integration/run-integration-suite.js @@ -18,6 +18,7 @@ const runtimeStatePath = path.resolve( process.env.DA_INTEGRATION_RUNTIME_STATE || path.join(runtimeRoot, "run-state.integration.json") ); const mockServerPath = path.join(testDir, "common", "mock-openai-server.js"); +const DEFAULT_DOCKER_DESKTOP_HOST_IP = "192.168.65.254"; function readJson(filePath) { return JSON.parse(fs.readFileSync(filePath, "utf8").replace(/^\uFEFF/, "")); @@ -84,7 +85,7 @@ function buildRuntimeConfig() { const port = readNonNegativeInteger(process.env.MOCK_OPENAI_PORT, readNonNegativeInteger(mockLlm.port, 18080)); const delayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_DELAY_MS, readNonNegativeInteger(mockLlm.delayMs, 5000)); const chunkDelayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_CHUNK_DELAY_MS, readNonNegativeInteger(mockLlm.chunkDelayMs, 0)); - const capacity = readNonNegativeInteger(process.env.MOCK_OPENAI_CAPACITY, readNonNegativeInteger(mockLlm.capacity, 1000)); + const capacity = readNonNegativeInteger(process.env.MOCK_OPENAI_CAPACITY, readNonNegativeInteger(mockLlm.capacity, 128)); const containerBaseUrl = readText( process.env.DA_MOCK_OPENAI_CONTAINER_BASE_URL, readText(process.env.MOCK_OPENAI_BASE_URL, readText(mockLlm.containerBaseUrl, `http://host.docker.internal:${port}/v1`)) @@ -105,6 +106,13 @@ function buildRuntimeConfig() { capacity, containerBaseUrl }; + config.agentNetworkAllowlist = readText( + process.env.DA_INTERNAL_NETWORK_ALLOWLIST, + readText( + process.env.MOCK_AGENT_NETWORK_ALLOWLIST, + readText(config.agentNetworkAllowlist, DEFAULT_DOCKER_DESKTOP_HOST_IP) + ) + ); writeJson(runtimeConfigPath, config); return { mockLlm: config.mockLlm }; } diff --git a/test/suite/stress/100a100c/config.stress.json b/test/suite/stress/100a100c/config.stress.json index b1d446231ba95adf7340330055cecbb4ff516ff1..56145e5839b263b751de28da703bc62bab58d8c1 100644 --- a/test/suite/stress/100a100c/config.stress.json +++ b/test/suite/stress/100a100c/config.stress.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18084/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": true, @@ -72,8 +72,8 @@ "blockIdle": 1500 }, "load": { - "warmupMs": 60000, - "durationMs": 600000, + "warmupMs": 30000, + "durationMs": 900000, "progressEveryMs": 30000 } } diff --git a/test/suite/stress/100a20c/config.stress.json b/test/suite/stress/100a20c/config.stress.json index 1074093934d67d522e2ac8630128e200d545de4e..bb5cdd0fdc957cf19f9b70efa2617bdf2461ff5c 100644 --- a/test/suite/stress/100a20c/config.stress.json +++ b/test/suite/stress/100a20c/config.stress.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18083/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": true, @@ -72,8 +72,8 @@ "blockIdle": 1500 }, "load": { - "warmupMs": 60000, - "durationMs": 600000, + "warmupMs": 30000, + "durationMs": 900000, "progressEveryMs": 30000 } } diff --git a/test/suite/stress/100a50c/config.stress.json b/test/suite/stress/100a50c/config.stress.json index 28d1536d360ff9cd16185d8dee49bed17dc35483..e91a2955452f5cea350dce26609d8bfa8dde8356 100644 --- a/test/suite/stress/100a50c/config.stress.json +++ b/test/suite/stress/100a50c/config.stress.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18080/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": true, @@ -72,8 +72,8 @@ "blockIdle": 1500 }, "load": { - "warmupMs": 60000, - "durationMs": 600000, + "warmupMs": 30000, + "durationMs": 900000, "progressEveryMs": 30000 } } diff --git a/test/suite/stress/20a10c/config.stress.json b/test/suite/stress/20a10c/config.stress.json index fae19766090d7bbea26f797e71814845a781438b..ae02887aacdc0473d11b2aebada8bd8728ff4437 100644 --- a/test/suite/stress/20a10c/config.stress.json +++ b/test/suite/stress/20a10c/config.stress.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18082/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": true, @@ -71,7 +71,7 @@ }, "load": { "warmupMs": 30000, - "durationMs": 300000, + "durationMs": 900000, "progressEveryMs": 30000 } } diff --git a/test/suite/stress/50a20c/config.stress.json b/test/suite/stress/50a20c/config.stress.json index c165f2c9d4d2b9da46a40e2ea410a5d5130975b6..94295b5882a60cf90626fb537f5492a4f7129f74 100644 --- a/test/suite/stress/50a20c/config.stress.json +++ b/test/suite/stress/50a20c/config.stress.json @@ -11,7 +11,7 @@ "delayMs": 5000, "chunkDelayMs": 0, "containerBaseUrl": "http://host.docker.internal:18081/v1", - "capacity": 1000 + "capacity": 128 }, "image": { "skipBuild": true, @@ -71,7 +71,7 @@ }, "load": { "warmupMs": 30000, - "durationMs": 300000, + "durationMs": 900000, "progressEveryMs": 30000 } } diff --git a/test/suite/stress/run-100a-reuse-concurrency-matrix.js b/test/suite/stress/run-100a-reuse-concurrency-matrix.js index 472ba88792c7dfde80b1ca63660d1a268d9839e9..f3fc11b5eea8c4e1f2db71e4b90b751248406901 100644 --- a/test/suite/stress/run-100a-reuse-concurrency-matrix.js +++ b/test/suite/stress/run-100a-reuse-concurrency-matrix.js @@ -2,6 +2,10 @@ import fs from "node:fs"; import path from "node:path"; import { spawn } from "node:child_process"; import { fileURLToPath } from "node:url"; +import { + DEFAULT_DELAY_JITTER_MS, + resolveDelayRange +} from "../../common/mock-delay.js"; const moduleDir = path.dirname(fileURLToPath(import.meta.url)); const testDir = path.resolve(moduleDir, "..", ".."); @@ -13,10 +17,16 @@ const configPath = path.join(runtimeRoot, "config.stress.json"); const runtimeStatePath = path.join(runtimeRoot, "run-state.stress.json"); const imageTag = readText(process.env.DA_STRESS_IMAGE_TAG, defaultImageTag()); +export const DEFAULT_MAX_CONCURRENT = 128; const mockOpenAiPort = readPositiveInteger(process.env.MOCK_OPENAI_PORT, 18085); const mockOpenAiDelayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_DELAY_MS, 5000); +const mockOpenAiDelayJitterMs = readNonNegativeInteger( + process.env.MOCK_OPENAI_DELAY_JITTER_MS, + DEFAULT_DELAY_JITTER_MS +); +const mockOpenAiDelayRange = resolveDelayRange(mockOpenAiDelayMs, mockOpenAiDelayJitterMs); const mockOpenAiChunkDelayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_CHUNK_DELAY_MS, 0); -const mockOpenAiCapacity = readPositiveInteger(process.env.MOCK_OPENAI_CAPACITY, 1000); +const mockOpenAiCapacity = readPositiveInteger(process.env.MOCK_OPENAI_CAPACITY, DEFAULT_MAX_CONCURRENT); const modelBaseUrl = readText(process.env.MOCK_OPENAI_BASE_URL, `http://host.docker.internal:${mockOpenAiPort}/v1`); const hostMockBaseUrl = readText(process.env.MOCK_OPENAI_HOST_BASE_URL, `http://127.0.0.1:${mockOpenAiPort}/v1`); const mountDir = path.resolve(readText(process.env.DA_STRESS_MOUNT_DIR, "/tmp/daedalus-agent-stress-100a-reuse")); @@ -30,9 +40,12 @@ const agentNetworkAllowlist = readText( process.env.DA_INTERNAL_NETWORK_ALLOWLIST, readText(process.env.MOCK_AGENT_NETWORK_ALLOWLIST, "192.168.65.254,127.0.0.1,::1") ); -const matrixValues = readMatrix(process.env.DA_STRESS_CONCURRENCY_MATRIX, [20, 50, 100]); -const warmupMs = readPositiveInteger(process.env.DA_STRESS_WARMUP_MS, 60000); -const durationMs = readPositiveInteger(process.env.DA_STRESS_DURATION_MS, 600000); +export const DEFAULT_CONCURRENCY_MATRIX = Object.freeze([5, 10, 30, 100]); +const matrixValues = readMatrix(process.env.DA_STRESS_CONCURRENCY_MATRIX, DEFAULT_CONCURRENCY_MATRIX); +export const DEFAULT_WARMUP_MS = 30000; +export const DEFAULT_DURATION_MS = 15 * 60 * 1000; +const warmupMs = readPositiveInteger(process.env.DA_STRESS_WARMUP_MS, DEFAULT_WARMUP_MS); +const durationMs = readPositiveInteger(process.env.DA_STRESS_DURATION_MS, DEFAULT_DURATION_MS); const progressEveryMs = readPositiveInteger(process.env.DA_STRESS_PROGRESS_EVERY_MS, 30000); const resourceLimits = { cpus: readText(process.env.DA_STRESS_CPUS, "2"), @@ -58,12 +71,12 @@ function readNonNegativeInteger(value, fallback) { return Number.isFinite(parsed) && parsed >= 0 ? Math.round(parsed) : fallback; } -function readMatrix(value, fallback) { - const items = String(value || "") +export function readMatrix(value, fallback) { + const items = [...new Set(String(value || "") .split(",") .map((item) => Number(item.trim())) - .filter((item) => Number.isFinite(item) && item > 0) - .map((item) => Math.round(item)); + .filter((item) => Number.isFinite(item) && item > 0 && item <= 100) + .map((item) => Math.round(item)))]; return items.length > 0 ? items : fallback; } @@ -142,6 +155,110 @@ async function waitForMockServer(port, timeoutMs = 15000) { throw new Error(`mock LLM did not become ready at ${url}: ${lastError}`); } +async function readContainerLogsSince(sinceIso) { + const result = await run("docker", ["logs", "--since", sinceIso, containerName], { capture: true }); + return `${result.stdout}\n${result.stderr}`; +} + +async function waitForOpenClawGatewayReady(sinceIso, timeoutMs = 240000) { + const deadline = Date.now() + timeoutMs; + let lastLog = ""; + while (Date.now() < deadline) { + lastLog = await readContainerLogsSince(sinceIso); + if (/\[gateway\] ready|http server listening/.test(lastLog)) { + return; + } + await sleep(2000); + } + throw new Error(`OpenClaw gateway did not become ready after hot maxConcurrent update. Recent logs:\n${lastLog.slice(-4000)}`); +} + +async function hotUpdateMaxConcurrent(maxConcurrent) { + const patchScript = [ + 'const fs = require("node:fs");', + 'const path = require("node:path");', + 'const maxConcurrent = Number(process.argv[1]);', + 'function writeJsonAtomic(filePath, value) {', + ' const mode = fs.statSync(filePath).mode & 0o777;', + ' const tempPath = `${filePath}.${process.pid}.tmp`;', + ' fs.writeFileSync(tempPath, `${JSON.stringify(value, null, 2)}\\n`, { mode });', + ' fs.renameSync(tempPath, filePath);', + ' fs.chmodSync(filePath, mode);', + '}', + 'for (const filePath of [', + ' "/var/daedalus-agent/.openclaw/openclaw.json",', + ' "/var/daedalus-agent/config.json",', + ' "/var/daedalus-agent/generated/runtime-config.json"', + ']) {', + ' if (!fs.existsSync(filePath)) continue;', + ' const config = JSON.parse(fs.readFileSync(filePath, "utf8"));', + ' config.agents = config.agents && typeof config.agents === "object" ? config.agents : {};', + ' config.agents.defaults = config.agents.defaults && typeof config.agents.defaults === "object" ? config.agents.defaults : {};', + ' config.agents.defaults.maxConcurrent = maxConcurrent;', + ' writeJsonAtomic(filePath, config);', + '}', + 'for (const filePath of [', + ' "/var/daedalus-agent/env.json",', + ' "/var/daedalus-agent/generated/runtime-env.json"', + ']) {', + ' if (!fs.existsSync(filePath)) continue;', + ' const env = JSON.parse(fs.readFileSync(filePath, "utf8"));', + ' env.DA_OPENCLAW_GATEWAY_MAX_CONCURRENT = maxConcurrent;', + ' writeJsonAtomic(filePath, env);', + '}' + ].join("\n"); + await runRequired("docker", [ + "exec", + containerName, + "node", + "-e", + patchScript, + String(maxConcurrent) + ]); + + const sinceIso = new Date().toISOString(); + const restartScript = [ + 'const fs = require("node:fs");', + 'const gatewayPids = fs.readdirSync("/proc")', + ' .filter((name) => /^\\d+$/.test(name))', + ' .filter((name) => {', + ' try {', + ' return fs.readFileSync(`/proc/${name}/cmdline`).toString().replace(/\\0/g, " ").trim() === "openclaw-gateway";', + ' } catch {', + ' return false;', + ' }', + ' })', + ' .map(Number);', + 'if (gatewayPids.length === 0) throw new Error("running openclaw-gateway process not found");', + 'for (const pid of gatewayPids) process.kill(pid, "SIGTERM");' + ].join("\n"); + await runRequired("docker", [ + "exec", + containerName, + "node", + "-e", + restartScript + ]); + await waitForOpenClawGatewayReady(sinceIso); + + const verifyScript = [ + 'const fs = require("node:fs");', + 'const config = JSON.parse(fs.readFileSync("/var/daedalus-agent/.openclaw/openclaw.json", "utf8"));', + 'process.stdout.write(String(config.agents?.defaults?.maxConcurrent ?? ""));' + ].join("\n"); + const verification = await runRequired("docker", [ + "exec", + containerName, + "node", + "-e", + verifyScript + ], { capture: true }); + if (Number(verification.stdout.trim()) !== maxConcurrent) { + throw new Error(`hot maxConcurrent verification failed: ${verification.stdout.trim()}`); + } + return sinceIso; +} + async function fetchMockStats(pathname = "/stats") { const url = new URL(hostMockBaseUrl); url.pathname = pathname; @@ -153,7 +270,7 @@ async function fetchMockStats(pathname = "/stats") { return await response.json(); } -function buildConfig(concurrency) { +export function buildConfig(concurrency) { const prefix = "mock-100a-reuse"; return { llm: { @@ -166,6 +283,7 @@ function buildConfig(concurrency) { host: "0.0.0.0", port: mockOpenAiPort, delayMs: mockOpenAiDelayMs, + delayJitterMs: mockOpenAiDelayJitterMs, chunkDelayMs: mockOpenAiChunkDelayMs, containerBaseUrl: modelBaseUrl, capacity: mockOpenAiCapacity @@ -199,13 +317,13 @@ function buildConfig(concurrency) { stress: { prefix, runId: `${prefix}-${concurrency}c`, - randomSeed: `${prefix}-${concurrency}c`, + randomSeed: prefix, senderId: `daedalus-agent-stress-100a-reuse-${concurrency}c`, startupMode: "docker", skipResourceCreation: true, skipSeededStartup: true, cleanupArtifacts: true, - gatewayMaxConcurrent: 100, + gatewayMaxConcurrent: DEFAULT_MAX_CONCURRENT, management: { username: "admin", password: "admin" @@ -422,7 +540,7 @@ function buildMarkdownReport(summary) { formatNumber(run.steadyThroughputRps, 2), formatMs(run.responseTimeMs?.avg), formatMs(run.responseTimeMs?.p50), - formatMs(run.responseTimeMs?.p95), + formatMs(run.responseTimeMs?.p90), formatMs(run.responseTimeMs?.p99), `${formatNumber(run.resources.cpuPercent.max, 2)}%`, formatBytes(run.resources.memoryUsedBytes.max) @@ -440,11 +558,11 @@ function buildMarkdownReport(summary) { `- 资源限制：${summary.resourceLimits.cpus} CPU / ${summary.resourceLimits.memory} memory`, `- 环境规模：100 Agent、100 模板、100 技能、100 业务系统、1000 用户、每 workspace 100 文件`, `- 负载：${matrixValues.join(" / ")} 并发；每组 ${formatMs(warmupMs)} 预热 + ${formatMs(durationMs)} 稳态`, - `- Mock LLM：${modelBaseUrl}，固定延迟 ${mockOpenAiDelayMs}ms，容量 ${mockOpenAiCapacity}`, + `- Mock LLM：${modelBaseUrl}，均匀随机延迟 ${mockOpenAiDelayRange.minMs}-${mockOpenAiDelayRange.maxMs}ms（理论均值/中位数 ${mockOpenAiDelayMs}ms），容量 ${mockOpenAiCapacity}`, "", "## 结果汇总", "", - "| 并发 | 结论 | 稳态成功/请求 | 失败 | 吞吐 req/s | Avg | P50 | P95 | P99 | CPU 峰值 | 内存峰值 |", + "| 并发 | 结论 | 稳态成功/请求 | 失败 | 吞吐 req/s | Avg | P50 | P90 | P99 | CPU 峰值 | 内存峰值 |", "| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", ...rows.map((row) => `| ${row.join(" | ")} |`), "", @@ -456,7 +574,7 @@ function buildMarkdownReport(summary) { "## 结论", "", summary.runs.every((run) => run.ok) - ? "三组并发压测均通过，稳态阶段未出现失败请求。" + ? `${summary.runs.length} 轮并发压测均通过，稳态阶段未出现失败请求。` : "存在未通过的并发轮次，详见对应原始报告的 error / failures 字段。", "" ].join("\n"); @@ -483,6 +601,7 @@ async function main() { MOCK_OPENAI_HOST: "0.0.0.0", MOCK_OPENAI_PORT: String(mockOpenAiPort), MOCK_OPENAI_DELAY_MS: String(mockOpenAiDelayMs), + MOCK_OPENAI_DELAY_JITTER_MS: String(mockOpenAiDelayJitterMs), MOCK_OPENAI_CHUNK_DELAY_MS: String(mockOpenAiChunkDelayMs), MOCK_OPENAI_CAPACITY: String(mockOpenAiCapacity) } @@ -497,6 +616,18 @@ async function main() { console.log("[100a-reuse] rebuild and start 100a container"); await runRequired("node", [prepareScriptPath, "--start-container"]); } + console.log(`[100a-reuse] hot update gateway maxConcurrent=${DEFAULT_MAX_CONCURRENT}`); + const lastGatewayRestartStartedAt = await hotUpdateMaxConcurrent(DEFAULT_MAX_CONCURRENT); + const runtimeState = fs.existsSync(runtimeStatePath) ? readJson(runtimeStatePath) : {}; + writeJson(runtimeStatePath, { + ...runtimeState, + lastGatewayRestartStartedAt, + gatewayMaxConcurrent: DEFAULT_MAX_CONCURRENT + }); + const initialMockStats = await fetchMockStats(); + if (initialMockStats.capacity !== mockOpenAiCapacity) { + throw new Error(`mock LLM capacity verification failed: ${initialMockStats.capacity}`); + } const containerInspect = await inspectJson(containerName); const imageInspect = await inspectJson(imageTag); @@ -544,7 +675,12 @@ async function main() { modelBaseUrl, hostMockBaseUrl, mockOpenAiDelayMs, + mockOpenAiDelayJitterMs, + mockOpenAiDelayMinMs: mockOpenAiDelayRange.minMs, + mockOpenAiDelayMaxMs: mockOpenAiDelayRange.maxMs, mockOpenAiCapacity, + gatewayMaxConcurrent: DEFAULT_MAX_CONCURRENT, + lastGatewayRestartStartedAt, matrixValues, warmupMs, durationMs, @@ -561,4 +697,8 @@ async function main() { } } -await main(); +const isMain = process.argv[1] + && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url); +if (isMain) { + await main(); +} diff --git a/test/suite/stress/run-existing-mock-lane-matrix.js b/test/suite/stress/run-existing-mock-lane-matrix.js index 5db13ded400bb727e4059530d97ed9e679aaf55b..13dc678565e886c7d9913ef4ac2b6ca7e029f9a9 100644 --- a/test/suite/stress/run-existing-mock-lane-matrix.js +++ b/test/suite/stress/run-existing-mock-lane-matrix.js @@ -25,7 +25,7 @@ const baseUrl = `http://127.0.0.1:${webPort}`; const mockOpenAiPort = readNonNegativeInteger(process.env.MOCK_OPENAI_PORT, 18080); const mockOpenAiDelayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_DELAY_MS, 5000); const mockOpenAiChunkDelayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_CHUNK_DELAY_MS, 0); -const mockOpenAiCapacity = readNonNegativeInteger(process.env.MOCK_OPENAI_CAPACITY, 1000); +const mockOpenAiCapacity = readNonNegativeInteger(process.env.MOCK_OPENAI_CAPACITY, 128); const modelBaseUrl = process.env.MOCK_OPENAI_BASE_URL || `http://host.docker.internal:${mockOpenAiPort}/v1`; const agentNetworkAllowlist = process.env.MOCK_AGENT_NETWORK_ALLOWLIST || "192.168.65.254"; const memoryBackend = normalizeMemoryBackend(process.env.MOCK_MEMORY_BACKEND); diff --git a/test/suite/stress/run-mock-lane-matrix.js b/test/suite/stress/run-mock-lane-matrix.js index 43105eb4f8ee4070e3a37be05d14df743e2bc2b2..02a0cb4c6a06a8e3b3ec82739476a08b67669759 100644 --- a/test/suite/stress/run-mock-lane-matrix.js +++ b/test/suite/stress/run-mock-lane-matrix.js @@ -40,7 +40,7 @@ function readMockSettings(config) { const port = readNonNegativeInteger(process.env.MOCK_OPENAI_PORT, readNonNegativeInteger(mockLlm.port, 18081)); const delayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_DELAY_MS, readNonNegativeInteger(mockLlm.delayMs, 5000)); const chunkDelayMs = readNonNegativeInteger(process.env.MOCK_OPENAI_CHUNK_DELAY_MS, readNonNegativeInteger(mockLlm.chunkDelayMs, 0)); - const capacity = readNonNegativeInteger(process.env.MOCK_OPENAI_CAPACITY, readNonNegativeInteger(mockLlm.capacity, 1000)); + const capacity = readNonNegativeInteger(process.env.MOCK_OPENAI_CAPACITY, readNonNegativeInteger(mockLlm.capacity, 128)); const containerBaseUrl = readText( process.env.MOCK_OPENAI_BASE_URL, readText(mockLlm.containerBaseUrl, `http://host.docker.internal:${port}/v1`) diff --git a/test/suite/stress/run-mock-stress-suite.js b/test/suite/stress/run-mock-stress-suite.js index 30c1fbcd9852deb23d45ce5a7d82b526ea640be6..3b5da713825baa814ac422432136e5525b23779c 100644 --- a/test/suite/stress/run-mock-stress-suite.js +++ b/test/suite/stress/run-mock-stress-suite.js @@ -14,6 +14,7 @@ const runtimeStatePath = path.resolve(process.env.DA_STRESS_RUNTIME_STATE || pat const mockServerPath = path.join(testDir, "common", "mock-openai-server.js"); const prepareScriptPath = path.join(testDir, "prepare", "stress", "rebuild-stress-env.js"); const stressCasePath = path.join(testDir, "case", "stress", "stress-flow.js"); +const DEFAULT_DOCKER_DESKTOP_HOST_IP = "192.168.65.254"; function readJson(filePath) { return JSON.parse(fs.readFileSync(filePath, "utf8").replace(/^\uFEFF/, "")); @@ -104,7 +105,7 @@ function buildRuntimeConfig() { const port = readPositiveInteger(process.env.MOCK_OPENAI_PORT, readPositiveInteger(mockLlm.port, 18080)); const delayMs = readPositiveInteger(process.env.MOCK_OPENAI_DELAY_MS, readPositiveInteger(mockLlm.delayMs, 5000)); const chunkDelayMs = readPositiveInteger(process.env.MOCK_OPENAI_CHUNK_DELAY_MS, readPositiveInteger(mockLlm.chunkDelayMs, 0)); - const capacity = readPositiveInteger(process.env.MOCK_OPENAI_CAPACITY, readPositiveInteger(mockLlm.capacity, 1000)); + const capacity = readPositiveInteger(process.env.MOCK_OPENAI_CAPACITY, readPositiveInteger(mockLlm.capacity, 128)); const containerBaseUrl = readText( process.env.DA_MOCK_OPENAI_CONTAINER_BASE_URL, readText(mockLlm.containerBaseUrl, `http://host.docker.internal:${port}/v1`) @@ -127,6 +128,13 @@ function buildRuntimeConfig() { capacity, containerBaseUrl }; + config.agentNetworkAllowlist = readText( + process.env.DA_INTERNAL_NETWORK_ALLOWLIST, + readText( + process.env.MOCK_AGENT_NETWORK_ALLOWLIST, + readText(config.agentNetworkAllowlist, DEFAULT_DOCKER_DESKTOP_HOST_IP) + ) + ); config.image = { ...(config.image && typeof config.image === "object" ? config.image : {}), skipBuild: true,