# SecLensRep

**Repository Path**: hickeyhsu/SecLensRep

## Basic Information

- **Project Name**: SecLensRep
- **Description**: 基于 SecLens 的纯大模型提示方法与工具调用式 LLM Agent/Harness 方法在漏洞检测任务上的可复现实证对比
- **Primary Language**: Python
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-05-19
- **Last Updated**: 2026-05-28

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# SecLensRep

SecLensRep is an experiment repository that follows the SecLens evaluation method to compare two LLM vulnerability-detection modes on SecRepoBench-derived tasks:

- Code-in-Prompt: pure LLM prompting, no repository tools.
- Tool-Use: LLM Agent/Harness with repository tools enabled by SecLens.

The project uses SecLens-style task execution and metrics while adapting SecRepoBench repair metadata into paired vulnerable/post-patch tasks for sampling, running, collecting, and scoring results.

## Environment

The project uses the micromamba environment named `srbr`.

```bash
micromamba activate srbr
python -m pip install -e ".[dev]"
```

Install upstream SecLens into the same environment before real preflight or live runs:

```bash
python -m pip install "engine-harness[all] @ git+https://github.com/mattersec-labs/engine-harness@v0.1.0"
python -m pip install -e "/home/hickey/python-workspace/homework/seclens"
```

## Local live configuration

Live API settings are read from `/home/hickey/python-workspace/homework/srbr.live.local.yaml` by default, or from `SRBR_LIVE_CONFIG`. The file is local-only and ignored by git.

Accepted YAML credential keys:

```yaml
openai_base_url: http://<openai-compatible-relay>/v1
openai_api_key: <redacted>
```

Environment variables `OPENAI_API_KEY` and `OPENAI_BASE_URL` override YAML credentials when both are set.
SecLensRep intentionally ignores benchmark-specific `upstream_path` and bare `model` values in this local file, because the same file may be shared by other homework repositories.

## Current commands

Offline preflight:

```bash
seclensrep preflight --offline --seclens-project /home/hickey/python-workspace/homework/seclens
```

Real preflight validates local paths and credential shape without making a network call:

```bash
seclensrep preflight --real --local-config /home/hickey/python-workspace/homework/srbr.live.local.yaml
```

Create a deterministic paired sample from local JSONL or a SecLens dataset id:

```bash
seclensrep sample --source tests/fixtures/dataset_min.jsonl --out .work/smoke/sample.jsonl --pairs 3
seclensrep sample --source enginesec/SecLens:test --out .work/main/sample.jsonl --pairs 10
```

The public dataset id may not be available in every environment. If that happens, use the
local JSONL fallback at `tests/fixtures/dataset_min.jsonl`.

## Reproducibility scripts and docs

- `scripts/offline_smoke.sh` runs local lint, tests, offline dry-run, fallback sampling,
  optional SecLens dry-run schema validation, fixture collect, and fixture metrics.
- `scripts/real_run_tmux.sh` starts a tmux session for a real run and writes logs to
  `.work/logs/<run_id>.log`.
- `docs/experiment_design.md`, `docs/reproducibility.md`, `docs/report_outline.md`, and
  `docs/slides_outline.md` give the course experiment skeleton.
- `.github/workflows/ci.yml` runs the offline CI path with micromamba and no secrets.

## Test commands

```bash
python -m ruff check .
python -m ruff format --check .
python -m pytest -q
```

No tests call the live API.