# knowledge-base

**Repository Path**: fdayu/knowledge-base

## Basic Information

- **Project Name**: knowledge-base
- **Description**: 维基本地知识库是一个本地优先的开源知识库系统，融合向量检索、重排与对话式问答，支持多种主流大模型 API，具备高性能本地存储与灵活扩展能力，适合智能问答、知识管理、企业知识中台等场景。MulanPSL2 许可证，欢迎共建！
- **Primary Language**: Python
- **License**: MulanPSL-2.0
- **Default Branch**: master
- **Homepage**: https://framewiki.com/knowledge-base.html
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 3
- **Created**: 2026-08-19
- **Last Updated**: 2026-08-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<div align="center">
 <img src="assets/logo2.png" alt="WIKI Knowledge Base" width="320" />
 <p><a href="https://framewiki.com/">https://framewiki.com/</a></p>
 <p>
        <a href="README.md">简体中文</a> | <a href="README-EN.md">English</a>
 </p>
 <p>
        <img src="https://img.shields.io/badge/License-MulanPSL2-blue" alt="License" />
        <img src="https://img.shields.io/badge/Chat-SSE-ff8c00" alt="SSE" />
        <img src="https://img.shields.io/badge/Vector-FAISS-6f42c1" alt="FAISS" />
        <img src="https://img.shields.io/badge/Status-Active-2e7d32" alt="Status" />
 </p>
</div>

# WIKI Knowledge Base

<<<<<<< HEAD
WIKI Knowledge Base is a local-first, self-hosted open-source knowledge base service that combines vector retrieval, reranking, conversational QA, model management, and a built-in Web console. It is suitable for internal enterprise knowledge assistants, team documentation search, and personal knowledge hubs.

## Navigation

- [Highlights](#highlights)
- [Use Cases](#use-cases)
- [Feature Comparison](#feature-comparison)
- [Architecture](#architecture)
- [Project Structure](#project-structure)
- [Quick Start](#quick-start)
- [Web Console](#web-console)
- [API Documentation](#api-documentation)
- [Configuration Guide](#configuration-guide)
- [Model and Deployment Guidance](#model-and-deployment-guidance)
- [Operations and Scripts](#operations-and-scripts)
- [Data and Security](#data-and-security)
- [Contribution and License](#contribution-and-license)
=======
Switch docs: [简体中文](README.md) | English

A local-first open-source knowledge base service that combines vector retrieval, reranking, and conversational Q&A. It ships with a lightweight HTTP API and a built-in Web console, supports SSE streaming, and is ideal for private knowledge Q&A applications.
>>>>>>> 85fbbd0d074f8c062bdec54ad64d49ff5026ca8a

## Highlights

- Local-first deployment with fully private knowledge, vector indexes, and session data.
- Built-in retrieval + rerank pipeline for better relevance and answer quality.
- SSE streaming responses for chat-style user experiences.
- Context-aware conversations with `user_id` and `session_id`.
- Multiple ingestion workflows including text input, file upload, batch import, and rebuild.
- Chunk management for viewing, editing, deleting, and rebuilding knowledge chunks.
- Built-in model configuration management compatible with OpenAI, DeepSeek, Qwen, Doubao, xAI, Gemini, Kimi, LM Studio, and more.
- Flexible storage backends for memory, databases, and Redis.

## Use Cases

- Internal enterprise knowledge assistant
- Team SOP, policy, and documentation search
- Personal notes, study materials, and document hub
- Private AI assistant prototypes
- Offline or intranet-based knowledge applications

## Feature Comparison

| Feature | Open-Source Edition | Commercial Edition (Planned/Available) |
| --- | --- | --- |
| Local knowledge storage | ✅ | ✅ |
| FAISS vector retrieval | ✅ | ✅ |
| Document import / batch upload | ✅ | ✅ |
| Document recognition | Text-only | DOC support + OCR for images/PDF/OFD (paddleOCR or Qwen/Qwen2-VL-2B-Instruct) |
| Document chunking | Fixed-length chunking | Semantic chunking |
| Rerank | ✅ | ✅ |
| Multi-model support | ✅ | ✅ |
| SSE streaming | ✅ | ✅ |
| Web UI | ✅ | ✅ |
| API | ✅ | ✅ |
| User/session context | ✅ | ✅ |
| Redis high concurrency | ✅ | ✅ |
| Permissions and multi-user | - | ✅ |
| MCP support | - | ✅ |
| Plugin / extension system | - | ✅ |
| Enterprise security | - | ✅ |
| Commercial support | - | ✅ |
| SLA guarantee | - | ✅ |
| Custom development | - | ✅ |

## Architecture

<<<<<<< HEAD
```text
Client / Browser
         |
         v
HTTP API (src/api.py)
         |
         +-- Web UI static files (web/)
         +-- Session / History Store (src/store/)
         |     +-- memory
         |     +-- db
         |     +-- redis
         |
         +-- Knowledge Base (src/knowledge_base.py)
                                 +-- document parser
                                 +-- chunking
                                 +-- embeddings
                                 +-- FAISS index
                                 +-- reranker
                                 +-- chat model / LLM client
```

**Overall System Architecture**
![](assets/系统整体架构图-en.png)

Typical execution flow:

1. Imported documents are chunked and stored in the local knowledge store.
2. Chunks are embedded and indexed in FAISS.
3. User queries go through vector recall and reranking.
4. Retrieved context plus session history is sent to the chat model.
5. Both the Web UI and external API are served by the same HTTP service.

**Core Data Flow (RAG Retrieval QA Pipeline)**
![](assets/rag-dataflow-en.png)

## Project Structure

```text
.
├─ src/
│  ├─ api.py                     # HTTP API and static resource routing
│  ├─ main.py                    # Application entry point
│  ├─ knowledge_base.py          # Core knowledge base logic
│  ├─ document_parser.py         # Document parsing
│  ├─ universal_llm_client.py    # Unified multi-provider LLM client
│  ├─ model_config_manager.py    # Model configuration management
│  ├─ windows_service.py         # Windows service entry
│  └─ store/                     # Session and history storage implementations
├─ web/                          # Built-in Web console
├─ docs/
│  └─ API.html                   # Offline API documentation
├─ assets/                       # README screenshots and assets
├─ config.json                   # Main configuration file
├─ config.multi-provider.example.json
├─ manage_service.ps1            # Windows service management script
├─ manage_service.sh             # Linux systemd service management script
├─ encrypt_secret.py             # Secret encryption utility
├─ ingest_to_kb_store.py         # Knowledge ingestion utility
├─ migrate_to_sessions_table.py  # Session data migration script
└─ tune_threshold.py             # Retrieval threshold tuning utility
```
=======
```mermaid
flowchart TD
        UI[Web UI<br/>web/app.js] --> API[HTTP API<br/>src/api.py]
        API --> STORE[Store Layer<br/>src/store]
        STORE --> DB[DB Backend<br/>history / session]
        STORE --> REDIS[Redis Backend<br/>session]

        API --> KB[Knowledge Base Core<br/>src/knowledge_base.py]
        KB --> EMB[Embeddings]
        KB --> FAISS[FAISS Index]
        KB --> RERANK[Reranker]
        KB --> CHAT[LLM Chat]
        KB --> PARSER[Document Parser Facade<br/>src/document_parsers/__init__.py]

        PARSER --> BASE[BaseDocumentParser]
        PARSER --> GENERIC[GenericDocumentParser]
        PARSER --> PDF[PdfDocumentParser]
        PARSER --> OFFICE[OfficeDocumentParser]
        PARSER --> IMAGE[ImageDocumentParser]
        PARSER --> OCR[OcrParser]

        KB --> LLM[Universal LLM Client<br/>src/universal_llm_client.py]
```

The current ingestion pipeline uses `KnowledgeBaseDocumentParser` to orchestrate specialized parsers, rather than the older single-entry parser modules.
>>>>>>> 85fbbd0d074f8c062bdec54ad64d49ff5026ca8a

## Quick Start

### Requirements

- Python 3.10+
- `venv` is recommended
- CUDA environment if you plan to run GPU inference
- Redis / MySQL / PostgreSQL if those backends are enabled

### 1. Install dependencies

```bash
pip install -r requirements.txt
```

For packaging/build:

```bash
pip install -r requirements.build.txt
```

For OCR or optional parsing features:

```bash
pip install -r requirements.optional-parser.txt
```

### 2. Configure the service

Edit `config.json` and review these sections first:

- `server`: HTTP bind host and port
- `db`: history storage backend and database connection
- `session`: session backend, `memory` or `redis`
- `knowledge_base.storage`: knowledge store directory and model cache directory
- `knowledge_base.embedding`: embedding model settings
- `knowledge_base.rerank`: reranker settings
- `knowledge_base.chat`: chat model settings
- `knowledge_base.lm_studio`: OpenAI-compatible model service endpoint
- `chat_context`: context enable flag and max turns

See `config.multi-provider.example.json` for a multi-provider example.

### 3. Start the service

Run directly:

```bash
python -m src.main
```

If installed as a package, you can also run:

```bash
knowledge-base
```

The default bind address is read from `config.json` via `server.host` and `server.port`, with `5000` as the default port.

### 4. Common startup arguments

```bash
python -m src.main --host 127.0.0.1 --port 5000
```

Available arguments:

- `--host`: bind host
- `--port`: bind port
- `--no-preload-embedding`: skip embedding warmup on startup
- `--no-preload-reranker`: skip reranker warmup on startup

### 5. Environment variables

- `KB_CONFIG_PATH`: path to a custom config file
- `KB_PRELOAD_EMBEDDING`: preload embedding model on startup
- `KB_PRELOAD_RERANKER`: preload reranker model on startup
- `KB_WARMUP_STRICT`: fail startup if model warmup fails

## Web Console

After startup, visit:

- `http://127.0.0.1:5000/ui/`: chat page
- `http://127.0.0.1:5000/ui/kb.html`: knowledge base management
- `http://127.0.0.1:5000/ui/model.html`: model management

Feature overview:

- Chat: new sessions, history, SSE streaming answers, deep-think toggle, answer copy, source viewer
- Knowledge base management: add text, upload files, batch import, document filters, chunk edit/delete/rebuild, retrieval parameter tuning, statistics
- Model management: add/edit model configs, enable/disable, set default, connectivity tests, bootstrap presets

**Chat**
![](assets/chat.png)

**Knowledge Base**
![](assets/kb.png)

**Chunk Management**
![](assets/kb-fp.png)

**Model Management**
![](assets/model.png)

## API Documentation

- Online docs: `http://127.0.0.1:5000/api-docs`, `http://127.0.0.1:5000/docs/`
- Offline docs in the repository: `docs/API.html`

**API Docs**
![](assets/api.png)

## Configuration Guide

### Core settings

Common `config.json` fields:

- `search.default_k`: default recall size
- `search.max_search_results`: max number of returned sources
- `search.min_source_similarity`: minimum source similarity threshold
- `db.backend`: history backend, `memory`, `mysql`, or `postgresql`
- `session.backend`: session backend, `memory` or `redis`
- `knowledge_base.chunking.size`: chunk size
- `knowledge_base.chunking.overlap`: chunk overlap
- `knowledge_base.retrieval.candidate_multiplier`: candidate recall multiplier
- `knowledge_base.retrieval.min_candidates`: minimum number of candidates
- `knowledge_base.retrieval.embed_weight`: embedding score weight
- `knowledge_base.retrieval.rerank_weight`: rerank score weight

### Multi-provider model access

`UniversalLLMClient` supports mainstream providers:

| Provider | Example Models | Example `base_url` |
| --- | --- | --- |
| OpenAI | GPT-4, GPT-3.5-Turbo | `https://api.openai.com/v1` |
| DeepSeek | deepseek-chat, deepseek-coder | `https://api.deepseek.com/v1` |
| Qwen | qwen-max, qwen-plus | `https://dashscope.aliyuncs.com/compatible-mode/v1` |
| Doubao | doubao-pro-32k | `https://ark.cn-beijing.volces.com/api/v3` |
| xAI | grok-beta | `https://api.x.ai/v1` |
| Gemini | gemini-pro | `https://generativelanguage.googleapis.com/v1beta` |
| Kimi | moonshot-v1-32k | `https://api.moonshot.cn/v1` |
| LM Studio | Local models | `http://localhost:1234/v1` |

Use `config.multi-provider.example.json` as the recommended reference.

### Supported document formats

| Format | Open-Source Edition | Commercial Edition | Notes |
| --- | --- | --- | --- |
| docx | ✅ | ✅ | Text parsing |
| doc | - | ✅ | Commercial only |
| xls | ✅ | ✅ | Text parsing |
| xlsx | ✅ | ✅ | Text parsing |
| txt | ✅ | ✅ | Text parsing |
| log | ✅ | ✅ | Text parsing |
| images | - | ✅ | OCR in commercial edition |
| PDF | Text-only | ✅ | OCR in commercial edition |
| OFD | Text-only | ✅ | OCR in commercial edition |

### Session ID uniqueness

When `session.backend=redis`, session IDs are generated using Redis atomic increments, which keeps them unique across concurrent and multi-instance deployments. If Redis is unavailable, the service falls back to in-memory generation, which is only unique within the current process.

## Model and Deployment Guidance

### Recommended embedding / reranker sizing

| Size | CPU | Memory | GPU | Typical use case | Notes |
| --- | --- | --- | --- | --- | --- |
| 0.6B | 8 cores | 16GB | Optional (>= 6GB) | Development, testing, medium-small datasets | Default local size |
| 4B | 16 cores | 32GB | Recommended >= 12GB | Single-node production, medium concurrency | GPU recommended |
| 8B | 24 cores | 64GB | Recommended >= 24GB | Higher quality retrieval, higher concurrency | GPU strongly recommended |

Deployment recommendations:

- The larger the model, the more important the GPU becomes.
- Put model caches and vector indexes on SSD storage.
- Use `redis` for `session.backend` in multi-instance deployments.
- Start with warmup disabled during initial deployment, then enable it after the end-to-end path is verified.

### Lightweight example

```json
{
        "knowledge_base": {
                "embedding": {
                        "model": "Qwen/Qwen3-Embedding-0.6B",
                        "device": "auto",
                        "local_files_only": true
                },
                "rerank": {
                        "model": "Qwen/Qwen3-Reranker-0.6B",
                        "use_lm_studio": false
                },
                "retrieval": {
                        "candidate_multiplier": 8,
                        "min_candidates": 30,
                        "embed_weight": 0.35,
                        "rerank_weight": 0.65
                }
        }
}
```

## Operations and Scripts

### Common scripts

| Script | Purpose |
| --- | --- |
| `download_models.py` | Download base models |
| `download_optional_parser_models.py` | Download optional parsing / OCR models |
| `ingest_to_kb_store.py` | Import documents into the knowledge store |
| `encrypt_secret.py` | Encrypt sensitive values in config |
| `migrate_to_sessions_table.py` | Migrate session data structures |
| `add_thinking_summary_column.py` | Add the thinking summary column to data tables |
| `tune_threshold.py` | Tune retrieval thresholds |

### Windows service

Manage the Windows service with `manage_service.ps1`:

```powershell
.\manage_service.ps1 -Command install
.\manage_service.ps1 -Command start
.\manage_service.ps1 -Command status
```

Supported commands: `install`, `uninstall`, `start`, `stop`, `restart`, `status`.

### Linux systemd service

Manage the Linux service with `manage_service.sh`:

```bash
./manage_service.sh install
./manage_service.sh start
./manage_service.sh status
```

### Build and distribution

Use the standard Python build flow:

```bash
python -m build
```

You can also use the bundled scripts:

- Windows PowerShell: `build.ps1`
- Windows CMD: `build.bat`
- Wheel builds: `build_wheel.ps1`, `build_wheel.bat`

Artifacts are generated in `dist/`.

## Data and Security

### Data directories

- Default knowledge store: `kb_store/`
- Model cache directory: `models/hf_cache/`
- Log directory: created automatically at runtime under the project root

### Security recommendations

- Expose the API only inside trusted networks.
- Use a reverse proxy, access controls, and HTTPS in production.
- Protect API keys and database passwords in `config.json` with `encrypt_secret.py`.
- Place Redis and database services in controlled network segments for multi-instance deployments.

## Contribution and License

### Contribution

Issues and pull requests are welcome. Please update documentation when features or configuration behavior change.

### License

MulanPSL-2.0