添加 A_Memorix 插件 v2.0.0（包含运行时与文档）
引入 A_Memorix 插件 v2.0.0：新增大量运行时组件、存储/模式更新、检索能力提升、管理工具、导入/调优工作流以及相关文档。关键新增内容包括：lifecycle_orchestrator、SDKMemoryKernel/运行时初始化器、新的存储层与 metadata_store 变更（SCHEMA_VERSION v8）、检索增强（双路径检索、图关系召回、稀疏 BM25），以及多种工具服务（episode/person_profile/relation/segmentation/tuning/search execution）。同时新增 Web 导入/摘要导入器及大量维护脚本。还更新了插件清单、embedding API 适配器、plugin.py、requirements/pyproject，以及主入口文件，使新插件接入项目。该变更为 2.0.0 版本发布做好准备，实现统一的 SDK Tool 接口并扩展整体运行能力。
This commit is contained in:
DawnARC
718
plugins/A_memorix/CHANGELOG.md
Normal file
718
plugins/A_memorix/CHANGELOG.md
Normal file
@@ -0,0 +1,718 @@
|
||||
# 更新日志 (Changelog)
|
||||
|
||||
## [2.0.0] - 2026-03-18
|
||||
|
||||
本次 `2.0.0` 为架构收敛版本，主线是 **SDK Tool 接口统一**、**管理工具能力补齐**、**元数据 schema 升级到 v8** 与 **文档口径同步到 2.0.0**。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`1.0.1` → `2.0.0`
|
||||
- 元数据 schema：`7` → `8`
|
||||
|
||||
### 🚀 重点能力
|
||||
|
||||
- Tool 接口统一：
|
||||
- `plugin.py` 统一通过 `SDKMemoryKernel` 对外提供 Tool 能力。
|
||||
- 保留基础工具：`search_memory / ingest_summary / ingest_text / get_person_profile / maintain_memory / memory_stats`。
|
||||
- 新增管理工具：`memory_graph_admin / memory_source_admin / memory_episode_admin / memory_profile_admin / memory_runtime_admin / memory_import_admin / memory_tuning_admin / memory_v5_admin / memory_delete_admin`。
|
||||
- 检索与写入治理增强：
|
||||
- 检索/写入链路支持 `respect_filter + user_id/group_id` 的聊天过滤语义。
|
||||
- `maintain_memory` 支持 `freeze` 与 `recycle_bin`，并统一到内核维护流程。
|
||||
- 导入与调优能力收敛：
|
||||
- `memory_import_admin` 提供任务化导入能力（上传、粘贴、扫描、OpenIE、LPMM 转换、时序回填、MaiBot 迁移）。
|
||||
- `memory_tuning_admin` 提供检索调优任务（创建、轮次查看、回滚、apply_best、报告导出）。
|
||||
- V5 与删除运维：
|
||||
- 新增 `memory_v5_admin`（`reinforce/weaken/remember_forever/forget/restore/status`）。
|
||||
- 新增 `memory_delete_admin`（`preview/execute/restore/list/get/purge`），支持操作审计与恢复。
|
||||
|
||||
### 🛠️ 存储与运行时
|
||||
|
||||
- `metadata_store` 升级到 `SCHEMA_VERSION = 8`。
|
||||
- 新增/完善外部引用与运维记录能力（包括 `external_memory_refs`、`memory_v5_operations`、`delete_operations` 相关数据结构）。
|
||||
- `SDKMemoryKernel` 增加统一后台任务编排（自动保存、Episode pending 处理、画像刷新、记忆维护）。
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- `README.md`、`QUICK_START.md`、`CONFIG_REFERENCE.md`、`IMPORT_GUIDE.md` 已切换到 `2.0.0` 口径。
|
||||
- 文档主入口统一为 SDK Tool 工作流，不再以旧版 slash 命令作为主说明路径。
|
||||
|
||||
## [1.0.1] - 2026-03-07
|
||||
|
||||
本次 `1.0.1` 为 `1.0.0` 发布后的热修复版本，主线是 **图谱 WebUI 取数稳定性修复**、**大图过滤性能修复** 与 **真实检索调优链路稳定性修复**。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`1.0.0` → `1.0.1`
|
||||
- 配置版本：`4.1.0`（不变）
|
||||
|
||||
### 🛠️ 代码修复
|
||||
|
||||
- 图谱接口稳定性：
|
||||
- 修复 `/api/graph` 在“磁盘已有图文件但运行时尚未装载入内存”场景下返回空图的问题，接口现在会自动补加载持久化图数据。
|
||||
- 修复问题数据集下 WebUI 打开图谱页时看似“没有任何节点”的现象；根因不是图数据消失，而是后端过滤路径过慢。
|
||||
- 图谱过滤性能：
|
||||
- 优化 `/api/graph?exclude_leaf=true` 的叶子过滤逻辑，改为预计算 hub 邻接关系，不再对每个节点反复做高成本边权查询。
|
||||
- 优化 `GraphStore.get_neighbors()` 并补充入邻居访问能力，避免稠密矩阵展开导致的大图性能退化。
|
||||
- 检索调优稳定性：
|
||||
- 修复真实调优任务在构建运行时配置时深拷贝 `plugin.config`，误复制注入的存储实例并触发 `cannot pickle '_thread.RLock' object` 的问题。
|
||||
- 调优评估改为跳过顶层运行时实例键，仅保留纯配置字段后再附加运行时依赖，真实 WebUI 调优任务可正常启动。
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- 同步更新 `README.md`、`CHANGELOG.md`、`CONFIG_REFERENCE.md` 与版本元数据（`plugin.py`、`__init__.py`、`_manifest.json`）。
|
||||
- README 新增 `v1.0.1` 修复说明，并补充“调优前先做 runtime self-check”的建议。
|
||||
|
||||
## [1.0.0] - 2026-03-06
|
||||
|
||||
本次 `1.0.0` 为主版本升级，主线是 **运行时架构模块化**、**Episode 情景记忆闭环**、**聚合检索与图召回增强**、**离线迁移 / 运行时自检 / 检索调优中心**。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.7.0` → `1.0.0`
|
||||
- 配置版本：`4.1.0`（不变）
|
||||
|
||||
### 🚀 重点能力
|
||||
|
||||
- 运行时重构：
|
||||
- `plugin.py` 大幅瘦身，生命周期、后台任务、请求路由、检索运行时初始化拆分到 `core/runtime/*`。
|
||||
- 配置 schema 抽离到 `core/config/plugin_config_schema.py`，`_manifest.json` 同步扩展新配置项。
|
||||
- 检索与查询增强：
|
||||
- `KnowledgeQueryTool` 拆分为 query mode + orchestrator，新增长 `aggregate` / `episode` 查询模式。
|
||||
- 新增图辅助关系召回、统一 forward/runtime 构建与请求去重桥接。
|
||||
- Episode / 运维能力：
|
||||
- `metadata_store` schema 升级到 `SCHEMA_VERSION = 7`，新增 `episodes` / `episode_paragraphs` / rebuild queue 等结构。
|
||||
- 新增 `release_vnext_migrate.py`、`runtime_self_check.py`、`rebuild_episodes.py` 与 Web 检索调优页 `web/tuning.html`。
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- 版本号同步到 `plugin.py`、`__init__.py`、`_manifest.json`、`README.md` 与 `CONFIG_REFERENCE.md`。
|
||||
- 新增 `RELEASE_SUMMARY_1.0.0.md`
|
||||
|
||||
## [0.7.0] - 2026-03-04
|
||||
|
||||
本次 `0.7.0` 为中版本升级，主线是 **关系向量化闭环（写入 + 状态机 + 回填 + 审计）**、**检索/命令链路增强** 与 **导入任务能力补齐**。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.6.1` → `0.7.0`
|
||||
- 配置版本：`4.1.0`（不变）
|
||||
|
||||
### 🚀 重点能力
|
||||
|
||||
- 关系向量化闭环：
|
||||
- 新增统一关系写入服务 `RelationWriteService`（metadata 先写、向量后写，失败进入状态机而非回滚主数据）。
|
||||
- `relations` 侧补齐 `vector_state/retry_count/last_error/updated_at` 等状态字段，支持 `none/pending/ready/failed` 统一治理。
|
||||
- 插件新增后台回填循环与统计接口，可持续修复关系向量缺失并暴露覆盖率指标。
|
||||
- 检索与命令链路增强：
|
||||
- 检索主链继续收敛到 `search/time` forward 路由，`legacy` 仅保留兼容别名。
|
||||
- relation 查询规格解析收口，结构化查询与语义回退边界更清晰。
|
||||
- `/query stats` 与 tool stats 补充关系向量化统计输出。
|
||||
- 导入与运维增强：
|
||||
- Web Import 新增 `temporal_backfill` 任务入口与编排处理。
|
||||
- 新增一致性审计与离线回填脚本，支持灰度修复历史数据。
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- 同步更新 `README.md`、`CONFIG_REFERENCE.md` 与本日志版本信息。
|
||||
- `README.md` 新增关系向量审计/回填脚本使用说明，并更新 `convert_lpmm.py` 的关系向量重建行为描述。
|
||||
|
||||
## [0.6.1] - 2026-03-03
|
||||
|
||||
本次 `0.6.1` 为热修复小版本，重点修复 WebUI 插件配置接口在 A_Memorix 场景下的 `tomlkit` 节点序列化兼容问题。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.6.0` → `0.6.1`
|
||||
- 配置版本：`4.1.0`（不变）
|
||||
|
||||
### 🛠️ 代码修复
|
||||
|
||||
- 新增运行时补丁 `_patch_webui_a_memorix_routes_for_tomlkit_serialization()`：
|
||||
- 仅包裹 `/api/webui/plugins/config/{plugin_id}` 及其 schema 的 `GET` 路由。
|
||||
- 仅在 `plugin_id == "A_Memorix"` 时，将返回中的 `config/schema` 通过 `to_builtin_data` 原生化。
|
||||
- 保持 `/api/webui/config/*` 全局接口行为不变，避免对其他插件或核心配置路径产生副作用。
|
||||
- 在插件初始化时执行该补丁，确保 WebUI 读取插件配置时返回结构可稳定序列化。
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- 同步更新 `README.md`、`CONFIG_REFERENCE.md` 与本日志中的版本信息及修复说明。
|
||||
|
||||
## [0.6.0] - 2026-03-02
|
||||
|
||||
本次 `0.6.0` 为中版本升级，主线是 **Web Import 导入中心上线与脚本能力对齐**、**失败重试机制升级**、**删除后 manifest 同步** 与 **导入链路稳定性增强**。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.5.1` → `0.6.0`
|
||||
- 配置版本：`4.0.1` → `4.1.0`
|
||||
|
||||
### 🚀 重点能力
|
||||
|
||||
- 新增 Web Import 导入中心（`/import`）：
|
||||
- 上传/粘贴/本地扫描/LPMM OpenIE/LPMM 转换/时序回填/MaiBot 迁移。
|
||||
- 任务/文件/分块三级状态展示，支持取消与失败重试。
|
||||
- 导入文档弹窗读取（远程优先，失败回退本地）。
|
||||
- 失败重试升级为“分块优先 + 文件回退”：
|
||||
- `POST /api/import/tasks/{task_id}/retry_failed` 保持原路径，语义升级。
|
||||
- 支持对 `extracting` 失败分块进行子集重试。
|
||||
- `writing`/JSON 解析失败自动回退为文件级重试。
|
||||
- 删除后 manifest 同步失效：
|
||||
- 覆盖 `/api/source/batch_delete` 与 `/api/source`。
|
||||
- 返回 `manifest_cleanup` 明细，避免误命中去重跳过重导入。
|
||||
|
||||
### 📂 变更文件清单（本次发布）
|
||||
|
||||
新增文件：
|
||||
|
||||
- `core/utils/web_import_manager.py`
|
||||
- `scripts/migrate_maibot_memory.py`
|
||||
- `web/import.html`
|
||||
|
||||
修改文件：
|
||||
|
||||
- `CHANGELOG.md`
|
||||
- `CONFIG_REFERENCE.md`
|
||||
- `IMPORT_GUIDE.md`
|
||||
- `QUICK_START.md`
|
||||
- `README.md`
|
||||
- `__init__.py`
|
||||
- `_manifest.json`
|
||||
- `components/commands/debug_server_command.py`
|
||||
- `core/embedding/api_adapter.py`
|
||||
- `core/storage/graph_store.py`
|
||||
- `core/utils/summary_importer.py`
|
||||
- `plugin.py`
|
||||
- `requirements.txt`
|
||||
- `server.py`
|
||||
- `web/index.html`
|
||||
|
||||
删除文件：
|
||||
|
||||
- 无
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- 同步更新 `README.md`、`QUICK_START.md`、`CONFIG_REFERENCE.md`、`IMPORT_GUIDE.md` 与本日志。
|
||||
- `IMPORT_GUIDE.md` 新增 “Web Import 导入中心” 专区，统一说明能力范围、状态语义与安全边界。
|
||||
|
||||
## [0.5.1] - 2026-02-23
|
||||
|
||||
本次 `0.5.1` 为热修订小版本，重点修复“随主程序启动的后台任务拉起”“空名单过滤语义”以及“知识抽取模型选择”。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.5.0` → `0.5.1`
|
||||
- 配置版本：`4.0.0` → `4.0.1`
|
||||
|
||||
### 🛠️ 代码修复
|
||||
|
||||
- 生命周期接入主程序事件：
|
||||
- 新增 `a_memorix_start_handler`（`ON_START`）调用 `plugin.on_enable()`；
|
||||
- 新增 `a_memorix_stop_handler`（`ON_STOP`）调用 `plugin.on_disable()`；
|
||||
- 解决仅注册插件但未触发生命周期时，定时导入任务不启动的问题。
|
||||
- 聊天过滤空列表策略调整：
|
||||
- `whitelist + []`：全部拒绝；
|
||||
- `blacklist + []`：全部放行。
|
||||
- 知识抽取模型选择逻辑调整（`import_command._select_model`）：
|
||||
- `advanced.extraction_model` 现在支持三种语义：任务名 / 模型名 / `auto`；
|
||||
- `auto` 优先抽取相关任务（`lpmm_entity_extract`、`lpmm_rdf_build` 等），并避免误落到 `embedding`；
|
||||
- 当配置无法识别时输出告警并回退自动选择，提高导入阶段的模型选择可预期性。
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- 同步更新 `README.md`、`CONFIG_REFERENCE.md` 与 `CHANGELOG.md`。
|
||||
- 同步修正文档中的空名单过滤行为描述，保持与当前代码一致。
|
||||
|
||||
## [0.5.0] - 2026-02-15
|
||||
|
||||
本次 `0.5.0` 以提交 `66ddc1b98547df3c866b19a3f5dc96e1c8eb7731` 为核心，主线是“人物画像能力上线 + 工具/命令接入 + 版本与文档同步”。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.4.0` → `0.5.0`
|
||||
- 配置版本：`3.1.0` → `4.0.0`
|
||||
|
||||
### 🚀 人物画像主特性（核心）
|
||||
|
||||
- 新增人物画像服务：`core/utils/person_profile_service.py`
|
||||
- 支持 `person_id/姓名/别名` 解析。
|
||||
- 聚合图关系证据 + 向量证据，生成画像文本并版本化快照。
|
||||
- 支持手工覆盖（override）与 TTL 快照复用。
|
||||
- 存储层新增人物画像相关表与 API：`core/storage/metadata_store.py`
|
||||
- `person_profile_switches`
|
||||
- `person_profile_snapshots`
|
||||
- `person_profile_active_persons`
|
||||
- `person_profile_overrides`
|
||||
- 新增命令：`/person_profile on|off|status`
|
||||
- 文件：`components/commands/person_profile_command.py`
|
||||
- 作用：按 `stream_id + user_id` 控制自动注入开关（opt-in 模式）。
|
||||
- 查询链路接入人物画像：
|
||||
- `knowledge_query_tool` 新增 `query_type=person`，支持 `person_id` 或别名查询。
|
||||
- `/query person` 与 `/query p` 接入画像查询输出。
|
||||
- 插件生命周期接入画像刷新任务：
|
||||
- 启动/停止统一管理 `person_profile_refresh` 后台任务。
|
||||
- 按活跃窗口自动刷新画像快照。
|
||||
|
||||
### 🛠️ 版本与 schema 同步
|
||||
|
||||
- `plugin.py`：`plugin_version` 更新为 `0.5.0`。
|
||||
- `plugin.py`：`plugin.config_version` 默认值更新为 `4.0.0`。
|
||||
- `config.toml`：`config_version` 基线同步为 `4.0.0`（本地配置文件）。
|
||||
- `__init__.py`：`__version__` 更新为 `0.5.0`。
|
||||
- `_manifest.json`：`version` 更新为 `0.5.0`，`manifest_version` 保持 `1` 。
|
||||
- `manifest_utils.py`：仓库内已兼容更高 manifest 版本；但插件发布默认保持 `manifest_version=1` 。
|
||||
|
||||
### 📚 文档同步
|
||||
|
||||
- 更新 `README.md`、`CONFIG_REFERENCE.md`、`QUICK_START.md`、`USAGE_ARCHITECTURE.md`。
|
||||
- 0.5.0 文档主线改为“人物画像能力 + 版本升级 + 检索链路补充说明”。
|
||||
|
||||
## [0.4.0] - 2026-02-13
|
||||
|
||||
本次 `0.4.0` 版本整合了时序检索增强与后续检索链路增强、稳定性修复和文档同步。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.3.3` → `0.4.0`
|
||||
- 配置版本：`3.0.0` → `3.1.0`
|
||||
|
||||
### 🚀 新增
|
||||
|
||||
- 新增 `core/retrieval/sparse_bm25.py`
|
||||
- `SparseBM25Config` / `SparseBM25Index`
|
||||
- FTS5 + BM25 稀疏检索
|
||||
- 支持 `jieba/mixed/char_2gram` 分词与懒加载
|
||||
- 支持 ngram 倒排回退与可选 LIKE 兜底
|
||||
- `DualPathRetriever` 新增 sparse/fusion 配置注入：
|
||||
- embedding 不可用时自动 sparse 回退；
|
||||
- `hybrid` 模式支持向量路 + sparse 路并行候选；
|
||||
- 新增 `FusionConfig` 与 `weighted_rrf` 融合。
|
||||
- `MetadataStore` 新增 FTS/倒排能力：
|
||||
- `paragraphs_fts`、`relations_fts` schema 与回填；
|
||||
- `paragraph_ngrams` 倒排索引与回填；
|
||||
- `fts_search_bm25` / `fts_search_relations_bm25` / `ngram_search_paragraphs`。
|
||||
|
||||
### 🛠️ 组件链路同步
|
||||
|
||||
- `plugin.py`
|
||||
- 新增 `[retrieval.sparse]`、`[retrieval.fusion]` 默认配置；
|
||||
- 初始化并向组件注入 `sparse_index`；
|
||||
- `on_disable` 支持按配置卸载 sparse 连接并释放缓存。
|
||||
- `knowledge_search_action.py` / `query_command.py` / `knowledge_query_tool.py`
|
||||
- 统一接入 sparse/fusion 配置；
|
||||
- 统一注入 `sparse_index`；
|
||||
- `stats` 输出新增 sparse 状态观测。
|
||||
- `requirements.txt`
|
||||
- 新增 `jieba>=0.42.1`（未安装时自动回退 char n-gram）。
|
||||
|
||||
### 🧯 修复与行为调整
|
||||
|
||||
- 修复 `retrieval.ppr_concurrency_limit` 不生效问题：
|
||||
- `DualPathRetriever` 使用配置值初始化 `_ppr_semaphore`，不再被固定值覆盖。
|
||||
- 修复 `char_2gram` 召回失效场景：
|
||||
- FTS miss 时增加 `_fallback_substring_search`，优先 ngram 倒排回退，按配置可选 LIKE 兜底。
|
||||
- 提升可观测性与兼容性：
|
||||
- `get_statistics()` 对向量规模字段兼容读取 `size -> num_vectors -> 0`，避免属性缺失导致异常。
|
||||
- `/query stats` 与 `knowledge_query` 输出包含 sparse 状态（enabled/loaded/tokenizer/doc_count）。
|
||||
|
||||
### 📚 文档
|
||||
|
||||
- `README.md`
|
||||
- 新增检索增强说明、稀疏行为说明、时序回填脚本入口。
|
||||
- `CONFIG_REFERENCE.md`
|
||||
- 补齐 sparse/fusion 参数与触发规则、回退链路、融合实现细节。
|
||||
|
||||
### ⏱️ 时序检索与导入增强
|
||||
|
||||
#### 时序检索能力（分钟级）
|
||||
|
||||
- 新增统一时序查询入口：
|
||||
- `/query time`（别名 `/query t`）
|
||||
- `knowledge_query(query_type=time)`
|
||||
- `knowledge_search(query_type=time|hybrid)`
|
||||
- 查询时间参数统一支持：
|
||||
- `YYYY/MM/DD`
|
||||
- `YYYY/MM/DD HH:mm`
|
||||
- 日期参数自动展开边界：
|
||||
- `from/time_from` -> `00:00`
|
||||
- `to/time_to` -> `23:59`
|
||||
- 查询结果统一回传 `metadata.time_meta`，包含命中时间窗口与命中依据（事件时间或 `created_at` 回退）。
|
||||
|
||||
#### 存储与检索链路
|
||||
|
||||
- 段落存储层支持时序字段：
|
||||
- `event_time`
|
||||
- `event_time_start`
|
||||
- `event_time_end`
|
||||
- `time_granularity`
|
||||
- `time_confidence`
|
||||
- 时序命中采用区间相交逻辑，并遵循“双层时间语义”：
|
||||
- 优先 `event_time/event_time_range`
|
||||
- 缺失时回退 `created_at`（可配置关闭）
|
||||
- 检索排序规则保持：语义优先，时间次排序（新到旧）。
|
||||
- `process_knowledge.py` 新增 `--chat-log` 参数：
|
||||
- 启用后强制使用 `narrative` 策略；
|
||||
- 使用 LLM 对聊天文本进行语义时间抽取（支持相对时间转绝对时间），写入 `event_time/event_time_start/event_time_end`。
|
||||
- 新增 `--chat-reference-time`，用于指定相对时间语义解析的参考时间点。
|
||||
|
||||
#### Schema 与文档同步
|
||||
|
||||
- `_manifest.json` 同步补齐 `retrieval.temporal` 配置 schema。
|
||||
- 配置 schema 版本升级：`config_version` 从 `3.0.0` 提升到 `3.1.0`（`plugin.py` / `config.toml` / 配置文档同步）。
|
||||
- 更新 `README.md`、`CONFIG_REFERENCE.md`、`IMPORT_GUIDE.md`，补充时序检索入口、参数格式与导入时间字段说明。
|
||||
|
||||
## [0.3.3] - 2026-02-11
|
||||
|
||||
本次更新为 **语言一致性补丁版本**，重点收敛知识抽取时的语言漂移问题，要求输出严格贴合原文语言，不做翻译改写。
|
||||
|
||||
### 🛠️ 关键修复
|
||||
|
||||
#### 抽取语言约束
|
||||
|
||||
- `BaseStrategy`:
|
||||
- 移除按 `zh/en/mixed` 分支的语言类型判定逻辑；
|
||||
- 统一为单一约束：抽取值保持原文语言、保留原始术语、禁止翻译。
|
||||
- `NarrativeStrategy` / `FactualStrategy`:
|
||||
- 抽取提示词统一接入上述语言约束；
|
||||
- 明确要求 JSON 键名固定、抽取值遵循原文语言表达。
|
||||
|
||||
#### 导入链路一致性
|
||||
|
||||
- `ImportCommand` 的 LLM 抽取提示词同步强化“优先原文语言、不要翻译”要求，避免脚本与指令导入行为不一致。
|
||||
|
||||
#### 测试与文档
|
||||
|
||||
- 更新 `test_strategies.py`，将语言判定测试调整为统一语言约束测试，并验证提示词中包含禁止翻译约束。
|
||||
- 同步更新注释与文档描述，确保实现与说明一致。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.3.2` → `0.3.3`
|
||||
|
||||
## [0.3.2] - 2026-02-11
|
||||
|
||||
本次更新为 **V5 稳定性与兼容性修复版本**，在保持原有业务设计（强化→衰减→冷冻→修剪→回收）的前提下，修复关键链路断裂与误判问题。
|
||||
|
||||
### 🛠️ 关键修复
|
||||
|
||||
#### V5 记忆系统契约与链路
|
||||
|
||||
- `MetadataStore`:
|
||||
- 统一 `mark_relations_inactive(hashes, inactive_since=None)` 调用契约，兼容不同调用方；
|
||||
- 补充 `has_table(table_name)`；
|
||||
- 增加 `restore_relation(hash)` 兼容别名，修复服务层恢复调用断裂；
|
||||
- 修正 `get_entity_gc_candidates` 对孤立节点参数的处理（支持节点名映射到实体 hash）。
|
||||
- `GraphStore`:
|
||||
- 清理 `deactivate_edges` 重复定义并统一返回冻结数量，保证上层日志与断言稳定。
|
||||
- `server.py`:
|
||||
- 修复 `/api/memory/restore` relation 恢复链路；
|
||||
- 清理不可达分支并统一异常路径；
|
||||
- 回收站查询在表检测场景下不再出现错误退空。
|
||||
|
||||
#### 命令与模型选择
|
||||
|
||||
- `/memory` 命令修复 hash 长度判定：以 64 位 `sha256` 为标准，同时兼容历史 32 位输入。
|
||||
- 总结模型选择修复：
|
||||
- 解决 `summarization.model_name = auto` 误命中 `embedding` 问题；
|
||||
- 支持数组与选择器语法（`task:model` / task / model）；
|
||||
- 兼容逗号分隔字符串写法（如 `"utils:model1","utils:model2",replyer`）。
|
||||
|
||||
#### 生命周期与脚本稳定性
|
||||
|
||||
- `plugin.py` 修复后台任务生命周期管理：
|
||||
- 增加 `_scheduled_import_task` / `_auto_save_task` / `_memory_maintenance_task` 句柄；
|
||||
- 避免重复启动；
|
||||
- 插件停用时统一 cancel + await 收敛。
|
||||
- `process_knowledge.py` 修复 tenacity 重试日志级别类型错误（`"WARNING"` → `logging.WARNING`），避免 `KeyError: 'WARNING'`。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.3.1` → `0.3.2`
|
||||
|
||||
## [0.3.1] - 2026-02-07
|
||||
|
||||
本次更新为 **稳定性补丁版本**，主要修复脚本导入链路、删除安全性与 LPMM 转换一致性问题。
|
||||
|
||||
### 🛠️ 关键修复
|
||||
|
||||
#### 新增功能
|
||||
|
||||
- 新增 `scripts/convert_lpmm.py`：
|
||||
- 支持将 LPMM 的 `parquet + graph` 数据直接转换为 A_Memorix 存储结构；
|
||||
- 提供 LPMM ID 到 A_Memorix ID 的映射能力，用于图节点/边重写；
|
||||
- 当前实现优先保证检索一致性，关系向量采用安全策略（不直接导入）。
|
||||
|
||||
#### 导入链路
|
||||
|
||||
- 修复 `import_lpmm_json.py` 依赖的 `AutoImporter.import_json_data` 公共入口缺失/不稳定问题，确保外部脚本可稳定调用 JSON 直导入流程。
|
||||
|
||||
#### 删除安全
|
||||
|
||||
- 修复按来源删除时“同一 `(subject, object)` 存在多关系”场景下的误删风险：
|
||||
- `MetadataStore.delete_paragraph_atomic` 新增 `relation_prune_ops`；
|
||||
- 仅在无兄弟关系时才回退删除整条边。
|
||||
- `delete_knowledge.py` 新增保守孤儿实体清理（仅对本次候选实体执行，且需同时满足无段落引用、无关系引用、图无邻居）。
|
||||
- `delete_knowledge.py` 改为读取向量元数据中的真实维度，避免 `dimension=1` 写回污染。
|
||||
|
||||
#### LPMM 转换修复
|
||||
|
||||
- 修复 `convert_lpmm.py` 中向量 ID 与 `MetadataStore` 哈希不一致导致的检索反查失败问题。
|
||||
- 为避免脏召回，转换阶段暂时跳过 `relation.parquet` 的直接向量导入（待关系元数据一一映射能力完善后再恢复）。
|
||||
|
||||
### 🔖 版本信息
|
||||
|
||||
- 插件版本：`0.3.0` → `0.3.1`
|
||||
|
||||
## [0.3.0] - 2026-01-30
|
||||
|
||||
本次更新引入了 **V5 动态记忆系统**，实现了符合生物学特性的记忆衰减、强化与全声明周期管理，并提供了配套的指令与工具。
|
||||
|
||||
### 🧠 记忆系统 (V5)
|
||||
|
||||
#### 核心机制
|
||||
|
||||
- **记忆衰减 (Decay)**: 引入"遗忘曲线"，随时间推移自动降低图谱连接权重。
|
||||
- **访问强化 (Reinforcement)**: "越用越强"，每次检索命中都会刷新记忆活跃度并增强权重。
|
||||
- **生命周期 (Lifecycle)**:
|
||||
- **活跃 (Active)**: 正常参与计算与检索。
|
||||
- **冷冻 (Inactive)**: 权重过低被冻结，不再参与 PPR 计算，但保留语义映射 (Mapping)。
|
||||
- **修剪 (Prune)**: 过期且无保护的冷冻记忆将被移入回收站。
|
||||
- **多重保护**: 支持 **永久锁定 (Pin)** 与 **限时保护 (TTL)**，防止关键记忆被误删。
|
||||
|
||||
#### GraphStore
|
||||
|
||||
- **多关系映射**: 实现 `(u,v) -> Set[Hash]` 映射，确保同一通道下的多重语义关系互不干扰。
|
||||
- **原子化操作**: 新增 `decay`, `deactivate_edges` (软删), `prune_relation_hashes` (硬删) 等原子操作。
|
||||
|
||||
### 🛠️ 指令与工具
|
||||
|
||||
#### Memory Command (`/memory`)
|
||||
|
||||
新增全套记忆维护指令：
|
||||
|
||||
- `/memory status`: 查看记忆系统健康状态（活跃/冷冻/回收站计数）。
|
||||
- `/memory protect <query> [hours]`: 保护记忆。不填时间为永久锁定(Pin)，填时间为临时保护(TTL)。
|
||||
- `/memory reinforce <query>`: 手动强化记忆（绕过冷却时间）。
|
||||
- `/memory restore <hash>`: 从回收站恢复误删记忆（仅当节点存在时重建连接）。
|
||||
|
||||
#### MemoryModifierTool
|
||||
|
||||
- **LLM 能力增强**: 更新工具逻辑，支持 LLM 自主触发 `reinforce`, `weaken`, `remember_forever`, `forget` 操作，并自动映射到 V5 底层逻辑。
|
||||
|
||||
### ⚙️ 配置 (`config.toml`)
|
||||
|
||||
新增 `[memory]` 配置节：
|
||||
|
||||
- `half_life_hours`: 记忆半衰期 (默认 24h)。
|
||||
- `enable_auto_reinforce`: 是否开启检索自动强化。
|
||||
- `prune_threshold`: 冷冻/修剪阈值 (默认 0.1)。
|
||||
|
||||
### 💻 WebUI (v1.4)
|
||||
|
||||
实现了与 V5 记忆系统深度集成的全生命周期管理界面：
|
||||
|
||||
- **可视化增强**:
|
||||
- **冷冻状态**: 非活跃记忆以 **虚线 + 灰色 (Slate-300)** 显示。
|
||||
- **保护状态**: 被 Pin 或保护的记忆带有 **金色 (Amber) 光晕**。
|
||||
- **交互升级**:
|
||||
- **记忆回收站**: 新增 Dock 入口与专用面板，支持浏览删除记录并一键恢复。
|
||||
- **快捷操作**: 边属性面板新增 **强化 (Reinforce)**、**保护 (Protect/Pin)**、**冷冻 (Freeze)** 按钮。
|
||||
- **实时反馈**: 操作后自动刷新图谱布局与样式。
|
||||
|
||||
---
|
||||
|
||||
## [0.2.3] - 2026-01-30
|
||||
|
||||
本次更新主要集中在 **WebUI 交互体验优化** 与 **文档/配置的规范化**。
|
||||
|
||||
### 🎨 WebUI (v1.3)
|
||||
|
||||
#### 加载与同步体验升级
|
||||
|
||||
- **沉浸式加载**: 全新设计的加载遮罩，采用磨砂玻璃背景 (`backdrop-filter`) 与呼吸灯文字动效，提升视觉质感。
|
||||
- **精准状态反馈**: 优化加载逻辑，明确区分“网络同步”与“拓扑计算”阶段，解决数据加载时的闪烁问题。
|
||||
- **新手引导**: 在加载界面新增基础操作提示，降低新用户上手门槛。
|
||||
|
||||
#### 全功能帮助面板
|
||||
|
||||
- **操作指南重构**: 全面翻新“操作指南”面板，新增 Dock 栏功能详解、编辑管理操作及视图配置说明。
|
||||
|
||||
### 🛠️ 工程与规范
|
||||
|
||||
#### plugin.py
|
||||
|
||||
- **配置描述补全**: 修复了 `config_section_descriptions` 中缺失 `summarization`, `schedule`, `filter` 节导致的问题。
|
||||
- **版本号**: `0.2.2` → `0.2.3`
|
||||
|
||||
### ⚙️ 核心与服务
|
||||
|
||||
#### Core
|
||||
|
||||
- **量化逻辑修正**: 修正了 `_scalar_quantize_int8` 函数，确保向量值正确映射到 `[-128, 127]` 区间，提高量化精度。
|
||||
|
||||
#### Server
|
||||
|
||||
- **缓存一致性**: 在执行删除节点/边等修改操作后，显式清除 `_relation_cache`，确保前端获取的关系数据实时更新。
|
||||
|
||||
### 🤖 脚本与数据处理
|
||||
|
||||
#### process_knowledge.py
|
||||
|
||||
- **策略模式重构**: 引入了 `Strategy-Aware` 架构，支持通过 `Narrative` (叙事), `Factual` (事实), `Quote` (引用) 三种策略差异化处理文本(准确说是确认实装)（默认采用 Narrative模式）。
|
||||
- **智能分块纠错**: 新增“分块拯救” (`Chunk Rescue`) 机制，可在长叙事文本中自动识别并提取内嵌的歌词或诗句。
|
||||
|
||||
#### import_lpmm_json.py
|
||||
|
||||
- **LPMM 迁移工具**: 增加了对 LPMM OpenIE JSON 格式的完整支持，能够自动计算 Hash 并迁移实体/关系数据，确保与 A_Memorix 存储格式兼容。
|
||||
|
||||
#### Project
|
||||
|
||||
- **构建清理**: 优化 `.gitignore` 规则
|
||||
|
||||
---
|
||||
|
||||
## [0.2.2] - 2026-01-27
|
||||
|
||||
本次更新专注于提高 **网络请求的鲁棒性**，特别是针对嵌入服务的调用。
|
||||
|
||||
### 🛠️ 稳定性与工程改进
|
||||
|
||||
#### EmbeddingAPI
|
||||
|
||||
- **可配置重试机制**: 新增 `[embedding.retry]` 配置项，允许自定义最大重试次数和等待时间。默认重试次数从 3 次增加到 10 次，以更好应对网络波动。
|
||||
- **配置项**:
|
||||
- `max_attempts`: 最大重试次数 (默认: 10)
|
||||
- `max_wait_seconds`: 最大等待时间 (默认: 30s)
|
||||
- `min_wait_seconds`: 最小等待时间 (默认: 2s)
|
||||
|
||||
#### plugin.py
|
||||
|
||||
- **版本号**: `0.2.1` → `0.2.2`
|
||||
|
||||
---
|
||||
|
||||
## [0.2.1] - 2026-01-26
|
||||
|
||||
本次更新重点在于 **可视化交互的全方位重构** 以及 **底层鲁棒性的进一步增强**。
|
||||
|
||||
### 🎨 可视化与交互重构
|
||||
|
||||
#### WebUI (Glassmorphism)
|
||||
|
||||
- **全新视觉设计**: 采用深色磨砂玻璃 (Glassmorphism) 风格，配合动态渐变背景。
|
||||
- **Dock 菜单栏**: 底部新增 macOS 风格 Dock 栏，聚合所有常用功能。
|
||||
- **显著性视图 (Saliency View)**: 基于 **PageRank** 算法的“信息密度”滑块，支持以此过滤叶子节点，仅展示核心骨干或全量细节。
|
||||
- **功能面板**:
|
||||
- **❓ 操作指南**: 内置交互说明与特性介绍。
|
||||
- **🔍 悬浮搜索**: 支持按拼音/ID 实时过滤节点。
|
||||
- **📂 记忆溯源**: 支持按源文件批量查看和删除记忆数据。
|
||||
- **📖 内容字典**: 列表化展示所有实体与关系，支持排序与筛选。
|
||||
|
||||
### 🛠️ 稳定性与工程改进
|
||||
|
||||
#### EmbeddingAPI
|
||||
|
||||
- **鲁棒性增强**: 引入 `tenacity` 实现指数退避重试机制。
|
||||
- **错误处理**: 失败时返回 `NaN` 向量而非零向量，允许上层逻辑安全跳过。
|
||||
|
||||
#### MetadataStore
|
||||
|
||||
- **自动修复**: 自动检测并修复 `vector_index` 列错位（文件名误存）的历史数据问题。
|
||||
- **数据统计**: 新增 `get_all_sources` 接口支持来源统计。
|
||||
|
||||
#### 脚本与工具
|
||||
|
||||
- **用户体验**: 引入 `rich` 库优化终端输出进度条与状态显示。
|
||||
- **接口开放**: `process_knowledge.py` 新增 `import_json_data` 供外部调用。
|
||||
- **LPMM 迁移**: 新增 `import_lpmm_json.py`，支持导入符合 LPMM 规范的 OpenIE JSON 数据。
|
||||
|
||||
#### plugin.py
|
||||
|
||||
- **版本号**: `0.2.0` → `0.2.1`
|
||||
|
||||
---
|
||||
|
||||
## [0.2.0] - 2026-01-22
|
||||
|
||||
> [!CAUTION]
|
||||
> **不完全兼容变更**：v0.2.0 版本重构了底层存储架构。由于数据结构的重大调整，**旧版本的导入数据无法在新版本中完全无损兼容**。
|
||||
> 虽然部分组件支持自动迁移，但为确保数据一致性和检索质量，**强烈建议在升级后重新使用 `process_knowledge.py` 导入原始数据**。
|
||||
|
||||
本次更新为**重大版本升级**，包含向量存储架构重写、检索逻辑强化及多项稳定性改进。
|
||||
|
||||
### 🚀 核心架构重写
|
||||
|
||||
#### VectorStore: SQ8 量化 + Append-Only 存储
|
||||
|
||||
- **全新存储格式**: 从 `.npy` 迁移至 `vectors.bin`（float16 增量追加）和 `vectors_ids.bin`，大幅减少内存占用。
|
||||
- **原生 SQ8 量化**: 使用 Faiss `IndexScalarQuantizer(QT_8bit)`，替代手动 int8 量化逻辑。
|
||||
- **L2 Normalization 强制化**: 所有向量在存储和检索时统一执行 L2 归一化，确保 Inner Product 等价于 Cosine 相似度。
|
||||
- **Fallback 索引机制**: 新增 `IndexFlatIP` 回退索引，在 SQ8 训练完成前提供检索能力，避免冷启动无结果问题。
|
||||
- **Reservoir Sampling 训练采样**: 使用蓄水池采样收集训练数据（上限 10k），保证小数据集和流式导入场景下的训练样本多样性。
|
||||
- **线程安全**: 新增 `threading.RLock` 保护并发读写操作。
|
||||
- **自动迁移**: 支持从旧版 `.npy` 格式自动迁移至新 `.bin` 格式。
|
||||
|
||||
### ✨ 检索功能增强
|
||||
|
||||
#### KnowledgeQueryTool: 智能回退与多跳路径搜索
|
||||
|
||||
- **Smart Fallback (智能回退)**: 当向量检索置信度低于阈值 (默认 0.6) 时，自动尝试提取查询中的实体进行多跳路径搜索（`_path_search`），增强对间接关系的召回能力。
|
||||
- **结果去重 (`_deduplicate_results`)**: 新增基于内容相似度的安全去重逻辑，防止冗余结果污染 LLM 上下文，同时确保至少保留一条结果。
|
||||
- **语义关系检索 (`_semantic_search_relation`)**: 支持自然语言查询关系（无需 `S|P|O` 格式），内部使用 `REL_ONLY` 策略进行向量检索。
|
||||
- **路径搜索 (`_path_search`)**: 新增 `GraphStore.find_paths` 调用，支持查找两个实体间的间接连接路径（最大深度 3，最多 5 条路径）。
|
||||
- **Clean Output**: LLM 上下文中不再包含原始相似度分数，避免模型偏见。
|
||||
|
||||
#### DualPathRetriever: 并发控制与调试模式
|
||||
|
||||
- **PPR 并发限制 (`ppr_concurrency_limit`)**: 新增 Semaphore 控制 PageRank 计算并发数，防止 CPU 峰值过载。
|
||||
- **Debug 模式**: 新增 `debug` 配置项，启用时打印检索结果原文到日志。
|
||||
- **Entity-Pivot 关系检索**: 优化 `_retrieve_relations_only` 策略，通过检索实体后扩展其关联关系，替代直接检索关系向量。
|
||||
|
||||
### ⚙️ 配置与 Schema 扩展
|
||||
|
||||
#### plugin.py
|
||||
|
||||
- **版本号**: `0.1.3` → `0.2.0`
|
||||
- **默认配置版本**: `config_version` 默认值更新为 `2.0.0`
|
||||
- **新增配置项**:
|
||||
- `retrieval.relation_semantic_fallback` (bool): 是否启用关系查询的语义回退。
|
||||
- `retrieval.relation_fallback_min_score` (float): 语义回退的最小相似度阈值。
|
||||
- **相对路径支持**: `storage.data_dir` 现在支持相对路径（相对于插件目录），默认值改为 `./data`。
|
||||
- **全局实例获取**: 新增 `A_MemorixPlugin.get_global_instance()` 静态方法，供组件可靠获取插件实例。
|
||||
|
||||
#### config.toml / \_manifest.json
|
||||
|
||||
- **新增 `ppr_concurrency_limit`**: 控制 PPR 算法并发数。
|
||||
- **新增训练阈值配置**: `embedding.min_train_threshold` 控制触发 SQ8 训练的最小样本数。
|
||||
|
||||
### 🛠️ 稳定性与工程改进
|
||||
|
||||
#### GraphStore
|
||||
|
||||
- **`find_paths` 方法**: 新增多跳路径查找功能，支持 BFS 搜索指定深度内的实体间路径。
|
||||
- **`find_node` 方法**: 新增大小写不敏感的节点查找。
|
||||
|
||||
#### MetadataStore
|
||||
|
||||
- **Schema 迁移**: 自动添加缺失的 `is_permanent`, `last_accessed`, `access_count` 字段。
|
||||
|
||||
#### 脚本与工具
|
||||
|
||||
- **新增脚本**:
|
||||
- `scripts/diagnose_relations_source.py`: 诊断关系溯源问题。
|
||||
- `scripts/verify_search_robustness.py`: 验证检索鲁棒性。
|
||||
- `scripts/run_stress_test.py`, `stress_test_data.py`: 压力测试套件。
|
||||
- `scripts/migrate_canonicalization.py`, `migrate_paragraph_relations.py`: 数据迁移工具。
|
||||
- **目录整理**: 将大量旧版测试脚本移动至 `deprecated/` 目录。
|
||||
|
||||
### 🗑️ 移除与废弃
|
||||
|
||||
- 废弃 `vectors.npy` 存储格式（自动迁移至 `.bin`）。
|
||||
|
||||
---
|
||||
|
||||
## [0.1.3] - 上一个稳定版本
|
||||
|
||||
- 初始发布，包含基础双路检索功能。
|
||||
- 手动 Int8 向量量化。
|
||||
- 基于 `.npy` 的向量存储。
|
||||
292
plugins/A_memorix/CONFIG_REFERENCE.md
Normal file
292
plugins/A_memorix/CONFIG_REFERENCE.md
Normal file
@@ -0,0 +1,292 @@
|
||||
# A_Memorix 配置参考 (v2.0.0)
|
||||
|
||||
本文档对应当前仓库代码（`__version__ = 2.0.0`、`SCHEMA_VERSION = 8`）。
|
||||
|
||||
说明：
|
||||
|
||||
- 本文只覆盖 **当前运行时实际读取** 的配置键。
|
||||
- 旧版 `/query`、`/memory`、`/visualize` 命令体系相关配置，不再作为主路径说明。
|
||||
- 未配置的键会回退到代码默认值。
|
||||
|
||||
## 最小可用配置
|
||||
|
||||
```toml
|
||||
[plugin]
|
||||
enabled = true
|
||||
|
||||
[storage]
|
||||
data_dir = "./data"
|
||||
|
||||
[embedding]
|
||||
model_name = "auto"
|
||||
dimension = 1024
|
||||
batch_size = 32
|
||||
max_concurrent = 5
|
||||
enable_cache = false
|
||||
quantization_type = "int8"
|
||||
|
||||
[retrieval]
|
||||
top_k_paragraphs = 20
|
||||
top_k_relations = 10
|
||||
top_k_final = 10
|
||||
alpha = 0.5
|
||||
enable_ppr = true
|
||||
ppr_alpha = 0.85
|
||||
ppr_timeout_seconds = 1.5
|
||||
ppr_concurrency_limit = 4
|
||||
enable_parallel = true
|
||||
|
||||
[retrieval.sparse]
|
||||
enabled = true
|
||||
|
||||
[episode]
|
||||
enabled = true
|
||||
generation_enabled = true
|
||||
pending_batch_size = 20
|
||||
pending_max_retry = 3
|
||||
|
||||
[person_profile]
|
||||
enabled = true
|
||||
|
||||
[memory]
|
||||
enabled = true
|
||||
half_life_hours = 24.0
|
||||
prune_threshold = 0.1
|
||||
|
||||
[advanced]
|
||||
enable_auto_save = true
|
||||
auto_save_interval_minutes = 5
|
||||
|
||||
[web.import]
|
||||
enabled = true
|
||||
|
||||
[web.tuning]
|
||||
enabled = true
|
||||
```
|
||||
|
||||
## 1. 存储与嵌入
|
||||
|
||||
### `storage`
|
||||
|
||||
- `storage.data_dir` (默认 `./data`)
|
||||
: 数据目录。相对路径按插件目录解析。
|
||||
|
||||
### `embedding`
|
||||
|
||||
- `embedding.model_name` (默认 `auto`)
|
||||
: embedding 模型选择。
|
||||
- `embedding.dimension` (默认 `1024`)
|
||||
: 期望维度（运行时会做真实探测并校验）。
|
||||
- `embedding.batch_size` (默认 `32`)
|
||||
- `embedding.max_concurrent` (默认 `5`)
|
||||
- `embedding.enable_cache` (默认 `false`)
|
||||
- `embedding.retry` (默认 `{}`)
|
||||
: embedding 调用重试策略。
|
||||
- `embedding.quantization_type`
|
||||
: 当前主路径仅建议 `int8`。
|
||||
|
||||
## 2. 检索
|
||||
|
||||
### `retrieval` 主键
|
||||
|
||||
- `retrieval.top_k_paragraphs` (默认 `20`)
|
||||
- `retrieval.top_k_relations` (默认 `10`)
|
||||
- `retrieval.top_k_final` (默认 `10`)
|
||||
- `retrieval.alpha` (默认 `0.5`)
|
||||
- `retrieval.enable_ppr` (默认 `true`)
|
||||
- `retrieval.ppr_alpha` (默认 `0.85`)
|
||||
- `retrieval.ppr_timeout_seconds` (默认 `1.5`)
|
||||
- `retrieval.ppr_concurrency_limit` (默认 `4`)
|
||||
- `retrieval.enable_parallel` (默认 `true`)
|
||||
- `retrieval.relation_vectorization.enabled` (默认 `false`)
|
||||
|
||||
### `retrieval.sparse` (`SparseBM25Config`)
|
||||
|
||||
常用键（默认值）：
|
||||
|
||||
- `enabled = true`
|
||||
- `backend = "fts5"`
|
||||
- `lazy_load = true`
|
||||
- `mode = "auto"` (`auto`/`fallback_only`/`hybrid`)
|
||||
- `tokenizer_mode = "jieba"` (`jieba`/`mixed`/`char_2gram`)
|
||||
- `char_ngram_n = 2`
|
||||
- `candidate_k = 80`
|
||||
- `relation_candidate_k = 60`
|
||||
- `enable_ngram_fallback_index = true`
|
||||
- `enable_relation_sparse_fallback = true`
|
||||
|
||||
### `retrieval.fusion` (`FusionConfig`)
|
||||
|
||||
- `method` (默认 `weighted_rrf`)
|
||||
- `rrf_k` (默认 `60`)
|
||||
- `vector_weight` (默认 `0.7`)
|
||||
- `bm25_weight` (默认 `0.3`)
|
||||
- `normalize_score` (默认 `true`)
|
||||
- `normalize_method` (默认 `minmax`)
|
||||
|
||||
### `retrieval.search.relation_intent` (`RelationIntentConfig`)
|
||||
|
||||
- `enabled` (默认 `true`)
|
||||
- `alpha_override` (默认 `0.35`)
|
||||
- `relation_candidate_multiplier` (默认 `4`)
|
||||
- `preserve_top_relations` (默认 `3`)
|
||||
- `force_relation_sparse` (默认 `true`)
|
||||
- `pair_predicate_rerank_enabled` (默认 `true`)
|
||||
- `pair_predicate_limit` (默认 `3`)
|
||||
|
||||
### `retrieval.search.graph_recall` (`GraphRelationRecallConfig`)
|
||||
|
||||
- `enabled` (默认 `true`)
|
||||
- `candidate_k` (默认 `24`)
|
||||
- `max_hop` (默认 `1`)
|
||||
- `allow_two_hop_pair` (默认 `true`)
|
||||
- `max_paths` (默认 `4`)
|
||||
|
||||
### `retrieval.aggregate`
|
||||
|
||||
- `retrieval.aggregate.rrf_k`
|
||||
- `retrieval.aggregate.weights`
|
||||
|
||||
用于聚合检索阶段混合策略；未配置时走代码默认行为。
|
||||
|
||||
## 3. 阈值过滤
|
||||
|
||||
### `threshold` (`ThresholdConfig`)
|
||||
|
||||
- `threshold.min_threshold` (默认 `0.3`)
|
||||
- `threshold.max_threshold` (默认 `0.95`)
|
||||
- `threshold.percentile` (默认 `75.0`)
|
||||
- `threshold.std_multiplier` (默认 `1.5`)
|
||||
- `threshold.min_results` (默认 `3`)
|
||||
- `threshold.enable_auto_adjust` (默认 `true`)
|
||||
|
||||
## 4. 聊天过滤
|
||||
|
||||
### `filter`
|
||||
|
||||
用于 `respect_filter=true` 场景（检索和写入都支持）。
|
||||
|
||||
```toml
|
||||
[filter]
|
||||
enabled = true
|
||||
mode = "blacklist" # blacklist / whitelist
|
||||
chats = ["group:123", "user:456", "stream:abc"]
|
||||
```
|
||||
|
||||
规则：
|
||||
|
||||
- `blacklist`：命中列表即拒绝
|
||||
- `whitelist`：仅列表内允许
|
||||
- 列表为空时：
|
||||
- `blacklist` => 全允许
|
||||
- `whitelist` => 全拒绝
|
||||
|
||||
## 5. Episode
|
||||
|
||||
### `episode`
|
||||
|
||||
- `episode.enabled` (默认 `true`)
|
||||
- `episode.generation_enabled` (默认 `true`)
|
||||
- `episode.pending_batch_size` (默认 `20`，部分路径默认 `12`)
|
||||
- `episode.pending_max_retry` (默认 `3`)
|
||||
- `episode.max_paragraphs_per_call` (默认 `20`)
|
||||
- `episode.max_chars_per_call` (默认 `6000`)
|
||||
- `episode.source_time_window_hours` (默认 `24`)
|
||||
- `episode.segmentation_model` (默认 `auto`)
|
||||
|
||||
## 6. 人物画像
|
||||
|
||||
### `person_profile`
|
||||
|
||||
- `person_profile.enabled` (默认 `true`)
|
||||
- `person_profile.refresh_interval_minutes` (默认 `30`)
|
||||
- `person_profile.active_window_hours` (默认 `72`)
|
||||
- `person_profile.max_refresh_per_cycle` (默认 `50`)
|
||||
- `person_profile.top_k_evidence` (默认 `12`)
|
||||
|
||||
## 7. 记忆演化与回收
|
||||
|
||||
### `memory`
|
||||
|
||||
- `memory.enabled` (默认 `true`)
|
||||
- `memory.half_life_hours` (默认 `24.0`)
|
||||
- `memory.base_decay_interval_hours` (默认 `1.0`)
|
||||
- `memory.prune_threshold` (默认 `0.1`)
|
||||
- `memory.freeze_duration_hours` (默认 `24.0`)
|
||||
|
||||
### `memory.orphan`
|
||||
|
||||
- `enable_soft_delete` (默认 `true`)
|
||||
- `entity_retention_days` (默认 `7.0`)
|
||||
- `paragraph_retention_days` (默认 `7.0`)
|
||||
- `sweep_grace_hours` (默认 `24.0`)
|
||||
|
||||
## 8. 高级运行时
|
||||
|
||||
### `advanced`
|
||||
|
||||
- `advanced.enable_auto_save` (默认 `true`)
|
||||
- `advanced.auto_save_interval_minutes` (默认 `5`)
|
||||
- `advanced.debug` (默认 `false`)
|
||||
- `advanced.extraction_model` (默认 `auto`)
|
||||
|
||||
## 9. 导入中心 (`web.import`)
|
||||
|
||||
### 开关与限流
|
||||
|
||||
- `web.import.enabled` (默认 `true`)
|
||||
- `web.import.max_queue_size` (默认 `20`)
|
||||
- `web.import.max_files_per_task` (默认 `200`)
|
||||
- `web.import.max_file_size_mb` (默认 `20`)
|
||||
- `web.import.max_paste_chars` (默认 `200000`)
|
||||
- `web.import.default_file_concurrency` (默认 `2`)
|
||||
- `web.import.default_chunk_concurrency` (默认 `4`)
|
||||
- `web.import.max_file_concurrency` (默认 `6`)
|
||||
- `web.import.max_chunk_concurrency` (默认 `12`)
|
||||
- `web.import.poll_interval_ms` (默认 `1000`)
|
||||
|
||||
### 重试与路径
|
||||
|
||||
- `web.import.llm_retry.max_attempts` (默认 `4`)
|
||||
- `web.import.llm_retry.min_wait_seconds` (默认 `3`)
|
||||
- `web.import.llm_retry.max_wait_seconds` (默认 `40`)
|
||||
- `web.import.llm_retry.backoff_multiplier` (默认 `3`)
|
||||
- `web.import.path_aliases` (默认内置 `raw/lpmm/plugin_data`)
|
||||
|
||||
### 转换阶段
|
||||
|
||||
- `web.import.convert.enable_staging_switch` (默认 `true`)
|
||||
- `web.import.convert.keep_backup_count` (默认 `3`)
|
||||
|
||||
## 10. 调优中心 (`web.tuning`)
|
||||
|
||||
- `web.tuning.enabled` (默认 `true`)
|
||||
- `web.tuning.max_queue_size` (默认 `8`)
|
||||
- `web.tuning.poll_interval_ms` (默认 `1200`)
|
||||
- `web.tuning.eval_query_timeout_seconds` (默认 `10.0`)
|
||||
- `web.tuning.default_intensity` (默认 `standard`)
|
||||
- `web.tuning.default_objective` (默认 `precision_priority`)
|
||||
- `web.tuning.default_top_k_eval` (默认 `20`)
|
||||
- `web.tuning.default_sample_size` (默认 `24`)
|
||||
- `web.tuning.llm_retry.max_attempts` (默认 `3`)
|
||||
- `web.tuning.llm_retry.min_wait_seconds` (默认 `2`)
|
||||
- `web.tuning.llm_retry.max_wait_seconds` (默认 `20`)
|
||||
- `web.tuning.llm_retry.backoff_multiplier` (默认 `2`)
|
||||
|
||||
## 11. 兼容性提示
|
||||
|
||||
- 若你从 `1.x` 升级，请优先运行：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/release_vnext_migrate.py preflight --strict
|
||||
python plugins/A_memorix/scripts/release_vnext_migrate.py migrate --verify-after
|
||||
python plugins/A_memorix/scripts/release_vnext_migrate.py verify --strict
|
||||
```
|
||||
|
||||
- 启动前再执行：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/runtime_self_check.py --json
|
||||
```
|
||||
|
||||
以避免 embedding 维度与向量库不匹配导致运行时异常。
|
||||
335
plugins/A_memorix/IMPORT_GUIDE.md
Normal file
335
plugins/A_memorix/IMPORT_GUIDE.md
Normal file
@@ -0,0 +1,335 @@
|
||||
# A_Memorix 导入指南 (v2.0.0)
|
||||
|
||||
本文档对应当前 `2.0.0` 代码路径，覆盖两类导入方式：
|
||||
|
||||
1. 脚本导入（离线批处理）
|
||||
2. `memory_import_admin` 任务导入（在线任务化）
|
||||
|
||||
## 1. 导入前检查
|
||||
|
||||
建议先执行：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/runtime_self_check.py --json
|
||||
```
|
||||
|
||||
再确认：
|
||||
|
||||
- `storage.data_dir` 路径可写
|
||||
- embedding 配置可用
|
||||
- 若是升级项目，先完成迁移脚本
|
||||
|
||||
## 2. 方式 A：脚本导入（推荐起步）
|
||||
|
||||
## 2.1 原始文本导入
|
||||
|
||||
将 `.txt` 文件放入：
|
||||
|
||||
```text
|
||||
plugins/A_memorix/data/raw/
|
||||
```
|
||||
|
||||
执行：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/process_knowledge.py
|
||||
```
|
||||
|
||||
常用参数：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/process_knowledge.py --force
|
||||
python plugins/A_memorix/scripts/process_knowledge.py --chat-log
|
||||
python plugins/A_memorix/scripts/process_knowledge.py --chat-log --chat-reference-time "2026/02/12 10:30"
|
||||
```
|
||||
|
||||
## 2.2 OpenIE JSON 导入
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/import_lpmm_json.py <json文件或目录>
|
||||
```
|
||||
|
||||
## 2.3 LPMM 数据转换
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/convert_lpmm.py -i <lpmm数据目录> -o plugins/A_memorix/data
|
||||
```
|
||||
|
||||
## 2.4 历史数据迁移
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/migrate_chat_history.py --help
|
||||
python plugins/A_memorix/scripts/migrate_maibot_memory.py --help
|
||||
python plugins/A_memorix/scripts/migrate_person_memory_points.py --help
|
||||
```
|
||||
|
||||
## 2.5 导入后修复与重建
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/backfill_temporal_metadata.py --dry-run
|
||||
python plugins/A_memorix/scripts/backfill_relation_vectors.py --limit 1000
|
||||
python plugins/A_memorix/scripts/rebuild_episodes.py --all --wait
|
||||
python plugins/A_memorix/scripts/audit_vector_consistency.py --json
|
||||
```
|
||||
|
||||
## 3. 方式 B：`memory_import_admin` 任务导入
|
||||
|
||||
`memory_import_admin` 是在线任务化导入入口，适合宿主侧面板或自动化管道。
|
||||
|
||||
### 3.1 常用 action
|
||||
|
||||
- `settings` / `get_settings` / `get_guide`
|
||||
- `path_aliases` / `get_path_aliases`
|
||||
- `resolve_path`
|
||||
- `create_upload`
|
||||
- `create_paste`
|
||||
- `create_raw_scan`
|
||||
- `create_lpmm_openie`
|
||||
- `create_lpmm_convert`
|
||||
- `create_temporal_backfill`
|
||||
- `create_maibot_migration`
|
||||
- `list`
|
||||
- `get`
|
||||
- `chunks` / `get_chunks`
|
||||
- `cancel`
|
||||
- `retry_failed`
|
||||
|
||||
### 3.2 调用示例
|
||||
|
||||
查看运行时设置：
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "memory_import_admin",
|
||||
"arguments": {
|
||||
"action": "settings"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
创建粘贴导入任务：
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "memory_import_admin",
|
||||
"arguments": {
|
||||
"action": "create_paste",
|
||||
"content": "今天完成了检索调优回归。",
|
||||
"input_mode": "plain_text",
|
||||
"source": "manual:worklog"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
查询任务列表：
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "memory_import_admin",
|
||||
"arguments": {
|
||||
"action": "list",
|
||||
"limit": 20
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
查看任务详情：
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "memory_import_admin",
|
||||
"arguments": {
|
||||
"action": "get",
|
||||
"task_id": "<task_id>",
|
||||
"include_chunks": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
重试失败任务：
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "memory_import_admin",
|
||||
"arguments": {
|
||||
"action": "retry_failed",
|
||||
"task_id": "<task_id>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 4. 直接写入 Tool（非任务化）
|
||||
|
||||
若你不需要任务编排，也可以直接调用：
|
||||
|
||||
- `ingest_summary`
|
||||
- `ingest_text`
|
||||
|
||||
示例：
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "ingest_text",
|
||||
"arguments": {
|
||||
"external_id": "note:2026-03-18:001",
|
||||
"source_type": "note",
|
||||
"text": "新的召回阈值方案已通过评审",
|
||||
"chat_id": "group:dev",
|
||||
"tags": ["worklog", "review"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`external_id` 建议全局唯一，用于幂等去重。
|
||||
|
||||
## 5. 时间字段建议
|
||||
|
||||
可用时间字段（按常见优先级）：
|
||||
|
||||
- `timestamp`
|
||||
- `time_start`
|
||||
- `time_end`
|
||||
|
||||
建议：
|
||||
|
||||
- 事件类记录优先写 `time_start/time_end`
|
||||
- 仅有单点时间时写 `timestamp`
|
||||
- 历史数据可先导入，再用 `backfill_temporal_metadata.py` 回填
|
||||
|
||||
## 6. source_type 建议
|
||||
|
||||
常见值：
|
||||
|
||||
- `chat_summary`
|
||||
- `note`
|
||||
- `person_fact`
|
||||
- `lpmm_openie`
|
||||
- `migration`
|
||||
|
||||
建议保持稳定枚举，便于后续按来源治理与重建 Episode。
|
||||
|
||||
## 7. 导入完成后的验证
|
||||
|
||||
建议执行以下顺序：
|
||||
|
||||
1. `memory_stats` 看总量是否增长
|
||||
2. `search_memory`（`mode=search`/`aggregate`）抽检召回
|
||||
3. `memory_episode_admin` 的 `status`/`query` 检查 Episode 生成
|
||||
4. `memory_runtime_admin` 的 `self_check` 再确认运行时健康
|
||||
|
||||
## 8. 常见问题
|
||||
|
||||
### Q1: 导入任务创建成功但无写入
|
||||
|
||||
- 检查聊天过滤配置 `filter`（若 `respect_filter=true` 可能被过滤）
|
||||
- 检查任务详情中的失败原因与分块状态
|
||||
|
||||
### Q2: 任务反复失败
|
||||
|
||||
- 检查 embedding 与 LLM 可用性
|
||||
- 降低并发（`web.import.default_*_concurrency`）
|
||||
- 调整重试参数（`web.import.llm_retry.*`）
|
||||
|
||||
### Q3: 导入后检索效果差
|
||||
|
||||
- 先做 `runtime_self_check`
|
||||
- 检查 `retrieval.sparse` 是否启用
|
||||
- 使用 `memory_tuning_admin` 创建调优任务做参数回归
|
||||
|
||||
## 9. 相关文档
|
||||
|
||||
- [QUICK_START.md](QUICK_START.md)
|
||||
- [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md)
|
||||
- [README.md](README.md)
|
||||
- [CHANGELOG.md](CHANGELOG.md)
|
||||
|
||||
## 10. 附录：策略模式参考
|
||||
|
||||
A_Memorix 导入链路仍然遵循策略模式（Strategy-Aware）。`process_knowledge.py` 会自动识别文本类型，也支持手动指定。
|
||||
|
||||
| 策略类型 | 适用场景 | 核心逻辑 | 自动识别特征 |
|
||||
| :-- | :-- | :-- | :-- |
|
||||
| `Narrative` (叙事) | 小说、同人文、剧本、长篇故事 | 按场景/章节切分，使用滑动窗口；提取事件与角色关系 | `#`、`Chapter`、`***` 等章节标记 |
|
||||
| `Factual` (事实) | 设定集、百科、说明书 | 按语义块切分，保留列表/定义结构；提取 SPO 三元组 | 列表符号、`术语: 解释` |
|
||||
| `Quote` (引用) | 歌词、诗歌、名言、台词 | 按双换行切分，原文即知识，不做概括 | 平均行长短、行数多 |
|
||||
|
||||
## 11. 附录：参考用例（已恢复）
|
||||
|
||||
以下样例可直接复制保存为文件测试，或作为 LLM few-shot 示例。
|
||||
|
||||
### 11.1 叙事文本 (`plugins/A_memorix/data/raw/story_demo.txt`)
|
||||
|
||||
```text
|
||||
# 第一章：星之子
|
||||
|
||||
艾瑞克在废墟中醒来，手中的星盘发出微弱的蓝光。他并不记得自己是如何来到这里的，只依稀记得莉莉丝最后的警告：“千万不要回头。”
|
||||
|
||||
远处传来了机械守卫的轰鸣声。艾瑞克迅速收起星盘，向着北方的废弃都市奔去。他知道，那里有反抗军唯一的据点。
|
||||
|
||||
***
|
||||
|
||||
# 第二章：重逢
|
||||
|
||||
在反抗军的地下掩体中，艾瑞克见到了那个熟悉的身影。莉莉丝正站在全息地图前，眉头紧锁。
|
||||
|
||||
“你还是来了。”莉莉丝没有回头，但声音中带着一丝颤抖。
|
||||
“我必须来，”艾瑞克握紧了拳头，“为了解开星盘的秘密，也为了你。”
|
||||
```
|
||||
|
||||
### 11.2 事实文本 (`plugins/A_memorix/data/raw/rules_demo.txt`)
|
||||
|
||||
```text
|
||||
# 联邦安全协议 v2.0
|
||||
|
||||
## 核心法则
|
||||
1. **第一公理**：任何人工智能不得伤害人类个体，或因不作为而使人类个体受到伤害。
|
||||
2. **第二公理**：人工智能必须服从人类的命令，除非该命令与第一公理冲突。
|
||||
|
||||
## 术语定义
|
||||
- **以太网络**：覆盖全联邦的高速量子通讯网络。
|
||||
- **黑色障壁**：用于隔离高危 AI 的物理防火墙设施。
|
||||
```
|
||||
|
||||
### 11.3 引用文本 (`plugins/A_memorix/data/raw/poem_demo.txt`)
|
||||
|
||||
```text
|
||||
致橡树
|
||||
|
||||
我如果爱你——
|
||||
绝不像攀援的凌霄花，
|
||||
借你的高枝炫耀自己；
|
||||
|
||||
我如果爱你——
|
||||
绝不学痴情的鸟儿，
|
||||
为绿荫重复单调的歌曲；
|
||||
|
||||
也不止像泉源，
|
||||
常年送来清凉的慰籍；
|
||||
也不止像险峰，
|
||||
增加你的高度，衬托你的威仪。
|
||||
```
|
||||
|
||||
### 11.4 LPMM JSON (`lpmm_data-openie.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"docs": [
|
||||
{
|
||||
"passage": "艾瑞克手中的星盘是打开遗迹的唯一钥匙。",
|
||||
"extracted_triples": [
|
||||
["星盘", "是", "唯一的钥匙"],
|
||||
["星盘", "属于", "艾瑞克"],
|
||||
["钥匙", "用于", "遗迹"]
|
||||
],
|
||||
"extracted_entities": ["星盘", "艾瑞克", "遗迹", "钥匙"]
|
||||
},
|
||||
{
|
||||
"passage": "莉莉丝是反抗军的现任领袖。",
|
||||
"extracted_triples": [
|
||||
["莉莉丝", "是", "领袖"],
|
||||
["领袖", "所属", "反抗军"]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
661
plugins/A_memorix/LICENSE
Normal file
661
plugins/A_memorix/LICENSE
Normal file
@@ -0,0 +1,661 @@
|
||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
||||
Version 3, 19 November 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU Affero General Public License is a free, copyleft license for
|
||||
software and other kinds of works, specifically designed to ensure
|
||||
cooperation with the community in the case of network server software.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
our General Public Licenses are intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
Developers that use our General Public Licenses protect your rights
|
||||
with two steps: (1) assert copyright on the software, and (2) offer
|
||||
you this License which gives you legal permission to copy, distribute
|
||||
and/or modify the software.
|
||||
|
||||
A secondary benefit of defending all users' freedom is that
|
||||
improvements made in alternate versions of the program, if they
|
||||
receive widespread use, become available for other developers to
|
||||
incorporate. Many developers of free software are heartened and
|
||||
encouraged by the resulting cooperation. However, in the case of
|
||||
software used on network servers, this result may fail to come about.
|
||||
The GNU General Public License permits making a modified version and
|
||||
letting the public access it on a server without ever releasing its
|
||||
source code to the public.
|
||||
|
||||
The GNU Affero General Public License is designed specifically to
|
||||
ensure that, in such cases, the modified source code becomes available
|
||||
to the community. It requires the operator of a network server to
|
||||
provide the source code of the modified version running there to the
|
||||
users of that server. Therefore, public use of a modified version, on
|
||||
a publicly accessible server, gives the public access to the source
|
||||
code of the modified version.
|
||||
|
||||
An older license, called the Affero General Public License and
|
||||
published by Affero, was designed to accomplish similar goals. This is
|
||||
a different license, not a version of the Affero GPL, but Affero has
|
||||
released a new version of the Affero GPL which permits relicensing under
|
||||
this license.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, if you modify the
|
||||
Program, your modified version must prominently offer all users
|
||||
interacting with it remotely through a computer network (if your version
|
||||
supports such interaction) an opportunity to receive the Corresponding
|
||||
Source of your version by providing access to the Corresponding Source
|
||||
from a network server at no charge, through some standard or customary
|
||||
means of facilitating copying of software. This Corresponding Source
|
||||
shall include the Corresponding Source for any work covered by version 3
|
||||
of the GNU General Public License that is incorporated pursuant to the
|
||||
following paragraph.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the work with which it is combined will remain governed by version
|
||||
3 of the GNU General Public License.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU Affero General Public License from time to time. Such new versions
|
||||
will be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU Affero General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU Affero General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU Affero General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published
|
||||
by the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If your software can interact with users remotely through a computer
|
||||
network, you should also make sure that it provides a way for users to
|
||||
get its source. For example, if your program is a web application, its
|
||||
interface could display a "Source" link that leads users to an archive
|
||||
of the code. There are many ways you could offer source, and different
|
||||
solutions will be better for different programs; see section 13 for the
|
||||
specific requirements.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
22
plugins/A_memorix/LICENSE-MAIBOT-GPL.md
Normal file
22
plugins/A_memorix/LICENSE-MAIBOT-GPL.md
Normal file
@@ -0,0 +1,22 @@
|
||||
Special GPL License Grant for MaiBot
|
||||
|
||||
Licensor
|
||||
- A_Dawn
|
||||
|
||||
Effective date
|
||||
- 2026-03-18
|
||||
|
||||
Default license
|
||||
- This repository is licensed under AGPL-3.0 by default (see `LICENSE`).
|
||||
|
||||
Additional grant for MaiBot
|
||||
- The copyright holder(s) of this repository grant an additional, non-exclusive permission to
|
||||
the project at `https://github.com/Mai-with-u/MaiBot` (including its maintainers and contributors)
|
||||
to use, modify, and redistribute code from this repository under GPL-3.0.
|
||||
|
||||
Scope
|
||||
- This additional GPL grant is intended for use in the MaiBot project context.
|
||||
- For all other uses not covered by the grant above, AGPL-3.0 remains the applicable license.
|
||||
|
||||
No warranty
|
||||
- This grant is provided without warranty, consistent with AGPL-3.0 and GPL-3.0.
|
||||
210
plugins/A_memorix/QUICK_START.md
Normal file
210
plugins/A_memorix/QUICK_START.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# A_Memorix Quick Start (v2.0.0)
|
||||
|
||||
本文档面向当前 `2.0.0` 架构（SDK Tool 接口）。
|
||||
|
||||
## 0. 版本与接口变更
|
||||
|
||||
- 当前插件版本：`2.0.0`
|
||||
- 接口形态：`memory_provider` + Tool 调用
|
||||
- 旧版 slash 命令（如 `/query`、`/memory`、`/visualize`）不再作为本分支主文档入口
|
||||
|
||||
## 1. 环境准备
|
||||
|
||||
- Python 3.10+
|
||||
- 与 MaiBot 主程序相同的运行环境
|
||||
- 可访问你配置的 embedding 服务
|
||||
|
||||
安装依赖：
|
||||
|
||||
```bash
|
||||
pip install -r plugins/A_memorix/requirements.txt --upgrade
|
||||
```
|
||||
|
||||
如果当前目录就是插件目录，也可以：
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt --upgrade
|
||||
```
|
||||
|
||||
## 2. 启用插件
|
||||
|
||||
在主程序插件配置中启用 `A_Memorix`。
|
||||
|
||||
若你使用 `plugins/A_memorix/config.toml` 方式，最小示例：
|
||||
|
||||
```toml
|
||||
[plugin]
|
||||
enabled = true
|
||||
|
||||
[storage]
|
||||
data_dir = "./data"
|
||||
|
||||
[embedding]
|
||||
model_name = "auto"
|
||||
dimension = 1024
|
||||
batch_size = 32
|
||||
max_concurrent = 5
|
||||
quantization_type = "int8"
|
||||
```
|
||||
|
||||
## 3. 运行时自检（强烈建议）
|
||||
|
||||
先确认 embedding 实际输出维度与向量库兼容：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/runtime_self_check.py --json
|
||||
```
|
||||
|
||||
如果结果 `ok=false`，先修复 embedding 配置或向量库，再继续导入。
|
||||
|
||||
## 4. 导入数据
|
||||
|
||||
### 4.1 文本批量导入
|
||||
|
||||
把文本放到：
|
||||
|
||||
```text
|
||||
plugins/A_memorix/data/raw/
|
||||
```
|
||||
|
||||
执行：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/process_knowledge.py
|
||||
```
|
||||
|
||||
常用参数：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/process_knowledge.py --force
|
||||
python plugins/A_memorix/scripts/process_knowledge.py --chat-log
|
||||
python plugins/A_memorix/scripts/process_knowledge.py --chat-log --chat-reference-time "2026/02/12 10:30"
|
||||
```
|
||||
|
||||
### 4.2 其他导入脚本
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/import_lpmm_json.py <json文件或目录>
|
||||
python plugins/A_memorix/scripts/convert_lpmm.py -i <lpmm数据目录> -o plugins/A_memorix/data
|
||||
python plugins/A_memorix/scripts/migrate_chat_history.py --help
|
||||
python plugins/A_memorix/scripts/migrate_maibot_memory.py --help
|
||||
python plugins/A_memorix/scripts/migrate_person_memory_points.py --help
|
||||
```
|
||||
|
||||
## 5. 核心 Tool 调用
|
||||
|
||||
### 5.1 检索
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "search_memory",
|
||||
"arguments": {
|
||||
"query": "项目复盘",
|
||||
"mode": "aggregate",
|
||||
"limit": 5,
|
||||
"chat_id": "group:dev"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`mode` 支持：`search/time/hybrid/episode/aggregate`
|
||||
|
||||
### 5.2 写入摘要
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "ingest_summary",
|
||||
"arguments": {
|
||||
"external_id": "chat_summary:group-dev:2026-03-18",
|
||||
"chat_id": "group:dev",
|
||||
"text": "今天完成了检索调优评审"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 写入普通记忆
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "ingest_text",
|
||||
"arguments": {
|
||||
"external_id": "note:2026-03-18:001",
|
||||
"source_type": "note",
|
||||
"text": "模型切换后召回质量更稳定",
|
||||
"chat_id": "group:dev",
|
||||
"tags": ["worklog"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.4 画像与维护
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "get_person_profile",
|
||||
"arguments": {
|
||||
"person_id": "Alice",
|
||||
"limit": 8
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "maintain_memory",
|
||||
"arguments": {
|
||||
"action": "protect",
|
||||
"target": "模型切换后召回质量更稳定",
|
||||
"hours": 24
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "memory_stats",
|
||||
"arguments": {}
|
||||
}
|
||||
```
|
||||
|
||||
## 6. 管理 Tool（进阶）
|
||||
|
||||
`2.0.0` 提供完整管理工具：
|
||||
|
||||
- `memory_graph_admin`
|
||||
- `memory_source_admin`
|
||||
- `memory_episode_admin`
|
||||
- `memory_profile_admin`
|
||||
- `memory_runtime_admin`
|
||||
- `memory_import_admin`
|
||||
- `memory_tuning_admin`
|
||||
- `memory_v5_admin`
|
||||
- `memory_delete_admin`
|
||||
|
||||
可先用 `action=list` / `action=status` 等只读动作验证链路。
|
||||
|
||||
## 7. 常见问题
|
||||
|
||||
### Q1: 检索为空
|
||||
|
||||
1. 先看 `memory_stats` 是否有段落/关系
|
||||
2. 检查 `chat_id`、`person_id` 过滤条件是否过严
|
||||
3. 运行 `runtime_self_check.py --json` 确认 embedding 维度无误
|
||||
|
||||
### Q2: 启动时报向量维度不一致
|
||||
|
||||
- 原因：现有向量库维度与当前 embedding 输出不一致
|
||||
- 处理：恢复原配置或重建向量数据后再启动
|
||||
|
||||
### Q3: Web 页面打不开
|
||||
|
||||
本分支不内置独立 `server.py`。
|
||||
|
||||
- `web/index.html`、`web/import.html`、`web/tuning.html` 由宿主侧路由/API 集成暴露
|
||||
- 请检查宿主是否已映射对应静态页与 `/api/*` 接口
|
||||
|
||||
## 8. 下一步
|
||||
|
||||
- 配置细节见 [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md)
|
||||
- 导入细节见 [IMPORT_GUIDE.md](IMPORT_GUIDE.md)
|
||||
- 版本历史见 [CHANGELOG.md](CHANGELOG.md)
|
||||
216
plugins/A_memorix/README.md
Normal file
216
plugins/A_memorix/README.md
Normal file
@@ -0,0 +1,216 @@
|
||||
# A_Memorix
|
||||
|
||||
**长期记忆与认知增强插件** (v2.0.0)
|
||||
|
||||
> 消えていかない感覚 , まだまだ足りてないみたい !
|
||||
|
||||
A_Memorix 是面向 MaiBot SDK 的 `memory_provider` 插件。
|
||||
它把文本、关系、Episode、人物画像和检索调优统一在一套运行时里，适合长期运行的 Agent 记忆场景。
|
||||
|
||||
## 快速导航
|
||||
|
||||
- [快速入门](QUICK_START.md)
|
||||
- [配置参数详解](CONFIG_REFERENCE.md)
|
||||
- [导入指南与最佳实践](IMPORT_GUIDE.md)
|
||||
- [更新日志](CHANGELOG.md)
|
||||
|
||||
## 2.0.0 版本定位
|
||||
|
||||
`v2.0.0` 是一次架构收敛版本，当前分支以 **SDK Tool 接口** 为主：
|
||||
|
||||
- 旧 `components/commands/*`、`components/tools/*` 与 `server.py` 已移除。
|
||||
- 统一入口为 [`plugin.py`](plugin.py) + [`core/runtime/sdk_memory_kernel.py`](core/runtime/sdk_memory_kernel.py)。
|
||||
- 元数据 schema 为 `v8`，新增外部引用与运维操作记录（如 `external_memory_refs`、`memory_v5_operations`、`delete_operations`）。
|
||||
|
||||
如果你还在使用旧版 slash 命令（如 `/query`、`/memory`、`/visualize`），需要按本文的 Tool 接口迁移。
|
||||
|
||||
## 核心能力
|
||||
|
||||
- 双路检索：向量 + 图谱关系联合召回，支持 `search/time/hybrid/episode/aggregate`。
|
||||
- 写入与去重：`external_id` 幂等、段落/关系联合写入、Episode pending 队列处理。
|
||||
- Episode 能力：按 source 重建、状态查询、批处理 pending。
|
||||
- 人物画像：自动快照 + 手动 override。
|
||||
- 管理能力：图谱、来源、Episode、画像、导入、调优、V5 运维、删除恢复全套管理工具。
|
||||
|
||||
## Tool 接口 (v2.0.0)
|
||||
|
||||
### 基础工具
|
||||
|
||||
| Tool | 说明 | 关键参数 |
|
||||
| --- | --- | --- |
|
||||
| `search_memory` | 检索长期记忆 | `query` `mode` `limit` `chat_id` `person_id` `time_start` `time_end` |
|
||||
| `ingest_summary` | 写入聊天摘要 | `external_id` `chat_id` `text` |
|
||||
| `ingest_text` | 写入普通文本记忆 | `external_id` `source_type` `text` |
|
||||
| `get_person_profile` | 获取人物画像 | `person_id` `chat_id` `limit` |
|
||||
| `maintain_memory` | 维护关系状态 | `action=reinforce/protect/restore/freeze/recycle_bin` |
|
||||
| `memory_stats` | 获取统计信息 | 无 |
|
||||
|
||||
### 管理工具
|
||||
|
||||
| Tool | 常用 action |
|
||||
| --- | --- |
|
||||
| `memory_graph_admin` | `get_graph/create_node/delete_node/rename_node/create_edge/delete_edge/update_edge_weight` |
|
||||
| `memory_source_admin` | `list/delete/batch_delete` |
|
||||
| `memory_episode_admin` | `query/list/get/status/rebuild/process_pending` |
|
||||
| `memory_profile_admin` | `query/list/set_override/delete_override` |
|
||||
| `memory_runtime_admin` | `save/get_config/self_check/refresh_self_check/set_auto_save` |
|
||||
| `memory_import_admin` | `settings/get_guide/create_upload/create_paste/create_raw_scan/create_lpmm_openie/create_lpmm_convert/create_temporal_backfill/create_maibot_migration/list/get/chunks/cancel/retry_failed` |
|
||||
| `memory_tuning_admin` | `settings/get_profile/apply_profile/rollback_profile/export_profile/create_task/list_tasks/get_task/get_rounds/cancel/apply_best/get_report` |
|
||||
| `memory_v5_admin` | `status/recycle_bin/restore/reinforce/weaken/remember_forever/forget` |
|
||||
| `memory_delete_admin` | `preview/execute/restore/get_operation/list_operations/purge` |
|
||||
|
||||
## 调用示例
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "search_memory",
|
||||
"arguments": {
|
||||
"query": "项目复盘",
|
||||
"mode": "aggregate",
|
||||
"limit": 5,
|
||||
"chat_id": "group:dev"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "ingest_text",
|
||||
"arguments": {
|
||||
"external_id": "note:2026-03-18:001",
|
||||
"source_type": "note",
|
||||
"text": "今天完成了检索调优评审",
|
||||
"chat_id": "group:dev",
|
||||
"tags": ["worklog"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "maintain_memory",
|
||||
"arguments": {
|
||||
"action": "protect",
|
||||
"target": "完成了 检索调优评审",
|
||||
"hours": 72
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 1. 安装依赖
|
||||
|
||||
在 MaiBot 主程序使用的同一个 Python 环境中执行：
|
||||
|
||||
```bash
|
||||
pip install -r plugins/A_memorix/requirements.txt --upgrade
|
||||
```
|
||||
|
||||
如果当前目录已经是插件目录，也可以执行：
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt --upgrade
|
||||
```
|
||||
|
||||
### 2. 启用插件
|
||||
|
||||
在 `config.toml` 中启用插件（路径取决于你的宿主部署）：
|
||||
|
||||
```toml
|
||||
[plugin]
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### 3. 先做运行时自检
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/runtime_self_check.py --json
|
||||
```
|
||||
|
||||
### 4. 导入文本并验证统计
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/process_knowledge.py
|
||||
```
|
||||
|
||||
然后调用 `memory_stats` 或 `search_memory` 检查是否有数据。
|
||||
|
||||
## Web 页面说明
|
||||
|
||||
仓库内保留了 Web 静态页面：
|
||||
|
||||
- `web/index.html`（图谱与记忆管理）
|
||||
- `web/import.html`（导入中心）
|
||||
- `web/tuning.html`（检索调优）
|
||||
|
||||
当前分支不再内置独立 `server.py`，页面路由与 API 暴露由宿主侧集成负责。
|
||||
|
||||
## 常用脚本
|
||||
|
||||
| 脚本 | 用途 |
|
||||
| --- | --- |
|
||||
| `process_knowledge.py` | 批量导入原始文本（策略感知） |
|
||||
| `import_lpmm_json.py` | 导入 OpenIE JSON |
|
||||
| `convert_lpmm.py` | 转换 LPMM 数据 |
|
||||
| `migrate_chat_history.py` | 迁移 chat_history |
|
||||
| `migrate_maibot_memory.py` | 迁移 MaiBot 历史记忆 |
|
||||
| `migrate_person_memory_points.py` | 迁移 person memory points |
|
||||
| `backfill_temporal_metadata.py` | 回填时间元数据 |
|
||||
| `audit_vector_consistency.py` | 审计向量一致性 |
|
||||
| `backfill_relation_vectors.py` | 回填关系向量 |
|
||||
| `rebuild_episodes.py` | 按 source 重建 Episode |
|
||||
| `release_vnext_migrate.py` | 升级预检/迁移/校验 |
|
||||
| `runtime_self_check.py` | 真实 embedding 运行时自检 |
|
||||
|
||||
## 配置重点
|
||||
|
||||
完整配置见 [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md)。
|
||||
|
||||
高频配置项：
|
||||
|
||||
- `storage.data_dir`
|
||||
- `embedding.dimension`
|
||||
- `embedding.quantization_type`（当前仅支持 `int8`）
|
||||
- `retrieval.*`
|
||||
- `retrieval.sparse.*`
|
||||
- `episode.*`
|
||||
- `person_profile.*`
|
||||
- `memory.*`
|
||||
- `web.import.*`
|
||||
- `web.tuning.*`
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SQLite 无 FTS5
|
||||
|
||||
如果环境中的 SQLite 未启用 `FTS5`，可关闭稀疏检索：
|
||||
|
||||
```toml
|
||||
[retrieval.sparse]
|
||||
enabled = false
|
||||
```
|
||||
|
||||
### 向量维度不一致
|
||||
|
||||
若日志提示当前 embedding 输出维度与既有向量库不一致，请先执行：
|
||||
|
||||
```bash
|
||||
python plugins/A_memorix/scripts/runtime_self_check.py --json
|
||||
```
|
||||
|
||||
必要时重建向量或调整 embedding 配置后再启动插件。
|
||||
|
||||
## 许可证
|
||||
|
||||
默认许可证为 [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0)（见 `LICENSE`）。
|
||||
|
||||
针对 `Mai-with-u/MaiBot` 项目的 GPL 额外授权见 `LICENSE-MAIBOT-GPL.md`。
|
||||
|
||||
除上述额外授权外，其他使用场景仍适用 AGPL-3.0。
|
||||
|
||||
## 贡献说明
|
||||
|
||||
当前不接受 PR，只接受 issue。
|
||||
|
||||
**作者**: `A_Dawn`
|
||||
@@ -55,6 +55,51 @@
|
||||
"type": "tool",
|
||||
"name": "memory_stats",
|
||||
"description": "查询记忆统计"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_graph_admin",
|
||||
"description": "图谱管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_source_admin",
|
||||
"description": "来源管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_episode_admin",
|
||||
"description": "Episode 管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_profile_admin",
|
||||
"description": "画像管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_runtime_admin",
|
||||
"description": "运行时管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_import_admin",
|
||||
"description": "导入管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_tuning_admin",
|
||||
"description": "调优管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_v5_admin",
|
||||
"description": "V5 记忆管理接口"
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "memory_delete_admin",
|
||||
"description": "删除管理接口"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
@@ -1,46 +1,55 @@
|
||||
"""
|
||||
Hash-based embedding adapter used by the SDK runtime.
|
||||
请求式嵌入 API 适配器。
|
||||
|
||||
The plugin runtime cannot import MaiBot host embedding internals from ``src.chat``
|
||||
or ``src.llm_models``. This adapter keeps A_Memorix self-contained and stable in
|
||||
Runner by generating deterministic dense vectors locally.
|
||||
恢复 v1.0.1 的真实 embedding 请求语义：
|
||||
- 通过宿主模型配置探测/请求 embedding
|
||||
- 支持 dimensions 参数
|
||||
- 支持批量与重试
|
||||
- 不再提供本地 hash fallback
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
import asyncio
|
||||
import time
|
||||
from typing import List, Optional, Union
|
||||
from typing import Any, List, Optional, Union
|
||||
|
||||
import aiohttp
|
||||
import numpy as np
|
||||
import openai
|
||||
|
||||
from src.common.logger import get_logger
|
||||
|
||||
from src.config.config import config_manager
|
||||
from src.config.model_configs import APIProvider, ModelInfo
|
||||
from src.llm_models.exceptions import NetworkConnectionError
|
||||
from src.llm_models.model_client.base_client import client_registry
|
||||
|
||||
logger = get_logger("A_Memorix.EmbeddingAPIAdapter")
|
||||
|
||||
_TOKEN_PATTERN = re.compile(r"[A-Za-z0-9_\u4e00-\u9fff]{1,}")
|
||||
|
||||
|
||||
class EmbeddingAPIAdapter:
|
||||
"""Deterministic local embedding adapter."""
|
||||
"""适配宿主 embedding 请求接口。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
batch_size: int = 32,
|
||||
max_concurrent: int = 5,
|
||||
default_dimension: int = 256,
|
||||
default_dimension: int = 1024,
|
||||
enable_cache: bool = False,
|
||||
model_name: str = "hash-v1",
|
||||
model_name: str = "auto",
|
||||
retry_config: Optional[dict] = None,
|
||||
) -> None:
|
||||
self.batch_size = max(1, int(batch_size))
|
||||
self.max_concurrent = max(1, int(max_concurrent))
|
||||
self.default_dimension = max(32, int(default_dimension))
|
||||
self.default_dimension = max(1, int(default_dimension))
|
||||
self.enable_cache = bool(enable_cache)
|
||||
self.model_name = str(model_name or "hash-v1")
|
||||
self.model_name = str(model_name or "auto")
|
||||
|
||||
self.retry_config = retry_config or {}
|
||||
self.max_attempts = max(1, int(self.retry_config.get("max_attempts", 5)))
|
||||
self.max_wait_seconds = max(0.1, float(self.retry_config.get("max_wait_seconds", 40)))
|
||||
self.min_wait_seconds = max(0.1, float(self.retry_config.get("min_wait_seconds", 3)))
|
||||
self.backoff_multiplier = max(1.0, float(self.retry_config.get("backoff_multiplier", 3)))
|
||||
|
||||
self._dimension: Optional[int] = None
|
||||
self._dimension_detected = False
|
||||
@@ -49,57 +58,164 @@ class EmbeddingAPIAdapter:
|
||||
self._total_time = 0.0
|
||||
|
||||
logger.info(
|
||||
"EmbeddingAPIAdapter 初始化: model=%s, batch_size=%s, dimension=%s",
|
||||
self.model_name,
|
||||
self.batch_size,
|
||||
self.default_dimension,
|
||||
"EmbeddingAPIAdapter 初始化: "
|
||||
f"batch_size={self.batch_size}, "
|
||||
f"max_concurrent={self.max_concurrent}, "
|
||||
f"default_dim={self.default_dimension}, "
|
||||
f"model={self.model_name}"
|
||||
)
|
||||
|
||||
def _get_current_model_config(self):
|
||||
return config_manager.get_model_config()
|
||||
|
||||
@staticmethod
|
||||
def _find_model_info(model_name: str) -> ModelInfo:
|
||||
model_cfg = config_manager.get_model_config()
|
||||
for item in model_cfg.models:
|
||||
if item.name == model_name:
|
||||
return item
|
||||
raise ValueError(f"未找到 embedding 模型: {model_name}")
|
||||
|
||||
@staticmethod
|
||||
def _find_provider(provider_name: str) -> APIProvider:
|
||||
model_cfg = config_manager.get_model_config()
|
||||
for item in model_cfg.api_providers:
|
||||
if item.name == provider_name:
|
||||
return item
|
||||
raise ValueError(f"未找到 embedding provider: {provider_name}")
|
||||
|
||||
def _resolve_candidate_model_names(self) -> List[str]:
|
||||
task_config = self._get_current_model_config().model_task_config.embedding
|
||||
configured = list(getattr(task_config, "model_list", []) or [])
|
||||
if self.model_name and self.model_name != "auto":
|
||||
return [self.model_name, *[name for name in configured if name != self.model_name]]
|
||||
return configured
|
||||
|
||||
@staticmethod
|
||||
def _validate_embedding_vector(embedding: Any, *, source: str) -> np.ndarray:
|
||||
array = np.asarray(embedding, dtype=np.float32)
|
||||
if array.ndim != 1:
|
||||
raise RuntimeError(f"{source} 返回的 embedding 维度非法: ndim={array.ndim}")
|
||||
if array.size <= 0:
|
||||
raise RuntimeError(f"{source} 返回了空 embedding")
|
||||
if not np.all(np.isfinite(array)):
|
||||
raise RuntimeError(f"{source} 返回了非有限 embedding 值")
|
||||
return array
|
||||
|
||||
async def _request_with_retry(self, client, model_info, text: str, extra_params: dict):
|
||||
retriable_exceptions = (
|
||||
openai.APIConnectionError,
|
||||
openai.APITimeoutError,
|
||||
aiohttp.ClientError,
|
||||
asyncio.TimeoutError,
|
||||
NetworkConnectionError,
|
||||
)
|
||||
|
||||
last_exc: Optional[BaseException] = None
|
||||
for attempt in range(1, self.max_attempts + 1):
|
||||
try:
|
||||
return await client.get_embedding(
|
||||
model_info=model_info,
|
||||
embedding_input=text,
|
||||
extra_params=extra_params,
|
||||
)
|
||||
except retriable_exceptions as exc:
|
||||
last_exc = exc
|
||||
if attempt >= self.max_attempts:
|
||||
raise
|
||||
wait_seconds = min(
|
||||
self.max_wait_seconds,
|
||||
self.min_wait_seconds * (self.backoff_multiplier ** (attempt - 1)),
|
||||
)
|
||||
logger.warning(
|
||||
"Embedding 请求失败，重试 "
|
||||
f"{attempt}/{max(1, self.max_attempts - 1)}，"
|
||||
f"{wait_seconds:.1f}s 后重试: {exc}"
|
||||
)
|
||||
await asyncio.sleep(wait_seconds)
|
||||
except Exception:
|
||||
raise
|
||||
|
||||
if last_exc is not None:
|
||||
raise last_exc
|
||||
raise RuntimeError("Embedding 请求失败：未知错误")
|
||||
|
||||
async def _get_embedding_direct(self, text: str, dimensions: Optional[int] = None) -> Optional[List[float]]:
|
||||
candidate_names = self._resolve_candidate_model_names()
|
||||
if not candidate_names:
|
||||
raise RuntimeError("embedding 任务未配置模型")
|
||||
|
||||
last_exc: Optional[BaseException] = None
|
||||
for candidate_name in candidate_names:
|
||||
try:
|
||||
model_info = self._find_model_info(candidate_name)
|
||||
api_provider = self._find_provider(model_info.api_provider)
|
||||
client = client_registry.get_client_class_instance(api_provider, force_new=True)
|
||||
|
||||
extra_params = dict(getattr(model_info, "extra_params", {}) or {})
|
||||
if dimensions is not None:
|
||||
extra_params["dimensions"] = int(dimensions)
|
||||
|
||||
response = await self._request_with_retry(
|
||||
client=client,
|
||||
model_info=model_info,
|
||||
text=text,
|
||||
extra_params=extra_params,
|
||||
)
|
||||
embedding = getattr(response, "embedding", None)
|
||||
if embedding is None:
|
||||
raise RuntimeError(f"模型 {candidate_name} 未返回 embedding")
|
||||
vector = self._validate_embedding_vector(
|
||||
embedding,
|
||||
source=f"embedding 模型 {candidate_name}",
|
||||
)
|
||||
return vector.tolist()
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
logger.warning(f"embedding 模型 {candidate_name} 请求失败: {exc}")
|
||||
|
||||
if last_exc is not None:
|
||||
logger.error(f"通过直接 Client 获取 Embedding 失败: {last_exc}")
|
||||
return None
|
||||
|
||||
async def _detect_dimension(self) -> int:
|
||||
if self._dimension_detected and self._dimension is not None:
|
||||
return self._dimension
|
||||
|
||||
logger.info("正在检测嵌入模型维度...")
|
||||
try:
|
||||
target_dim = self.default_dimension
|
||||
logger.debug(f"尝试请求指定维度: {target_dim}")
|
||||
test_embedding = await self._get_embedding_direct("test", dimensions=target_dim)
|
||||
if test_embedding and isinstance(test_embedding, list):
|
||||
detected_dim = len(test_embedding)
|
||||
if detected_dim == target_dim:
|
||||
logger.info(f"嵌入维度检测成功 (匹配配置): {detected_dim}")
|
||||
else:
|
||||
logger.warning(
|
||||
f"请求维度 {target_dim} 但模型返回 {detected_dim}，将使用模型自然维度"
|
||||
)
|
||||
self._dimension = detected_dim
|
||||
self._dimension_detected = True
|
||||
return detected_dim
|
||||
except Exception as exc:
|
||||
logger.debug(f"带维度参数探测失败: {exc}，尝试不带参数探测")
|
||||
|
||||
try:
|
||||
test_embedding = await self._get_embedding_direct("test", dimensions=None)
|
||||
if test_embedding and isinstance(test_embedding, list):
|
||||
detected_dim = len(test_embedding)
|
||||
self._dimension = detected_dim
|
||||
self._dimension_detected = True
|
||||
logger.info(f"嵌入维度检测成功 (自然维度): {detected_dim}")
|
||||
return detected_dim
|
||||
logger.warning(f"嵌入维度检测失败，使用默认值: {self.default_dimension}")
|
||||
except Exception as exc:
|
||||
logger.error(f"嵌入维度检测异常: {exc}，使用默认值: {self.default_dimension}")
|
||||
|
||||
self._dimension = self.default_dimension
|
||||
self._dimension_detected = True
|
||||
return self._dimension
|
||||
|
||||
@staticmethod
|
||||
def _tokenize(text: str) -> List[str]:
|
||||
clean = str(text or "").strip().lower()
|
||||
if not clean:
|
||||
return []
|
||||
return _TOKEN_PATTERN.findall(clean)
|
||||
|
||||
@staticmethod
|
||||
def _feature_weight(token: str) -> float:
|
||||
digest = hashlib.sha256(token.encode("utf-8")).digest()
|
||||
return 1.0 + (digest[10] / 255.0) * 0.5
|
||||
|
||||
def _encode_single(self, text: str, dimension: int) -> np.ndarray:
|
||||
vector = np.zeros(dimension, dtype=np.float32)
|
||||
content = str(text or "").strip()
|
||||
tokens = self._tokenize(content)
|
||||
if not tokens and content:
|
||||
tokens = [content.lower()]
|
||||
if not tokens:
|
||||
vector[0] = 1.0
|
||||
return vector
|
||||
|
||||
for token in tokens:
|
||||
digest = hashlib.sha256(token.encode("utf-8")).digest()
|
||||
bucket = int.from_bytes(digest[:8], byteorder="big", signed=False) % dimension
|
||||
sign = 1.0 if digest[8] % 2 == 0 else -1.0
|
||||
vector[bucket] += sign * self._feature_weight(token)
|
||||
|
||||
second_bucket = int.from_bytes(digest[12:20], byteorder="big", signed=False) % dimension
|
||||
if second_bucket != bucket:
|
||||
vector[second_bucket] += (sign * 0.35)
|
||||
|
||||
norm = float(np.linalg.norm(vector))
|
||||
if norm > 1e-8:
|
||||
vector /= norm
|
||||
else:
|
||||
vector[0] = 1.0
|
||||
return vector
|
||||
return self.default_dimension
|
||||
|
||||
async def encode(
|
||||
self,
|
||||
@@ -109,59 +225,137 @@ class EmbeddingAPIAdapter:
|
||||
normalize: bool = True,
|
||||
dimensions: Optional[int] = None,
|
||||
) -> np.ndarray:
|
||||
_ = batch_size
|
||||
_ = show_progress
|
||||
_ = normalize
|
||||
del show_progress
|
||||
del normalize
|
||||
|
||||
started_at = time.time()
|
||||
target_dimension = max(32, int(dimensions or await self._detect_dimension()))
|
||||
start_time = time.time()
|
||||
target_dim = int(dimensions) if dimensions is not None else int(await self._detect_dimension())
|
||||
|
||||
if isinstance(texts, str):
|
||||
single_input = True
|
||||
normalized_texts = [texts]
|
||||
single_input = True
|
||||
else:
|
||||
single_input = False
|
||||
normalized_texts = list(texts or [])
|
||||
single_input = False
|
||||
|
||||
if not normalized_texts:
|
||||
empty = np.zeros((0, target_dimension), dtype=np.float32)
|
||||
empty = np.zeros((0, target_dim), dtype=np.float32)
|
||||
return empty[0] if single_input else empty
|
||||
|
||||
if batch_size is None:
|
||||
batch_size = self.batch_size
|
||||
|
||||
try:
|
||||
matrix = np.vstack([self._encode_single(item, target_dimension) for item in normalized_texts])
|
||||
embeddings = await self._encode_batch_internal(
|
||||
normalized_texts,
|
||||
batch_size=max(1, int(batch_size)),
|
||||
dimensions=dimensions,
|
||||
)
|
||||
if embeddings.ndim == 1:
|
||||
embeddings = embeddings.reshape(1, -1)
|
||||
self._total_encoded += len(normalized_texts)
|
||||
self._total_time += time.time() - started_at
|
||||
except Exception:
|
||||
elapsed = time.time() - start_time
|
||||
self._total_time += elapsed
|
||||
logger.debug(
|
||||
"编码完成: "
|
||||
f"{len(normalized_texts)} 个文本, "
|
||||
f"耗时 {elapsed:.2f}s, "
|
||||
f"平均 {elapsed / max(1, len(normalized_texts)):.3f}s/文本"
|
||||
)
|
||||
return embeddings[0] if single_input else embeddings
|
||||
except Exception as exc:
|
||||
self._total_errors += 1
|
||||
raise
|
||||
logger.error(f"编码失败: {exc}")
|
||||
raise RuntimeError(f"embedding encode failed: {exc}") from exc
|
||||
|
||||
return matrix[0] if single_input else matrix
|
||||
async def _encode_batch_internal(
|
||||
self,
|
||||
texts: List[str],
|
||||
batch_size: int,
|
||||
dimensions: Optional[int] = None,
|
||||
) -> np.ndarray:
|
||||
all_embeddings: List[np.ndarray] = []
|
||||
for offset in range(0, len(texts), batch_size):
|
||||
batch = texts[offset : offset + batch_size]
|
||||
semaphore = asyncio.Semaphore(self.max_concurrent)
|
||||
|
||||
def get_statistics(self) -> dict:
|
||||
avg_time = self._total_time / self._total_encoded if self._total_encoded else 0.0
|
||||
async def encode_with_semaphore(text: str, index: int):
|
||||
async with semaphore:
|
||||
embedding = await self._get_embedding_direct(text, dimensions=dimensions)
|
||||
if embedding is None:
|
||||
raise RuntimeError(f"文本 {index} 编码失败：embedding 返回为空")
|
||||
vector = self._validate_embedding_vector(
|
||||
embedding,
|
||||
source=f"文本 {index}",
|
||||
)
|
||||
return index, vector
|
||||
|
||||
tasks = [
|
||||
encode_with_semaphore(text, offset + index)
|
||||
for index, text in enumerate(batch)
|
||||
]
|
||||
results = await asyncio.gather(*tasks)
|
||||
results.sort(key=lambda item: item[0])
|
||||
all_embeddings.extend(emb for _, emb in results)
|
||||
|
||||
return np.array(all_embeddings, dtype=np.float32)
|
||||
|
||||
async def encode_batch(
|
||||
self,
|
||||
texts: List[str],
|
||||
batch_size: Optional[int] = None,
|
||||
num_workers: Optional[int] = None,
|
||||
show_progress: bool = False,
|
||||
dimensions: Optional[int] = None,
|
||||
) -> np.ndarray:
|
||||
del show_progress
|
||||
if num_workers is not None:
|
||||
previous = self.max_concurrent
|
||||
self.max_concurrent = max(1, int(num_workers))
|
||||
try:
|
||||
return await self.encode(texts, batch_size=batch_size, dimensions=dimensions)
|
||||
finally:
|
||||
self.max_concurrent = previous
|
||||
return await self.encode(texts, batch_size=batch_size, dimensions=dimensions)
|
||||
|
||||
def get_embedding_dimension(self) -> int:
|
||||
if self._dimension is not None:
|
||||
return self._dimension
|
||||
logger.warning(f"维度尚未检测，返回默认值: {self.default_dimension}")
|
||||
return self.default_dimension
|
||||
|
||||
def get_model_info(self) -> dict:
|
||||
return {
|
||||
"model_name": self.model_name,
|
||||
"dimension": self._dimension or self.default_dimension,
|
||||
"dimension_detected": self._dimension_detected,
|
||||
"batch_size": self.batch_size,
|
||||
"max_concurrent": self.max_concurrent,
|
||||
"total_encoded": self._total_encoded,
|
||||
"total_errors": self._total_errors,
|
||||
"total_time": self._total_time,
|
||||
"avg_time_per_text": avg_time,
|
||||
"avg_time_per_text": self._total_time / self._total_encoded if self._total_encoded else 0.0,
|
||||
}
|
||||
|
||||
def get_statistics(self) -> dict:
|
||||
return self.get_model_info()
|
||||
|
||||
@property
|
||||
def is_model_loaded(self) -> bool:
|
||||
return True
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return (
|
||||
f"EmbeddingAPIAdapter(model_name={self.model_name}, "
|
||||
f"dimension={self._dimension or self.default_dimension}, "
|
||||
f"total_encoded={self._total_encoded})"
|
||||
f"EmbeddingAPIAdapter(dim={self._dimension or self.default_dimension}, "
|
||||
f"detected={self._dimension_detected}, encoded={self._total_encoded})"
|
||||
)
|
||||
|
||||
|
||||
def create_embedding_api_adapter(
|
||||
batch_size: int = 32,
|
||||
max_concurrent: int = 5,
|
||||
default_dimension: int = 256,
|
||||
default_dimension: int = 1024,
|
||||
enable_cache: bool = False,
|
||||
model_name: str = "hash-v1",
|
||||
model_name: str = "auto",
|
||||
retry_config: Optional[dict] = None,
|
||||
) -> EmbeddingAPIAdapter:
|
||||
return EmbeddingAPIAdapter(
|
||||
|
||||
@@ -285,10 +285,10 @@ class DualPathRetriever:
|
||||
relation_intent_ctx = self._build_relation_intent_context(query=query, top_k=top_k)
|
||||
|
||||
logger.info(
|
||||
"执行检索: query='%s...', strategy=%s, relation_intent=%s",
|
||||
query[:50],
|
||||
strategy.value,
|
||||
relation_intent_ctx.get("enabled", False),
|
||||
"执行检索: "
|
||||
f"query='{query[:50]}...', "
|
||||
f"strategy={strategy.value}, "
|
||||
f"relation_intent={relation_intent_ctx.get('enabled', False)}"
|
||||
)
|
||||
|
||||
if temporal and not (query or "").strip():
|
||||
@@ -1408,10 +1408,10 @@ class DualPathRetriever:
|
||||
return results
|
||||
|
||||
logger.debug(
|
||||
"relation_rerank_applied=1 relation_pair_groups=%s relation_pair_overflow_count=%s relation_pair_limit=%s",
|
||||
len(ordered_groups),
|
||||
len(overflow),
|
||||
pair_limit,
|
||||
"relation_rerank_applied=1 "
|
||||
f"relation_pair_groups={len(ordered_groups)} "
|
||||
f"relation_pair_overflow_count={len(overflow)} "
|
||||
f"relation_pair_limit={pair_limit}"
|
||||
)
|
||||
|
||||
rebuilt = list(results)
|
||||
@@ -1455,9 +1455,9 @@ class DualPathRetriever:
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning(
|
||||
"metric.ppr_timeout_skip_count=1 timeout_s=%s entities=%s",
|
||||
ppr_timeout_s,
|
||||
len(entities),
|
||||
"metric.ppr_timeout_skip_count=1 "
|
||||
f"timeout_s={ppr_timeout_s} "
|
||||
f"entities={len(entities)}"
|
||||
)
|
||||
return results
|
||||
except Exception as e:
|
||||
|
||||
@@ -170,7 +170,7 @@ class GraphRelationRecallService:
|
||||
max_paths=self.config.max_paths,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug("graph two-hop recall skipped: %s", e)
|
||||
logger.debug(f"graph two-hop recall skipped: {e}")
|
||||
return
|
||||
|
||||
for path_nodes in paths:
|
||||
@@ -210,7 +210,7 @@ class GraphRelationRecallService:
|
||||
limit=self.config.candidate_k,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug("graph one-hop recall skipped: %s", e)
|
||||
logger.debug(f"graph one-hop recall skipped: {e}")
|
||||
return
|
||||
self._append_relation_hashes(
|
||||
relation_hashes=relation_hashes,
|
||||
|
||||
@@ -123,9 +123,8 @@ class SparseBM25Index:
|
||||
self._loaded = True
|
||||
self._prepare_tokenizer()
|
||||
logger.info(
|
||||
"SparseBM25Index loaded: backend=fts5, tokenizer=%s, mode=%s",
|
||||
self.config.tokenizer_mode,
|
||||
self.config.mode,
|
||||
"SparseBM25Index loaded: "
|
||||
f"backend=fts5, tokenizer={self.config.tokenizer_mode}, mode={self.config.mode}"
|
||||
)
|
||||
return True
|
||||
|
||||
@@ -141,9 +140,9 @@ class SparseBM25Index:
|
||||
if user_dict:
|
||||
try:
|
||||
jieba.load_userdict(user_dict) # type: ignore[union-attr]
|
||||
logger.info("已加载 jieba 用户词典: %s", user_dict)
|
||||
logger.info(f"已加载 jieba 用户词典: {user_dict}")
|
||||
except Exception as e:
|
||||
logger.warning("加载 jieba 用户词典失败: %s", e)
|
||||
logger.warning(f"加载 jieba 用户词典失败: {e}")
|
||||
self._jieba_dict_loaded = True
|
||||
|
||||
def _tokenize_jieba(self, text: str) -> List[str]:
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
"""SDK runtime exports for A_Memorix."""
|
||||
|
||||
from .search_runtime_initializer import (
|
||||
SearchRuntimeBundle,
|
||||
SearchRuntimeInitializer,
|
||||
build_search_runtime,
|
||||
)
|
||||
from .sdk_memory_kernel import KernelSearchRequest, SDKMemoryKernel
|
||||
|
||||
__all__ = [
|
||||
"SearchRuntimeBundle",
|
||||
"SearchRuntimeInitializer",
|
||||
"build_search_runtime",
|
||||
"KernelSearchRequest",
|
||||
"SDKMemoryKernel",
|
||||
]
|
||||
|
||||
268
plugins/A_memorix/core/runtime/lifecycle_orchestrator.py
Normal file
268
plugins/A_memorix/core/runtime/lifecycle_orchestrator.py
Normal file
@@ -0,0 +1,268 @@
|
||||
"""Lifecycle bootstrap/teardown helpers extracted from plugin.py."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from src.common.logger import get_logger
|
||||
|
||||
from ..embedding import create_embedding_api_adapter
|
||||
from ..retrieval import SparseBM25Config, SparseBM25Index
|
||||
from ..storage import (
|
||||
GraphStore,
|
||||
MetadataStore,
|
||||
QuantizationType,
|
||||
SparseMatrixFormat,
|
||||
VectorStore,
|
||||
)
|
||||
from ..utils.runtime_self_check import ensure_runtime_self_check
|
||||
from ..utils.relation_write_service import RelationWriteService
|
||||
|
||||
logger = get_logger("A_Memorix.LifecycleOrchestrator")
|
||||
|
||||
|
||||
async def ensure_initialized(plugin: Any) -> None:
|
||||
if plugin._initialized:
|
||||
plugin._runtime_ready = plugin._check_storage_ready()
|
||||
return
|
||||
|
||||
async with plugin._init_lock:
|
||||
if plugin._initialized:
|
||||
plugin._runtime_ready = plugin._check_storage_ready()
|
||||
return
|
||||
|
||||
logger.info("A_Memorix 插件正在异步初始化存储组件...")
|
||||
plugin._validate_runtime_config()
|
||||
await initialize_storage_async(plugin)
|
||||
report = await ensure_runtime_self_check(plugin, force=True)
|
||||
if not bool(report.get("ok", False)):
|
||||
logger.error(
|
||||
"A_Memorix runtime self-check failed: "
|
||||
f"{report.get('message', 'unknown')}; "
|
||||
"建议执行 python plugins/A_memorix/scripts/runtime_self_check.py --json"
|
||||
)
|
||||
|
||||
if plugin.graph_store and plugin.metadata_store:
|
||||
relation_count = plugin.metadata_store.count_relations()
|
||||
if relation_count > 0 and not plugin.graph_store.has_edge_hash_map():
|
||||
raise RuntimeError(
|
||||
"检测到 relations 数据存在但 edge-hash-map 为空。"
|
||||
" 请先执行 scripts/release_vnext_migrate.py migrate。"
|
||||
)
|
||||
|
||||
plugin._initialized = True
|
||||
plugin._runtime_ready = plugin._check_storage_ready()
|
||||
plugin._update_plugin_config()
|
||||
logger.info("A_Memorix 插件异步初始化成功")
|
||||
|
||||
|
||||
def start_background_tasks(plugin: Any) -> None:
|
||||
"""Start background tasks idempotently."""
|
||||
if not hasattr(plugin, "_episode_generation_task"):
|
||||
plugin._episode_generation_task = None
|
||||
|
||||
if (
|
||||
plugin.get_config("summarization.enabled", True)
|
||||
and plugin.get_config("schedule.enabled", True)
|
||||
and (plugin._scheduled_import_task is None or plugin._scheduled_import_task.done())
|
||||
):
|
||||
plugin._scheduled_import_task = asyncio.create_task(plugin._scheduled_import_loop())
|
||||
|
||||
if (
|
||||
plugin.get_config("advanced.enable_auto_save", True)
|
||||
and (plugin._auto_save_task is None or plugin._auto_save_task.done())
|
||||
):
|
||||
plugin._auto_save_task = asyncio.create_task(plugin._auto_save_loop())
|
||||
|
||||
if (
|
||||
plugin.get_config("person_profile.enabled", True)
|
||||
and (plugin._person_profile_refresh_task is None or plugin._person_profile_refresh_task.done())
|
||||
):
|
||||
plugin._person_profile_refresh_task = asyncio.create_task(plugin._person_profile_refresh_loop())
|
||||
|
||||
if plugin._memory_maintenance_task is None or plugin._memory_maintenance_task.done():
|
||||
plugin._memory_maintenance_task = asyncio.create_task(plugin._memory_maintenance_loop())
|
||||
|
||||
rv_cfg = plugin.get_config("retrieval.relation_vectorization", {}) or {}
|
||||
if isinstance(rv_cfg, dict):
|
||||
rv_enabled = bool(rv_cfg.get("enabled", False))
|
||||
rv_backfill = bool(rv_cfg.get("backfill_enabled", False))
|
||||
else:
|
||||
rv_enabled = False
|
||||
rv_backfill = False
|
||||
if rv_enabled and rv_backfill and (
|
||||
plugin._relation_vector_backfill_task is None or plugin._relation_vector_backfill_task.done()
|
||||
):
|
||||
plugin._relation_vector_backfill_task = asyncio.create_task(plugin._relation_vector_backfill_loop())
|
||||
|
||||
episode_task = getattr(plugin, "_episode_generation_task", None)
|
||||
episode_loop = getattr(plugin, "_episode_generation_loop", None)
|
||||
if (
|
||||
callable(episode_loop)
|
||||
and bool(plugin.get_config("episode.enabled", True))
|
||||
and bool(plugin.get_config("episode.generation_enabled", True))
|
||||
and (episode_task is None or episode_task.done())
|
||||
):
|
||||
plugin._episode_generation_task = asyncio.create_task(episode_loop())
|
||||
|
||||
|
||||
async def cancel_background_tasks(plugin: Any) -> None:
|
||||
"""Cancel all background tasks and wait for cleanup."""
|
||||
tasks = [
|
||||
("scheduled_import", plugin._scheduled_import_task),
|
||||
("auto_save", plugin._auto_save_task),
|
||||
("person_profile_refresh", plugin._person_profile_refresh_task),
|
||||
("memory_maintenance", plugin._memory_maintenance_task),
|
||||
("relation_vector_backfill", plugin._relation_vector_backfill_task),
|
||||
("episode_generation", getattr(plugin, "_episode_generation_task", None)),
|
||||
]
|
||||
for _, task in tasks:
|
||||
if task and not task.done():
|
||||
task.cancel()
|
||||
|
||||
for name, task in tasks:
|
||||
if not task:
|
||||
continue
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning(f"后台任务 {name} 退出异常: {e}")
|
||||
|
||||
plugin._scheduled_import_task = None
|
||||
plugin._auto_save_task = None
|
||||
plugin._person_profile_refresh_task = None
|
||||
plugin._memory_maintenance_task = None
|
||||
plugin._relation_vector_backfill_task = None
|
||||
plugin._episode_generation_task = None
|
||||
|
||||
|
||||
async def initialize_storage_async(plugin: Any) -> None:
|
||||
"""Initialize storage components asynchronously."""
|
||||
data_dir_str = plugin.get_config("storage.data_dir", "./data")
|
||||
if data_dir_str.startswith("."):
|
||||
plugin_dir = Path(__file__).resolve().parents[2]
|
||||
data_dir = (plugin_dir / data_dir_str).resolve()
|
||||
else:
|
||||
data_dir = Path(data_dir_str)
|
||||
|
||||
logger.info(f"A_Memorix 数据存储路径: {data_dir}")
|
||||
data_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
plugin.embedding_manager = create_embedding_api_adapter(
|
||||
batch_size=plugin.get_config("embedding.batch_size", 32),
|
||||
max_concurrent=plugin.get_config("embedding.max_concurrent", 5),
|
||||
default_dimension=plugin.get_config("embedding.dimension", 1024),
|
||||
model_name=plugin.get_config("embedding.model_name", "auto"),
|
||||
retry_config=plugin.get_config("embedding.retry", {}),
|
||||
)
|
||||
logger.info("嵌入 API 适配器初始化完成")
|
||||
|
||||
try:
|
||||
detected_dimension = await plugin.embedding_manager._detect_dimension()
|
||||
logger.info(f"嵌入维度检测成功: {detected_dimension}")
|
||||
except Exception as e:
|
||||
logger.warning(f"嵌入维度检测失败: {e}，使用默认值")
|
||||
detected_dimension = plugin.embedding_manager.default_dimension
|
||||
|
||||
quantization_str = plugin.get_config("embedding.quantization_type", "int8")
|
||||
if str(quantization_str or "").strip().lower() != "int8":
|
||||
raise ValueError("embedding.quantization_type 在 vNext 仅允许 int8(SQ8)。")
|
||||
quantization_type = QuantizationType.INT8
|
||||
|
||||
plugin.vector_store = VectorStore(
|
||||
dimension=detected_dimension,
|
||||
quantization_type=quantization_type,
|
||||
data_dir=data_dir / "vectors",
|
||||
)
|
||||
plugin.vector_store.min_train_threshold = plugin.get_config("embedding.min_train_threshold", 40)
|
||||
logger.info(
|
||||
"向量存储初始化完成（"
|
||||
f"维度: {detected_dimension}, "
|
||||
f"训练阈值: {plugin.vector_store.min_train_threshold}）"
|
||||
)
|
||||
|
||||
matrix_format_str = plugin.get_config("graph.sparse_matrix_format", "csr")
|
||||
matrix_format_map = {
|
||||
"csr": SparseMatrixFormat.CSR,
|
||||
"csc": SparseMatrixFormat.CSC,
|
||||
}
|
||||
matrix_format = matrix_format_map.get(matrix_format_str, SparseMatrixFormat.CSR)
|
||||
|
||||
plugin.graph_store = GraphStore(
|
||||
matrix_format=matrix_format,
|
||||
data_dir=data_dir / "graph",
|
||||
)
|
||||
logger.info("图存储初始化完成")
|
||||
|
||||
plugin.metadata_store = MetadataStore(data_dir=data_dir / "metadata")
|
||||
plugin.metadata_store.connect()
|
||||
logger.info("元数据存储初始化完成")
|
||||
|
||||
plugin.relation_write_service = RelationWriteService(
|
||||
metadata_store=plugin.metadata_store,
|
||||
graph_store=plugin.graph_store,
|
||||
vector_store=plugin.vector_store,
|
||||
embedding_manager=plugin.embedding_manager,
|
||||
)
|
||||
logger.info("关系写入服务初始化完成")
|
||||
|
||||
sparse_cfg_raw = plugin.get_config("retrieval.sparse", {}) or {}
|
||||
if not isinstance(sparse_cfg_raw, dict):
|
||||
sparse_cfg_raw = {}
|
||||
try:
|
||||
sparse_cfg = SparseBM25Config(**sparse_cfg_raw)
|
||||
except Exception as e:
|
||||
logger.warning(f"sparse 配置非法，回退默认配置: {e}")
|
||||
sparse_cfg = SparseBM25Config()
|
||||
plugin.sparse_index = SparseBM25Index(
|
||||
metadata_store=plugin.metadata_store,
|
||||
config=sparse_cfg,
|
||||
)
|
||||
logger.info(
|
||||
"稀疏检索组件初始化完成: "
|
||||
f"enabled={sparse_cfg.enabled}, "
|
||||
f"lazy_load={sparse_cfg.lazy_load}, "
|
||||
f"mode={sparse_cfg.mode}, "
|
||||
f"tokenizer={sparse_cfg.tokenizer_mode}"
|
||||
)
|
||||
if sparse_cfg.enabled and not sparse_cfg.lazy_load:
|
||||
plugin.sparse_index.ensure_loaded()
|
||||
|
||||
if plugin.vector_store.has_data():
|
||||
try:
|
||||
plugin.vector_store.load()
|
||||
logger.info(f"向量数据已加载，共 {plugin.vector_store.num_vectors} 个向量")
|
||||
except Exception as e:
|
||||
logger.warning(f"加载向量数据失败: {e}")
|
||||
|
||||
try:
|
||||
warmup_summary = plugin.vector_store.warmup_index(force_train=True)
|
||||
if warmup_summary.get("ok"):
|
||||
logger.info(
|
||||
"向量索引预热完成: "
|
||||
f"trained={warmup_summary.get('trained')}, "
|
||||
f"index_ntotal={warmup_summary.get('index_ntotal')}, "
|
||||
f"fallback_ntotal={warmup_summary.get('fallback_ntotal')}, "
|
||||
f"bin_count={warmup_summary.get('bin_count')}, "
|
||||
f"duration_ms={float(warmup_summary.get('duration_ms', 0.0)):.2f}"
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"向量索引预热失败，继续启用 sparse 降级路径: "
|
||||
f"{warmup_summary.get('error', 'unknown')}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"向量索引预热异常，继续启用 sparse 降级路径: {e}")
|
||||
|
||||
if plugin.graph_store.has_data():
|
||||
try:
|
||||
plugin.graph_store.load()
|
||||
logger.info(f"图数据已加载，共 {plugin.graph_store.num_nodes} 个节点")
|
||||
except Exception as e:
|
||||
logger.warning(f"加载图数据失败: {e}")
|
||||
|
||||
logger.info(f"知识库数据目录: {data_dir}")
|
||||
File diff suppressed because it is too large
Load Diff
240
plugins/A_memorix/core/runtime/search_runtime_initializer.py
Normal file
240
plugins/A_memorix/core/runtime/search_runtime_initializer.py
Normal file
@@ -0,0 +1,240 @@
|
||||
"""Shared runtime initializer for Action/Tool/Command retrieval components."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from src.common.logger import get_logger
|
||||
|
||||
from ..retrieval import (
|
||||
DualPathRetriever,
|
||||
DualPathRetrieverConfig,
|
||||
DynamicThresholdFilter,
|
||||
FusionConfig,
|
||||
GraphRelationRecallConfig,
|
||||
RelationIntentConfig,
|
||||
RetrievalStrategy,
|
||||
SparseBM25Config,
|
||||
ThresholdConfig,
|
||||
ThresholdMethod,
|
||||
)
|
||||
|
||||
_logger = get_logger("A_Memorix.SearchRuntimeInitializer")
|
||||
|
||||
_REQUIRED_COMPONENT_KEYS = (
|
||||
"vector_store",
|
||||
"graph_store",
|
||||
"metadata_store",
|
||||
"embedding_manager",
|
||||
)
|
||||
|
||||
|
||||
def _get_config_value(config: Optional[dict], key: str, default: Any = None) -> Any:
|
||||
if not isinstance(config, dict):
|
||||
return default
|
||||
current: Any = config
|
||||
for part in key.split("."):
|
||||
if isinstance(current, dict) and part in current:
|
||||
current = current[part]
|
||||
else:
|
||||
return default
|
||||
return current
|
||||
|
||||
|
||||
def _safe_dict(value: Any) -> Dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _resolve_debug_enabled(plugin_config: Optional[dict]) -> bool:
|
||||
advanced = _get_config_value(plugin_config, "advanced", {})
|
||||
if isinstance(advanced, dict):
|
||||
return bool(advanced.get("debug", False))
|
||||
return bool(_get_config_value(plugin_config, "debug", False))
|
||||
|
||||
|
||||
@dataclass
|
||||
class SearchRuntimeBundle:
|
||||
"""Resolved runtime components and initialized retriever/filter."""
|
||||
|
||||
vector_store: Optional[Any] = None
|
||||
graph_store: Optional[Any] = None
|
||||
metadata_store: Optional[Any] = None
|
||||
embedding_manager: Optional[Any] = None
|
||||
sparse_index: Optional[Any] = None
|
||||
retriever: Optional[DualPathRetriever] = None
|
||||
threshold_filter: Optional[DynamicThresholdFilter] = None
|
||||
error: str = ""
|
||||
|
||||
@property
|
||||
def ready(self) -> bool:
|
||||
return (
|
||||
self.retriever is not None
|
||||
and self.vector_store is not None
|
||||
and self.graph_store is not None
|
||||
and self.metadata_store is not None
|
||||
and self.embedding_manager is not None
|
||||
)
|
||||
|
||||
|
||||
def _resolve_runtime_components(plugin_config: Optional[dict]) -> SearchRuntimeBundle:
|
||||
bundle = SearchRuntimeBundle(
|
||||
vector_store=_get_config_value(plugin_config, "vector_store"),
|
||||
graph_store=_get_config_value(plugin_config, "graph_store"),
|
||||
metadata_store=_get_config_value(plugin_config, "metadata_store"),
|
||||
embedding_manager=_get_config_value(plugin_config, "embedding_manager"),
|
||||
sparse_index=_get_config_value(plugin_config, "sparse_index"),
|
||||
)
|
||||
|
||||
missing_required = any(
|
||||
getattr(bundle, key) is None for key in _REQUIRED_COMPONENT_KEYS
|
||||
)
|
||||
if not missing_required:
|
||||
return bundle
|
||||
|
||||
try:
|
||||
from ...plugin import AMemorixPlugin
|
||||
|
||||
instances = AMemorixPlugin.get_storage_instances()
|
||||
except Exception:
|
||||
instances = {}
|
||||
|
||||
if not isinstance(instances, dict) or not instances:
|
||||
return bundle
|
||||
|
||||
if bundle.vector_store is None:
|
||||
bundle.vector_store = instances.get("vector_store")
|
||||
if bundle.graph_store is None:
|
||||
bundle.graph_store = instances.get("graph_store")
|
||||
if bundle.metadata_store is None:
|
||||
bundle.metadata_store = instances.get("metadata_store")
|
||||
if bundle.embedding_manager is None:
|
||||
bundle.embedding_manager = instances.get("embedding_manager")
|
||||
if bundle.sparse_index is None:
|
||||
bundle.sparse_index = instances.get("sparse_index")
|
||||
return bundle
|
||||
|
||||
|
||||
def build_search_runtime(
|
||||
plugin_config: Optional[dict],
|
||||
logger_obj: Optional[Any],
|
||||
owner_tag: str,
|
||||
*,
|
||||
log_prefix: str = "",
|
||||
) -> SearchRuntimeBundle:
|
||||
"""Build retriever + threshold filter with unified fallback/config parsing."""
|
||||
|
||||
log = logger_obj or _logger
|
||||
owner = str(owner_tag or "runtime").strip().lower() or "runtime"
|
||||
prefix = str(log_prefix or "").strip()
|
||||
prefix_text = f"{prefix} " if prefix else ""
|
||||
|
||||
runtime = _resolve_runtime_components(plugin_config)
|
||||
if any(getattr(runtime, key) is None for key in _REQUIRED_COMPONENT_KEYS):
|
||||
runtime.error = "存储组件未完全初始化"
|
||||
log.warning(f"{prefix_text}[{owner}] 存储组件未完全初始化，无法使用检索功能")
|
||||
return runtime
|
||||
|
||||
sparse_cfg_raw = _safe_dict(_get_config_value(plugin_config, "retrieval.sparse", {}) or {})
|
||||
fusion_cfg_raw = _safe_dict(_get_config_value(plugin_config, "retrieval.fusion", {}) or {})
|
||||
relation_intent_cfg_raw = _safe_dict(
|
||||
_get_config_value(plugin_config, "retrieval.search.relation_intent", {}) or {}
|
||||
)
|
||||
graph_recall_cfg_raw = _safe_dict(
|
||||
_get_config_value(plugin_config, "retrieval.search.graph_recall", {}) or {}
|
||||
)
|
||||
|
||||
try:
|
||||
sparse_cfg = SparseBM25Config(**sparse_cfg_raw)
|
||||
except Exception as e:
|
||||
log.warning(f"{prefix_text}[{owner}] sparse 配置非法，回退默认: {e}")
|
||||
sparse_cfg = SparseBM25Config()
|
||||
|
||||
try:
|
||||
fusion_cfg = FusionConfig(**fusion_cfg_raw)
|
||||
except Exception as e:
|
||||
log.warning(f"{prefix_text}[{owner}] fusion 配置非法，回退默认: {e}")
|
||||
fusion_cfg = FusionConfig()
|
||||
|
||||
try:
|
||||
relation_intent_cfg = RelationIntentConfig(**relation_intent_cfg_raw)
|
||||
except Exception as e:
|
||||
log.warning(f"{prefix_text}[{owner}] relation_intent 配置非法，回退默认: {e}")
|
||||
relation_intent_cfg = RelationIntentConfig()
|
||||
|
||||
try:
|
||||
graph_recall_cfg = GraphRelationRecallConfig(**graph_recall_cfg_raw)
|
||||
except Exception as e:
|
||||
log.warning(f"{prefix_text}[{owner}] graph_recall 配置非法，回退默认: {e}")
|
||||
graph_recall_cfg = GraphRelationRecallConfig()
|
||||
|
||||
try:
|
||||
config = DualPathRetrieverConfig(
|
||||
top_k_paragraphs=_get_config_value(plugin_config, "retrieval.top_k_paragraphs", 20),
|
||||
top_k_relations=_get_config_value(plugin_config, "retrieval.top_k_relations", 10),
|
||||
top_k_final=_get_config_value(plugin_config, "retrieval.top_k_final", 10),
|
||||
alpha=_get_config_value(plugin_config, "retrieval.alpha", 0.5),
|
||||
enable_ppr=_get_config_value(plugin_config, "retrieval.enable_ppr", True),
|
||||
ppr_alpha=_get_config_value(plugin_config, "retrieval.ppr_alpha", 0.85),
|
||||
ppr_timeout_seconds=_get_config_value(
|
||||
plugin_config, "retrieval.ppr_timeout_seconds", 1.5
|
||||
),
|
||||
ppr_concurrency_limit=_get_config_value(
|
||||
plugin_config, "retrieval.ppr_concurrency_limit", 4
|
||||
),
|
||||
enable_parallel=_get_config_value(plugin_config, "retrieval.enable_parallel", True),
|
||||
retrieval_strategy=RetrievalStrategy.DUAL_PATH,
|
||||
debug=_resolve_debug_enabled(plugin_config),
|
||||
sparse=sparse_cfg,
|
||||
fusion=fusion_cfg,
|
||||
relation_intent=relation_intent_cfg,
|
||||
graph_recall=graph_recall_cfg,
|
||||
)
|
||||
|
||||
runtime.retriever = DualPathRetriever(
|
||||
vector_store=runtime.vector_store,
|
||||
graph_store=runtime.graph_store,
|
||||
metadata_store=runtime.metadata_store,
|
||||
embedding_manager=runtime.embedding_manager,
|
||||
sparse_index=runtime.sparse_index,
|
||||
config=config,
|
||||
)
|
||||
|
||||
threshold_config = ThresholdConfig(
|
||||
method=ThresholdMethod.ADAPTIVE,
|
||||
min_threshold=_get_config_value(plugin_config, "threshold.min_threshold", 0.3),
|
||||
max_threshold=_get_config_value(plugin_config, "threshold.max_threshold", 0.95),
|
||||
percentile=_get_config_value(plugin_config, "threshold.percentile", 75.0),
|
||||
std_multiplier=_get_config_value(plugin_config, "threshold.std_multiplier", 1.5),
|
||||
min_results=_get_config_value(plugin_config, "threshold.min_results", 3),
|
||||
enable_auto_adjust=_get_config_value(plugin_config, "threshold.enable_auto_adjust", True),
|
||||
)
|
||||
runtime.threshold_filter = DynamicThresholdFilter(threshold_config)
|
||||
runtime.error = ""
|
||||
log.info(f"{prefix_text}[{owner}] 检索运行时初始化完成")
|
||||
except Exception as e:
|
||||
runtime.retriever = None
|
||||
runtime.threshold_filter = None
|
||||
runtime.error = str(e)
|
||||
log.error(f"{prefix_text}[{owner}] 检索运行时初始化失败: {e}")
|
||||
|
||||
return runtime
|
||||
|
||||
|
||||
class SearchRuntimeInitializer:
|
||||
"""Compatibility wrapper around the function style initializer."""
|
||||
|
||||
@staticmethod
|
||||
def build_search_runtime(
|
||||
plugin_config: Optional[dict],
|
||||
logger_obj: Optional[Any],
|
||||
owner_tag: str,
|
||||
*,
|
||||
log_prefix: str = "",
|
||||
) -> SearchRuntimeBundle:
|
||||
return build_search_runtime(
|
||||
plugin_config=plugin_config,
|
||||
logger_obj=logger_obj,
|
||||
owner_tag=owner_tag,
|
||||
log_prefix=log_prefix,
|
||||
)
|
||||
@@ -24,6 +24,20 @@ try:
|
||||
from scipy.sparse.linalg import norm
|
||||
HAS_SCIPY = True
|
||||
except ImportError:
|
||||
class _SparseMatrixPlaceholder:
|
||||
pass
|
||||
|
||||
def _scipy_missing(*args, **kwargs):
|
||||
raise ImportError("SciPy 未安装，请安装: pip install scipy")
|
||||
|
||||
csr_matrix = _SparseMatrixPlaceholder
|
||||
csc_matrix = _SparseMatrixPlaceholder
|
||||
lil_matrix = _SparseMatrixPlaceholder
|
||||
triu = _scipy_missing
|
||||
save_npz = _scipy_missing
|
||||
load_npz = _scipy_missing
|
||||
bmat = _scipy_missing
|
||||
norm = _scipy_missing
|
||||
HAS_SCIPY = False
|
||||
|
||||
import contextlib
|
||||
|
||||
@@ -7,6 +7,8 @@
|
||||
import sqlite3
|
||||
import pickle
|
||||
import json
|
||||
import uuid
|
||||
import re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Union, List, Dict, Any, Tuple
|
||||
@@ -24,7 +26,7 @@ from .knowledge_types import (
|
||||
logger = get_logger("A_Memorix.MetadataStore")
|
||||
|
||||
|
||||
SCHEMA_VERSION = 7
|
||||
SCHEMA_VERSION = 8
|
||||
|
||||
|
||||
class MetadataStore:
|
||||
@@ -500,6 +502,63 @@ class MetadataStore:
|
||||
CREATE INDEX IF NOT EXISTS idx_external_memory_refs_paragraph
|
||||
ON external_memory_refs(paragraph_hash)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS memory_v5_operations (
|
||||
operation_id TEXT PRIMARY KEY,
|
||||
action TEXT NOT NULL,
|
||||
target TEXT,
|
||||
reason TEXT,
|
||||
updated_by TEXT,
|
||||
created_at REAL NOT NULL,
|
||||
resolved_hashes_json TEXT,
|
||||
result_json TEXT
|
||||
)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_memory_v5_operations_created
|
||||
ON memory_v5_operations(created_at DESC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS delete_operations (
|
||||
operation_id TEXT PRIMARY KEY,
|
||||
mode TEXT NOT NULL,
|
||||
selector TEXT,
|
||||
reason TEXT,
|
||||
requested_by TEXT,
|
||||
status TEXT NOT NULL,
|
||||
created_at REAL NOT NULL,
|
||||
restored_at REAL,
|
||||
summary_json TEXT
|
||||
)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operations_created
|
||||
ON delete_operations(created_at DESC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operations_mode
|
||||
ON delete_operations(mode, created_at DESC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS delete_operation_items (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
operation_id TEXT NOT NULL,
|
||||
item_type TEXT NOT NULL,
|
||||
item_hash TEXT,
|
||||
item_key TEXT,
|
||||
payload_json TEXT,
|
||||
created_at REAL NOT NULL,
|
||||
FOREIGN KEY (operation_id) REFERENCES delete_operations(operation_id) ON DELETE CASCADE
|
||||
)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operation_items_operation
|
||||
ON delete_operation_items(operation_id, id ASC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operation_items_hash
|
||||
ON delete_operation_items(item_hash)
|
||||
""")
|
||||
# 新版 schema 包含完整字段，直接写入版本信息
|
||||
cursor.execute("INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES (?, ?)", (SCHEMA_VERSION, datetime.now().timestamp()))
|
||||
self._conn.commit()
|
||||
@@ -618,6 +677,63 @@ class MetadataStore:
|
||||
CREATE INDEX IF NOT EXISTS idx_external_memory_refs_paragraph
|
||||
ON external_memory_refs(paragraph_hash)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS memory_v5_operations (
|
||||
operation_id TEXT PRIMARY KEY,
|
||||
action TEXT NOT NULL,
|
||||
target TEXT,
|
||||
reason TEXT,
|
||||
updated_by TEXT,
|
||||
created_at REAL NOT NULL,
|
||||
resolved_hashes_json TEXT,
|
||||
result_json TEXT
|
||||
)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_memory_v5_operations_created
|
||||
ON memory_v5_operations(created_at DESC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS delete_operations (
|
||||
operation_id TEXT PRIMARY KEY,
|
||||
mode TEXT NOT NULL,
|
||||
selector TEXT,
|
||||
reason TEXT,
|
||||
requested_by TEXT,
|
||||
status TEXT NOT NULL,
|
||||
created_at REAL NOT NULL,
|
||||
restored_at REAL,
|
||||
summary_json TEXT
|
||||
)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operations_created
|
||||
ON delete_operations(created_at DESC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operations_mode
|
||||
ON delete_operations(mode, created_at DESC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS delete_operation_items (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
operation_id TEXT NOT NULL,
|
||||
item_type TEXT NOT NULL,
|
||||
item_hash TEXT,
|
||||
item_key TEXT,
|
||||
payload_json TEXT,
|
||||
created_at REAL NOT NULL,
|
||||
FOREIGN KEY (operation_id) REFERENCES delete_operations(operation_id) ON DELETE CASCADE
|
||||
)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operation_items_operation
|
||||
ON delete_operation_items(operation_id, id ASC)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_delete_operation_items_hash
|
||||
ON delete_operation_items(item_hash)
|
||||
""")
|
||||
|
||||
# 检查paragraphs表是否有knowledge_type列
|
||||
cursor.execute("PRAGMA table_info(paragraphs)")
|
||||
@@ -2595,6 +2711,328 @@ class MetadataStore:
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _json_dumps(value: Any) -> str:
|
||||
return json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
|
||||
@staticmethod
|
||||
def _json_loads(value: Any, default: Any) -> Any:
|
||||
if value in {None, ""}:
|
||||
return default
|
||||
try:
|
||||
return json.loads(value)
|
||||
except Exception:
|
||||
return default
|
||||
|
||||
def list_external_memory_refs_by_paragraphs(self, paragraph_hashes: List[str]) -> List[Dict[str, Any]]:
|
||||
hashes = [str(item or "").strip() for item in (paragraph_hashes or []) if str(item or "").strip()]
|
||||
if not hashes:
|
||||
return []
|
||||
placeholders = ",".join(["?"] * len(hashes))
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
f"""
|
||||
SELECT external_id, paragraph_hash, source_type, created_at, metadata_json
|
||||
FROM external_memory_refs
|
||||
WHERE paragraph_hash IN ({placeholders})
|
||||
ORDER BY created_at ASC, external_id ASC
|
||||
""",
|
||||
tuple(hashes),
|
||||
)
|
||||
items: List[Dict[str, Any]] = []
|
||||
for row in cursor.fetchall():
|
||||
payload = dict(row)
|
||||
payload["metadata"] = self._json_loads(payload.get("metadata_json"), {})
|
||||
items.append(payload)
|
||||
return items
|
||||
|
||||
def delete_external_memory_refs_by_paragraphs(self, paragraph_hashes: List[str]) -> List[Dict[str, Any]]:
|
||||
items = self.list_external_memory_refs_by_paragraphs(paragraph_hashes)
|
||||
hashes = [str(item or "").strip() for item in (paragraph_hashes or []) if str(item or "").strip()]
|
||||
if not hashes:
|
||||
return items
|
||||
placeholders = ",".join(["?"] * len(hashes))
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
f"DELETE FROM external_memory_refs WHERE paragraph_hash IN ({placeholders})",
|
||||
tuple(hashes),
|
||||
)
|
||||
self._conn.commit()
|
||||
return items
|
||||
|
||||
def restore_external_memory_refs(self, refs: List[Dict[str, Any]]) -> int:
|
||||
count = 0
|
||||
for item in refs or []:
|
||||
external_id = str(item.get("external_id", "") or "").strip()
|
||||
paragraph_hash = str(item.get("paragraph_hash", "") or "").strip()
|
||||
if not external_id or not paragraph_hash:
|
||||
continue
|
||||
created_at = float(item.get("created_at") or datetime.now().timestamp())
|
||||
metadata_json = self._json_dumps(item.get("metadata") or {})
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO external_memory_refs (
|
||||
external_id, paragraph_hash, source_type, created_at, metadata_json
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(external_id) DO UPDATE SET
|
||||
paragraph_hash = excluded.paragraph_hash,
|
||||
source_type = excluded.source_type,
|
||||
created_at = excluded.created_at,
|
||||
metadata_json = excluded.metadata_json
|
||||
""",
|
||||
(
|
||||
external_id,
|
||||
paragraph_hash,
|
||||
str(item.get("source_type", "") or "").strip() or None,
|
||||
created_at,
|
||||
metadata_json,
|
||||
),
|
||||
)
|
||||
count += max(0, int(cursor.rowcount or 0))
|
||||
self._conn.commit()
|
||||
return count
|
||||
|
||||
def record_v5_operation(
|
||||
self,
|
||||
*,
|
||||
action: str,
|
||||
target: str,
|
||||
resolved_hashes: List[str],
|
||||
reason: str = "",
|
||||
updated_by: str = "",
|
||||
result: Optional[Dict[str, Any]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
operation_id = f"v5_{uuid.uuid4().hex}"
|
||||
created_at = datetime.now().timestamp()
|
||||
payload = {
|
||||
"operation_id": operation_id,
|
||||
"action": str(action or "").strip(),
|
||||
"target": str(target or "").strip(),
|
||||
"reason": str(reason or "").strip(),
|
||||
"updated_by": str(updated_by or "").strip(),
|
||||
"created_at": created_at,
|
||||
"resolved_hashes": [str(item or "").strip() for item in (resolved_hashes or []) if str(item or "").strip()],
|
||||
"result": result or {},
|
||||
}
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO memory_v5_operations (
|
||||
operation_id, action, target, reason, updated_by, created_at, resolved_hashes_json, result_json
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
operation_id,
|
||||
payload["action"],
|
||||
payload["target"] or None,
|
||||
payload["reason"] or None,
|
||||
payload["updated_by"] or None,
|
||||
created_at,
|
||||
self._json_dumps(payload["resolved_hashes"]),
|
||||
self._json_dumps(payload["result"]),
|
||||
),
|
||||
)
|
||||
self._conn.commit()
|
||||
return payload
|
||||
|
||||
def create_delete_operation(
|
||||
self,
|
||||
*,
|
||||
mode: str,
|
||||
selector: Any,
|
||||
items: List[Dict[str, Any]],
|
||||
reason: str = "",
|
||||
requested_by: str = "",
|
||||
status: str = "executed",
|
||||
summary: Optional[Dict[str, Any]] = None,
|
||||
operation_id: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
op_id = str(operation_id or f"del_{uuid.uuid4().hex}").strip()
|
||||
created_at = datetime.now().timestamp()
|
||||
normalized_items: List[Dict[str, Any]] = []
|
||||
for item in items or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
item_type = str(item.get("item_type", "") or "").strip()
|
||||
if not item_type:
|
||||
continue
|
||||
normalized_items.append(
|
||||
{
|
||||
"item_type": item_type,
|
||||
"item_hash": str(item.get("item_hash", "") or "").strip() or None,
|
||||
"item_key": str(item.get("item_key", "") or item.get("item_hash", "") or "").strip() or None,
|
||||
"payload": item.get("payload") if isinstance(item.get("payload"), dict) else {},
|
||||
}
|
||||
)
|
||||
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO delete_operations (
|
||||
operation_id, mode, selector, reason, requested_by, status, created_at, restored_at, summary_json
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, NULL, ?)
|
||||
""",
|
||||
(
|
||||
op_id,
|
||||
str(mode or "").strip(),
|
||||
self._json_dumps(selector if selector is not None else {}),
|
||||
str(reason or "").strip() or None,
|
||||
str(requested_by or "").strip() or None,
|
||||
str(status or "executed").strip(),
|
||||
created_at,
|
||||
self._json_dumps(summary or {}),
|
||||
),
|
||||
)
|
||||
if normalized_items:
|
||||
cursor.executemany(
|
||||
"""
|
||||
INSERT INTO delete_operation_items (
|
||||
operation_id, item_type, item_hash, item_key, payload_json, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
(
|
||||
op_id,
|
||||
item["item_type"],
|
||||
item["item_hash"],
|
||||
item["item_key"],
|
||||
self._json_dumps(item["payload"]),
|
||||
created_at,
|
||||
)
|
||||
for item in normalized_items
|
||||
],
|
||||
)
|
||||
self._conn.commit()
|
||||
return self.get_delete_operation(op_id) or {
|
||||
"operation_id": op_id,
|
||||
"mode": str(mode or "").strip(),
|
||||
"selector": selector,
|
||||
"reason": str(reason or "").strip(),
|
||||
"requested_by": str(requested_by or "").strip(),
|
||||
"status": str(status or "executed").strip(),
|
||||
"created_at": created_at,
|
||||
"summary": summary or {},
|
||||
"items": normalized_items,
|
||||
}
|
||||
|
||||
def mark_delete_operation_restored(
|
||||
self,
|
||||
operation_id: str,
|
||||
*,
|
||||
summary: Optional[Dict[str, Any]] = None,
|
||||
) -> bool:
|
||||
token = str(operation_id or "").strip()
|
||||
if not token:
|
||||
return False
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
UPDATE delete_operations
|
||||
SET status = ?, restored_at = ?, summary_json = ?
|
||||
WHERE operation_id = ?
|
||||
""",
|
||||
(
|
||||
"restored",
|
||||
datetime.now().timestamp(),
|
||||
self._json_dumps(summary or {}),
|
||||
token,
|
||||
),
|
||||
)
|
||||
self._conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
|
||||
def list_delete_operations(self, *, limit: int = 50, mode: str = "") -> List[Dict[str, Any]]:
|
||||
cursor = self._conn.cursor()
|
||||
params: List[Any] = []
|
||||
where = ""
|
||||
mode_token = str(mode or "").strip().lower()
|
||||
if mode_token:
|
||||
where = "WHERE LOWER(mode) = ?"
|
||||
params.append(mode_token)
|
||||
params.append(max(1, int(limit or 50)))
|
||||
cursor.execute(
|
||||
f"""
|
||||
SELECT operation_id, mode, selector, reason, requested_by, status, created_at, restored_at, summary_json
|
||||
FROM delete_operations
|
||||
{where}
|
||||
ORDER BY created_at DESC
|
||||
LIMIT ?
|
||||
""",
|
||||
tuple(params),
|
||||
)
|
||||
items: List[Dict[str, Any]] = []
|
||||
for row in cursor.fetchall():
|
||||
payload = dict(row)
|
||||
payload["selector"] = self._json_loads(payload.get("selector"), {})
|
||||
payload["summary"] = self._json_loads(payload.get("summary_json"), {})
|
||||
items.append(payload)
|
||||
return items
|
||||
|
||||
def get_delete_operation(self, operation_id: str) -> Optional[Dict[str, Any]]:
|
||||
token = str(operation_id or "").strip()
|
||||
if not token:
|
||||
return None
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT operation_id, mode, selector, reason, requested_by, status, created_at, restored_at, summary_json
|
||||
FROM delete_operations
|
||||
WHERE operation_id = ?
|
||||
LIMIT 1
|
||||
""",
|
||||
(token,),
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
payload = dict(row)
|
||||
payload["selector"] = self._json_loads(payload.get("selector"), {})
|
||||
payload["summary"] = self._json_loads(payload.get("summary_json"), {})
|
||||
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT item_type, item_hash, item_key, payload_json, created_at
|
||||
FROM delete_operation_items
|
||||
WHERE operation_id = ?
|
||||
ORDER BY id ASC
|
||||
""",
|
||||
(token,),
|
||||
)
|
||||
payload["items"] = [
|
||||
{
|
||||
"item_type": str(item["item_type"] or ""),
|
||||
"item_hash": str(item["item_hash"] or ""),
|
||||
"item_key": str(item["item_key"] or ""),
|
||||
"payload": self._json_loads(item["payload_json"], {}),
|
||||
"created_at": item["created_at"],
|
||||
}
|
||||
for item in cursor.fetchall()
|
||||
]
|
||||
return payload
|
||||
|
||||
def purge_deleted_relations(self, *, cutoff_time: float, limit: int = 1000) -> List[str]:
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT hash
|
||||
FROM deleted_relations
|
||||
WHERE deleted_at IS NOT NULL AND deleted_at < ?
|
||||
ORDER BY deleted_at ASC
|
||||
LIMIT ?
|
||||
""",
|
||||
(float(cutoff_time), max(1, int(limit or 1000))),
|
||||
)
|
||||
hashes = [str(row[0] or "").strip() for row in cursor.fetchall() if str(row[0] or "").strip()]
|
||||
if not hashes:
|
||||
return []
|
||||
placeholders = ",".join(["?"] * len(hashes))
|
||||
cursor.execute(f"DELETE FROM deleted_relations WHERE hash IN ({placeholders})", tuple(hashes))
|
||||
self._conn.commit()
|
||||
return hashes
|
||||
|
||||
def get_statistics(self) -> Dict[str, int]:
|
||||
"""
|
||||
获取统计信息
|
||||
@@ -2956,6 +3394,18 @@ class MetadataStore:
|
||||
self._conn.commit()
|
||||
return changed
|
||||
|
||||
def restore_paragraph_by_hash(self, paragraph_hash: str) -> bool:
|
||||
"""恢复软删除段落。"""
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"UPDATE paragraphs SET is_deleted=0, deleted_at=NULL WHERE hash=?",
|
||||
(str(paragraph_hash),),
|
||||
)
|
||||
changed = cursor.rowcount > 0
|
||||
if changed:
|
||||
self._conn.commit()
|
||||
return changed
|
||||
|
||||
def backfill_temporal_metadata_from_created_at(
|
||||
self,
|
||||
*,
|
||||
@@ -4698,6 +5148,29 @@ class MetadataStore:
|
||||
)
|
||||
self._conn.commit()
|
||||
|
||||
def get_episode_pending_status_counts(self, source: str) -> Dict[str, int]:
|
||||
"""统计某个 source 当前 pending 队列中的状态分布。"""
|
||||
token = self._normalize_episode_source(source)
|
||||
if not token:
|
||||
return {"pending": 0, "running": 0, "failed": 0, "done": 0}
|
||||
|
||||
cursor = self._conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT status, COUNT(*) AS count
|
||||
FROM episode_pending_paragraphs
|
||||
WHERE TRIM(COALESCE(source, '')) = ?
|
||||
GROUP BY status
|
||||
""",
|
||||
(token,),
|
||||
)
|
||||
counts = {"pending": 0, "running": 0, "failed": 0, "done": 0}
|
||||
for row in cursor.fetchall():
|
||||
status = str(row["status"] or "").strip().lower()
|
||||
if status in counts:
|
||||
counts[status] = int(row["count"] or 0)
|
||||
return counts
|
||||
|
||||
def _episode_row_to_dict(self, row: sqlite3.Row) -> Dict[str, Any]:
|
||||
data = dict(row)
|
||||
|
||||
@@ -4904,7 +5377,7 @@ class MetadataStore:
|
||||
SELECT 1
|
||||
FROM episode_rebuild_sources ers
|
||||
WHERE ers.source = TRIM(COALESCE(e.source, ''))
|
||||
AND ers.status IN ('pending', 'running', 'failed')
|
||||
AND ers.status IN ('pending', 'running')
|
||||
)
|
||||
"""
|
||||
)
|
||||
@@ -4948,6 +5421,26 @@ class MetadataStore:
|
||||
|
||||
return source_expr, effective_start, effective_end, conditions, params
|
||||
|
||||
@staticmethod
|
||||
def _tokenize_episode_query(query: str) -> Tuple[str, List[str]]:
|
||||
"""将 episode 查询归一化为短语和 token。"""
|
||||
normalized = normalize_text(str(query or "")).strip().lower()
|
||||
if not normalized:
|
||||
return "", []
|
||||
|
||||
token_pattern = re.compile(r"[A-Za-z0-9_\u4e00-\u9fff]{2,}")
|
||||
tokens: List[str] = []
|
||||
seen = set()
|
||||
for token in token_pattern.findall(normalized):
|
||||
if token in seen:
|
||||
continue
|
||||
seen.add(token)
|
||||
tokens.append(token)
|
||||
|
||||
if not tokens and len(normalized) >= 2:
|
||||
tokens = [normalized]
|
||||
return normalized, tokens
|
||||
|
||||
def get_episode_rows_by_paragraph_hashes(
|
||||
self,
|
||||
paragraph_hashes: List[str],
|
||||
@@ -5097,28 +5590,58 @@ class MetadataStore:
|
||||
source=source,
|
||||
)
|
||||
|
||||
q = str(query or "").strip().lower()
|
||||
q, tokens = self._tokenize_episode_query(query)
|
||||
select_score_sql = "0.0 AS lexical_score"
|
||||
order_sql = f"{effective_end} DESC, e.updated_at DESC"
|
||||
select_params: List[Any] = []
|
||||
query_params: List[Any] = []
|
||||
if q:
|
||||
like = f"%{q}%"
|
||||
title_expr = "LOWER(COALESCE(e.title, '')) LIKE ?"
|
||||
summary_expr = "LOWER(COALESCE(e.summary, '')) LIKE ?"
|
||||
keywords_expr = "LOWER(COALESCE(e.keywords_json, '')) LIKE ?"
|
||||
participants_expr = "LOWER(COALESCE(e.participants_json, '')) LIKE ?"
|
||||
conditions.append(
|
||||
f"({title_expr} OR {summary_expr} OR {keywords_expr} OR {participants_expr})"
|
||||
field_exprs = {
|
||||
"title": "LOWER(COALESCE(e.title, ''))",
|
||||
"summary": "LOWER(COALESCE(e.summary, ''))",
|
||||
"keywords": "LOWER(COALESCE(e.keywords_json, ''))",
|
||||
"participants": "LOWER(COALESCE(e.participants_json, ''))",
|
||||
}
|
||||
|
||||
score_parts: List[str] = []
|
||||
phrase_like = f"%{q}%"
|
||||
score_parts.extend(
|
||||
[
|
||||
f"CASE WHEN {field_exprs['title']} LIKE ? THEN 6.0 ELSE 0.0 END",
|
||||
f"CASE WHEN {field_exprs['keywords']} LIKE ? THEN 4.5 ELSE 0.0 END",
|
||||
f"CASE WHEN {field_exprs['summary']} LIKE ? THEN 3.0 ELSE 0.0 END",
|
||||
f"CASE WHEN {field_exprs['participants']} LIKE ? THEN 2.0 ELSE 0.0 END",
|
||||
]
|
||||
)
|
||||
select_score_sql = (
|
||||
f"(CASE WHEN {title_expr} THEN 4.0 ELSE 0.0 END + "
|
||||
f"CASE WHEN {keywords_expr} THEN 3.0 ELSE 0.0 END + "
|
||||
f"CASE WHEN {summary_expr} THEN 2.0 ELSE 0.0 END + "
|
||||
f"CASE WHEN {participants_expr} THEN 1.0 ELSE 0.0 END) AS lexical_score"
|
||||
)
|
||||
select_params.extend([like, like, like, like])
|
||||
query_params.extend([like, like, like, like])
|
||||
select_params.extend([phrase_like, phrase_like, phrase_like, phrase_like])
|
||||
|
||||
token_predicates: List[str] = []
|
||||
for token in tokens:
|
||||
like = f"%{token}%"
|
||||
token_any = (
|
||||
f"({field_exprs['title']} LIKE ? OR "
|
||||
f"{field_exprs['summary']} LIKE ? OR "
|
||||
f"{field_exprs['keywords']} LIKE ? OR "
|
||||
f"{field_exprs['participants']} LIKE ?)"
|
||||
)
|
||||
token_predicates.append(token_any)
|
||||
query_params.extend([like, like, like, like])
|
||||
|
||||
score_parts.append(
|
||||
"("
|
||||
f"CASE WHEN {field_exprs['title']} LIKE ? THEN 3.0 ELSE 0.0 END + "
|
||||
f"CASE WHEN {field_exprs['keywords']} LIKE ? THEN 2.5 ELSE 0.0 END + "
|
||||
f"CASE WHEN {field_exprs['summary']} LIKE ? THEN 2.0 ELSE 0.0 END + "
|
||||
f"CASE WHEN {field_exprs['participants']} LIKE ? THEN 1.5 ELSE 0.0 END + "
|
||||
f"CASE WHEN {token_any.replace('?', '?')} THEN 2.0 ELSE 0.0 END"
|
||||
")"
|
||||
)
|
||||
select_params.extend([like, like, like, like, like, like, like, like])
|
||||
|
||||
if token_predicates:
|
||||
conditions.append("(" + " OR ".join(token_predicates) + ")")
|
||||
|
||||
select_score_sql = f"({' + '.join(score_parts)}) AS lexical_score"
|
||||
order_sql = f"lexical_score DESC, {effective_end} DESC, e.updated_at DESC"
|
||||
|
||||
where_sql = ("WHERE " + " AND ".join(conditions)) if conditions else ""
|
||||
|
||||
@@ -302,7 +302,7 @@ class AggregateQueryService:
|
||||
)
|
||||
for (branch_name, _), payload in zip(scheduled, done):
|
||||
if isinstance(payload, Exception):
|
||||
logger.error("aggregate branch failed: branch=%s error=%s", branch_name, payload)
|
||||
logger.error(f"aggregate branch failed: branch={branch_name} error={payload}")
|
||||
normalized = self._normalize_branch_payload(
|
||||
branch_name,
|
||||
{
|
||||
|
||||
@@ -70,7 +70,7 @@ class EpisodeRetrievalService:
|
||||
temporal=temporal,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("episode evidence retrieval failed, fallback to lexical only: %s", exc)
|
||||
logger.warning(f"episode evidence retrieval failed, fallback to lexical only: {exc}")
|
||||
else:
|
||||
paragraph_rank_map: Dict[str, int] = {}
|
||||
relation_rank_map: Dict[str, int] = {}
|
||||
|
||||
304
plugins/A_memorix/core/utils/episode_segmentation_service.py
Normal file
304
plugins/A_memorix/core/utils/episode_segmentation_service.py
Normal file
@@ -0,0 +1,304 @@
|
||||
"""
|
||||
Episode 语义切分服务（LLM 主路径）。
|
||||
|
||||
职责：
|
||||
1. 组装语义切分提示词
|
||||
2. 调用 LLM 生成结构化 episode JSON
|
||||
3. 严格校验输出结构，返回标准化结果
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from src.common.logger import get_logger
|
||||
from src.config.model_configs import TaskConfig
|
||||
from src.config.config import model_config as host_model_config
|
||||
from src.services import llm_service as llm_api
|
||||
|
||||
logger = get_logger("A_Memorix.EpisodeSegmentationService")
|
||||
|
||||
|
||||
class EpisodeSegmentationService:
|
||||
"""基于 LLM 的 episode 语义切分服务。"""
|
||||
|
||||
SEGMENTATION_VERSION = "episode_mvp_v1"
|
||||
|
||||
def __init__(self, plugin_config: Optional[dict] = None):
|
||||
self.plugin_config = plugin_config or {}
|
||||
|
||||
def _cfg(self, key: str, default: Any = None) -> Any:
|
||||
current: Any = self.plugin_config
|
||||
for part in key.split("."):
|
||||
if isinstance(current, dict) and part in current:
|
||||
current = current[part]
|
||||
else:
|
||||
return default
|
||||
return current
|
||||
|
||||
@staticmethod
|
||||
def _is_task_config(obj: Any) -> bool:
|
||||
return hasattr(obj, "model_list") and bool(getattr(obj, "model_list", []))
|
||||
|
||||
def _build_single_model_task(self, model_name: str, template: TaskConfig) -> TaskConfig:
|
||||
return TaskConfig(
|
||||
model_list=[model_name],
|
||||
max_tokens=template.max_tokens,
|
||||
temperature=template.temperature,
|
||||
slow_threshold=template.slow_threshold,
|
||||
selection_strategy=template.selection_strategy,
|
||||
)
|
||||
|
||||
def _pick_template_task(self, available_tasks: Dict[str, Any]) -> Optional[TaskConfig]:
|
||||
preferred = ("utils", "replyer", "planner", "tool_use")
|
||||
for task_name in preferred:
|
||||
cfg = available_tasks.get(task_name)
|
||||
if self._is_task_config(cfg):
|
||||
return cfg
|
||||
for task_name, cfg in available_tasks.items():
|
||||
if task_name != "embedding" and self._is_task_config(cfg):
|
||||
return cfg
|
||||
for cfg in available_tasks.values():
|
||||
if self._is_task_config(cfg):
|
||||
return cfg
|
||||
return None
|
||||
|
||||
def _resolve_model_config(self) -> Tuple[Optional[Any], str]:
|
||||
available_tasks = llm_api.get_available_models() or {}
|
||||
if not available_tasks:
|
||||
return None, "unavailable"
|
||||
|
||||
selector = str(self._cfg("episode.segmentation_model", "auto") or "auto").strip()
|
||||
model_dict = getattr(host_model_config, "models_dict", {}) or {}
|
||||
|
||||
if selector and selector.lower() != "auto":
|
||||
direct_task = available_tasks.get(selector)
|
||||
if self._is_task_config(direct_task):
|
||||
return direct_task, selector
|
||||
|
||||
if selector in model_dict:
|
||||
template = self._pick_template_task(available_tasks)
|
||||
if template is not None:
|
||||
return self._build_single_model_task(selector, template), selector
|
||||
|
||||
logger.warning(f"episode.segmentation_model='{selector}' 不可用，回退 auto")
|
||||
|
||||
for task_name in ("utils", "replyer", "planner", "tool_use"):
|
||||
cfg = available_tasks.get(task_name)
|
||||
if self._is_task_config(cfg):
|
||||
return cfg, task_name
|
||||
|
||||
fallback = self._pick_template_task(available_tasks)
|
||||
if fallback is not None:
|
||||
return fallback, "auto"
|
||||
return None, "unavailable"
|
||||
|
||||
@staticmethod
|
||||
def _clamp_score(value: Any, default: float = 0.0) -> float:
|
||||
try:
|
||||
num = float(value)
|
||||
except Exception:
|
||||
num = default
|
||||
if num < 0.0:
|
||||
return 0.0
|
||||
if num > 1.0:
|
||||
return 1.0
|
||||
return num
|
||||
|
||||
@staticmethod
|
||||
def _safe_json_loads(text: str) -> Dict[str, Any]:
|
||||
raw = str(text or "").strip()
|
||||
if not raw:
|
||||
raise ValueError("empty_response")
|
||||
|
||||
if "```" in raw:
|
||||
raw = raw.replace("```json", "```").replace("```JSON", "```")
|
||||
parts = raw.split("```")
|
||||
for part in parts:
|
||||
part = part.strip()
|
||||
if part.startswith("{") and part.endswith("}"):
|
||||
raw = part
|
||||
break
|
||||
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
if isinstance(data, dict):
|
||||
return data
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
start = raw.find("{")
|
||||
end = raw.rfind("}")
|
||||
if start >= 0 and end > start:
|
||||
candidate = raw[start : end + 1]
|
||||
data = json.loads(candidate)
|
||||
if isinstance(data, dict):
|
||||
return data
|
||||
|
||||
raise ValueError("invalid_json_response")
|
||||
|
||||
def _build_prompt(
|
||||
self,
|
||||
*,
|
||||
source: str,
|
||||
window_start: Optional[float],
|
||||
window_end: Optional[float],
|
||||
paragraphs: List[Dict[str, Any]],
|
||||
) -> str:
|
||||
rows: List[str] = []
|
||||
for idx, item in enumerate(paragraphs, 1):
|
||||
p_hash = str(item.get("hash", "") or "").strip()
|
||||
content = str(item.get("content", "") or "").strip().replace("\r\n", "\n")
|
||||
content = content[:800]
|
||||
event_start = item.get("event_time_start")
|
||||
event_end = item.get("event_time_end")
|
||||
event_time = item.get("event_time")
|
||||
rows.append(
|
||||
(
|
||||
f"[{idx}] hash={p_hash}\n"
|
||||
f"event_time={event_time}\n"
|
||||
f"event_time_start={event_start}\n"
|
||||
f"event_time_end={event_end}\n"
|
||||
f"content={content}"
|
||||
)
|
||||
)
|
||||
|
||||
source_text = str(source or "").strip() or "unknown"
|
||||
return (
|
||||
"You are an episode segmentation engine.\n"
|
||||
"Group the given paragraphs into one or more coherent episodes.\n"
|
||||
"Return JSON ONLY. No markdown, no explanation.\n"
|
||||
"\n"
|
||||
"Hard JSON schema:\n"
|
||||
"{\n"
|
||||
' "episodes": [\n'
|
||||
" {\n"
|
||||
' "title": "string",\n'
|
||||
' "summary": "string",\n'
|
||||
' "paragraph_hashes": ["hash1", "hash2"],\n'
|
||||
' "participants": ["person1", "person2"],\n'
|
||||
' "keywords": ["kw1", "kw2"],\n'
|
||||
' "time_confidence": 0.0,\n'
|
||||
' "llm_confidence": 0.0\n'
|
||||
" }\n"
|
||||
" ]\n"
|
||||
"}\n"
|
||||
"\n"
|
||||
"Rules:\n"
|
||||
"1) paragraph_hashes must come from input only.\n"
|
||||
"2) title and summary must be non-empty.\n"
|
||||
"3) keep participants/keywords concise and deduplicated.\n"
|
||||
"4) if uncertain, still provide best effort confidence values.\n"
|
||||
"\n"
|
||||
f"source={source_text}\n"
|
||||
f"window_start={window_start}\n"
|
||||
f"window_end={window_end}\n"
|
||||
"paragraphs:\n"
|
||||
+ "\n\n".join(rows)
|
||||
)
|
||||
|
||||
def _normalize_episodes(
|
||||
self,
|
||||
*,
|
||||
payload: Dict[str, Any],
|
||||
input_hashes: List[str],
|
||||
) -> List[Dict[str, Any]]:
|
||||
raw_episodes = payload.get("episodes")
|
||||
if not isinstance(raw_episodes, list):
|
||||
raise ValueError("episodes_missing_or_not_list")
|
||||
|
||||
valid_hashes = set(input_hashes)
|
||||
normalized: List[Dict[str, Any]] = []
|
||||
for item in raw_episodes:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
title = str(item.get("title", "") or "").strip()
|
||||
summary = str(item.get("summary", "") or "").strip()
|
||||
if not title or not summary:
|
||||
continue
|
||||
|
||||
raw_hashes = item.get("paragraph_hashes")
|
||||
if not isinstance(raw_hashes, list):
|
||||
continue
|
||||
|
||||
dedup_hashes: List[str] = []
|
||||
seen_hashes = set()
|
||||
for h in raw_hashes:
|
||||
token = str(h or "").strip()
|
||||
if not token or token in seen_hashes or token not in valid_hashes:
|
||||
continue
|
||||
seen_hashes.add(token)
|
||||
dedup_hashes.append(token)
|
||||
|
||||
if not dedup_hashes:
|
||||
continue
|
||||
|
||||
participants = []
|
||||
for p in item.get("participants", []) or []:
|
||||
token = str(p or "").strip()
|
||||
if token:
|
||||
participants.append(token)
|
||||
|
||||
keywords = []
|
||||
for kw in item.get("keywords", []) or []:
|
||||
token = str(kw or "").strip()
|
||||
if token:
|
||||
keywords.append(token)
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"title": title,
|
||||
"summary": summary,
|
||||
"paragraph_hashes": dedup_hashes,
|
||||
"participants": participants[:16],
|
||||
"keywords": keywords[:20],
|
||||
"time_confidence": self._clamp_score(item.get("time_confidence"), default=1.0),
|
||||
"llm_confidence": self._clamp_score(item.get("llm_confidence"), default=0.5),
|
||||
}
|
||||
)
|
||||
|
||||
if not normalized:
|
||||
raise ValueError("episodes_all_invalid")
|
||||
return normalized
|
||||
|
||||
async def segment(
|
||||
self,
|
||||
*,
|
||||
source: str,
|
||||
window_start: Optional[float],
|
||||
window_end: Optional[float],
|
||||
paragraphs: List[Dict[str, Any]],
|
||||
) -> Dict[str, Any]:
|
||||
if not paragraphs:
|
||||
raise ValueError("paragraphs_empty")
|
||||
|
||||
model_config, model_label = self._resolve_model_config()
|
||||
if model_config is None:
|
||||
raise RuntimeError("episode segmentation model unavailable")
|
||||
|
||||
prompt = self._build_prompt(
|
||||
source=source,
|
||||
window_start=window_start,
|
||||
window_end=window_end,
|
||||
paragraphs=paragraphs,
|
||||
)
|
||||
success, response, _, _ = await llm_api.generate_with_model(
|
||||
prompt=prompt,
|
||||
model_config=model_config,
|
||||
request_type="A_Memorix.EpisodeSegmentation",
|
||||
)
|
||||
if not success or not response:
|
||||
raise RuntimeError("llm_generate_failed")
|
||||
|
||||
payload = self._safe_json_loads(str(response))
|
||||
input_hashes = [str(p.get("hash", "") or "").strip() for p in paragraphs]
|
||||
episodes = self._normalize_episodes(payload=payload, input_hashes=input_hashes)
|
||||
|
||||
return {
|
||||
"episodes": episodes,
|
||||
"segmentation_model": model_label,
|
||||
"segmentation_version": self.SEGMENTATION_VERSION,
|
||||
}
|
||||
|
||||
558
plugins/A_memorix/core/utils/episode_service.py
Normal file
558
plugins/A_memorix/core/utils/episode_service.py
Normal file
@@ -0,0 +1,558 @@
|
||||
"""
|
||||
Episode 聚合与落库服务。
|
||||
|
||||
流程：
|
||||
1. 从 pending 队列读取段落并组批
|
||||
2. 按 source + 时间窗口切组
|
||||
3. 调用 LLM 语义切分
|
||||
4. 写入 episodes + episode_paragraphs
|
||||
5. LLM 失败时使用确定性 fallback
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from collections import Counter
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from src.common.logger import get_logger
|
||||
|
||||
from .episode_segmentation_service import EpisodeSegmentationService
|
||||
from .hash import compute_hash
|
||||
|
||||
logger = get_logger("A_Memorix.EpisodeService")
|
||||
|
||||
|
||||
class EpisodeService:
|
||||
"""Episode MVP 后台处理服务。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
metadata_store: Any,
|
||||
plugin_config: Optional[Any] = None,
|
||||
segmentation_service: Optional[EpisodeSegmentationService] = None,
|
||||
):
|
||||
self.metadata_store = metadata_store
|
||||
self.plugin_config = plugin_config or {}
|
||||
self.segmentation_service = segmentation_service or EpisodeSegmentationService(
|
||||
plugin_config=self._config_dict(),
|
||||
)
|
||||
|
||||
def _config_dict(self) -> Dict[str, Any]:
|
||||
if isinstance(self.plugin_config, dict):
|
||||
return self.plugin_config
|
||||
return {}
|
||||
|
||||
def _cfg(self, key: str, default: Any = None) -> Any:
|
||||
getter = getattr(self.plugin_config, "get_config", None)
|
||||
if callable(getter):
|
||||
return getter(key, default)
|
||||
|
||||
current: Any = self.plugin_config
|
||||
for part in key.split("."):
|
||||
if isinstance(current, dict) and part in current:
|
||||
current = current[part]
|
||||
else:
|
||||
return default
|
||||
return current
|
||||
|
||||
@staticmethod
|
||||
def _to_optional_float(value: Any) -> Optional[float]:
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _clamp_score(value: Any, default: float = 1.0) -> float:
|
||||
try:
|
||||
num = float(value)
|
||||
except Exception:
|
||||
num = default
|
||||
if num < 0.0:
|
||||
return 0.0
|
||||
if num > 1.0:
|
||||
return 1.0
|
||||
return num
|
||||
|
||||
@staticmethod
|
||||
def _paragraph_anchor(paragraph: Dict[str, Any]) -> float:
|
||||
for key in ("event_time_end", "event_time_start", "event_time", "created_at"):
|
||||
value = paragraph.get(key)
|
||||
try:
|
||||
if value is not None:
|
||||
return float(value)
|
||||
except Exception:
|
||||
continue
|
||||
return 0.0
|
||||
|
||||
@staticmethod
|
||||
def _paragraph_sort_key(paragraph: Dict[str, Any]) -> Tuple[float, str]:
|
||||
return (
|
||||
EpisodeService._paragraph_anchor(paragraph),
|
||||
str(paragraph.get("hash", "") or ""),
|
||||
)
|
||||
|
||||
def load_pending_paragraphs(
|
||||
self,
|
||||
pending_rows: List[Dict[str, Any]],
|
||||
) -> Tuple[List[Dict[str, Any]], List[str]]:
|
||||
"""
|
||||
将 pending 行展开为段落上下文。
|
||||
|
||||
Returns:
|
||||
(loaded_paragraphs, missing_hashes)
|
||||
"""
|
||||
loaded: List[Dict[str, Any]] = []
|
||||
missing: List[str] = []
|
||||
for row in pending_rows or []:
|
||||
p_hash = str(row.get("paragraph_hash", "") or "").strip()
|
||||
if not p_hash:
|
||||
continue
|
||||
|
||||
paragraph = self.metadata_store.get_paragraph(p_hash)
|
||||
if not paragraph:
|
||||
missing.append(p_hash)
|
||||
continue
|
||||
|
||||
loaded.append(
|
||||
{
|
||||
"hash": p_hash,
|
||||
"source": str(row.get("source") or paragraph.get("source") or "").strip(),
|
||||
"content": str(paragraph.get("content", "") or ""),
|
||||
"created_at": self._to_optional_float(paragraph.get("created_at"))
|
||||
or self._to_optional_float(row.get("created_at"))
|
||||
or 0.0,
|
||||
"event_time": self._to_optional_float(paragraph.get("event_time")),
|
||||
"event_time_start": self._to_optional_float(paragraph.get("event_time_start")),
|
||||
"event_time_end": self._to_optional_float(paragraph.get("event_time_end")),
|
||||
"time_granularity": str(paragraph.get("time_granularity", "") or "").strip() or None,
|
||||
"time_confidence": self._clamp_score(paragraph.get("time_confidence"), default=1.0),
|
||||
}
|
||||
)
|
||||
return loaded, missing
|
||||
|
||||
def group_paragraphs(self, paragraphs: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
按 source + 时间邻近窗口组批，并受段落数/字符数上限约束。
|
||||
"""
|
||||
if not paragraphs:
|
||||
return []
|
||||
|
||||
max_paragraphs = max(1, int(self._cfg("episode.max_paragraphs_per_call", 20)))
|
||||
max_chars = max(200, int(self._cfg("episode.max_chars_per_call", 6000)))
|
||||
window_seconds = max(
|
||||
60.0,
|
||||
float(self._cfg("episode.source_time_window_hours", 24)) * 3600.0,
|
||||
)
|
||||
|
||||
by_source: Dict[str, List[Dict[str, Any]]] = {}
|
||||
for paragraph in paragraphs:
|
||||
source = str(paragraph.get("source", "") or "").strip()
|
||||
by_source.setdefault(source, []).append(paragraph)
|
||||
|
||||
groups: List[Dict[str, Any]] = []
|
||||
for source, items in by_source.items():
|
||||
ordered = sorted(items, key=self._paragraph_sort_key)
|
||||
|
||||
current: List[Dict[str, Any]] = []
|
||||
current_chars = 0
|
||||
last_anchor: Optional[float] = None
|
||||
|
||||
def flush() -> None:
|
||||
nonlocal current, current_chars, last_anchor
|
||||
if not current:
|
||||
return
|
||||
sorted_current = sorted(current, key=self._paragraph_sort_key)
|
||||
groups.append(
|
||||
{
|
||||
"source": source,
|
||||
"paragraphs": sorted_current,
|
||||
}
|
||||
)
|
||||
current = []
|
||||
current_chars = 0
|
||||
last_anchor = None
|
||||
|
||||
for paragraph in ordered:
|
||||
anchor = self._paragraph_anchor(paragraph)
|
||||
content_len = len(str(paragraph.get("content", "") or ""))
|
||||
|
||||
need_flush = False
|
||||
if current:
|
||||
if len(current) >= max_paragraphs:
|
||||
need_flush = True
|
||||
elif current_chars + content_len > max_chars:
|
||||
need_flush = True
|
||||
elif last_anchor is not None and abs(anchor - last_anchor) > window_seconds:
|
||||
need_flush = True
|
||||
|
||||
if need_flush:
|
||||
flush()
|
||||
|
||||
current.append(paragraph)
|
||||
current_chars += content_len
|
||||
last_anchor = anchor
|
||||
|
||||
flush()
|
||||
|
||||
groups.sort(
|
||||
key=lambda g: self._paragraph_anchor(g["paragraphs"][0]) if g.get("paragraphs") else 0.0
|
||||
)
|
||||
return groups
|
||||
|
||||
def _compute_time_meta(self, paragraphs: List[Dict[str, Any]]) -> Tuple[Optional[float], Optional[float], Optional[str], float]:
|
||||
starts: List[float] = []
|
||||
ends: List[float] = []
|
||||
granularity_priority = {
|
||||
"minute": 4,
|
||||
"hour": 3,
|
||||
"day": 2,
|
||||
"month": 1,
|
||||
"year": 0,
|
||||
}
|
||||
granularity = None
|
||||
granularity_rank = -1
|
||||
conf_values: List[float] = []
|
||||
|
||||
for p in paragraphs:
|
||||
s = self._to_optional_float(p.get("event_time_start"))
|
||||
e = self._to_optional_float(p.get("event_time_end"))
|
||||
t = self._to_optional_float(p.get("event_time"))
|
||||
c = self._to_optional_float(p.get("created_at"))
|
||||
|
||||
start_candidate = s if s is not None else (t if t is not None else (e if e is not None else c))
|
||||
end_candidate = e if e is not None else (t if t is not None else (s if s is not None else c))
|
||||
|
||||
if start_candidate is not None:
|
||||
starts.append(start_candidate)
|
||||
if end_candidate is not None:
|
||||
ends.append(end_candidate)
|
||||
|
||||
g = str(p.get("time_granularity", "") or "").strip().lower()
|
||||
if g in granularity_priority and granularity_priority[g] > granularity_rank:
|
||||
granularity_rank = granularity_priority[g]
|
||||
granularity = g
|
||||
|
||||
conf_values.append(self._clamp_score(p.get("time_confidence"), default=1.0))
|
||||
|
||||
time_start = min(starts) if starts else None
|
||||
time_end = max(ends) if ends else None
|
||||
time_conf = sum(conf_values) / len(conf_values) if conf_values else 1.0
|
||||
return time_start, time_end, granularity, self._clamp_score(time_conf, default=1.0)
|
||||
|
||||
def _collect_participants(self, paragraph_hashes: List[str], limit: int = 16) -> List[str]:
|
||||
seen = set()
|
||||
participants: List[str] = []
|
||||
for p_hash in paragraph_hashes:
|
||||
try:
|
||||
entities = self.metadata_store.get_paragraph_entities(p_hash)
|
||||
except Exception:
|
||||
entities = []
|
||||
for item in entities:
|
||||
name = str(item.get("name", "") or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
key = name.lower()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
participants.append(name)
|
||||
if len(participants) >= limit:
|
||||
return participants
|
||||
return participants
|
||||
|
||||
@staticmethod
|
||||
def _derive_keywords(paragraphs: List[Dict[str, Any]], limit: int = 12) -> List[str]:
|
||||
token_counter: Counter[str] = Counter()
|
||||
token_pattern = re.compile(r"[A-Za-z0-9_\u4e00-\u9fff]{2,}")
|
||||
stop_words = {
|
||||
"the",
|
||||
"and",
|
||||
"that",
|
||||
"this",
|
||||
"with",
|
||||
"from",
|
||||
"for",
|
||||
"have",
|
||||
"will",
|
||||
"your",
|
||||
"you",
|
||||
"我们",
|
||||
"你们",
|
||||
"他们",
|
||||
"以及",
|
||||
"一个",
|
||||
"这个",
|
||||
"那个",
|
||||
"然后",
|
||||
"因为",
|
||||
"所以",
|
||||
}
|
||||
for p in paragraphs:
|
||||
text = str(p.get("content", "") or "").lower()
|
||||
for token in token_pattern.findall(text):
|
||||
if token in stop_words:
|
||||
continue
|
||||
token_counter[token] += 1
|
||||
|
||||
return [token for token, _ in token_counter.most_common(limit)]
|
||||
|
||||
def _build_fallback_episode(self, group: Dict[str, Any]) -> Dict[str, Any]:
|
||||
paragraphs = group.get("paragraphs", []) or []
|
||||
source = str(group.get("source", "") or "").strip()
|
||||
hashes = [str(p.get("hash", "") or "").strip() for p in paragraphs if str(p.get("hash", "") or "").strip()]
|
||||
snippets = []
|
||||
for p in paragraphs[:3]:
|
||||
text = str(p.get("content", "") or "").strip().replace("\n", " ")
|
||||
if text:
|
||||
snippets.append(text[:140])
|
||||
summary = "；".join(snippets)[:500] if snippets else "自动回退生成的情景记忆。"
|
||||
|
||||
time_start, time_end, granularity, time_conf = self._compute_time_meta(paragraphs)
|
||||
participants = self._collect_participants(hashes, limit=12)
|
||||
keywords = self._derive_keywords(paragraphs, limit=10)
|
||||
|
||||
if time_start is not None:
|
||||
day_text = datetime.fromtimestamp(time_start).strftime("%Y-%m-%d")
|
||||
title = f"{source or 'unknown'} {day_text} 情景片段"
|
||||
else:
|
||||
title = f"{source or 'unknown'} 情景片段"
|
||||
|
||||
return {
|
||||
"title": title[:80],
|
||||
"summary": summary,
|
||||
"paragraph_hashes": hashes,
|
||||
"participants": participants,
|
||||
"keywords": keywords,
|
||||
"time_confidence": time_conf,
|
||||
"llm_confidence": 0.0,
|
||||
"event_time_start": time_start,
|
||||
"event_time_end": time_end,
|
||||
"time_granularity": granularity,
|
||||
"segmentation_model": "fallback_rule",
|
||||
"segmentation_version": EpisodeSegmentationService.SEGMENTATION_VERSION,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _normalize_episode_hashes(episode_hashes: List[str], group_hashes_ordered: List[str]) -> List[str]:
|
||||
in_group = set(group_hashes_ordered)
|
||||
dedup: List[str] = []
|
||||
seen = set()
|
||||
for h in episode_hashes or []:
|
||||
token = str(h or "").strip()
|
||||
if not token or token not in in_group or token in seen:
|
||||
continue
|
||||
seen.add(token)
|
||||
dedup.append(token)
|
||||
return dedup
|
||||
|
||||
async def _build_episode_payloads_for_group(self, group: Dict[str, Any]) -> Dict[str, Any]:
|
||||
paragraphs = group.get("paragraphs", []) or []
|
||||
if not paragraphs:
|
||||
return {
|
||||
"payloads": [],
|
||||
"done_hashes": [],
|
||||
"episode_count": 0,
|
||||
"fallback_count": 0,
|
||||
}
|
||||
|
||||
source = str(group.get("source", "") or "").strip()
|
||||
group_hashes = [str(p.get("hash", "") or "").strip() for p in paragraphs if str(p.get("hash", "") or "").strip()]
|
||||
group_start, group_end, _, _ = self._compute_time_meta(paragraphs)
|
||||
|
||||
fallback_used = False
|
||||
segmentation_model = "fallback_rule"
|
||||
segmentation_version = EpisodeSegmentationService.SEGMENTATION_VERSION
|
||||
|
||||
try:
|
||||
llm_result = await self.segmentation_service.segment(
|
||||
source=source,
|
||||
window_start=group_start,
|
||||
window_end=group_end,
|
||||
paragraphs=paragraphs,
|
||||
)
|
||||
episodes = list(llm_result.get("episodes") or [])
|
||||
segmentation_model = str(llm_result.get("segmentation_model", "") or "").strip() or "auto"
|
||||
segmentation_version = str(llm_result.get("segmentation_version", "") or "").strip() or EpisodeSegmentationService.SEGMENTATION_VERSION
|
||||
if not episodes:
|
||||
raise ValueError("llm_empty_episodes")
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Episode segmentation fallback: "
|
||||
f"source={source} "
|
||||
f"size={len(group_hashes)} "
|
||||
f"err={e}"
|
||||
)
|
||||
episodes = [self._build_fallback_episode(group)]
|
||||
fallback_used = True
|
||||
|
||||
stored_payloads: List[Dict[str, Any]] = []
|
||||
for episode in episodes:
|
||||
ordered_hashes = self._normalize_episode_hashes(
|
||||
episode_hashes=episode.get("paragraph_hashes", []),
|
||||
group_hashes_ordered=group_hashes,
|
||||
)
|
||||
if not ordered_hashes:
|
||||
continue
|
||||
|
||||
sub_paragraphs = [p for p in paragraphs if str(p.get("hash", "") or "") in set(ordered_hashes)]
|
||||
event_start, event_end, granularity, time_conf_default = self._compute_time_meta(sub_paragraphs)
|
||||
|
||||
participants = [str(x).strip() for x in (episode.get("participants", []) or []) if str(x).strip()]
|
||||
keywords = [str(x).strip() for x in (episode.get("keywords", []) or []) if str(x).strip()]
|
||||
if not participants:
|
||||
participants = self._collect_participants(ordered_hashes, limit=16)
|
||||
if not keywords:
|
||||
keywords = self._derive_keywords(sub_paragraphs, limit=12)
|
||||
|
||||
title = str(episode.get("title", "") or "").strip()[:120]
|
||||
summary = str(episode.get("summary", "") or "").strip()[:2000]
|
||||
if not title or not summary:
|
||||
continue
|
||||
|
||||
seed = json.dumps(
|
||||
{
|
||||
"source": source,
|
||||
"hashes": ordered_hashes,
|
||||
"version": segmentation_version,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
)
|
||||
episode_id = compute_hash(seed)
|
||||
|
||||
payload = {
|
||||
"episode_id": episode_id,
|
||||
"source": source or None,
|
||||
"title": title,
|
||||
"summary": summary,
|
||||
"event_time_start": episode.get("event_time_start", event_start),
|
||||
"event_time_end": episode.get("event_time_end", event_end),
|
||||
"time_granularity": episode.get("time_granularity", granularity),
|
||||
"time_confidence": self._clamp_score(
|
||||
episode.get("time_confidence"),
|
||||
default=time_conf_default,
|
||||
),
|
||||
"participants": participants[:16],
|
||||
"keywords": keywords[:20],
|
||||
"evidence_ids": ordered_hashes,
|
||||
"paragraph_count": len(ordered_hashes),
|
||||
"llm_confidence": self._clamp_score(
|
||||
episode.get("llm_confidence"),
|
||||
default=0.0 if fallback_used else 0.6,
|
||||
),
|
||||
"segmentation_model": (
|
||||
str(episode.get("segmentation_model", "") or "").strip()
|
||||
or ("fallback_rule" if fallback_used else segmentation_model)
|
||||
),
|
||||
"segmentation_version": (
|
||||
str(episode.get("segmentation_version", "") or "").strip()
|
||||
or segmentation_version
|
||||
),
|
||||
}
|
||||
stored_payloads.append(payload)
|
||||
|
||||
return {
|
||||
"payloads": stored_payloads,
|
||||
"done_hashes": group_hashes,
|
||||
"episode_count": len(stored_payloads),
|
||||
"fallback_count": 1 if fallback_used else 0,
|
||||
}
|
||||
|
||||
async def process_group(self, group: Dict[str, Any]) -> Dict[str, Any]:
|
||||
result = await self._build_episode_payloads_for_group(group)
|
||||
stored_count = 0
|
||||
for payload in result.get("payloads") or []:
|
||||
stored = self.metadata_store.upsert_episode(payload)
|
||||
final_id = str(stored.get("episode_id") or payload.get("episode_id") or "")
|
||||
if final_id:
|
||||
self.metadata_store.bind_episode_paragraphs(
|
||||
final_id,
|
||||
list(payload.get("evidence_ids") or []),
|
||||
)
|
||||
stored_count += 1
|
||||
|
||||
result["episode_count"] = stored_count
|
||||
return {
|
||||
"done_hashes": list(result.get("done_hashes") or []),
|
||||
"episode_count": stored_count,
|
||||
"fallback_count": int(result.get("fallback_count") or 0),
|
||||
}
|
||||
|
||||
async def process_pending_rows(self, pending_rows: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
loaded, missing_hashes = self.load_pending_paragraphs(pending_rows)
|
||||
groups = self.group_paragraphs(loaded)
|
||||
|
||||
done_hashes: List[str] = list(missing_hashes)
|
||||
failed_hashes: Dict[str, str] = {}
|
||||
episode_count = 0
|
||||
fallback_count = 0
|
||||
|
||||
for group in groups:
|
||||
group_hashes = [str(p.get("hash", "") or "").strip() for p in (group.get("paragraphs") or [])]
|
||||
try:
|
||||
result = await self.process_group(group)
|
||||
done_hashes.extend(result.get("done_hashes") or [])
|
||||
episode_count += int(result.get("episode_count") or 0)
|
||||
fallback_count += int(result.get("fallback_count") or 0)
|
||||
except Exception as e:
|
||||
err = str(e)[:500]
|
||||
for h in group_hashes:
|
||||
if h:
|
||||
failed_hashes[h] = err
|
||||
|
||||
dedup_done = list(dict.fromkeys([h for h in done_hashes if h]))
|
||||
return {
|
||||
"done_hashes": dedup_done,
|
||||
"failed_hashes": failed_hashes,
|
||||
"episode_count": episode_count,
|
||||
"fallback_count": fallback_count,
|
||||
"missing_count": len(missing_hashes),
|
||||
"group_count": len(groups),
|
||||
}
|
||||
|
||||
async def rebuild_source(self, source: str) -> Dict[str, Any]:
|
||||
token = str(source or "").strip()
|
||||
if not token:
|
||||
return {
|
||||
"source": "",
|
||||
"episode_count": 0,
|
||||
"fallback_count": 0,
|
||||
"group_count": 0,
|
||||
"paragraph_count": 0,
|
||||
}
|
||||
|
||||
paragraphs = self.metadata_store.get_live_paragraphs_by_source(token)
|
||||
if not paragraphs:
|
||||
replace_result = self.metadata_store.replace_episodes_for_source(token, [])
|
||||
return {
|
||||
"source": token,
|
||||
"episode_count": int(replace_result.get("episode_count") or 0),
|
||||
"fallback_count": 0,
|
||||
"group_count": 0,
|
||||
"paragraph_count": 0,
|
||||
}
|
||||
|
||||
groups = self.group_paragraphs(paragraphs)
|
||||
payloads: List[Dict[str, Any]] = []
|
||||
fallback_count = 0
|
||||
|
||||
for group in groups:
|
||||
result = await self._build_episode_payloads_for_group(group)
|
||||
payloads.extend(list(result.get("payloads") or []))
|
||||
fallback_count += int(result.get("fallback_count") or 0)
|
||||
|
||||
replace_result = self.metadata_store.replace_episodes_for_source(token, payloads)
|
||||
return {
|
||||
"source": token,
|
||||
"episode_count": int(replace_result.get("episode_count") or 0),
|
||||
"fallback_count": fallback_count,
|
||||
"group_count": len(groups),
|
||||
"paragraph_count": len(paragraphs),
|
||||
}
|
||||
@@ -9,7 +9,11 @@ import json
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlmodel import select
|
||||
|
||||
from src.common.logger import get_logger
|
||||
from src.common.database.database import get_db_session
|
||||
from src.common.database.database_model import PersonInfo
|
||||
|
||||
from ..embedding import EmbeddingAPIAdapter
|
||||
@@ -120,31 +124,40 @@ class PersonProfileService:
|
||||
if not key:
|
||||
return ""
|
||||
|
||||
try:
|
||||
with get_db_session(auto_commit=False) as session:
|
||||
record = session.exec(
|
||||
select(PersonInfo.person_id).where(PersonInfo.person_id == key).limit(1)
|
||||
).first()
|
||||
if record:
|
||||
return str(record)
|
||||
|
||||
record = session.exec(
|
||||
select(PersonInfo.person_id)
|
||||
.where(
|
||||
or_(
|
||||
PersonInfo.person_name == key,
|
||||
PersonInfo.user_nickname == key,
|
||||
)
|
||||
)
|
||||
.limit(1)
|
||||
).first()
|
||||
if record:
|
||||
return str(record)
|
||||
|
||||
record = session.exec(
|
||||
select(PersonInfo.person_id)
|
||||
.where(PersonInfo.group_cardname.contains(key))
|
||||
.limit(1)
|
||||
).first()
|
||||
if record:
|
||||
return str(record)
|
||||
except Exception as e:
|
||||
logger.warning(f"按别名解析 person_id 失败: identifier={key}, err={e}")
|
||||
|
||||
if len(key) == 32 and all(ch in "0123456789abcdefABCDEF" for ch in key):
|
||||
return key.lower()
|
||||
|
||||
try:
|
||||
record = (
|
||||
PersonInfo.select(PersonInfo.person_id)
|
||||
.where((PersonInfo.person_name == key) | (PersonInfo.nickname == key))
|
||||
.first()
|
||||
)
|
||||
if record and record.person_id:
|
||||
return str(record.person_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
record = (
|
||||
PersonInfo.select(PersonInfo.person_id)
|
||||
.where(PersonInfo.group_nick_name.contains(key))
|
||||
.first()
|
||||
)
|
||||
if record and record.person_id:
|
||||
return str(record.person_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return ""
|
||||
|
||||
def _parse_group_nicks(self, raw_value: Any) -> List[str]:
|
||||
@@ -160,7 +173,7 @@ class PersonProfileService:
|
||||
names: List[str] = []
|
||||
for item in items:
|
||||
if isinstance(item, dict):
|
||||
value = str(item.get("group_nick_name", "")).strip()
|
||||
value = str(item.get("group_cardname") or item.get("group_nick_name") or "").strip()
|
||||
if value:
|
||||
names.append(value)
|
||||
elif isinstance(item, str):
|
||||
@@ -193,6 +206,42 @@ class PersonProfileService:
|
||||
traits.append(text)
|
||||
return traits[:10]
|
||||
|
||||
def _recover_aliases_from_memory(self, person_id: str) -> Tuple[List[str], str]:
|
||||
"""当人物主档案缺失时，从已有记忆证据里回捞可用别名。"""
|
||||
if not person_id:
|
||||
return [], ""
|
||||
|
||||
aliases: List[str] = []
|
||||
primary_name = ""
|
||||
seen = set()
|
||||
|
||||
try:
|
||||
paragraphs = self.metadata_store.get_paragraphs_by_entity(person_id)
|
||||
except Exception as e:
|
||||
logger.warning(f"从记忆证据回捞人物别名失败: person_id={person_id}, err={e}")
|
||||
return [], ""
|
||||
|
||||
for paragraph in paragraphs[:20]:
|
||||
paragraph_hash = str(paragraph.get("hash", "") or "").strip()
|
||||
if not paragraph_hash:
|
||||
continue
|
||||
try:
|
||||
paragraph_entities = self.metadata_store.get_paragraph_entities(paragraph_hash)
|
||||
except Exception:
|
||||
paragraph_entities = []
|
||||
for entity in paragraph_entities:
|
||||
name = str(entity.get("name", "") or "").strip()
|
||||
if not name or name == person_id:
|
||||
continue
|
||||
key = name.lower()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
aliases.append(name)
|
||||
if not primary_name:
|
||||
primary_name = name
|
||||
return aliases, primary_name
|
||||
|
||||
def get_person_aliases(self, person_id: str) -> Tuple[List[str], str, List[str]]:
|
||||
"""获取人物别名集合、主展示名、记忆特征。"""
|
||||
aliases: List[str] = []
|
||||
@@ -200,18 +249,28 @@ class PersonProfileService:
|
||||
memory_traits: List[str] = []
|
||||
if not person_id:
|
||||
return aliases, primary_name, memory_traits
|
||||
recovered_aliases, recovered_primary_name = self._recover_aliases_from_memory(person_id)
|
||||
try:
|
||||
record = PersonInfo.get_or_none(PersonInfo.person_id == person_id)
|
||||
if not record:
|
||||
return aliases, primary_name, memory_traits
|
||||
with get_db_session(auto_commit=False) as session:
|
||||
record = session.exec(
|
||||
select(PersonInfo).where(PersonInfo.person_id == person_id).limit(1)
|
||||
).first()
|
||||
if not record:
|
||||
return recovered_aliases, recovered_primary_name or person_id, memory_traits
|
||||
person_name = str(getattr(record, "person_name", "") or "").strip()
|
||||
nickname = str(getattr(record, "nickname", "") or "").strip()
|
||||
group_nicks = self._parse_group_nicks(getattr(record, "group_nick_name", None))
|
||||
nickname = str(getattr(record, "user_nickname", "") or "").strip()
|
||||
group_nicks = self._parse_group_nicks(getattr(record, "group_cardname", None))
|
||||
memory_traits = self._parse_memory_traits(getattr(record, "memory_points", None))
|
||||
|
||||
primary_name = person_name or nickname or str(getattr(record, "user_id", "") or "").strip() or person_id
|
||||
primary_name = (
|
||||
person_name
|
||||
or nickname
|
||||
or recovered_primary_name
|
||||
or str(getattr(record, "user_id", "") or "").strip()
|
||||
or person_id
|
||||
)
|
||||
|
||||
candidates = [person_name, nickname] + group_nicks
|
||||
candidates = [person_name, nickname] + group_nicks + recovered_aliases
|
||||
seen = set()
|
||||
for item in candidates:
|
||||
norm = str(item or "").strip()
|
||||
|
||||
@@ -82,8 +82,9 @@ class RelationWriteService:
|
||||
)
|
||||
self.metadata_store.set_relation_vector_state(hash_value, "ready")
|
||||
logger.info(
|
||||
"metric.relation_vector_write_success=1 metric.relation_vector_write_success_count=1 hash=%s",
|
||||
hash_value[:16],
|
||||
"metric.relation_vector_write_success=1 "
|
||||
"metric.relation_vector_write_success_count=1 "
|
||||
f"hash={hash_value[:16]}"
|
||||
)
|
||||
return RelationWriteResult(
|
||||
hash_value=hash_value,
|
||||
@@ -109,9 +110,10 @@ class RelationWriteService:
|
||||
bump_retry=True,
|
||||
)
|
||||
logger.warning(
|
||||
"metric.relation_vector_write_fail=1 metric.relation_vector_write_fail_count=1 hash=%s err=%s",
|
||||
hash_value[:16],
|
||||
err,
|
||||
"metric.relation_vector_write_fail=1 "
|
||||
"metric.relation_vector_write_fail_count=1 "
|
||||
f"hash={hash_value[:16]} "
|
||||
f"err={err}"
|
||||
)
|
||||
return RelationWriteResult(
|
||||
hash_value=hash_value,
|
||||
|
||||
1857
plugins/A_memorix/core/utils/retrieval_tuning_manager.py
Normal file
1857
plugins/A_memorix/core/utils/retrieval_tuning_manager.py
Normal file
File diff suppressed because it is too large
Load Diff
@@ -61,6 +61,29 @@ def _build_report(
|
||||
}
|
||||
|
||||
|
||||
def _normalize_encoded_vector(encoded: Any) -> np.ndarray:
|
||||
if encoded is None:
|
||||
raise ValueError("embedding encode returned None")
|
||||
|
||||
if isinstance(encoded, np.ndarray):
|
||||
array = encoded
|
||||
else:
|
||||
array = np.asarray(encoded, dtype=np.float32)
|
||||
|
||||
if array.ndim == 2:
|
||||
if array.shape[0] != 1:
|
||||
raise ValueError(f"embedding encode returned batched output: shape={tuple(array.shape)}")
|
||||
array = array[0]
|
||||
|
||||
if array.ndim != 1:
|
||||
raise ValueError(f"embedding encode returned invalid ndim={array.ndim}")
|
||||
if array.size <= 0:
|
||||
raise ValueError("embedding encode returned empty vector")
|
||||
if not np.all(np.isfinite(array)):
|
||||
raise ValueError("embedding encode returned non-finite values")
|
||||
return array.astype(np.float32, copy=False)
|
||||
|
||||
|
||||
async def run_embedding_runtime_self_check(
|
||||
*,
|
||||
config: Any,
|
||||
@@ -91,13 +114,11 @@ async def run_embedding_runtime_self_check(
|
||||
try:
|
||||
detected_dimension = _safe_int(await embedding_manager._detect_dimension(), 0)
|
||||
encoded = await embedding_manager.encode(sample_text)
|
||||
if isinstance(encoded, np.ndarray):
|
||||
encoded_dimension = int(encoded.shape[0]) if encoded.ndim == 1 else int(encoded.shape[-1])
|
||||
else:
|
||||
encoded_dimension = len(encoded) if encoded is not None else 0
|
||||
encoded_array = _normalize_encoded_vector(encoded)
|
||||
encoded_dimension = int(encoded_array.shape[0])
|
||||
except Exception as exc:
|
||||
elapsed_ms = (time.perf_counter() - start) * 1000.0
|
||||
logger.warning("embedding runtime self-check failed: %s", exc)
|
||||
logger.warning(f"embedding runtime self-check failed: {exc}")
|
||||
return _build_report(
|
||||
ok=False,
|
||||
code="embedding_probe_failed",
|
||||
|
||||
442
plugins/A_memorix/core/utils/search_execution_service.py
Normal file
442
plugins/A_memorix/core/utils/search_execution_service.py
Normal file
@@ -0,0 +1,442 @@
|
||||
"""
|
||||
统一检索执行服务。
|
||||
|
||||
用于收敛 Action/Tool 在 search/time 上的核心执行流程，避免重复实现。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from src.common.logger import get_logger
|
||||
|
||||
from ..retrieval import TemporalQueryOptions
|
||||
from .search_postprocess import (
|
||||
apply_safe_content_dedup,
|
||||
maybe_apply_smart_path_fallback,
|
||||
)
|
||||
from .time_parser import parse_query_time_range
|
||||
|
||||
logger = get_logger("A_Memorix.SearchExecutionService")
|
||||
|
||||
|
||||
def _get_config_value(config: Optional[dict], key: str, default: Any = None) -> Any:
|
||||
if not isinstance(config, dict):
|
||||
return default
|
||||
current: Any = config
|
||||
for part in key.split("."):
|
||||
if isinstance(current, dict) and part in current:
|
||||
current = current[part]
|
||||
else:
|
||||
return default
|
||||
return current
|
||||
|
||||
|
||||
def _sanitize_text(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
|
||||
@dataclass
|
||||
class SearchExecutionRequest:
|
||||
caller: str
|
||||
stream_id: Optional[str] = None
|
||||
group_id: Optional[str] = None
|
||||
user_id: Optional[str] = None
|
||||
query_type: str = "search" # search|semantic|time|hybrid
|
||||
query: str = ""
|
||||
top_k: Optional[int] = None
|
||||
time_from: Optional[str] = None
|
||||
time_to: Optional[str] = None
|
||||
person: Optional[str] = None
|
||||
source: Optional[str] = None
|
||||
use_threshold: bool = True
|
||||
enable_ppr: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class SearchExecutionResult:
|
||||
success: bool
|
||||
error: str = ""
|
||||
query_type: str = "search"
|
||||
query: str = ""
|
||||
top_k: int = 10
|
||||
time_from: Optional[str] = None
|
||||
time_to: Optional[str] = None
|
||||
person: Optional[str] = None
|
||||
source: Optional[str] = None
|
||||
temporal: Optional[TemporalQueryOptions] = None
|
||||
results: List[Any] = field(default_factory=list)
|
||||
elapsed_ms: float = 0.0
|
||||
chat_filtered: bool = False
|
||||
dedup_hit: bool = False
|
||||
|
||||
@property
|
||||
def count(self) -> int:
|
||||
return len(self.results)
|
||||
|
||||
|
||||
class SearchExecutionService:
|
||||
"""统一检索执行服务。"""
|
||||
|
||||
@staticmethod
|
||||
def _resolve_plugin_instance(plugin_config: Optional[dict]) -> Optional[Any]:
|
||||
if isinstance(plugin_config, dict):
|
||||
plugin_instance = plugin_config.get("plugin_instance")
|
||||
if plugin_instance is not None:
|
||||
return plugin_instance
|
||||
|
||||
try:
|
||||
from ...plugin import AMemorixPlugin
|
||||
|
||||
return getattr(AMemorixPlugin, "get_global_instance", lambda: None)()
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _normalize_query_type(raw_query_type: str) -> str:
|
||||
query_type = _sanitize_text(raw_query_type).lower() or "search"
|
||||
if query_type == "semantic":
|
||||
return "search"
|
||||
return query_type
|
||||
|
||||
@staticmethod
|
||||
def _resolve_runtime_component(
|
||||
plugin_config: Optional[dict],
|
||||
plugin_instance: Optional[Any],
|
||||
key: str,
|
||||
) -> Optional[Any]:
|
||||
if isinstance(plugin_config, dict):
|
||||
value = plugin_config.get(key)
|
||||
if value is not None:
|
||||
return value
|
||||
if plugin_instance is not None:
|
||||
value = getattr(plugin_instance, key, None)
|
||||
if value is not None:
|
||||
return value
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _resolve_top_k(
|
||||
plugin_config: Optional[dict],
|
||||
query_type: str,
|
||||
top_k_raw: Optional[Any],
|
||||
) -> Tuple[bool, int, str]:
|
||||
temporal_default_top_k = int(
|
||||
_get_config_value(plugin_config, "retrieval.temporal.default_top_k", 10)
|
||||
)
|
||||
default_top_k = temporal_default_top_k if query_type in {"time", "hybrid"} else 10
|
||||
if top_k_raw is None:
|
||||
return True, max(1, min(50, default_top_k)), ""
|
||||
try:
|
||||
top_k = int(top_k_raw)
|
||||
except (TypeError, ValueError):
|
||||
return False, 0, "top_k 参数必须为整数"
|
||||
return True, max(1, min(50, top_k)), ""
|
||||
|
||||
@staticmethod
|
||||
def _build_temporal(
|
||||
plugin_config: Optional[dict],
|
||||
query_type: str,
|
||||
time_from_raw: Optional[str],
|
||||
time_to_raw: Optional[str],
|
||||
person: Optional[str],
|
||||
source: Optional[str],
|
||||
) -> Tuple[bool, Optional[TemporalQueryOptions], str]:
|
||||
if query_type not in {"time", "hybrid"}:
|
||||
return True, None, ""
|
||||
|
||||
temporal_enabled = bool(_get_config_value(plugin_config, "retrieval.temporal.enabled", True))
|
||||
if not temporal_enabled:
|
||||
return False, None, "时序检索已禁用（retrieval.temporal.enabled=false）"
|
||||
|
||||
if not time_from_raw and not time_to_raw:
|
||||
return False, None, "time/hybrid 模式至少需要 time_from 或 time_to"
|
||||
|
||||
try:
|
||||
ts_from, ts_to = parse_query_time_range(
|
||||
str(time_from_raw) if time_from_raw is not None else None,
|
||||
str(time_to_raw) if time_to_raw is not None else None,
|
||||
)
|
||||
except ValueError as e:
|
||||
return False, None, f"时间参数错误: {e}"
|
||||
|
||||
temporal = TemporalQueryOptions(
|
||||
time_from=ts_from,
|
||||
time_to=ts_to,
|
||||
person=_sanitize_text(person) or None,
|
||||
source=_sanitize_text(source) or None,
|
||||
allow_created_fallback=bool(
|
||||
_get_config_value(plugin_config, "retrieval.temporal.allow_created_fallback", True)
|
||||
),
|
||||
candidate_multiplier=int(
|
||||
_get_config_value(plugin_config, "retrieval.temporal.candidate_multiplier", 8)
|
||||
),
|
||||
max_scan=int(_get_config_value(plugin_config, "retrieval.temporal.max_scan", 1000)),
|
||||
)
|
||||
return True, temporal, ""
|
||||
|
||||
@staticmethod
|
||||
def _build_request_key(
|
||||
request: SearchExecutionRequest,
|
||||
query_type: str,
|
||||
top_k: int,
|
||||
temporal: Optional[TemporalQueryOptions],
|
||||
) -> str:
|
||||
payload = {
|
||||
"stream_id": _sanitize_text(request.stream_id),
|
||||
"query_type": query_type,
|
||||
"query": _sanitize_text(request.query),
|
||||
"time_from": _sanitize_text(request.time_from),
|
||||
"time_to": _sanitize_text(request.time_to),
|
||||
"time_from_ts": temporal.time_from if temporal else None,
|
||||
"time_to_ts": temporal.time_to if temporal else None,
|
||||
"person": _sanitize_text(request.person),
|
||||
"source": _sanitize_text(request.source),
|
||||
"top_k": int(top_k),
|
||||
"use_threshold": bool(request.use_threshold),
|
||||
"enable_ppr": bool(request.enable_ppr),
|
||||
}
|
||||
payload_json = json.dumps(payload, ensure_ascii=False, sort_keys=True)
|
||||
return hashlib.sha1(payload_json.encode("utf-8")).hexdigest()
|
||||
|
||||
@staticmethod
|
||||
async def execute(
|
||||
*,
|
||||
retriever: Any,
|
||||
threshold_filter: Optional[Any],
|
||||
plugin_config: Optional[dict],
|
||||
request: SearchExecutionRequest,
|
||||
enforce_chat_filter: bool = True,
|
||||
reinforce_access: bool = True,
|
||||
) -> SearchExecutionResult:
|
||||
if retriever is None:
|
||||
return SearchExecutionResult(success=False, error="知识检索器未初始化")
|
||||
|
||||
query_type = SearchExecutionService._normalize_query_type(request.query_type)
|
||||
query = _sanitize_text(request.query)
|
||||
if query_type not in {"search", "time", "hybrid"}:
|
||||
return SearchExecutionResult(
|
||||
success=False,
|
||||
error=f"query_type 无效: {query_type}（仅支持 search/time/hybrid）",
|
||||
)
|
||||
|
||||
if query_type in {"search", "hybrid"} and not query:
|
||||
return SearchExecutionResult(
|
||||
success=False,
|
||||
error="search/hybrid 模式必须提供 query",
|
||||
)
|
||||
|
||||
top_k_ok, top_k, top_k_error = SearchExecutionService._resolve_top_k(
|
||||
plugin_config, query_type, request.top_k
|
||||
)
|
||||
if not top_k_ok:
|
||||
return SearchExecutionResult(success=False, error=top_k_error)
|
||||
|
||||
temporal_ok, temporal, temporal_error = SearchExecutionService._build_temporal(
|
||||
plugin_config=plugin_config,
|
||||
query_type=query_type,
|
||||
time_from_raw=request.time_from,
|
||||
time_to_raw=request.time_to,
|
||||
person=request.person,
|
||||
source=request.source,
|
||||
)
|
||||
if not temporal_ok:
|
||||
return SearchExecutionResult(success=False, error=temporal_error)
|
||||
|
||||
plugin_instance = SearchExecutionService._resolve_plugin_instance(plugin_config)
|
||||
if (
|
||||
enforce_chat_filter
|
||||
and plugin_instance is not None
|
||||
and hasattr(plugin_instance, "is_chat_enabled")
|
||||
):
|
||||
if not plugin_instance.is_chat_enabled(
|
||||
stream_id=request.stream_id,
|
||||
group_id=request.group_id,
|
||||
user_id=request.user_id,
|
||||
):
|
||||
logger.info(
|
||||
"检索请求被聊天过滤拦截: "
|
||||
f"caller={request.caller}, "
|
||||
f"stream_id={request.stream_id}"
|
||||
)
|
||||
return SearchExecutionResult(
|
||||
success=True,
|
||||
query_type=query_type,
|
||||
query=query,
|
||||
top_k=top_k,
|
||||
time_from=request.time_from,
|
||||
time_to=request.time_to,
|
||||
person=request.person,
|
||||
source=request.source,
|
||||
temporal=temporal,
|
||||
results=[],
|
||||
elapsed_ms=0.0,
|
||||
chat_filtered=True,
|
||||
dedup_hit=False,
|
||||
)
|
||||
|
||||
request_key = SearchExecutionService._build_request_key(
|
||||
request=request,
|
||||
query_type=query_type,
|
||||
top_k=top_k,
|
||||
temporal=temporal,
|
||||
)
|
||||
|
||||
async def _executor() -> Dict[str, Any]:
|
||||
original_ppr = bool(getattr(retriever.config, "enable_ppr", True))
|
||||
setattr(retriever.config, "enable_ppr", bool(request.enable_ppr))
|
||||
started_at = time.time()
|
||||
try:
|
||||
retrieved = await retriever.retrieve(
|
||||
query=query,
|
||||
top_k=top_k,
|
||||
temporal=temporal,
|
||||
)
|
||||
|
||||
should_apply_threshold = bool(request.use_threshold) and threshold_filter is not None
|
||||
if (
|
||||
query_type == "time"
|
||||
and not query
|
||||
and bool(
|
||||
_get_config_value(
|
||||
plugin_config,
|
||||
"retrieval.time.skip_threshold_when_query_empty",
|
||||
True,
|
||||
)
|
||||
)
|
||||
):
|
||||
should_apply_threshold = False
|
||||
|
||||
if should_apply_threshold:
|
||||
retrieved = threshold_filter.filter(retrieved)
|
||||
|
||||
if (
|
||||
reinforce_access
|
||||
and plugin_instance is not None
|
||||
and hasattr(plugin_instance, "reinforce_access")
|
||||
):
|
||||
relation_hashes = [
|
||||
item.hash_value
|
||||
for item in retrieved
|
||||
if getattr(item, "result_type", "") == "relation"
|
||||
]
|
||||
if relation_hashes:
|
||||
await plugin_instance.reinforce_access(relation_hashes)
|
||||
|
||||
if query_type == "search":
|
||||
graph_store = SearchExecutionService._resolve_runtime_component(
|
||||
plugin_config, plugin_instance, "graph_store"
|
||||
)
|
||||
metadata_store = SearchExecutionService._resolve_runtime_component(
|
||||
plugin_config, plugin_instance, "metadata_store"
|
||||
)
|
||||
fallback_enabled = bool(
|
||||
_get_config_value(
|
||||
plugin_config,
|
||||
"retrieval.search.smart_fallback.enabled",
|
||||
True,
|
||||
)
|
||||
)
|
||||
fallback_threshold = float(
|
||||
_get_config_value(
|
||||
plugin_config,
|
||||
"retrieval.search.smart_fallback.threshold",
|
||||
0.6,
|
||||
)
|
||||
)
|
||||
retrieved, fallback_triggered, fallback_added = maybe_apply_smart_path_fallback(
|
||||
query=query,
|
||||
results=list(retrieved),
|
||||
graph_store=graph_store,
|
||||
metadata_store=metadata_store,
|
||||
enabled=fallback_enabled,
|
||||
threshold=fallback_threshold,
|
||||
)
|
||||
if fallback_triggered:
|
||||
logger.info(
|
||||
"metric.smart_fallback_triggered_count=1 "
|
||||
f"caller={request.caller} "
|
||||
f"added={fallback_added}"
|
||||
)
|
||||
|
||||
dedup_enabled = bool(
|
||||
_get_config_value(
|
||||
plugin_config,
|
||||
"retrieval.search.safe_content_dedup.enabled",
|
||||
True,
|
||||
)
|
||||
)
|
||||
if dedup_enabled:
|
||||
retrieved, removed_count = apply_safe_content_dedup(list(retrieved))
|
||||
if removed_count > 0:
|
||||
logger.info(
|
||||
f"metric.safe_dedup_removed_count={removed_count} "
|
||||
f"caller={request.caller}"
|
||||
)
|
||||
|
||||
elapsed_ms = (time.time() - started_at) * 1000.0
|
||||
return {"results": retrieved, "elapsed_ms": elapsed_ms}
|
||||
finally:
|
||||
setattr(retriever.config, "enable_ppr", original_ppr)
|
||||
|
||||
dedup_hit = False
|
||||
try:
|
||||
# 调优评估需要逐轮真实执行，且应避免额外 dedup 锁竞争。
|
||||
bypass_request_dedup = str(request.caller or "").strip().lower() == "retrieval_tuning"
|
||||
if (
|
||||
not bypass_request_dedup
|
||||
and
|
||||
plugin_instance is not None
|
||||
and hasattr(plugin_instance, "execute_request_with_dedup")
|
||||
):
|
||||
dedup_hit, payload = await plugin_instance.execute_request_with_dedup(
|
||||
request_key,
|
||||
_executor,
|
||||
)
|
||||
else:
|
||||
payload = await _executor()
|
||||
except Exception as e:
|
||||
return SearchExecutionResult(success=False, error=f"知识检索失败: {e}")
|
||||
|
||||
if dedup_hit:
|
||||
logger.info(f"metric.search_execution_dedup_hit_count=1 caller={request.caller}")
|
||||
|
||||
return SearchExecutionResult(
|
||||
success=True,
|
||||
query_type=query_type,
|
||||
query=query,
|
||||
top_k=top_k,
|
||||
time_from=request.time_from,
|
||||
time_to=request.time_to,
|
||||
person=request.person,
|
||||
source=request.source,
|
||||
temporal=temporal,
|
||||
results=payload.get("results", []),
|
||||
elapsed_ms=float(payload.get("elapsed_ms", 0.0)),
|
||||
chat_filtered=False,
|
||||
dedup_hit=bool(dedup_hit),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def to_serializable_results(results: List[Any]) -> List[Dict[str, Any]]:
|
||||
serialized: List[Dict[str, Any]] = []
|
||||
for item in results:
|
||||
metadata = dict(getattr(item, "metadata", {}) or {})
|
||||
if "time_meta" not in metadata:
|
||||
metadata["time_meta"] = {}
|
||||
serialized.append(
|
||||
{
|
||||
"hash": getattr(item, "hash_value", ""),
|
||||
"type": getattr(item, "result_type", ""),
|
||||
"score": float(getattr(item, "score", 0.0)),
|
||||
"content": getattr(item, "content", ""),
|
||||
"metadata": metadata,
|
||||
}
|
||||
)
|
||||
return serialized
|
||||
425
plugins/A_memorix/core/utils/summary_importer.py
Normal file
425
plugins/A_memorix/core/utils/summary_importer.py
Normal file
@@ -0,0 +1,425 @@
|
||||
"""
|
||||
聊天总结与知识导入工具
|
||||
|
||||
该模块负责从聊天记录中提取信息，生成总结，并将总结内容及提取的实体/关系
|
||||
导入到 A_memorix 的存储组件中。
|
||||
"""
|
||||
|
||||
import time
|
||||
import json
|
||||
import re
|
||||
import traceback
|
||||
from typing import List, Dict, Any, Tuple, Optional
|
||||
from pathlib import Path
|
||||
|
||||
from src.common.logger import get_logger
|
||||
from src.services import llm_service as llm_api
|
||||
from src.services import message_service as message_api
|
||||
from src.config.config import global_config, model_config as host_model_config
|
||||
from src.config.model_configs import TaskConfig
|
||||
|
||||
from ..storage import (
|
||||
KnowledgeType,
|
||||
VectorStore,
|
||||
GraphStore,
|
||||
MetadataStore,
|
||||
resolve_stored_knowledge_type,
|
||||
)
|
||||
from ..embedding import EmbeddingAPIAdapter
|
||||
from .relation_write_service import RelationWriteService
|
||||
from .runtime_self_check import ensure_runtime_self_check, run_embedding_runtime_self_check
|
||||
|
||||
logger = get_logger("A_Memorix.SummaryImporter")
|
||||
|
||||
# 默认总结提示词模版
|
||||
SUMMARY_PROMPT_TEMPLATE = """
|
||||
你是 {bot_name}。{personality_context}
|
||||
现在你需要对以下一段聊天记录进行总结，并提取其中的重要知识。
|
||||
|
||||
聊天记录内容：
|
||||
{chat_history}
|
||||
|
||||
请完成以下任务：
|
||||
1. **生成总结**：以第三人称或机器人的视角，简洁明了地总结这段对话的主要内容、发生的事件或讨论的主题。
|
||||
2. **提取实体与关系**：识别并提取对话中提到的重要实体以及它们之间的关系。
|
||||
|
||||
请严格以 JSON 格式输出，格式如下：
|
||||
{{
|
||||
"summary": "总结文本内容",
|
||||
"entities": ["张三", "李四"],
|
||||
"relations": [
|
||||
{{"subject": "张三", "predicate": "认识", "object": "李四"}}
|
||||
]
|
||||
}}
|
||||
|
||||
注意：总结应具有叙事性，能够作为长程记忆的一部分。直接使用实体的实际名称，不要使用 e1/e2 等代号。
|
||||
"""
|
||||
|
||||
class SummaryImporter:
|
||||
"""总结并导入知识的工具类"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vector_store: VectorStore,
|
||||
graph_store: GraphStore,
|
||||
metadata_store: MetadataStore,
|
||||
embedding_manager: EmbeddingAPIAdapter,
|
||||
plugin_config: dict
|
||||
):
|
||||
self.vector_store = vector_store
|
||||
self.graph_store = graph_store
|
||||
self.metadata_store = metadata_store
|
||||
self.embedding_manager = embedding_manager
|
||||
self.plugin_config = plugin_config
|
||||
self.relation_write_service: Optional[RelationWriteService] = (
|
||||
plugin_config.get("relation_write_service")
|
||||
if isinstance(plugin_config, dict)
|
||||
else None
|
||||
)
|
||||
|
||||
def _normalize_summary_model_selectors(self, raw_value: Any) -> List[str]:
|
||||
"""标准化 summarization.model_name 配置（vNext 仅接受字符串数组）。"""
|
||||
if raw_value is None:
|
||||
return ["auto"]
|
||||
if isinstance(raw_value, list):
|
||||
selectors = [str(x).strip() for x in raw_value if str(x).strip()]
|
||||
return selectors or ["auto"]
|
||||
raise ValueError(
|
||||
"summarization.model_name 在 vNext 必须为 List[str]。"
|
||||
" 请执行 scripts/release_vnext_migrate.py migrate。"
|
||||
)
|
||||
|
||||
def _pick_default_summary_task(self, available_tasks: Dict[str, TaskConfig]) -> Tuple[Optional[str], Optional[TaskConfig]]:
|
||||
"""
|
||||
选择总结默认任务，避免错误落到 embedding 任务。
|
||||
优先级：replyer > utils > planner > tool_use > 其他非 embedding。
|
||||
"""
|
||||
preferred = ("replyer", "utils", "planner", "tool_use")
|
||||
for name in preferred:
|
||||
cfg = available_tasks.get(name)
|
||||
if cfg and cfg.model_list:
|
||||
return name, cfg
|
||||
|
||||
for name, cfg in available_tasks.items():
|
||||
if name != "embedding" and cfg.model_list:
|
||||
return name, cfg
|
||||
|
||||
for name, cfg in available_tasks.items():
|
||||
if cfg.model_list:
|
||||
return name, cfg
|
||||
|
||||
return None, None
|
||||
|
||||
def _resolve_summary_model_config(self) -> Optional[TaskConfig]:
|
||||
"""
|
||||
解析 summarization.model_name 为 TaskConfig。
|
||||
支持：
|
||||
- "auto"
|
||||
- "replyer"（任务名）
|
||||
- "some-model-name"（具体模型名）
|
||||
- ["utils:model1", "utils:model2", "replyer"]（数组混合语法）
|
||||
"""
|
||||
available_tasks = llm_api.get_available_models()
|
||||
if not available_tasks:
|
||||
return None
|
||||
|
||||
raw_cfg = self.plugin_config.get("summarization", {}).get("model_name", "auto")
|
||||
selectors = self._normalize_summary_model_selectors(raw_cfg)
|
||||
default_task_name, default_task_cfg = self._pick_default_summary_task(available_tasks)
|
||||
|
||||
selected_models: List[str] = []
|
||||
base_cfg: Optional[TaskConfig] = None
|
||||
model_dict = getattr(host_model_config, "models_dict", {})
|
||||
|
||||
def _append_models(models: List[str]):
|
||||
for model_name in models:
|
||||
if model_name and model_name not in selected_models:
|
||||
selected_models.append(model_name)
|
||||
|
||||
for raw_selector in selectors:
|
||||
selector = raw_selector.strip()
|
||||
if not selector:
|
||||
continue
|
||||
|
||||
if selector.lower() == "auto":
|
||||
if default_task_cfg:
|
||||
_append_models(default_task_cfg.model_list)
|
||||
if base_cfg is None:
|
||||
base_cfg = default_task_cfg
|
||||
continue
|
||||
|
||||
if ":" in selector:
|
||||
task_name, model_name = selector.split(":", 1)
|
||||
task_name = task_name.strip()
|
||||
model_name = model_name.strip()
|
||||
task_cfg = available_tasks.get(task_name)
|
||||
if not task_cfg:
|
||||
logger.warning(f"总结模型选择器 '{selector}' 的任务 '{task_name}' 不存在，已跳过")
|
||||
continue
|
||||
|
||||
if base_cfg is None:
|
||||
base_cfg = task_cfg
|
||||
|
||||
if not model_name or model_name.lower() == "auto":
|
||||
_append_models(task_cfg.model_list)
|
||||
continue
|
||||
|
||||
if model_name in model_dict or model_name in task_cfg.model_list:
|
||||
_append_models([model_name])
|
||||
else:
|
||||
logger.warning(f"总结模型选择器 '{selector}' 的模型 '{model_name}' 不存在，已跳过")
|
||||
continue
|
||||
|
||||
task_cfg = available_tasks.get(selector)
|
||||
if task_cfg:
|
||||
_append_models(task_cfg.model_list)
|
||||
if base_cfg is None:
|
||||
base_cfg = task_cfg
|
||||
continue
|
||||
|
||||
if selector in model_dict:
|
||||
_append_models([selector])
|
||||
continue
|
||||
|
||||
logger.warning(f"总结模型选择器 '{selector}' 无法识别，已跳过")
|
||||
|
||||
if not selected_models:
|
||||
if default_task_cfg:
|
||||
_append_models(default_task_cfg.model_list)
|
||||
if base_cfg is None:
|
||||
base_cfg = default_task_cfg
|
||||
else:
|
||||
first_cfg = next(iter(available_tasks.values()))
|
||||
_append_models(first_cfg.model_list)
|
||||
if base_cfg is None:
|
||||
base_cfg = first_cfg
|
||||
|
||||
if not selected_models:
|
||||
return None
|
||||
|
||||
template_cfg = base_cfg or default_task_cfg or next(iter(available_tasks.values()))
|
||||
return TaskConfig(
|
||||
model_list=selected_models,
|
||||
max_tokens=template_cfg.max_tokens,
|
||||
temperature=template_cfg.temperature,
|
||||
slow_threshold=template_cfg.slow_threshold,
|
||||
selection_strategy=template_cfg.selection_strategy,
|
||||
)
|
||||
|
||||
async def import_from_stream(
|
||||
self,
|
||||
stream_id: str,
|
||||
context_length: Optional[int] = None,
|
||||
include_personality: Optional[bool] = None
|
||||
) -> Tuple[bool, str]:
|
||||
"""
|
||||
从指定的聊天流中提取记录并执行总结导入
|
||||
|
||||
Args:
|
||||
stream_id: 聊天流 ID
|
||||
context_length: 总结的历史消息条数
|
||||
include_personality: 是否包含人设
|
||||
|
||||
Returns:
|
||||
Tuple[bool, str]: (是否成功, 结果消息)
|
||||
"""
|
||||
try:
|
||||
self_check_ok, self_check_msg = await self._ensure_runtime_self_check()
|
||||
if not self_check_ok:
|
||||
return False, f"导入前自检失败: {self_check_msg}"
|
||||
|
||||
# 1. 获取配置
|
||||
if context_length is None:
|
||||
context_length = self.plugin_config.get("summarization", {}).get("context_length", 50)
|
||||
|
||||
if include_personality is None:
|
||||
include_personality = self.plugin_config.get("summarization", {}).get("include_personality", True)
|
||||
|
||||
# 2. 获取历史消息
|
||||
# 获取当前时间之前的消息
|
||||
now = time.time()
|
||||
messages = message_api.get_messages_before_time_in_chat(
|
||||
chat_id=stream_id,
|
||||
timestamp=now,
|
||||
limit=context_length
|
||||
)
|
||||
|
||||
if not messages:
|
||||
return False, "未找到有效的聊天记录进行总结"
|
||||
|
||||
# 转换为可读文本
|
||||
chat_history_text = message_api.build_readable_messages(messages)
|
||||
|
||||
# 3. 准备提示词内容
|
||||
bot_name = global_config.bot.nickname or "机器人"
|
||||
personality_context = ""
|
||||
if include_personality:
|
||||
personality = getattr(global_config.bot, "personality", "")
|
||||
if personality:
|
||||
personality_context = f"你的性格设定是：{personality}"
|
||||
|
||||
# 4. 调用 LLM
|
||||
prompt = SUMMARY_PROMPT_TEMPLATE.format(
|
||||
bot_name=bot_name,
|
||||
personality_context=personality_context,
|
||||
chat_history=chat_history_text
|
||||
)
|
||||
|
||||
model_config_to_use = self._resolve_summary_model_config()
|
||||
if model_config_to_use is None:
|
||||
return False, "未找到可用的总结模型配置"
|
||||
|
||||
logger.info(f"正在为流 {stream_id} 执行总结，消息条数: {len(messages)}")
|
||||
logger.info(f"总结模型候选列表: {model_config_to_use.model_list}")
|
||||
|
||||
success, response, _, _ = await llm_api.generate_with_model(
|
||||
prompt=prompt,
|
||||
model_config=model_config_to_use,
|
||||
request_type="A_Memorix.ChatSummarization"
|
||||
)
|
||||
|
||||
if not success or not response:
|
||||
return False, "LLM 生成总结失败"
|
||||
|
||||
# 5. 解析结果
|
||||
data = self._parse_llm_response(response)
|
||||
if not data or "summary" not in data:
|
||||
return False, "解析 LLM 响应失败或总结为空"
|
||||
|
||||
summary_text = data["summary"]
|
||||
entities = data.get("entities", [])
|
||||
relations = data.get("relations", [])
|
||||
msg_times = [
|
||||
float(getattr(getattr(msg, "timestamp", None), "timestamp", lambda: 0.0)())
|
||||
for msg in messages
|
||||
if getattr(msg, "time", None) is not None
|
||||
]
|
||||
time_meta = {}
|
||||
if msg_times:
|
||||
time_meta = {
|
||||
"event_time_start": min(msg_times),
|
||||
"event_time_end": max(msg_times),
|
||||
"time_granularity": "minute",
|
||||
"time_confidence": 0.95,
|
||||
}
|
||||
|
||||
# 6. 执行导入
|
||||
await self._execute_import(summary_text, entities, relations, stream_id, time_meta=time_meta)
|
||||
|
||||
# 7. 持久化
|
||||
self.vector_store.save()
|
||||
self.graph_store.save()
|
||||
|
||||
result_msg = (
|
||||
f"✅ 总结导入成功\n"
|
||||
f"📝 总结长度: {len(summary_text)}\n"
|
||||
f"📌 提取实体: {len(entities)}\n"
|
||||
f"🔗 提取关系: {len(relations)}"
|
||||
)
|
||||
return True, result_msg
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"总结导入过程中出错: {e}\n{traceback.format_exc()}")
|
||||
return False, f"错误: {str(e)}"
|
||||
|
||||
async def _ensure_runtime_self_check(self) -> Tuple[bool, str]:
|
||||
plugin_instance = self.plugin_config.get("plugin_instance") if isinstance(self.plugin_config, dict) else None
|
||||
if plugin_instance is not None:
|
||||
report = await ensure_runtime_self_check(plugin_instance)
|
||||
else:
|
||||
report = await run_embedding_runtime_self_check(
|
||||
config=self.plugin_config,
|
||||
vector_store=self.vector_store,
|
||||
embedding_manager=self.embedding_manager,
|
||||
)
|
||||
if bool(report.get("ok", False)):
|
||||
return True, ""
|
||||
return (
|
||||
False,
|
||||
f"{report.get('message', 'unknown')} "
|
||||
f"(configured={report.get('configured_dimension', 0)}, "
|
||||
f"store={report.get('vector_store_dimension', 0)}, "
|
||||
f"encoded={report.get('encoded_dimension', 0)})",
|
||||
)
|
||||
|
||||
def _parse_llm_response(self, response: str) -> Dict[str, Any]:
|
||||
"""解析 LLM 返回的 JSON"""
|
||||
try:
|
||||
# 尝试查找 JSON
|
||||
json_match = re.search(r"\{.*\}", response, re.DOTALL)
|
||||
if json_match:
|
||||
return json.loads(json_match.group())
|
||||
return {}
|
||||
except Exception as e:
|
||||
logger.warning(f"解析总结 JSON 失败: {e}")
|
||||
return {}
|
||||
|
||||
async def _execute_import(
|
||||
self,
|
||||
summary: str,
|
||||
entities: List[str],
|
||||
relations: List[Dict[str, str]],
|
||||
stream_id: str,
|
||||
time_meta: Optional[Dict[str, Any]] = None,
|
||||
):
|
||||
"""将数据写入存储"""
|
||||
# 获取默认知识类型
|
||||
type_str = self.plugin_config.get("summarization", {}).get("default_knowledge_type", "narrative")
|
||||
try:
|
||||
knowledge_type = resolve_stored_knowledge_type(type_str, content=summary)
|
||||
except ValueError:
|
||||
logger.warning(f"非法 summarization.default_knowledge_type={type_str}，回退 narrative")
|
||||
knowledge_type = KnowledgeType.NARRATIVE
|
||||
|
||||
# 导入总结文本
|
||||
hash_value = self.metadata_store.add_paragraph(
|
||||
content=summary,
|
||||
source=f"chat_summary:{stream_id}",
|
||||
knowledge_type=knowledge_type.value,
|
||||
time_meta=time_meta,
|
||||
)
|
||||
|
||||
embedding = await self.embedding_manager.encode(summary)
|
||||
self.vector_store.add(
|
||||
vectors=embedding.reshape(1, -1),
|
||||
ids=[hash_value]
|
||||
)
|
||||
|
||||
# 导入实体
|
||||
if entities:
|
||||
self.graph_store.add_nodes(entities)
|
||||
|
||||
# 导入关系
|
||||
rv_cfg = self.plugin_config.get("retrieval", {}).get("relation_vectorization", {})
|
||||
if not isinstance(rv_cfg, dict):
|
||||
rv_cfg = {}
|
||||
write_vector = bool(rv_cfg.get("enabled", False)) and bool(rv_cfg.get("write_on_import", True))
|
||||
for rel in relations:
|
||||
s, p, o = rel.get("subject"), rel.get("predicate"), rel.get("object")
|
||||
if all([s, p, o]):
|
||||
if self.relation_write_service is not None:
|
||||
await self.relation_write_service.upsert_relation_with_vector(
|
||||
subject=s,
|
||||
predicate=p,
|
||||
obj=o,
|
||||
confidence=1.0,
|
||||
source_paragraph=summary,
|
||||
write_vector=write_vector,
|
||||
)
|
||||
else:
|
||||
# 写入元数据
|
||||
rel_hash = self.metadata_store.add_relation(
|
||||
subject=s,
|
||||
predicate=p,
|
||||
obj=o,
|
||||
confidence=1.0,
|
||||
source_paragraph=summary
|
||||
)
|
||||
# 写入图数据库（写入 relation_hashes，确保后续可按关系精确修剪）
|
||||
self.graph_store.add_edges([(s, o)], relation_hashes=[rel_hash])
|
||||
try:
|
||||
self.metadata_store.set_relation_vector_state(rel_hash, "none")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
logger.info(f"总结导入完成: hash={hash_value[:8]}")
|
||||
3522
plugins/A_memorix/core/utils/web_import_manager.py
Normal file
3522
plugins/A_memorix/core/utils/web_import_manager.py
Normal file
File diff suppressed because it is too large
Load Diff
@@ -15,6 +15,12 @@ def _tool_param(name: str, param_type: ToolParamType, description: str, required
|
||||
return ToolParameterInfo(name=name, param_type=param_type, description=description, required=required)
|
||||
|
||||
|
||||
_ADMIN_TOOL_PARAMS = [
|
||||
_tool_param("action", ToolParamType.STRING, "管理动作", True),
|
||||
_tool_param("target", ToolParamType.STRING, "可选目标标识", False),
|
||||
]
|
||||
|
||||
|
||||
class AMemorixPlugin(MaiBotPlugin):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
@@ -33,7 +39,11 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
|
||||
async def on_unload(self):
|
||||
if self._kernel is not None:
|
||||
self._kernel.close()
|
||||
shutdown = getattr(self._kernel, "shutdown", None)
|
||||
if callable(shutdown):
|
||||
await shutdown()
|
||||
else:
|
||||
self._kernel.close()
|
||||
self._kernel = None
|
||||
|
||||
async def _get_kernel(self) -> SDKMemoryKernel:
|
||||
@@ -42,6 +52,11 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
await self._kernel.initialize()
|
||||
return self._kernel
|
||||
|
||||
async def _dispatch_admin_tool(self, method_name: str, action: str, **kwargs):
|
||||
kernel = await self._get_kernel()
|
||||
handler = getattr(kernel, method_name)
|
||||
return await handler(action=action, **kwargs)
|
||||
|
||||
@Tool(
|
||||
"search_memory",
|
||||
description="搜索长期记忆",
|
||||
@@ -53,6 +68,7 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
_tool_param("person_id", ToolParamType.STRING, "人物 ID", False),
|
||||
_tool_param("time_start", ToolParamType.FLOAT, "起始时间戳", False),
|
||||
_tool_param("time_end", ToolParamType.FLOAT, "结束时间戳", False),
|
||||
_tool_param("respect_filter", ToolParamType.BOOLEAN, "是否应用聊天过滤配置", False),
|
||||
],
|
||||
)
|
||||
async def handle_search_memory(
|
||||
@@ -62,11 +78,11 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
mode: str = "hybrid",
|
||||
chat_id: str = "",
|
||||
person_id: str = "",
|
||||
time_start: float | None = None,
|
||||
time_end: float | None = None,
|
||||
time_start: str | float | None = None,
|
||||
time_end: str | float | None = None,
|
||||
respect_filter: bool = True,
|
||||
**kwargs,
|
||||
):
|
||||
_ = kwargs
|
||||
kernel = await self._get_kernel()
|
||||
return await kernel.search_memory(
|
||||
KernelSearchRequest(
|
||||
@@ -77,6 +93,9 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
person_id=person_id,
|
||||
time_start=time_start,
|
||||
time_end=time_end,
|
||||
respect_filter=respect_filter,
|
||||
user_id=str(kwargs.get("user_id", "") or "").strip(),
|
||||
group_id=str(kwargs.get("group_id", "") or "").strip(),
|
||||
)
|
||||
)
|
||||
|
||||
@@ -89,6 +108,7 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
_tool_param("text", ToolParamType.STRING, "摘要文本", True),
|
||||
_tool_param("time_start", ToolParamType.FLOAT, "起始时间戳", False),
|
||||
_tool_param("time_end", ToolParamType.FLOAT, "结束时间戳", False),
|
||||
_tool_param("respect_filter", ToolParamType.BOOLEAN, "是否应用聊天过滤配置", False),
|
||||
],
|
||||
)
|
||||
async def handle_ingest_summary(
|
||||
@@ -101,9 +121,9 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
time_end: float | None = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
respect_filter: bool = True,
|
||||
**kwargs,
|
||||
):
|
||||
_ = kwargs
|
||||
kernel = await self._get_kernel()
|
||||
return await kernel.ingest_summary(
|
||||
external_id=external_id,
|
||||
@@ -114,6 +134,9 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
time_end=time_end,
|
||||
tags=tags,
|
||||
metadata=metadata,
|
||||
respect_filter=respect_filter,
|
||||
user_id=str(kwargs.get("user_id", "") or "").strip(),
|
||||
group_id=str(kwargs.get("group_id", "") or "").strip(),
|
||||
)
|
||||
|
||||
@Tool(
|
||||
@@ -125,6 +148,7 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
_tool_param("text", ToolParamType.STRING, "原始文本", True),
|
||||
_tool_param("chat_id", ToolParamType.STRING, "聊天流 ID", False),
|
||||
_tool_param("timestamp", ToolParamType.FLOAT, "时间戳", False),
|
||||
_tool_param("respect_filter", ToolParamType.BOOLEAN, "是否应用聊天过滤配置", False),
|
||||
],
|
||||
)
|
||||
async def handle_ingest_text(
|
||||
@@ -140,6 +164,7 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
time_end: float | None = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
respect_filter: bool = True,
|
||||
**kwargs,
|
||||
):
|
||||
relations = kwargs.get("relations")
|
||||
@@ -159,6 +184,9 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
metadata=metadata,
|
||||
entities=entities,
|
||||
relations=relations,
|
||||
respect_filter=respect_filter,
|
||||
user_id=str(kwargs.get("user_id", "") or "").strip(),
|
||||
group_id=str(kwargs.get("group_id", "") or "").strip(),
|
||||
)
|
||||
|
||||
@Tool(
|
||||
@@ -179,22 +207,24 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
"maintain_memory",
|
||||
description="维护长期记忆关系状态",
|
||||
parameters=[
|
||||
_tool_param("action", ToolParamType.STRING, "reinforce/protect/restore", True),
|
||||
_tool_param("target", ToolParamType.STRING, "目标哈希或查询文本", True),
|
||||
_tool_param("action", ToolParamType.STRING, "reinforce/protect/restore/freeze/recycle_bin", True),
|
||||
_tool_param("target", ToolParamType.STRING, "目标哈希或查询文本", False),
|
||||
_tool_param("hours", ToolParamType.FLOAT, "保护时长（小时）", False),
|
||||
_tool_param("limit", ToolParamType.INTEGER, "查询条数（用于 recycle_bin）", False),
|
||||
],
|
||||
)
|
||||
async def handle_maintain_memory(
|
||||
self,
|
||||
action: str,
|
||||
target: str,
|
||||
target: str = "",
|
||||
hours: float | None = None,
|
||||
reason: str = "",
|
||||
limit: int = 50,
|
||||
**kwargs,
|
||||
):
|
||||
_ = kwargs
|
||||
kernel = await self._get_kernel()
|
||||
return await kernel.maintain_memory(action=action, target=target, hours=hours, reason=reason)
|
||||
return await kernel.maintain_memory(action=action, target=target, hours=hours, reason=reason, limit=limit)
|
||||
|
||||
@Tool("memory_stats", description="获取长期记忆统计", parameters=[])
|
||||
async def handle_memory_stats(self, **kwargs):
|
||||
@@ -202,6 +232,42 @@ class AMemorixPlugin(MaiBotPlugin):
|
||||
kernel = await self._get_kernel()
|
||||
return kernel.memory_stats()
|
||||
|
||||
@Tool("memory_graph_admin", description="长期记忆图谱管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_graph_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_graph_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_source_admin", description="长期记忆来源管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_source_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_source_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_episode_admin", description="Episode 管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_episode_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_episode_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_profile_admin", description="人物画像管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_profile_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_profile_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_runtime_admin", description="长期记忆运行时管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_runtime_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_runtime_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_import_admin", description="长期记忆导入管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_import_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_import_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_tuning_admin", description="长期记忆调优管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_tuning_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_tuning_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_v5_admin", description="长期记忆 V5 管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_v5_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_v5_admin", action=action, **kwargs)
|
||||
|
||||
@Tool("memory_delete_admin", description="长期记忆删除管理接口", parameters=_ADMIN_TOOL_PARAMS)
|
||||
async def handle_memory_delete_admin(self, action: str, **kwargs):
|
||||
return await self._dispatch_admin_tool("memory_delete_admin", action=action, **kwargs)
|
||||
|
||||
|
||||
def create_plugin():
|
||||
return AMemorixPlugin()
|
||||
|
||||
52
plugins/A_memorix/requirements.txt
Normal file
52
plugins/A_memorix/requirements.txt
Normal file
@@ -0,0 +1,52 @@
|
||||
# A_Memorix 插件依赖
|
||||
#
|
||||
# 核心依赖 (必需)
|
||||
# ==================
|
||||
|
||||
# 数值计算 - 用于向量操作、矩阵计算
|
||||
numpy>=1.20.0
|
||||
|
||||
# 稀疏矩阵 - 用于图存储的邻接矩阵
|
||||
scipy>=1.7.0
|
||||
|
||||
# 图结构处理（LPMM 转换）
|
||||
networkx>=3.0.0
|
||||
|
||||
# Parquet 读取（LPMM 转换）
|
||||
pyarrow>=10.0.0
|
||||
|
||||
# DataFrame 处理（LPMM 转换）
|
||||
pandas>=1.5.0
|
||||
|
||||
# 异步事件循环嵌套 - 用于插件初始化时的异步操作
|
||||
nest-asyncio>=1.5.0
|
||||
|
||||
# 向量索引 - 用于向量存储和检索
|
||||
faiss-cpu>=1.7.0
|
||||
|
||||
# Web 服务器依赖 (可视化功能需要)
|
||||
# ==================
|
||||
|
||||
# ASGI 服务器
|
||||
uvicorn>=0.20.0
|
||||
|
||||
# Web 框架
|
||||
fastapi>=0.100.0
|
||||
|
||||
# 数据验证
|
||||
pydantic>=2.0.0
|
||||
python-multipart>=0.0.9
|
||||
|
||||
# 注意事项
|
||||
# ==================
|
||||
#
|
||||
# 1. sqlite3 是 Python 标准库，无需安装
|
||||
# 2. json, re, time, pathlib 等都是标准库
|
||||
# 3. sentence-transformers 不需要（使用主程序 Embedding API）
|
||||
|
||||
# UI 交互
|
||||
rich>=14.0.0
|
||||
tenacity>=8.0.0
|
||||
|
||||
# 稀疏检索中文分词（可选，未安装时自动回退 char n-gram）
|
||||
jieba>=0.42.1
|
||||
213
plugins/A_memorix/scripts/audit_vector_consistency.py
Normal file
213
plugins/A_memorix/scripts/audit_vector_consistency.py
Normal file
@@ -0,0 +1,213 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
A_Memorix 一致性审计脚本。
|
||||
|
||||
输出内容：
|
||||
1. paragraph/entity/relation 向量覆盖率
|
||||
2. relation vector_state 分布
|
||||
3. 孤儿向量数量（向量存在但 metadata 不存在）
|
||||
4. 状态与向量文件不一致统计
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pickle
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Set
|
||||
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
PLUGIN_ROOT = CURRENT_DIR.parent
|
||||
PROJECT_ROOT = PLUGIN_ROOT.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
||||
|
||||
def _build_arg_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="审计 A_Memorix 向量一致性")
|
||||
parser.add_argument(
|
||||
"--data-dir",
|
||||
default=str(PLUGIN_ROOT / "data"),
|
||||
help="A_Memorix 数据目录（默认: plugins/A_memorix/data）",
|
||||
)
|
||||
parser.add_argument("--json-out", default="", help="可选：输出 JSON 文件路径")
|
||||
parser.add_argument(
|
||||
"--strict",
|
||||
action="store_true",
|
||||
help="若发现一致性异常则返回非 0 退出码",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
# --help/-h fast path: avoid heavy host/plugin bootstrap
|
||||
if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
sys.exit(0)
|
||||
|
||||
try:
|
||||
from core.storage.vector_store import VectorStore
|
||||
from core.storage.metadata_store import MetadataStore
|
||||
from core.storage import QuantizationType
|
||||
except Exception as e: # pragma: no cover
|
||||
print(f"❌ 导入核心模块失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def _safe_ratio(numerator: int, denominator: int) -> float:
|
||||
if denominator <= 0:
|
||||
return 0.0
|
||||
return float(numerator) / float(denominator)
|
||||
|
||||
|
||||
def _load_vector_store(data_dir: Path) -> VectorStore:
|
||||
meta_path = data_dir / "vectors" / "vectors_metadata.pkl"
|
||||
if not meta_path.exists():
|
||||
raise FileNotFoundError(f"未找到向量元数据文件: {meta_path}")
|
||||
|
||||
with open(meta_path, "rb") as f:
|
||||
meta = pickle.load(f)
|
||||
dimension = int(meta.get("dimension", 1024))
|
||||
|
||||
store = VectorStore(
|
||||
dimension=max(1, dimension),
|
||||
quantization_type=QuantizationType.INT8,
|
||||
data_dir=data_dir / "vectors",
|
||||
)
|
||||
if store.has_data():
|
||||
store.load()
|
||||
return store
|
||||
|
||||
|
||||
def _load_metadata_store(data_dir: Path) -> MetadataStore:
|
||||
store = MetadataStore(data_dir=data_dir / "metadata")
|
||||
store.connect()
|
||||
return store
|
||||
|
||||
|
||||
def _hash_set(metadata_store: MetadataStore, table: str) -> Set[str]:
|
||||
return {str(h) for h in metadata_store.list_hashes(table)}
|
||||
|
||||
|
||||
def _relation_state_stats(metadata_store: MetadataStore) -> Dict[str, int]:
|
||||
return metadata_store.count_relations_by_vector_state()
|
||||
|
||||
|
||||
def run_audit(data_dir: Path) -> Dict[str, Any]:
|
||||
vector_store = _load_vector_store(data_dir)
|
||||
metadata_store = _load_metadata_store(data_dir)
|
||||
try:
|
||||
paragraph_hashes = _hash_set(metadata_store, "paragraphs")
|
||||
entity_hashes = _hash_set(metadata_store, "entities")
|
||||
relation_hashes = _hash_set(metadata_store, "relations")
|
||||
|
||||
known_hashes = set(getattr(vector_store, "_known_hashes", set()))
|
||||
live_vector_hashes = {h for h in known_hashes if h in vector_store}
|
||||
|
||||
para_vector_hits = len(paragraph_hashes & live_vector_hashes)
|
||||
ent_vector_hits = len(entity_hashes & live_vector_hashes)
|
||||
rel_vector_hits = len(relation_hashes & live_vector_hashes)
|
||||
|
||||
orphan_vector_hashes = sorted(
|
||||
live_vector_hashes - paragraph_hashes - entity_hashes - relation_hashes
|
||||
)
|
||||
|
||||
relation_rows = metadata_store.get_relations()
|
||||
ready_but_missing = 0
|
||||
not_ready_but_present = 0
|
||||
for row in relation_rows:
|
||||
h = str(row.get("hash") or "")
|
||||
state = str(row.get("vector_state") or "none").lower()
|
||||
in_vector = h in live_vector_hashes
|
||||
if state == "ready" and not in_vector:
|
||||
ready_but_missing += 1
|
||||
if state != "ready" and in_vector:
|
||||
not_ready_but_present += 1
|
||||
|
||||
relation_states = _relation_state_stats(metadata_store)
|
||||
rel_total = max(0, int(relation_states.get("total", len(relation_hashes))))
|
||||
ready_count = max(0, int(relation_states.get("ready", 0)))
|
||||
|
||||
result = {
|
||||
"counts": {
|
||||
"paragraphs": len(paragraph_hashes),
|
||||
"entities": len(entity_hashes),
|
||||
"relations": len(relation_hashes),
|
||||
"vectors_live": len(live_vector_hashes),
|
||||
},
|
||||
"coverage": {
|
||||
"paragraph_vector_coverage": _safe_ratio(para_vector_hits, len(paragraph_hashes)),
|
||||
"entity_vector_coverage": _safe_ratio(ent_vector_hits, len(entity_hashes)),
|
||||
"relation_vector_coverage": _safe_ratio(rel_vector_hits, len(relation_hashes)),
|
||||
"relation_ready_coverage": _safe_ratio(ready_count, rel_total),
|
||||
},
|
||||
"relation_states": relation_states,
|
||||
"orphans": {
|
||||
"vector_only_count": len(orphan_vector_hashes),
|
||||
"vector_only_sample": orphan_vector_hashes[:30],
|
||||
},
|
||||
"consistency_checks": {
|
||||
"ready_but_missing_vector": ready_but_missing,
|
||||
"not_ready_but_vector_present": not_ready_but_present,
|
||||
},
|
||||
}
|
||||
return result
|
||||
finally:
|
||||
metadata_store.close()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = _build_arg_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
data_dir = Path(args.data_dir).resolve()
|
||||
if not data_dir.exists():
|
||||
print(f"❌ 数据目录不存在: {data_dir}")
|
||||
return 2
|
||||
|
||||
try:
|
||||
result = run_audit(data_dir)
|
||||
except Exception as e:
|
||||
print(f"❌ 审计失败: {e}")
|
||||
return 2
|
||||
|
||||
print("=== A_Memorix Vector Consistency Audit ===")
|
||||
print(f"data_dir: {data_dir}")
|
||||
print(f"paragraphs: {result['counts']['paragraphs']}")
|
||||
print(f"entities: {result['counts']['entities']}")
|
||||
print(f"relations: {result['counts']['relations']}")
|
||||
print(f"vectors_live: {result['counts']['vectors_live']}")
|
||||
print(
|
||||
"coverage: "
|
||||
f"paragraph={result['coverage']['paragraph_vector_coverage']:.3f}, "
|
||||
f"entity={result['coverage']['entity_vector_coverage']:.3f}, "
|
||||
f"relation={result['coverage']['relation_vector_coverage']:.3f}, "
|
||||
f"relation_ready={result['coverage']['relation_ready_coverage']:.3f}"
|
||||
)
|
||||
print(f"relation_states: {result['relation_states']}")
|
||||
print(
|
||||
"consistency_checks: "
|
||||
f"ready_but_missing_vector={result['consistency_checks']['ready_but_missing_vector']}, "
|
||||
f"not_ready_but_vector_present={result['consistency_checks']['not_ready_but_vector_present']}"
|
||||
)
|
||||
print(f"orphan_vectors: {result['orphans']['vector_only_count']}")
|
||||
|
||||
if args.json_out:
|
||||
out_path = Path(args.json_out).resolve()
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
print(f"json_out: {out_path}")
|
||||
|
||||
has_anomaly = (
|
||||
result["orphans"]["vector_only_count"] > 0
|
||||
or result["consistency_checks"]["ready_but_missing_vector"] > 0
|
||||
)
|
||||
if args.strict and has_anomaly:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
270
plugins/A_memorix/scripts/backfill_relation_vectors.py
Normal file
270
plugins/A_memorix/scripts/backfill_relation_vectors.py
Normal file
@@ -0,0 +1,270 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
关系向量一次性回填脚本（灰度/离线执行）。
|
||||
|
||||
用途：
|
||||
1. 对 relations 中 vector_state in (none, failed, pending) 的记录补齐向量。
|
||||
2. 支持并发控制，降低总耗时。
|
||||
3. 可作为灰度阶段验证工具，与 audit_vector_consistency.py 配合使用。
|
||||
4. 可选自动纳入“ready 但向量缺失”的漂移记录进行修复。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
import tomlkit
|
||||
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
PLUGIN_ROOT = CURRENT_DIR.parent
|
||||
PROJECT_ROOT = PLUGIN_ROOT.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
||||
|
||||
def _build_arg_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="关系向量一次性回填")
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
default=str(PLUGIN_ROOT / "config.toml"),
|
||||
help="配置文件路径（默认 plugins/A_memorix/config.toml）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--data-dir",
|
||||
default=str(PLUGIN_ROOT / "data"),
|
||||
help="数据目录（默认 plugins/A_memorix/data）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--states",
|
||||
default="none,failed,pending",
|
||||
help="待处理状态列表，逗号分隔",
|
||||
)
|
||||
parser.add_argument("--limit", type=int, default=50000, help="最大处理数量")
|
||||
parser.add_argument("--concurrency", type=int, default=8, help="并发数")
|
||||
parser.add_argument("--max-retry", type=int, default=None, help="最大重试次数过滤")
|
||||
parser.add_argument(
|
||||
"--include-ready-missing",
|
||||
action="store_true",
|
||||
help="额外纳入 vector_state=ready 但向量缺失的关系",
|
||||
)
|
||||
parser.add_argument("--dry-run", action="store_true", help="仅统计候选，不写入")
|
||||
return parser
|
||||
|
||||
|
||||
# --help/-h fast path: avoid heavy host/plugin bootstrap
|
||||
if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
raise SystemExit(0)
|
||||
|
||||
from core.storage import (
|
||||
VectorStore,
|
||||
GraphStore,
|
||||
MetadataStore,
|
||||
QuantizationType,
|
||||
SparseMatrixFormat,
|
||||
)
|
||||
from core.embedding import create_embedding_api_adapter
|
||||
from core.utils.relation_write_service import RelationWriteService
|
||||
|
||||
|
||||
def _load_config(config_path: Path) -> Dict[str, Any]:
|
||||
with open(config_path, "r", encoding="utf-8") as f:
|
||||
raw = tomlkit.load(f)
|
||||
return dict(raw) if isinstance(raw, dict) else {}
|
||||
|
||||
|
||||
def _build_vector_store(data_dir: Path, emb_cfg: Dict[str, Any]) -> VectorStore:
|
||||
q_type = str(emb_cfg.get("quantization_type", "int8")).lower()
|
||||
if q_type != "int8":
|
||||
raise ValueError(
|
||||
"embedding.quantization_type 在 vNext 仅允许 int8(SQ8)。"
|
||||
" 请先执行 scripts/release_vnext_migrate.py migrate。"
|
||||
)
|
||||
dim = int(emb_cfg.get("dimension", 1024))
|
||||
store = VectorStore(
|
||||
dimension=max(1, dim),
|
||||
quantization_type=QuantizationType.INT8,
|
||||
data_dir=data_dir / "vectors",
|
||||
)
|
||||
if store.has_data():
|
||||
store.load()
|
||||
return store
|
||||
|
||||
|
||||
def _build_graph_store(data_dir: Path, graph_cfg: Dict[str, Any]) -> GraphStore:
|
||||
fmt = str(graph_cfg.get("sparse_matrix_format", "csr")).lower()
|
||||
fmt_map = {
|
||||
"csr": SparseMatrixFormat.CSR,
|
||||
"csc": SparseMatrixFormat.CSC,
|
||||
}
|
||||
store = GraphStore(
|
||||
matrix_format=fmt_map.get(fmt, SparseMatrixFormat.CSR),
|
||||
data_dir=data_dir / "graph",
|
||||
)
|
||||
if store.has_data():
|
||||
store.load()
|
||||
return store
|
||||
|
||||
|
||||
def _build_metadata_store(data_dir: Path) -> MetadataStore:
|
||||
store = MetadataStore(data_dir=data_dir / "metadata")
|
||||
store.connect()
|
||||
return store
|
||||
|
||||
|
||||
def _build_embedding_manager(emb_cfg: Dict[str, Any]):
|
||||
retry_cfg = emb_cfg.get("retry", {})
|
||||
if not isinstance(retry_cfg, dict):
|
||||
retry_cfg = {}
|
||||
return create_embedding_api_adapter(
|
||||
batch_size=int(emb_cfg.get("batch_size", 32)),
|
||||
max_concurrent=int(emb_cfg.get("max_concurrent", 5)),
|
||||
default_dimension=int(emb_cfg.get("dimension", 1024)),
|
||||
model_name=str(emb_cfg.get("model_name", "auto")),
|
||||
retry_config=retry_cfg,
|
||||
)
|
||||
|
||||
|
||||
async def _process_rows(
|
||||
service: RelationWriteService,
|
||||
rows: List[Dict[str, Any]],
|
||||
concurrency: int,
|
||||
) -> Dict[str, int]:
|
||||
semaphore = asyncio.Semaphore(max(1, int(concurrency)))
|
||||
stat = {"success": 0, "failed": 0, "skipped": 0}
|
||||
|
||||
async def _worker(row: Dict[str, Any]) -> None:
|
||||
async with semaphore:
|
||||
result = await service.ensure_relation_vector(
|
||||
hash_value=str(row["hash"]),
|
||||
subject=str(row.get("subject", "")),
|
||||
predicate=str(row.get("predicate", "")),
|
||||
obj=str(row.get("object", "")),
|
||||
)
|
||||
if result.vector_state == "ready":
|
||||
if result.vector_written:
|
||||
stat["success"] += 1
|
||||
else:
|
||||
stat["skipped"] += 1
|
||||
else:
|
||||
stat["failed"] += 1
|
||||
|
||||
await asyncio.gather(*[_worker(row) for row in rows])
|
||||
return stat
|
||||
|
||||
|
||||
async def main_async(args: argparse.Namespace) -> int:
|
||||
config_path = Path(args.config).resolve()
|
||||
if not config_path.exists():
|
||||
print(f"❌ 配置文件不存在: {config_path}")
|
||||
return 2
|
||||
|
||||
cfg = _load_config(config_path)
|
||||
emb_cfg = cfg.get("embedding", {}) if isinstance(cfg, dict) else {}
|
||||
graph_cfg = cfg.get("graph", {}) if isinstance(cfg, dict) else {}
|
||||
retrieval_cfg = cfg.get("retrieval", {}) if isinstance(cfg, dict) else {}
|
||||
rv_cfg = retrieval_cfg.get("relation_vectorization", {}) if isinstance(retrieval_cfg, dict) else {}
|
||||
if not isinstance(emb_cfg, dict):
|
||||
emb_cfg = {}
|
||||
if not isinstance(graph_cfg, dict):
|
||||
graph_cfg = {}
|
||||
if not isinstance(rv_cfg, dict):
|
||||
rv_cfg = {}
|
||||
|
||||
data_dir = Path(args.data_dir).resolve()
|
||||
if not data_dir.exists():
|
||||
print(f"❌ 数据目录不存在: {data_dir}")
|
||||
return 2
|
||||
|
||||
print(f"data_dir: {data_dir}")
|
||||
print(f"config: {config_path}")
|
||||
|
||||
vector_store = _build_vector_store(data_dir, emb_cfg)
|
||||
graph_store = _build_graph_store(data_dir, graph_cfg)
|
||||
metadata_store = _build_metadata_store(data_dir)
|
||||
embedding_manager = _build_embedding_manager(emb_cfg)
|
||||
service = RelationWriteService(
|
||||
metadata_store=metadata_store,
|
||||
graph_store=graph_store,
|
||||
vector_store=vector_store,
|
||||
embedding_manager=embedding_manager,
|
||||
)
|
||||
|
||||
try:
|
||||
states = [s.strip() for s in str(args.states).split(",") if s.strip()]
|
||||
if not states:
|
||||
states = ["none", "failed", "pending"]
|
||||
max_retry = int(args.max_retry) if args.max_retry is not None else int(rv_cfg.get("max_retry", 3))
|
||||
limit = int(args.limit)
|
||||
|
||||
rows = metadata_store.list_relations_by_vector_state(
|
||||
states=states,
|
||||
limit=max(1, limit),
|
||||
max_retry=max(1, max_retry),
|
||||
)
|
||||
added_ready_missing = 0
|
||||
if args.include_ready_missing:
|
||||
ready_rows = metadata_store.list_relations_by_vector_state(
|
||||
states=["ready"],
|
||||
limit=max(1, limit),
|
||||
max_retry=max(1, max_retry),
|
||||
)
|
||||
ready_missing_rows = [
|
||||
row for row in ready_rows if str(row.get("hash", "")) not in vector_store
|
||||
]
|
||||
added_ready_missing = len(ready_missing_rows)
|
||||
if ready_missing_rows:
|
||||
dedup: Dict[str, Dict[str, Any]] = {}
|
||||
for row in rows:
|
||||
dedup[str(row.get("hash", ""))] = row
|
||||
for row in ready_missing_rows:
|
||||
dedup.setdefault(str(row.get("hash", "")), row)
|
||||
rows = list(dedup.values())[: max(1, limit)]
|
||||
print(f"candidates: {len(rows)} (states={states}, max_retry={max_retry})")
|
||||
if args.include_ready_missing:
|
||||
print(f"ready_missing_candidates_added: {added_ready_missing}")
|
||||
if not rows:
|
||||
return 0
|
||||
|
||||
if args.dry_run:
|
||||
print("dry_run=true，未执行写入。")
|
||||
return 0
|
||||
|
||||
started = time.time()
|
||||
stat = await _process_rows(
|
||||
service=service,
|
||||
rows=rows,
|
||||
concurrency=int(args.concurrency),
|
||||
)
|
||||
elapsed = (time.time() - started) * 1000.0
|
||||
|
||||
vector_store.save()
|
||||
graph_store.save()
|
||||
state_stats = metadata_store.count_relations_by_vector_state()
|
||||
output = {
|
||||
"processed": len(rows),
|
||||
"success": int(stat["success"]),
|
||||
"failed": int(stat["failed"]),
|
||||
"skipped": int(stat["skipped"]),
|
||||
"elapsed_ms": elapsed,
|
||||
"state_stats": state_stats,
|
||||
}
|
||||
print(json.dumps(output, ensure_ascii=False, indent=2))
|
||||
return 0 if stat["failed"] == 0 else 1
|
||||
finally:
|
||||
metadata_store.close()
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
return _build_arg_parser().parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
arguments = parse_args()
|
||||
raise SystemExit(asyncio.run(main_async(arguments)))
|
||||
73
plugins/A_memorix/scripts/backfill_temporal_metadata.py
Normal file
73
plugins/A_memorix/scripts/backfill_temporal_metadata.py
Normal file
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
回填段落时序字段。
|
||||
|
||||
默认策略：
|
||||
1. 若段落缺失 event_time/event_time_start/event_time_end
|
||||
2. 且存在 created_at
|
||||
3. 写入 event_time=created_at, time_granularity=day, time_confidence=0.2
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
PLUGIN_ROOT = CURRENT_DIR.parent
|
||||
PROJECT_ROOT = PLUGIN_ROOT.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from plugins.A_memorix.core.storage import MetadataStore # noqa: E402
|
||||
|
||||
|
||||
def backfill(
|
||||
data_dir: Path,
|
||||
dry_run: bool,
|
||||
limit: int,
|
||||
no_created_fallback: bool,
|
||||
) -> int:
|
||||
store = MetadataStore(data_dir=data_dir)
|
||||
store.connect()
|
||||
summary = store.backfill_temporal_metadata_from_created_at(
|
||||
limit=limit,
|
||||
dry_run=dry_run,
|
||||
no_created_fallback=no_created_fallback,
|
||||
)
|
||||
store.close()
|
||||
if dry_run:
|
||||
print(f"[dry-run] candidates={summary['candidates']}")
|
||||
return int(summary["candidates"])
|
||||
if no_created_fallback:
|
||||
print(f"skip update (no-created-fallback), candidates={summary['candidates']}")
|
||||
return 0
|
||||
print(f"updated={summary['updated']}")
|
||||
return int(summary["updated"])
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Backfill temporal metadata for A_Memorix paragraphs")
|
||||
parser.add_argument("--data-dir", default=str(PLUGIN_ROOT / "data"), help="数据目录")
|
||||
parser.add_argument("--dry-run", action="store_true", help="仅统计，不写入")
|
||||
parser.add_argument("--limit", type=int, default=100000, help="最大处理条数")
|
||||
parser.add_argument(
|
||||
"--no-created-fallback",
|
||||
action="store_true",
|
||||
help="不使用 created_at 回填，仅输出候选数量",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
backfill(
|
||||
data_dir=Path(args.data_dir),
|
||||
dry_run=args.dry_run,
|
||||
limit=max(1, int(args.limit)),
|
||||
no_created_fallback=args.no_created_fallback,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
||||
@@ -46,9 +46,14 @@ if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
sys.exit(0)
|
||||
|
||||
# 设置日志
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
|
||||
logger = logging.getLogger("LPMM_Converter")
|
||||
# 设置日志：优先复用 MaiBot 统一日志体系，失败时回退到标准 logging。
|
||||
try:
|
||||
from src.common.logger import get_logger
|
||||
|
||||
logger = get_logger("A_Memorix.LPMMConverter")
|
||||
except Exception:
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
|
||||
logger = logging.getLogger("A_Memorix.LPMMConverter")
|
||||
|
||||
try:
|
||||
import networkx as nx
|
||||
@@ -225,11 +230,11 @@ class LPMMConverter:
|
||||
failed += 1
|
||||
|
||||
logger.info(
|
||||
"关系向量重建完成: total=%s success=%s skipped=%s failed=%s",
|
||||
len(rows),
|
||||
success,
|
||||
skipped,
|
||||
failed,
|
||||
"关系向量重建完成: "
|
||||
f"total={len(rows)} "
|
||||
f"success={success} "
|
||||
f"skipped={skipped} "
|
||||
f"failed={failed}"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
@@ -317,8 +322,8 @@ class LPMMConverter:
|
||||
if p_type == "relation":
|
||||
relation_count = self._import_relation_metadata_from_parquet(p_path)
|
||||
logger.warning(
|
||||
"跳过 relation.parquet 向量导入（保持一致性）；已导入关系元数据: %s",
|
||||
relation_count,
|
||||
"跳过 relation.parquet 向量导入（保持一致性）；"
|
||||
f"已导入关系元数据: {relation_count}"
|
||||
)
|
||||
continue
|
||||
|
||||
|
||||
172
plugins/A_memorix/scripts/import_lpmm_json.py
Normal file
172
plugins/A_memorix/scripts/import_lpmm_json.py
Normal file
@@ -0,0 +1,172 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LPMM OpenIE JSON 导入工具。
|
||||
|
||||
功能：
|
||||
1. 读取符合 LPMM 规范的 OpenIE JSON 文件
|
||||
2. 转换为 A_Memorix 的统一导入格式
|
||||
3. 复用 `process_knowledge.py` 中的 `AutoImporter` 直接入库
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from rich.console import Console
|
||||
from rich.progress import BarColumn, Progress, SpinnerColumn, TextColumn, TimeElapsedColumn
|
||||
|
||||
console = Console()
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
PLUGIN_ROOT = CURRENT_DIR.parent
|
||||
WORKSPACE_ROOT = PLUGIN_ROOT.parent
|
||||
MAIBOT_ROOT = WORKSPACE_ROOT / "MaiBot"
|
||||
for path in (CURRENT_DIR, WORKSPACE_ROOT, MAIBOT_ROOT, PLUGIN_ROOT):
|
||||
path_str = str(path)
|
||||
if path_str not in sys.path:
|
||||
sys.path.insert(0, path_str)
|
||||
|
||||
|
||||
def _build_arg_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="将 LPMM OpenIE JSON 导入 A_Memorix")
|
||||
parser.add_argument("path", help="LPMM JSON 文件路径或目录")
|
||||
parser.add_argument("--force", action="store_true", help="强制重新导入")
|
||||
parser.add_argument("--concurrency", "-c", type=int, default=5, help="并发数")
|
||||
return parser
|
||||
|
||||
|
||||
if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
raise SystemExit(0)
|
||||
|
||||
|
||||
try:
|
||||
from process_knowledge import AutoImporter
|
||||
from A_memorix.core.utils.hash import compute_paragraph_hash
|
||||
from src.common.logger import get_logger
|
||||
except ImportError as exc: # pragma: no cover - script bootstrap
|
||||
print(f"导入模块失败，请确认 PYTHONPATH 与工作区结构: {exc}")
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
logger = get_logger("A_Memorix.LPMMImport")
|
||||
|
||||
|
||||
class LPMMConverter:
|
||||
def convert_lpmm_to_memorix(self, lpmm_data: Dict[str, Any], filename: str) -> Dict[str, Any]:
|
||||
memorix_data = {"paragraphs": [], "entities": []}
|
||||
docs = lpmm_data.get("docs", []) or []
|
||||
if not docs:
|
||||
logger.warning(f"文件中未找到 docs 字段: {filename}")
|
||||
return memorix_data
|
||||
|
||||
all_entities = set()
|
||||
for doc in docs:
|
||||
content = str(doc.get("passage", "") or "").strip()
|
||||
if not content:
|
||||
continue
|
||||
|
||||
relations: List[Dict[str, str]] = []
|
||||
for triple in doc.get("extracted_triples", []) or []:
|
||||
if isinstance(triple, list) and len(triple) == 3:
|
||||
relations.append(
|
||||
{
|
||||
"subject": str(triple[0] or "").strip(),
|
||||
"predicate": str(triple[1] or "").strip(),
|
||||
"object": str(triple[2] or "").strip(),
|
||||
}
|
||||
)
|
||||
|
||||
entities = [str(item or "").strip() for item in doc.get("extracted_entities", []) or [] if str(item or "").strip()]
|
||||
all_entities.update(entities)
|
||||
for relation in relations:
|
||||
if relation["subject"]:
|
||||
all_entities.add(relation["subject"])
|
||||
if relation["object"]:
|
||||
all_entities.add(relation["object"])
|
||||
|
||||
memorix_data["paragraphs"].append(
|
||||
{
|
||||
"hash": compute_paragraph_hash(content),
|
||||
"content": content,
|
||||
"source": filename,
|
||||
"entities": entities,
|
||||
"relations": relations,
|
||||
}
|
||||
)
|
||||
|
||||
memorix_data["entities"] = sorted(all_entities)
|
||||
return memorix_data
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
parser = _build_arg_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
target_path = Path(args.path)
|
||||
if not target_path.exists():
|
||||
logger.error(f"路径不存在: {target_path}")
|
||||
return
|
||||
|
||||
if target_path.is_dir():
|
||||
files_to_process = list(target_path.glob("*-openie.json")) or list(target_path.glob("*.json"))
|
||||
else:
|
||||
files_to_process = [target_path]
|
||||
|
||||
if not files_to_process:
|
||||
logger.error("未找到可处理的 JSON 文件")
|
||||
return
|
||||
|
||||
importer = AutoImporter(force=bool(args.force), concurrency=int(args.concurrency))
|
||||
if not await importer.initialize():
|
||||
logger.error("初始化存储失败")
|
||||
return
|
||||
|
||||
converter = LPMMConverter()
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
TimeElapsedColumn(),
|
||||
console=console,
|
||||
transient=False,
|
||||
) as progress:
|
||||
for json_file in files_to_process:
|
||||
logger.info(f"正在转换并导入: {json_file.name}")
|
||||
try:
|
||||
with open(json_file, "r", encoding="utf-8") as handle:
|
||||
lpmm_data = json.load(handle)
|
||||
memorix_data = converter.convert_lpmm_to_memorix(lpmm_data, json_file.name)
|
||||
total_items = len(memorix_data.get("paragraphs", []))
|
||||
if total_items <= 0:
|
||||
logger.warning(f"转换结果为空: {json_file.name}")
|
||||
continue
|
||||
|
||||
task_id = progress.add_task(f"Importing {json_file.name}", total=total_items)
|
||||
|
||||
def update_progress(step: int = 1) -> None:
|
||||
progress.advance(task_id, advance=step)
|
||||
|
||||
await importer.import_json_data(
|
||||
memorix_data,
|
||||
filename=f"lpmm_{json_file.name}",
|
||||
progress_callback=update_progress,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.error(f"处理文件 {json_file.name} 失败: {exc}\n{traceback.format_exc()}")
|
||||
|
||||
await importer.close()
|
||||
logger.info("全部处理完成")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.platform == "win32": # pragma: no cover
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
|
||||
asyncio.run(main())
|
||||
1714
plugins/A_memorix/scripts/migrate_maibot_memory.py
Normal file
1714
plugins/A_memorix/scripts/migrate_maibot_memory.py
Normal file
File diff suppressed because it is too large
Load Diff
728
plugins/A_memorix/scripts/process_knowledge.py
Normal file
728
plugins/A_memorix/scripts/process_knowledge.py
Normal file
@@ -0,0 +1,728 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
知识库自动导入脚本 (Strategy-Aware Version)
|
||||
|
||||
功能：
|
||||
1. 扫描 plugins/A_memorix/data/raw 下的 .txt 文件
|
||||
2. 检查 data/import_manifest.json 确认是否已导入
|
||||
3. 使用 Strategy 模式处理文件 (Narrative/Factual/Quote)
|
||||
4. 将生成的数据直接存入 VectorStore/GraphStore/MetadataStore
|
||||
5. 更新 manifest
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import asyncio
|
||||
import time
|
||||
import random
|
||||
import hashlib
|
||||
import tomlkit
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import List, Dict, Any, Optional
|
||||
from rich.progress import Progress, SpinnerColumn, BarColumn, TextColumn, TimeElapsedColumn
|
||||
from rich.console import Console
|
||||
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
|
||||
|
||||
console = Console()
|
||||
|
||||
class LLMGenerationError(Exception):
|
||||
pass
|
||||
|
||||
# 路径设置
|
||||
current_dir = Path(__file__).resolve().parent
|
||||
plugin_root = current_dir.parent
|
||||
workspace_root = plugin_root.parent
|
||||
maibot_root = workspace_root / "MaiBot"
|
||||
for path in (workspace_root, maibot_root, plugin_root):
|
||||
path_str = str(path)
|
||||
if path_str not in sys.path:
|
||||
sys.path.insert(0, path_str)
|
||||
|
||||
# 数据目录
|
||||
DATA_DIR = plugin_root / "data"
|
||||
RAW_DIR = DATA_DIR / "raw"
|
||||
PROCESSED_DIR = DATA_DIR / "processed"
|
||||
MANIFEST_PATH = DATA_DIR / "import_manifest.json"
|
||||
|
||||
|
||||
def _build_arg_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="A_Memorix Knowledge Importer (Strategy-Aware)")
|
||||
parser.add_argument("--force", action="store_true", help="Force re-import")
|
||||
parser.add_argument("--clear-manifest", action="store_true", help="Clear manifest")
|
||||
parser.add_argument(
|
||||
"--type",
|
||||
"-t",
|
||||
default="auto",
|
||||
help="Target import strategy override (auto/narrative/factual/quote)",
|
||||
)
|
||||
parser.add_argument("--concurrency", "-c", type=int, default=5)
|
||||
parser.add_argument(
|
||||
"--chat-log",
|
||||
action="store_true",
|
||||
help="聊天记录导入模式：强制 narrative 策略，并使用 LLM 语义抽取 event_time/event_time_range",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--chat-reference-time",
|
||||
default=None,
|
||||
help="chat_log 模式的相对时间参考点（如 2026/02/12 10:30）；不传则使用当前本地时间",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
# --help/-h fast path: avoid heavy host/plugin bootstrap
|
||||
if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
try:
|
||||
import A_memorix.core as core_module
|
||||
import A_memorix.core.storage as storage_module
|
||||
from src.common.logger import get_logger
|
||||
from src.services import llm_service as llm_api
|
||||
from src.config.config import global_config, model_config
|
||||
|
||||
VectorStore = core_module.VectorStore
|
||||
GraphStore = core_module.GraphStore
|
||||
MetadataStore = core_module.MetadataStore
|
||||
ImportStrategy = core_module.ImportStrategy
|
||||
create_embedding_api_adapter = core_module.create_embedding_api_adapter
|
||||
RelationWriteService = getattr(core_module, "RelationWriteService", None)
|
||||
|
||||
looks_like_quote_text = storage_module.looks_like_quote_text
|
||||
parse_import_strategy = storage_module.parse_import_strategy
|
||||
resolve_stored_knowledge_type = storage_module.resolve_stored_knowledge_type
|
||||
select_import_strategy = storage_module.select_import_strategy
|
||||
|
||||
from A_memorix.core.utils.time_parser import normalize_time_meta
|
||||
from A_memorix.core.utils.import_payloads import normalize_paragraph_import_item
|
||||
from A_memorix.core.strategies.base import BaseStrategy, ProcessedChunk, KnowledgeType as StratKnowledgeType
|
||||
from A_memorix.core.strategies.narrative import NarrativeStrategy
|
||||
from A_memorix.core.strategies.factual import FactualStrategy
|
||||
from A_memorix.core.strategies.quote import QuoteStrategy
|
||||
|
||||
except ImportError as e:
|
||||
print(f"❌ 无法导入模块: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
logger = get_logger("A_Memorix.AutoImport")
|
||||
|
||||
|
||||
def _log_before_retry(retry_state) -> None:
|
||||
"""使用项目统一日志风格记录重试信息。"""
|
||||
exc = None
|
||||
if getattr(retry_state, "outcome", None) is not None and retry_state.outcome.failed:
|
||||
exc = retry_state.outcome.exception()
|
||||
next_sleep = getattr(getattr(retry_state, "next_action", None), "sleep", None)
|
||||
logger.warning(
|
||||
"LLM 调用即将重试: "
|
||||
f"attempt={getattr(retry_state, 'attempt_number', '?')} "
|
||||
f"next_sleep={next_sleep} "
|
||||
f"error={exc}"
|
||||
)
|
||||
|
||||
class AutoImporter:
|
||||
def __init__(
|
||||
self,
|
||||
force: bool = False,
|
||||
clear_manifest: bool = False,
|
||||
target_type: str = "auto",
|
||||
concurrency: int = 5,
|
||||
chat_log: bool = False,
|
||||
chat_reference_time: Optional[str] = None,
|
||||
):
|
||||
self.vector_store: Optional[VectorStore] = None
|
||||
self.graph_store: Optional[GraphStore] = None
|
||||
self.metadata_store: Optional[MetadataStore] = None
|
||||
self.embedding_manager = None
|
||||
self.relation_write_service = None
|
||||
self.plugin_config = {}
|
||||
self.manifest = {}
|
||||
self.force = force
|
||||
self.clear_manifest = clear_manifest
|
||||
self.chat_log = chat_log
|
||||
parsed_target_type = parse_import_strategy(target_type, default=ImportStrategy.AUTO)
|
||||
self.target_type = ImportStrategy.NARRATIVE.value if chat_log else parsed_target_type.value
|
||||
self.chat_reference_dt = self._parse_reference_time(chat_reference_time)
|
||||
if self.chat_log and parsed_target_type not in {ImportStrategy.AUTO, ImportStrategy.NARRATIVE}:
|
||||
logger.warning(
|
||||
f"chat_log 模式已启用，target_type={target_type} 将被覆盖为 narrative"
|
||||
)
|
||||
self.concurrency_limit = concurrency
|
||||
self.semaphore = None
|
||||
self.storage_lock = None
|
||||
|
||||
async def initialize(self):
|
||||
logger.info(f"正在初始化... (并发数: {self.concurrency_limit})")
|
||||
self.semaphore = asyncio.Semaphore(self.concurrency_limit)
|
||||
self.storage_lock = asyncio.Lock()
|
||||
|
||||
RAW_DIR.mkdir(parents=True, exist_ok=True)
|
||||
PROCESSED_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if self.clear_manifest:
|
||||
logger.info("🧹 清理 Mainfest")
|
||||
self.manifest = {}
|
||||
self._save_manifest()
|
||||
elif MANIFEST_PATH.exists():
|
||||
try:
|
||||
with open(MANIFEST_PATH, "r", encoding="utf-8") as f:
|
||||
self.manifest = json.load(f)
|
||||
except Exception:
|
||||
self.manifest = {}
|
||||
|
||||
config_path = plugin_root / "config.toml"
|
||||
try:
|
||||
with open(config_path, "r", encoding="utf-8") as f:
|
||||
self.plugin_config = tomlkit.load(f)
|
||||
except Exception as e:
|
||||
logger.error(f"加载插件配置失败: {e}")
|
||||
return False
|
||||
|
||||
try:
|
||||
await self._init_stores()
|
||||
except Exception as e:
|
||||
logger.error(f"初始化存储失败: {e}")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
async def _init_stores(self):
|
||||
# ... (Same as original)
|
||||
self.embedding_manager = create_embedding_api_adapter(
|
||||
batch_size=self.plugin_config.get("embedding", {}).get("batch_size", 32),
|
||||
default_dimension=self.plugin_config.get("embedding", {}).get("dimension", 384),
|
||||
model_name=self.plugin_config.get("embedding", {}).get("model_name", "auto"),
|
||||
retry_config=self.plugin_config.get("embedding", {}).get("retry", {}),
|
||||
)
|
||||
try:
|
||||
dim = await self.embedding_manager._detect_dimension()
|
||||
except:
|
||||
dim = self.embedding_manager.default_dimension
|
||||
|
||||
q_type_str = str(self.plugin_config.get("embedding", {}).get("quantization_type", "int8") or "int8").lower()
|
||||
# Need to access QuantizationType from storage_module if not imported globally
|
||||
QuantizationType = storage_module.QuantizationType
|
||||
if q_type_str != "int8":
|
||||
raise ValueError(
|
||||
"embedding.quantization_type 在 vNext 仅允许 int8(SQ8)。"
|
||||
" 请先执行 scripts/release_vnext_migrate.py migrate。"
|
||||
)
|
||||
|
||||
self.vector_store = VectorStore(
|
||||
dimension=dim,
|
||||
quantization_type=QuantizationType.INT8,
|
||||
data_dir=DATA_DIR / "vectors"
|
||||
)
|
||||
|
||||
SparseMatrixFormat = storage_module.SparseMatrixFormat
|
||||
m_fmt_str = self.plugin_config.get("graph", {}).get("sparse_matrix_format", "csr")
|
||||
m_map = {"csr": SparseMatrixFormat.CSR, "csc": SparseMatrixFormat.CSC}
|
||||
|
||||
self.graph_store = GraphStore(
|
||||
matrix_format=m_map.get(m_fmt_str, SparseMatrixFormat.CSR),
|
||||
data_dir=DATA_DIR / "graph"
|
||||
)
|
||||
|
||||
self.metadata_store = MetadataStore(data_dir=DATA_DIR / "metadata")
|
||||
self.metadata_store.connect()
|
||||
|
||||
if RelationWriteService is not None:
|
||||
self.relation_write_service = RelationWriteService(
|
||||
metadata_store=self.metadata_store,
|
||||
graph_store=self.graph_store,
|
||||
vector_store=self.vector_store,
|
||||
embedding_manager=self.embedding_manager,
|
||||
)
|
||||
|
||||
if self.vector_store.has_data(): self.vector_store.load()
|
||||
if self.graph_store.has_data(): self.graph_store.load()
|
||||
|
||||
def _should_write_relation_vectors(self) -> bool:
|
||||
retrieval_cfg = self.plugin_config.get("retrieval", {})
|
||||
if not isinstance(retrieval_cfg, dict):
|
||||
return False
|
||||
rv_cfg = retrieval_cfg.get("relation_vectorization", {})
|
||||
if not isinstance(rv_cfg, dict):
|
||||
return False
|
||||
return bool(rv_cfg.get("enabled", False)) and bool(rv_cfg.get("write_on_import", True))
|
||||
|
||||
def load_file(self, file_path: Path) -> str:
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
return f.read()
|
||||
|
||||
def get_file_hash(self, content: str) -> str:
|
||||
return hashlib.md5(content.encode("utf-8")).hexdigest()
|
||||
|
||||
def _parse_reference_time(self, value: Optional[str]) -> datetime:
|
||||
"""解析 chat_log 模式的参考时间（用于相对时间语义解析）。"""
|
||||
if not value:
|
||||
return datetime.now()
|
||||
formats = [
|
||||
"%Y/%m/%d %H:%M:%S",
|
||||
"%Y/%m/%d %H:%M",
|
||||
"%Y-%m-%d %H:%M:%S",
|
||||
"%Y-%m-%d %H:%M",
|
||||
"%Y/%m/%d",
|
||||
"%Y-%m-%d",
|
||||
]
|
||||
text = str(value).strip()
|
||||
for fmt in formats:
|
||||
try:
|
||||
return datetime.strptime(text, fmt)
|
||||
except ValueError:
|
||||
continue
|
||||
logger.warning(
|
||||
f"无法解析 chat_reference_time={value}，将回退为当前本地时间"
|
||||
)
|
||||
return datetime.now()
|
||||
|
||||
async def _extract_chat_time_meta_with_llm(
|
||||
self,
|
||||
text: str,
|
||||
model_config: Any,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
使用 LLM 从聊天文本语义中抽取时间信息。
|
||||
支持将相对时间表达转换为绝对时间。
|
||||
"""
|
||||
if not text.strip():
|
||||
return None
|
||||
|
||||
reference_now = self.chat_reference_dt.strftime("%Y/%m/%d %H:%M")
|
||||
prompt = f"""You are a time extraction engine for chat logs.
|
||||
Extract temporal information from the following chat paragraph.
|
||||
|
||||
Rules:
|
||||
1. Use semantic understanding, not regex matching.
|
||||
2. Convert relative expressions (e.g., yesterday evening, last Friday morning) to absolute local datetime using reference_now.
|
||||
3. If a time span exists, return event_time_start/event_time_end.
|
||||
4. If only one point in time exists, return event_time.
|
||||
5. If no reliable time can be inferred, return all time fields as null.
|
||||
6. Output ONLY valid JSON. No markdown, no explanation.
|
||||
|
||||
reference_now: {reference_now}
|
||||
timezone: local system timezone
|
||||
|
||||
Allowed output formats for time values:
|
||||
- "YYYY/MM/DD"
|
||||
- "YYYY/MM/DD HH:mm"
|
||||
|
||||
JSON schema:
|
||||
{{
|
||||
"event_time": null,
|
||||
"event_time_start": null,
|
||||
"event_time_end": null,
|
||||
"time_range": null,
|
||||
"time_granularity": "day",
|
||||
"time_confidence": 0.0
|
||||
}}
|
||||
|
||||
Chat paragraph:
|
||||
\"\"\"{text}\"\"\"
|
||||
"""
|
||||
try:
|
||||
result = await self._llm_call(prompt, model_config)
|
||||
except Exception as e:
|
||||
logger.warning(f"chat_log 时间语义抽取失败: {e}")
|
||||
return None
|
||||
|
||||
if not isinstance(result, dict):
|
||||
return None
|
||||
|
||||
raw_time_meta = {
|
||||
"event_time": result.get("event_time"),
|
||||
"event_time_start": result.get("event_time_start"),
|
||||
"event_time_end": result.get("event_time_end"),
|
||||
"time_range": result.get("time_range"),
|
||||
"time_granularity": result.get("time_granularity"),
|
||||
"time_confidence": result.get("time_confidence"),
|
||||
}
|
||||
try:
|
||||
normalized = normalize_time_meta(raw_time_meta)
|
||||
except Exception as e:
|
||||
logger.warning(f"chat_log 时间语义抽取结果不可用，已忽略: {e}")
|
||||
return None
|
||||
|
||||
has_effective_time = any(
|
||||
key in normalized
|
||||
for key in ("event_time", "event_time_start", "event_time_end")
|
||||
)
|
||||
if not has_effective_time:
|
||||
return None
|
||||
|
||||
return normalized
|
||||
|
||||
def _determine_strategy(self, filename: str, content: str) -> BaseStrategy:
|
||||
"""Layer 1: Global Strategy Routing"""
|
||||
strategy = select_import_strategy(
|
||||
content,
|
||||
override=self.target_type,
|
||||
chat_log=self.chat_log,
|
||||
)
|
||||
if self.chat_log:
|
||||
logger.info(f"chat_log 模式: {filename} 强制使用 NarrativeStrategy")
|
||||
elif strategy == ImportStrategy.QUOTE:
|
||||
logger.info(f"Auto-detected Quote/Lyric type for {filename}")
|
||||
|
||||
if strategy == ImportStrategy.FACTUAL:
|
||||
return FactualStrategy(filename)
|
||||
if strategy == ImportStrategy.QUOTE:
|
||||
return QuoteStrategy(filename)
|
||||
return NarrativeStrategy(filename)
|
||||
|
||||
def _chunk_rescue(self, chunk: ProcessedChunk, filename: str) -> Optional[BaseStrategy]:
|
||||
"""Layer 2: Chunk-level rescue strategies"""
|
||||
# If we are already in Quote strategy, no need to rescue
|
||||
if chunk.type == StratKnowledgeType.QUOTE:
|
||||
return None
|
||||
|
||||
if looks_like_quote_text(chunk.chunk.text):
|
||||
logger.info(f" > Rescuing chunk {chunk.chunk.index} as Quote")
|
||||
return QuoteStrategy(filename)
|
||||
|
||||
return None
|
||||
|
||||
async def process_and_import(self):
|
||||
if not await self.initialize(): return
|
||||
|
||||
files = list(RAW_DIR.glob("*.txt"))
|
||||
logger.info(f"扫描到 {len(files)} 个文件 in {RAW_DIR}")
|
||||
|
||||
if not files: return
|
||||
|
||||
tasks = []
|
||||
for file_path in files:
|
||||
tasks.append(asyncio.create_task(self._process_single_file(file_path)))
|
||||
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
success_count = sum(1 for r in results if r is True)
|
||||
logger.info(f"本次主处理完成，共成功处理 {success_count}/{len(files)} 个文件")
|
||||
|
||||
if self.vector_store: self.vector_store.save()
|
||||
if self.graph_store: self.graph_store.save()
|
||||
|
||||
async def _process_single_file(self, file_path: Path) -> bool:
|
||||
filename = file_path.name
|
||||
async with self.semaphore:
|
||||
try:
|
||||
content = self.load_file(file_path)
|
||||
file_hash = self.get_file_hash(content)
|
||||
|
||||
if not self.force and filename in self.manifest:
|
||||
record = self.manifest[filename]
|
||||
if record.get("hash") == file_hash and record.get("imported"):
|
||||
logger.info(f"跳过已导入文件: {filename}")
|
||||
return False
|
||||
|
||||
logger.info(f">>> 开始处理: {filename}")
|
||||
|
||||
# 1. Strategy Selection
|
||||
strategy = self._determine_strategy(filename, content)
|
||||
logger.info(f" 策略: {strategy.__class__.__name__}")
|
||||
|
||||
# 2. Split (Strategy-Aware)
|
||||
initial_chunks = strategy.split(content)
|
||||
logger.info(f" 初步分块: {len(initial_chunks)}")
|
||||
|
||||
processed_data = {"paragraphs": [], "entities": [], "relations": []}
|
||||
|
||||
# 3. Extract Loop
|
||||
model_config = await self._select_model()
|
||||
|
||||
for i, chunk in enumerate(initial_chunks):
|
||||
current_strategy = strategy
|
||||
# Layer 2: Chunk Rescue
|
||||
rescue_strategy = self._chunk_rescue(chunk, filename)
|
||||
if rescue_strategy:
|
||||
# Re-split? No, just re-process this text as a single chunk using the rescue strategy
|
||||
# But rescue strategy might want to split it further?
|
||||
# Simplification: Treat the whole chunk text as one block for the rescue strategy
|
||||
# OR create a single chunk object for it.
|
||||
# Creating a new chunk using rescue strategy logic might be complex if split behavior differs.
|
||||
# Let's just instantiate a chunk of the new type manually
|
||||
chunk.type = StratKnowledgeType.QUOTE
|
||||
chunk.flags.verbatim = True
|
||||
chunk.flags.requires_llm = False # Quotes don't usually need LLM
|
||||
current_strategy = rescue_strategy
|
||||
|
||||
# Extraction
|
||||
if chunk.flags.requires_llm:
|
||||
result_chunk = await current_strategy.extract(chunk, lambda p: self._llm_call(p, model_config))
|
||||
else:
|
||||
# For quotes, extract might be just pass through or regex
|
||||
result_chunk = await current_strategy.extract(chunk)
|
||||
|
||||
time_meta = None
|
||||
if self.chat_log:
|
||||
time_meta = await self._extract_chat_time_meta_with_llm(
|
||||
result_chunk.chunk.text,
|
||||
model_config,
|
||||
)
|
||||
|
||||
# Normalize Data
|
||||
self._normalize_and_aggregate(
|
||||
result_chunk,
|
||||
processed_data,
|
||||
time_meta=time_meta,
|
||||
)
|
||||
|
||||
logger.info(f" 已处理块 {i+1}/{len(initial_chunks)}")
|
||||
|
||||
# 4. Save Json
|
||||
json_path = PROCESSED_DIR / f"{file_path.stem}.json"
|
||||
with open(json_path, "w", encoding="utf-8") as f:
|
||||
json.dump(processed_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
# 5. Import to DB
|
||||
async with self.storage_lock:
|
||||
await self._import_to_db(processed_data)
|
||||
|
||||
self.manifest[filename] = {
|
||||
"hash": file_hash,
|
||||
"timestamp": time.time(),
|
||||
"imported": True
|
||||
}
|
||||
self._save_manifest()
|
||||
self.vector_store.save()
|
||||
self.graph_store.save()
|
||||
logger.info(f"✅ 文件 {filename} 处理并导入完成")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"❌ 处理失败 {filename}: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
def _normalize_and_aggregate(
|
||||
self,
|
||||
chunk: ProcessedChunk,
|
||||
all_data: Dict,
|
||||
time_meta: Optional[Dict[str, Any]] = None,
|
||||
):
|
||||
"""Convert strategy-specific data to unified generic format for storage."""
|
||||
# Generic fields
|
||||
para_item = {
|
||||
"content": chunk.chunk.text,
|
||||
"source": chunk.source.file,
|
||||
"knowledge_type": resolve_stored_knowledge_type(
|
||||
chunk.type.value,
|
||||
content=chunk.chunk.text,
|
||||
).value,
|
||||
"entities": [],
|
||||
"relations": []
|
||||
}
|
||||
|
||||
data = chunk.data
|
||||
|
||||
# 1. Triples (Factual)
|
||||
if "triples" in data:
|
||||
for t in data["triples"]:
|
||||
para_item["relations"].append({
|
||||
"subject": t.get("subject"),
|
||||
"predicate": t.get("predicate"),
|
||||
"object": t.get("object")
|
||||
})
|
||||
# Auto-add entities from triples
|
||||
para_item["entities"].extend([t.get("subject"), t.get("object")])
|
||||
|
||||
# 2. Events & Relations (Narrative)
|
||||
if "events" in data:
|
||||
# Store events as content/metadata? Or entities?
|
||||
# For now maybe just keep them in logic, or add as 'Event' entities?
|
||||
# Creating entities for events is good.
|
||||
para_item["entities"].extend(data["events"])
|
||||
|
||||
if "relations" in data: # Narrative also outputs relations list
|
||||
para_item["relations"].extend(data["relations"])
|
||||
for r in data["relations"]:
|
||||
para_item["entities"].extend([r.get("subject"), r.get("object")])
|
||||
|
||||
# 3. Verbatim Entities (Quote)
|
||||
if "verbatim_entities" in data:
|
||||
para_item["entities"].extend(data["verbatim_entities"])
|
||||
|
||||
# Dedupe per paragraph
|
||||
para_item["entities"] = list(set([e for e in para_item["entities"] if e]))
|
||||
|
||||
if time_meta:
|
||||
para_item["time_meta"] = time_meta
|
||||
|
||||
all_data["paragraphs"].append(para_item)
|
||||
all_data["entities"].extend(para_item["entities"])
|
||||
if "relations" in para_item:
|
||||
all_data["relations"].extend(para_item["relations"])
|
||||
|
||||
@retry(
|
||||
retry=retry_if_exception_type((LLMGenerationError, json.JSONDecodeError)),
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential(multiplier=1, min=2, max=10),
|
||||
before_sleep=_log_before_retry
|
||||
)
|
||||
async def _llm_call(self, prompt: str, model_config: Any) -> Dict:
|
||||
"""Generic LLM Caller"""
|
||||
success, response, _, _ = await llm_api.generate_with_model(
|
||||
prompt=prompt,
|
||||
model_config=model_config,
|
||||
request_type="Script.ProcessKnowledge"
|
||||
)
|
||||
if success:
|
||||
txt = response.strip()
|
||||
if "```" in txt:
|
||||
txt = txt.split("```json")[-1].split("```")[0].strip()
|
||||
try:
|
||||
return json.loads(txt)
|
||||
except json.JSONDecodeError:
|
||||
# Fallback: try to find first { and last }
|
||||
start = txt.find('{')
|
||||
end = txt.rfind('}')
|
||||
if start != -1 and end != -1:
|
||||
return json.loads(txt[start:end+1])
|
||||
raise
|
||||
else:
|
||||
raise LLMGenerationError("LLM generation failed")
|
||||
|
||||
async def _select_model(self) -> Any:
|
||||
models = llm_api.get_available_models()
|
||||
if not models: raise ValueError("No LLM models")
|
||||
|
||||
config_model = self.plugin_config.get("advanced", {}).get("extraction_model", "auto")
|
||||
if config_model != "auto" and config_model in models:
|
||||
return models[config_model]
|
||||
|
||||
for task_key in ["lpmm_entity_extract", "lpmm_rdf_build", "embedding"]:
|
||||
if task_key in models: return models[task_key]
|
||||
|
||||
return models[list(models.keys())[0]]
|
||||
|
||||
# Re-use existing methods
|
||||
async def _add_entity_with_vector(self, name: str, source_paragraph: Optional[str] = None) -> str:
|
||||
# Same as before
|
||||
hash_value = self.metadata_store.add_entity(name, source_paragraph=source_paragraph)
|
||||
self.graph_store.add_nodes([name])
|
||||
try:
|
||||
emb = await self.embedding_manager.encode(name)
|
||||
try:
|
||||
self.vector_store.add(emb.reshape(1, -1), [hash_value])
|
||||
except ValueError: pass
|
||||
except Exception: pass
|
||||
return hash_value
|
||||
|
||||
async def import_json_data(self, data: Dict, filename: str = "script_import", progress_callback=None):
|
||||
"""Public import entrypoint for pre-processed JSON payloads."""
|
||||
if not self.storage_lock:
|
||||
raise RuntimeError("Importer is not initialized. Call initialize() first.")
|
||||
|
||||
async with self.storage_lock:
|
||||
await self._import_to_db(data, progress_callback=progress_callback)
|
||||
self.manifest[filename] = {
|
||||
"hash": self.get_file_hash(json.dumps(data, ensure_ascii=False, sort_keys=True)),
|
||||
"timestamp": time.time(),
|
||||
"imported": True,
|
||||
}
|
||||
self._save_manifest()
|
||||
self.vector_store.save()
|
||||
self.graph_store.save()
|
||||
|
||||
async def _import_to_db(self, data: Dict, progress_callback=None):
|
||||
# Same logic, but ensure robust
|
||||
with self.graph_store.batch_update():
|
||||
for item in data.get("paragraphs", []):
|
||||
paragraph = normalize_paragraph_import_item(
|
||||
item,
|
||||
default_source="script",
|
||||
)
|
||||
content = paragraph["content"]
|
||||
source = paragraph["source"]
|
||||
k_type_val = paragraph["knowledge_type"]
|
||||
|
||||
h_val = self.metadata_store.add_paragraph(
|
||||
content=content,
|
||||
source=source,
|
||||
knowledge_type=k_type_val,
|
||||
time_meta=paragraph["time_meta"],
|
||||
)
|
||||
|
||||
if h_val not in self.vector_store:
|
||||
try:
|
||||
emb = await self.embedding_manager.encode(content)
|
||||
self.vector_store.add(emb.reshape(1, -1), [h_val])
|
||||
except Exception as e:
|
||||
logger.error(f" Vector fail: {e}")
|
||||
|
||||
para_entities = paragraph["entities"]
|
||||
for entity in para_entities:
|
||||
if entity:
|
||||
await self._add_entity_with_vector(entity, source_paragraph=h_val)
|
||||
|
||||
para_relations = paragraph["relations"]
|
||||
for rel in para_relations:
|
||||
s, p, o = rel.get("subject"), rel.get("predicate"), rel.get("object")
|
||||
if s and p and o:
|
||||
await self._add_entity_with_vector(s, source_paragraph=h_val)
|
||||
await self._add_entity_with_vector(o, source_paragraph=h_val)
|
||||
confidence = float(rel.get("confidence", 1.0) or 1.0)
|
||||
rel_meta = rel.get("metadata", {})
|
||||
write_vector = self._should_write_relation_vectors()
|
||||
if self.relation_write_service is not None:
|
||||
await self.relation_write_service.upsert_relation_with_vector(
|
||||
subject=s,
|
||||
predicate=p,
|
||||
obj=o,
|
||||
confidence=confidence,
|
||||
source_paragraph=h_val,
|
||||
metadata=rel_meta if isinstance(rel_meta, dict) else {},
|
||||
write_vector=write_vector,
|
||||
)
|
||||
else:
|
||||
rel_hash = self.metadata_store.add_relation(
|
||||
s,
|
||||
p,
|
||||
o,
|
||||
confidence=confidence,
|
||||
source_paragraph=h_val,
|
||||
metadata=rel_meta if isinstance(rel_meta, dict) else {},
|
||||
)
|
||||
self.graph_store.add_edges([(s, o)], relation_hashes=[rel_hash])
|
||||
try:
|
||||
self.metadata_store.set_relation_vector_state(rel_hash, "none")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if progress_callback: progress_callback(1)
|
||||
|
||||
async def close(self):
|
||||
if self.metadata_store: self.metadata_store.close()
|
||||
|
||||
def _save_manifest(self):
|
||||
with open(MANIFEST_PATH, "w", encoding="utf-8") as f:
|
||||
json.dump(self.manifest, f, ensure_ascii=False, indent=2)
|
||||
|
||||
async def main():
|
||||
parser = _build_arg_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
if not global_config: return
|
||||
|
||||
importer = AutoImporter(
|
||||
force=args.force,
|
||||
clear_manifest=args.clear_manifest,
|
||||
target_type=args.type,
|
||||
concurrency=args.concurrency,
|
||||
chat_log=args.chat_log,
|
||||
chat_reference_time=args.chat_reference_time,
|
||||
)
|
||||
await importer.process_and_import()
|
||||
await importer.close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.platform == "win32":
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
|
||||
asyncio.run(main())
|
||||
127
plugins/A_memorix/scripts/rebuild_episodes.py
Normal file
127
plugins/A_memorix/scripts/rebuild_episodes.py
Normal file
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Episode source 级重建工具。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
PLUGIN_ROOT = CURRENT_DIR.parent
|
||||
WORKSPACE_ROOT = PLUGIN_ROOT.parent
|
||||
MAIBOT_ROOT = WORKSPACE_ROOT / "MaiBot"
|
||||
for path in (WORKSPACE_ROOT, MAIBOT_ROOT, PLUGIN_ROOT):
|
||||
path_str = str(path)
|
||||
if path_str not in sys.path:
|
||||
sys.path.insert(0, path_str)
|
||||
|
||||
try:
|
||||
import tomlkit # type: ignore
|
||||
except Exception: # pragma: no cover
|
||||
tomlkit = None
|
||||
|
||||
from A_memorix.core.storage import MetadataStore
|
||||
from A_memorix.core.utils.episode_service import EpisodeService
|
||||
|
||||
|
||||
def _build_arg_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="Rebuild A_Memorix episodes by source")
|
||||
parser.add_argument("--data-dir", default=str(PLUGIN_ROOT / "data"), help="插件数据目录")
|
||||
parser.add_argument("--source", type=str, help="指定单个 source 入队/重建")
|
||||
parser.add_argument("--all", action="store_true", help="对所有 source 入队/重建")
|
||||
parser.add_argument("--wait", action="store_true", help="在脚本内同步执行重建")
|
||||
return parser
|
||||
|
||||
|
||||
if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
raise SystemExit(0)
|
||||
|
||||
|
||||
def _load_plugin_config() -> Dict[str, Any]:
|
||||
config_path = PLUGIN_ROOT / "config.toml"
|
||||
if tomlkit is None or not config_path.exists():
|
||||
return {}
|
||||
try:
|
||||
with open(config_path, "r", encoding="utf-8") as handle:
|
||||
parsed = tomlkit.load(handle)
|
||||
return dict(parsed) if isinstance(parsed, dict) else {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _resolve_sources(store: MetadataStore, *, source: str | None, rebuild_all: bool) -> List[str]:
|
||||
if rebuild_all:
|
||||
return list(store.list_episode_sources_for_rebuild())
|
||||
token = str(source or "").strip()
|
||||
if not token:
|
||||
raise ValueError("必须提供 --source 或 --all")
|
||||
return [token]
|
||||
|
||||
|
||||
async def _run_rebuilds(store: MetadataStore, plugin_config: Dict[str, Any], sources: List[str]) -> int:
|
||||
service = EpisodeService(metadata_store=store, plugin_config=plugin_config)
|
||||
failures: List[str] = []
|
||||
for source in sources:
|
||||
started = store.mark_episode_source_running(source)
|
||||
if not started:
|
||||
failures.append(f"{source}: unable_to_mark_running")
|
||||
continue
|
||||
try:
|
||||
result = await service.rebuild_source(source)
|
||||
store.mark_episode_source_done(source)
|
||||
print(
|
||||
"rebuilt"
|
||||
f" source={source}"
|
||||
f" paragraphs={int(result.get('paragraph_count') or 0)}"
|
||||
f" groups={int(result.get('group_count') or 0)}"
|
||||
f" episodes={int(result.get('episode_count') or 0)}"
|
||||
f" fallback={int(result.get('fallback_count') or 0)}"
|
||||
)
|
||||
except Exception as exc:
|
||||
err = str(exc)[:500]
|
||||
store.mark_episode_source_failed(source, err)
|
||||
failures.append(f"{source}: {err}")
|
||||
print(f"failed source={source} error={err}")
|
||||
|
||||
if failures:
|
||||
for item in failures:
|
||||
print(item)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = _build_arg_parser()
|
||||
args = parser.parse_args()
|
||||
if bool(args.all) == bool(args.source):
|
||||
parser.error("必须且只能选择一个：--source 或 --all")
|
||||
|
||||
store = MetadataStore(data_dir=Path(args.data_dir) / "metadata")
|
||||
store.connect()
|
||||
try:
|
||||
sources = _resolve_sources(store, source=args.source, rebuild_all=bool(args.all))
|
||||
if not sources:
|
||||
print("no sources to rebuild")
|
||||
return 0
|
||||
|
||||
enqueued = 0
|
||||
reason = "script_rebuild_all" if args.all else "script_rebuild_source"
|
||||
for source in sources:
|
||||
enqueued += int(store.enqueue_episode_source_rebuild(source, reason=reason))
|
||||
print(f"enqueued={enqueued} sources={len(sources)}")
|
||||
|
||||
if not args.wait:
|
||||
return 0
|
||||
|
||||
plugin_config = _load_plugin_config()
|
||||
return asyncio.run(_run_rebuilds(store, plugin_config, sources))
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
731
plugins/A_memorix/scripts/release_vnext_migrate.py
Normal file
731
plugins/A_memorix/scripts/release_vnext_migrate.py
Normal file
@@ -0,0 +1,731 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
vNext release migration entrypoint for A_Memorix.
|
||||
|
||||
Subcommands:
|
||||
- preflight: detect legacy config/data/schema risks
|
||||
- migrate: offline migrate config + vectors + metadata schema + graph edge hash map
|
||||
- verify: strict post-migration consistency checks
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pickle
|
||||
import sqlite3
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
|
||||
|
||||
import tomlkit
|
||||
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
PLUGIN_ROOT = CURRENT_DIR.parent
|
||||
PROJECT_ROOT = PLUGIN_ROOT.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
||||
|
||||
def _build_arg_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="A_Memorix vNext release migration tool")
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
default=str(PLUGIN_ROOT / "config.toml"),
|
||||
help="config.toml path (default: plugins/A_memorix/config.toml)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--data-dir",
|
||||
default="",
|
||||
help="optional data dir override; default resolved from config.storage.data_dir",
|
||||
)
|
||||
parser.add_argument("--json-out", default="", help="optional JSON report output path")
|
||||
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_preflight = sub.add_parser("preflight", help="scan legacy risks")
|
||||
p_preflight.add_argument("--strict", action="store_true", help="return 1 if any error check exists")
|
||||
|
||||
p_migrate = sub.add_parser("migrate", help="run offline migration")
|
||||
p_migrate.add_argument("--dry-run", action="store_true", help="only print planned changes")
|
||||
p_migrate.add_argument(
|
||||
"--verify-after",
|
||||
action="store_true",
|
||||
help="run verify automatically after migrate",
|
||||
)
|
||||
|
||||
p_verify = sub.add_parser("verify", help="post-migration verification")
|
||||
p_verify.add_argument("--strict", action="store_true", help="return 1 if any error check exists")
|
||||
return parser
|
||||
|
||||
|
||||
# --help/-h fast path: avoid heavy host/plugin bootstrap
|
||||
if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
raise SystemExit(0)
|
||||
|
||||
try:
|
||||
from core.storage import GraphStore, KnowledgeType, MetadataStore, QuantizationType, VectorStore
|
||||
from core.storage.metadata_store import SCHEMA_VERSION
|
||||
except Exception as e: # pragma: no cover
|
||||
print(f"❌ failed to import storage modules: {e}")
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CheckItem:
|
||||
code: str
|
||||
level: str
|
||||
message: str
|
||||
details: Optional[Dict[str, Any]] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
out = {
|
||||
"code": self.code,
|
||||
"level": self.level,
|
||||
"message": self.message,
|
||||
}
|
||||
if self.details:
|
||||
out["details"] = self.details
|
||||
return out
|
||||
|
||||
|
||||
def _read_toml(path: Path) -> Dict[str, Any]:
|
||||
text = path.read_text(encoding="utf-8")
|
||||
return tomlkit.parse(text)
|
||||
|
||||
|
||||
def _write_toml(path: Path, data: Dict[str, Any]) -> None:
|
||||
path.write_text(tomlkit.dumps(data), encoding="utf-8")
|
||||
|
||||
|
||||
def _get_nested(obj: Dict[str, Any], keys: Sequence[str], default: Any = None) -> Any:
|
||||
cur: Any = obj
|
||||
for k in keys:
|
||||
if not isinstance(cur, dict) or k not in cur:
|
||||
return default
|
||||
cur = cur[k]
|
||||
return cur
|
||||
|
||||
|
||||
def _ensure_table(obj: Dict[str, Any], key: str) -> Dict[str, Any]:
|
||||
if key not in obj or not isinstance(obj[key], dict):
|
||||
obj[key] = tomlkit.table()
|
||||
return obj[key]
|
||||
|
||||
|
||||
def _resolve_data_dir(config_doc: Dict[str, Any], explicit_data_dir: Optional[str]) -> Path:
|
||||
if explicit_data_dir:
|
||||
return Path(explicit_data_dir).expanduser().resolve()
|
||||
raw = str(_get_nested(config_doc, ("storage", "data_dir"), "./data") or "./data").strip()
|
||||
if raw.startswith("."):
|
||||
return (PLUGIN_ROOT / raw).resolve()
|
||||
return Path(raw).expanduser().resolve()
|
||||
|
||||
|
||||
def _sqlite_table_exists(conn: sqlite3.Connection, table: str) -> bool:
|
||||
row = conn.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name=? LIMIT 1",
|
||||
(table,),
|
||||
).fetchone()
|
||||
return row is not None
|
||||
|
||||
|
||||
def _collect_hash_alias_conflicts(conn: sqlite3.Connection) -> Dict[str, List[str]]:
|
||||
hashes: List[str] = []
|
||||
if _sqlite_table_exists(conn, "relations"):
|
||||
rows = conn.execute("SELECT hash FROM relations").fetchall()
|
||||
hashes.extend(str(r[0]) for r in rows if r and r[0])
|
||||
if _sqlite_table_exists(conn, "deleted_relations"):
|
||||
rows = conn.execute("SELECT hash FROM deleted_relations").fetchall()
|
||||
hashes.extend(str(r[0]) for r in rows if r and r[0])
|
||||
|
||||
alias_map: Dict[str, str] = {}
|
||||
conflicts: Dict[str, set[str]] = {}
|
||||
for h in hashes:
|
||||
if len(h) != 64:
|
||||
continue
|
||||
alias = h[:32]
|
||||
old = alias_map.get(alias)
|
||||
if old is None:
|
||||
alias_map[alias] = h
|
||||
continue
|
||||
if old != h:
|
||||
conflicts.setdefault(alias, set()).update({old, h})
|
||||
return {k: sorted(v) for k, v in conflicts.items()}
|
||||
|
||||
|
||||
def _collect_invalid_knowledge_types(conn: sqlite3.Connection) -> List[str]:
|
||||
if not _sqlite_table_exists(conn, "paragraphs"):
|
||||
return []
|
||||
|
||||
allowed = {item.value for item in KnowledgeType}
|
||||
rows = conn.execute("SELECT DISTINCT knowledge_type FROM paragraphs").fetchall()
|
||||
invalid: List[str] = []
|
||||
for row in rows:
|
||||
raw = row[0]
|
||||
value = str(raw).strip().lower() if raw is not None else ""
|
||||
if value not in allowed:
|
||||
invalid.append(str(raw) if raw is not None else "")
|
||||
return sorted(set(invalid))
|
||||
|
||||
|
||||
def _guess_vector_dimension(config_doc: Dict[str, Any], vectors_dir: Path) -> int:
|
||||
meta_path = vectors_dir / "vectors_metadata.pkl"
|
||||
if meta_path.exists():
|
||||
try:
|
||||
with open(meta_path, "rb") as f:
|
||||
meta = pickle.load(f)
|
||||
dim = int(meta.get("dimension", 0))
|
||||
if dim > 0:
|
||||
return dim
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
dim_cfg = int(_get_nested(config_doc, ("embedding", "dimension"), 1024))
|
||||
if dim_cfg > 0:
|
||||
return dim_cfg
|
||||
except Exception:
|
||||
pass
|
||||
return 1024
|
||||
|
||||
|
||||
def _preflight_impl(config_path: Path, data_dir: Path) -> Dict[str, Any]:
|
||||
checks: List[CheckItem] = []
|
||||
facts: Dict[str, Any] = {
|
||||
"config_path": str(config_path),
|
||||
"data_dir": str(data_dir),
|
||||
}
|
||||
|
||||
if not config_path.exists():
|
||||
checks.append(CheckItem("CFG-00", "error", f"config not found: {config_path}"))
|
||||
return {"ok": False, "checks": [c.to_dict() for c in checks], "facts": facts}
|
||||
|
||||
config_doc = _read_toml(config_path)
|
||||
tool_mode = str(_get_nested(config_doc, ("routing", "tool_search_mode"), "forward") or "").strip().lower()
|
||||
summary_model = _get_nested(config_doc, ("summarization", "model_name"), ["auto"])
|
||||
summary_knowledge_type = str(
|
||||
_get_nested(config_doc, ("summarization", "default_knowledge_type"), "narrative") or "narrative"
|
||||
).strip().lower()
|
||||
quantization = str(_get_nested(config_doc, ("embedding", "quantization_type"), "int8") or "").strip().lower()
|
||||
|
||||
facts["routing.tool_search_mode"] = tool_mode
|
||||
facts["summarization.model_name_type"] = type(summary_model).__name__
|
||||
facts["summarization.default_knowledge_type"] = summary_knowledge_type
|
||||
facts["embedding.quantization_type"] = quantization
|
||||
|
||||
if tool_mode == "legacy":
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-04",
|
||||
"error",
|
||||
"routing.tool_search_mode=legacy is no longer accepted at runtime",
|
||||
)
|
||||
)
|
||||
elif tool_mode not in {"forward", "disabled"}:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-04",
|
||||
"error",
|
||||
f"routing.tool_search_mode invalid value: {tool_mode}",
|
||||
)
|
||||
)
|
||||
|
||||
if isinstance(summary_model, str):
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-11",
|
||||
"error",
|
||||
"summarization.model_name must be List[str], string legacy format detected",
|
||||
)
|
||||
)
|
||||
elif not isinstance(summary_model, list) or any(not isinstance(x, str) for x in summary_model):
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-11",
|
||||
"error",
|
||||
"summarization.model_name must be List[str]",
|
||||
)
|
||||
)
|
||||
|
||||
if summary_knowledge_type not in {item.value for item in KnowledgeType}:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-13",
|
||||
"error",
|
||||
f"invalid summarization.default_knowledge_type: {summary_knowledge_type}",
|
||||
)
|
||||
)
|
||||
|
||||
if quantization != "int8":
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"UG-07",
|
||||
"error",
|
||||
"embedding.quantization_type must be int8 in vNext",
|
||||
)
|
||||
)
|
||||
|
||||
vectors_dir = data_dir / "vectors"
|
||||
npy_path = vectors_dir / "vectors.npy"
|
||||
bin_path = vectors_dir / "vectors.bin"
|
||||
ids_bin_path = vectors_dir / "vectors_ids.bin"
|
||||
facts["vectors.npy_exists"] = npy_path.exists()
|
||||
facts["vectors.bin_exists"] = bin_path.exists()
|
||||
facts["vectors_ids.bin_exists"] = ids_bin_path.exists()
|
||||
|
||||
if npy_path.exists() and not (bin_path.exists() and ids_bin_path.exists()):
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-07",
|
||||
"error",
|
||||
"legacy vectors.npy detected; offline migrate required",
|
||||
{"npy_path": str(npy_path)},
|
||||
)
|
||||
)
|
||||
|
||||
metadata_db = data_dir / "metadata" / "metadata.db"
|
||||
facts["metadata_db_exists"] = metadata_db.exists()
|
||||
relation_count = 0
|
||||
if metadata_db.exists():
|
||||
conn = sqlite3.connect(str(metadata_db))
|
||||
try:
|
||||
has_schema_table = _sqlite_table_exists(conn, "schema_migrations")
|
||||
facts["schema_migrations_exists"] = has_schema_table
|
||||
if not has_schema_table:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-08",
|
||||
"error",
|
||||
"schema_migrations table missing (legacy metadata schema)",
|
||||
)
|
||||
)
|
||||
else:
|
||||
row = conn.execute("SELECT MAX(version) FROM schema_migrations").fetchone()
|
||||
version = int(row[0]) if row and row[0] is not None else 0
|
||||
facts["schema_version"] = version
|
||||
if version != SCHEMA_VERSION:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-08",
|
||||
"error",
|
||||
f"schema version mismatch: current={version}, expected={SCHEMA_VERSION}",
|
||||
)
|
||||
)
|
||||
|
||||
if _sqlite_table_exists(conn, "relations"):
|
||||
row = conn.execute("SELECT COUNT(*) FROM relations").fetchone()
|
||||
relation_count = int(row[0]) if row and row[0] is not None else 0
|
||||
facts["relations_count"] = relation_count
|
||||
|
||||
conflicts = _collect_hash_alias_conflicts(conn)
|
||||
facts["alias_conflict_count"] = len(conflicts)
|
||||
if conflicts:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-05",
|
||||
"error",
|
||||
"32-bit relation hash alias conflict detected",
|
||||
{"aliases": sorted(conflicts.keys())[:20], "total": len(conflicts)},
|
||||
)
|
||||
)
|
||||
|
||||
invalid_knowledge_types = _collect_invalid_knowledge_types(conn)
|
||||
facts["invalid_knowledge_type_values"] = invalid_knowledge_types
|
||||
if invalid_knowledge_types:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-12",
|
||||
"error",
|
||||
"invalid paragraph knowledge_type values detected",
|
||||
{"values": invalid_knowledge_types[:20], "total": len(invalid_knowledge_types)},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
conn.close()
|
||||
else:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"META-00",
|
||||
"warning",
|
||||
"metadata.db not found, schema checks skipped",
|
||||
)
|
||||
)
|
||||
|
||||
graph_meta_path = data_dir / "graph" / "graph_metadata.pkl"
|
||||
facts["graph_metadata_exists"] = graph_meta_path.exists()
|
||||
if relation_count > 0:
|
||||
if not graph_meta_path.exists():
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-06",
|
||||
"error",
|
||||
"relations exist but graph metadata missing",
|
||||
)
|
||||
)
|
||||
else:
|
||||
try:
|
||||
with open(graph_meta_path, "rb") as f:
|
||||
graph_meta = pickle.load(f)
|
||||
edge_hash_map = graph_meta.get("edge_hash_map", {})
|
||||
edge_hash_map_size = len(edge_hash_map) if isinstance(edge_hash_map, dict) else 0
|
||||
facts["edge_hash_map_size"] = edge_hash_map_size
|
||||
if edge_hash_map_size <= 0:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-06",
|
||||
"error",
|
||||
"edge_hash_map missing/empty while relations exist",
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-06",
|
||||
"error",
|
||||
f"failed to read graph metadata: {e}",
|
||||
)
|
||||
)
|
||||
|
||||
has_error = any(c.level == "error" for c in checks)
|
||||
return {
|
||||
"ok": not has_error,
|
||||
"checks": [c.to_dict() for c in checks],
|
||||
"facts": facts,
|
||||
}
|
||||
|
||||
|
||||
def _migrate_config(config_doc: Dict[str, Any]) -> Dict[str, Any]:
|
||||
changes: Dict[str, Any] = {}
|
||||
|
||||
routing = _ensure_table(config_doc, "routing")
|
||||
mode_raw = str(routing.get("tool_search_mode", "forward") or "").strip().lower()
|
||||
mode_new = mode_raw
|
||||
if mode_raw == "legacy" or mode_raw not in {"forward", "disabled"}:
|
||||
mode_new = "forward"
|
||||
if mode_new != mode_raw:
|
||||
routing["tool_search_mode"] = mode_new
|
||||
changes["routing.tool_search_mode"] = {"old": mode_raw, "new": mode_new}
|
||||
|
||||
summary = _ensure_table(config_doc, "summarization")
|
||||
summary_model = summary.get("model_name", ["auto"])
|
||||
if isinstance(summary_model, str):
|
||||
normalized = [summary_model.strip() or "auto"]
|
||||
summary["model_name"] = normalized
|
||||
changes["summarization.model_name"] = {"old": summary_model, "new": normalized}
|
||||
elif not isinstance(summary_model, list):
|
||||
normalized = ["auto"]
|
||||
summary["model_name"] = normalized
|
||||
changes["summarization.model_name"] = {"old": str(type(summary_model)), "new": normalized}
|
||||
elif any(not isinstance(x, str) for x in summary_model):
|
||||
normalized = [str(x).strip() for x in summary_model if str(x).strip()]
|
||||
if not normalized:
|
||||
normalized = ["auto"]
|
||||
summary["model_name"] = normalized
|
||||
changes["summarization.model_name"] = {"old": summary_model, "new": normalized}
|
||||
|
||||
default_knowledge_type = str(summary.get("default_knowledge_type", "narrative") or "").strip().lower()
|
||||
allowed_knowledge_types = {item.value for item in KnowledgeType}
|
||||
if default_knowledge_type not in allowed_knowledge_types:
|
||||
summary["default_knowledge_type"] = "narrative"
|
||||
changes["summarization.default_knowledge_type"] = {
|
||||
"old": default_knowledge_type,
|
||||
"new": "narrative",
|
||||
}
|
||||
|
||||
embedding = _ensure_table(config_doc, "embedding")
|
||||
quantization = str(embedding.get("quantization_type", "int8") or "").strip().lower()
|
||||
if quantization != "int8":
|
||||
embedding["quantization_type"] = "int8"
|
||||
changes["embedding.quantization_type"] = {"old": quantization, "new": "int8"}
|
||||
|
||||
return changes
|
||||
|
||||
|
||||
def _migrate_impl(config_path: Path, data_dir: Path, dry_run: bool) -> Dict[str, Any]:
|
||||
config_doc = _read_toml(config_path)
|
||||
result: Dict[str, Any] = {
|
||||
"config_path": str(config_path),
|
||||
"data_dir": str(data_dir),
|
||||
"dry_run": bool(dry_run),
|
||||
"steps": {},
|
||||
}
|
||||
|
||||
config_changes = _migrate_config(config_doc)
|
||||
result["steps"]["config"] = {"changed": bool(config_changes), "changes": config_changes}
|
||||
if config_changes and not dry_run:
|
||||
_write_toml(config_path, config_doc)
|
||||
|
||||
vectors_dir = data_dir / "vectors"
|
||||
vectors_dir.mkdir(parents=True, exist_ok=True)
|
||||
npy_path = vectors_dir / "vectors.npy"
|
||||
bin_path = vectors_dir / "vectors.bin"
|
||||
ids_bin_path = vectors_dir / "vectors_ids.bin"
|
||||
if npy_path.exists() and not (bin_path.exists() and ids_bin_path.exists()):
|
||||
if dry_run:
|
||||
result["steps"]["vector"] = {"migrated": False, "reason": "dry_run"}
|
||||
else:
|
||||
dim = _guess_vector_dimension(config_doc, vectors_dir)
|
||||
store = VectorStore(
|
||||
dimension=max(1, int(dim)),
|
||||
quantization_type=QuantizationType.INT8,
|
||||
data_dir=vectors_dir,
|
||||
)
|
||||
result["steps"]["vector"] = store.migrate_legacy_npy(vectors_dir)
|
||||
else:
|
||||
result["steps"]["vector"] = {"migrated": False, "reason": "not_required"}
|
||||
|
||||
metadata_dir = data_dir / "metadata"
|
||||
metadata_dir.mkdir(parents=True, exist_ok=True)
|
||||
metadata_db = metadata_dir / "metadata.db"
|
||||
triples: List[Tuple[str, str, str, str]] = []
|
||||
relation_count = 0
|
||||
|
||||
metadata_result: Dict[str, Any] = {"migrated": False, "reason": "not_required"}
|
||||
if metadata_db.exists():
|
||||
store = MetadataStore(data_dir=metadata_dir)
|
||||
store.connect(enforce_schema=False)
|
||||
try:
|
||||
if dry_run:
|
||||
metadata_result = {"migrated": False, "reason": "dry_run"}
|
||||
else:
|
||||
metadata_result = store.run_legacy_migration_for_vnext()
|
||||
relation_count = int(store.count_relations())
|
||||
if relation_count > 0:
|
||||
triples = [(str(s), str(p), str(o), str(h)) for s, p, o, h in store.get_all_triples()]
|
||||
finally:
|
||||
store.close()
|
||||
result["steps"]["metadata"] = metadata_result
|
||||
|
||||
graph_dir = data_dir / "graph"
|
||||
graph_dir.mkdir(parents=True, exist_ok=True)
|
||||
graph_matrix_format = str(_get_nested(config_doc, ("graph", "sparse_matrix_format"), "csr") or "csr")
|
||||
graph_store = GraphStore(matrix_format=graph_matrix_format, data_dir=graph_dir)
|
||||
graph_step: Dict[str, Any] = {
|
||||
"rebuilt": False,
|
||||
"mapped_hashes": 0,
|
||||
"relation_count": relation_count,
|
||||
"topology_rebuilt_from_relations": False,
|
||||
}
|
||||
if relation_count > 0:
|
||||
if dry_run:
|
||||
graph_step["reason"] = "dry_run"
|
||||
else:
|
||||
if graph_store.has_data():
|
||||
graph_store.load()
|
||||
|
||||
mapped = graph_store.rebuild_edge_hash_map(triples)
|
||||
|
||||
# 兜底：历史数据里 graph 节点/边与 relations 脱节时，直接从 relations 重建图。
|
||||
if mapped <= 0 or not graph_store.has_edge_hash_map():
|
||||
nodes = sorted({s for s, _, o, _ in triples} | {o for _, _, o, _ in triples})
|
||||
edges = [(s, o) for s, _, o, _ in triples]
|
||||
hashes = [h for _, _, _, h in triples]
|
||||
|
||||
graph_store.clear()
|
||||
if nodes:
|
||||
graph_store.add_nodes(nodes)
|
||||
if edges:
|
||||
mapped = graph_store.add_edges(edges, relation_hashes=hashes)
|
||||
else:
|
||||
mapped = 0
|
||||
graph_step.update(
|
||||
{
|
||||
"topology_rebuilt_from_relations": True,
|
||||
"rebuilt_nodes": len(nodes),
|
||||
"rebuilt_edges": int(graph_store.num_edges),
|
||||
}
|
||||
)
|
||||
|
||||
graph_store.save()
|
||||
graph_step.update({"rebuilt": True, "mapped_hashes": int(mapped)})
|
||||
else:
|
||||
graph_step["reason"] = "no_relations"
|
||||
result["steps"]["graph"] = graph_step
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _verify_impl(config_path: Path, data_dir: Path) -> Dict[str, Any]:
|
||||
checks: List[CheckItem] = []
|
||||
facts: Dict[str, Any] = {
|
||||
"config_path": str(config_path),
|
||||
"data_dir": str(data_dir),
|
||||
}
|
||||
|
||||
if not config_path.exists():
|
||||
checks.append(CheckItem("CFG-00", "error", f"config not found: {config_path}"))
|
||||
return {"ok": False, "checks": [c.to_dict() for c in checks], "facts": facts}
|
||||
|
||||
config_doc = _read_toml(config_path)
|
||||
mode = str(_get_nested(config_doc, ("routing", "tool_search_mode"), "forward") or "").strip().lower()
|
||||
if mode not in {"forward", "disabled"}:
|
||||
checks.append(CheckItem("CP-04", "error", f"invalid routing.tool_search_mode: {mode}"))
|
||||
|
||||
summary_model = _get_nested(config_doc, ("summarization", "model_name"), ["auto"])
|
||||
if not isinstance(summary_model, list) or any(not isinstance(x, str) for x in summary_model):
|
||||
checks.append(CheckItem("CP-11", "error", "summarization.model_name must be List[str]"))
|
||||
summary_knowledge_type = str(
|
||||
_get_nested(config_doc, ("summarization", "default_knowledge_type"), "narrative") or "narrative"
|
||||
).strip().lower()
|
||||
if summary_knowledge_type not in {item.value for item in KnowledgeType}:
|
||||
checks.append(
|
||||
CheckItem("CP-13", "error", f"invalid summarization.default_knowledge_type: {summary_knowledge_type}")
|
||||
)
|
||||
|
||||
quantization = str(_get_nested(config_doc, ("embedding", "quantization_type"), "int8") or "").strip().lower()
|
||||
if quantization != "int8":
|
||||
checks.append(CheckItem("UG-07", "error", "embedding.quantization_type must be int8"))
|
||||
|
||||
vectors_dir = data_dir / "vectors"
|
||||
npy_path = vectors_dir / "vectors.npy"
|
||||
bin_path = vectors_dir / "vectors.bin"
|
||||
ids_bin_path = vectors_dir / "vectors_ids.bin"
|
||||
if npy_path.exists() and not (bin_path.exists() and ids_bin_path.exists()):
|
||||
checks.append(CheckItem("CP-07", "error", "legacy vectors.npy still exists without bin migration"))
|
||||
|
||||
metadata_dir = data_dir / "metadata"
|
||||
store = MetadataStore(data_dir=metadata_dir)
|
||||
try:
|
||||
store.connect(enforce_schema=True)
|
||||
schema_version = store.get_schema_version()
|
||||
facts["schema_version"] = schema_version
|
||||
if schema_version != SCHEMA_VERSION:
|
||||
checks.append(CheckItem("CP-08", "error", f"schema version mismatch: {schema_version}"))
|
||||
|
||||
relation_count = int(store.count_relations())
|
||||
facts["relations_count"] = relation_count
|
||||
|
||||
conflicts = {}
|
||||
invalid_knowledge_types: List[str] = []
|
||||
db_path = metadata_dir / "metadata.db"
|
||||
if db_path.exists():
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
try:
|
||||
conflicts = _collect_hash_alias_conflicts(conn)
|
||||
invalid_knowledge_types = _collect_invalid_knowledge_types(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
if conflicts:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-05",
|
||||
"error",
|
||||
"alias conflicts still exist after migration",
|
||||
{"aliases": sorted(conflicts.keys())[:20], "total": len(conflicts)},
|
||||
)
|
||||
)
|
||||
if invalid_knowledge_types:
|
||||
checks.append(
|
||||
CheckItem(
|
||||
"CP-12",
|
||||
"error",
|
||||
"invalid paragraph knowledge_type values remain after migration",
|
||||
{"values": invalid_knowledge_types[:20], "total": len(invalid_knowledge_types)},
|
||||
)
|
||||
)
|
||||
|
||||
if relation_count > 0:
|
||||
graph_dir = data_dir / "graph"
|
||||
if not (graph_dir / "graph_metadata.pkl").exists():
|
||||
checks.append(CheckItem("CP-06", "error", "graph metadata missing while relations exist"))
|
||||
else:
|
||||
matrix_format = str(_get_nested(config_doc, ("graph", "sparse_matrix_format"), "csr") or "csr")
|
||||
graph_store = GraphStore(matrix_format=matrix_format, data_dir=graph_dir)
|
||||
graph_store.load()
|
||||
if not graph_store.has_edge_hash_map():
|
||||
checks.append(CheckItem("CP-06", "error", "edge_hash_map is empty"))
|
||||
except Exception as e:
|
||||
checks.append(CheckItem("CP-08", "error", f"metadata strict connect failed: {e}"))
|
||||
finally:
|
||||
try:
|
||||
store.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
has_error = any(c.level == "error" for c in checks)
|
||||
return {
|
||||
"ok": not has_error,
|
||||
"checks": [c.to_dict() for c in checks],
|
||||
"facts": facts,
|
||||
}
|
||||
|
||||
|
||||
def _print_report(title: str, report: Dict[str, Any]) -> None:
|
||||
print(f"=== {title} ===")
|
||||
print(f"ok: {bool(report.get('ok', True))}")
|
||||
facts = report.get("facts", {})
|
||||
if facts:
|
||||
print("facts:")
|
||||
for k in sorted(facts.keys()):
|
||||
print(f" - {k}: {facts[k]}")
|
||||
checks = report.get("checks", [])
|
||||
if checks:
|
||||
print("checks:")
|
||||
for item in checks:
|
||||
print(f" - [{item.get('level')}] {item.get('code')}: {item.get('message')}")
|
||||
else:
|
||||
print("checks: none")
|
||||
|
||||
|
||||
def _write_json_if_needed(path: str, payload: Dict[str, Any]) -> None:
|
||||
if not path:
|
||||
return
|
||||
out = Path(path).expanduser().resolve()
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"json_out: {out}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = _build_arg_parser()
|
||||
args = parser.parse_args()
|
||||
config_path = Path(args.config).expanduser().resolve()
|
||||
if not config_path.exists():
|
||||
print(f"❌ config not found: {config_path}")
|
||||
return 2
|
||||
config_doc = _read_toml(config_path)
|
||||
data_dir = _resolve_data_dir(config_doc, args.data_dir)
|
||||
|
||||
if args.command == "preflight":
|
||||
report = _preflight_impl(config_path, data_dir)
|
||||
_print_report("vNext Preflight", report)
|
||||
_write_json_if_needed(args.json_out, report)
|
||||
has_error = any(item.get("level") == "error" for item in report.get("checks", []))
|
||||
if args.strict and has_error:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
if args.command == "migrate":
|
||||
payload = _migrate_impl(config_path, data_dir, dry_run=bool(args.dry_run))
|
||||
print("=== vNext Migrate ===")
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
|
||||
verify_report = None
|
||||
if args.verify_after and not args.dry_run:
|
||||
verify_report = _verify_impl(config_path, data_dir)
|
||||
_print_report("vNext Verify (after migrate)", verify_report)
|
||||
payload["verify_after"] = verify_report
|
||||
|
||||
_write_json_if_needed(args.json_out, payload)
|
||||
if verify_report is not None:
|
||||
has_error = any(item.get("level") == "error" for item in verify_report.get("checks", []))
|
||||
if has_error:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
if args.command == "verify":
|
||||
report = _verify_impl(config_path, data_dir)
|
||||
_print_report("vNext Verify", report)
|
||||
_write_json_if_needed(args.json_out, report)
|
||||
has_error = any(item.get("level") == "error" for item in report.get("checks", []))
|
||||
if args.strict and has_error:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
152
plugins/A_memorix/scripts/runtime_self_check.py
Normal file
152
plugins/A_memorix/scripts/runtime_self_check.py
Normal file
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run A_Memorix runtime self-check against real embedding/runtime configuration."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import tomlkit
|
||||
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
PLUGIN_ROOT = CURRENT_DIR.parent
|
||||
PROJECT_ROOT = PLUGIN_ROOT.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
sys.path.insert(0, str(PLUGIN_ROOT))
|
||||
|
||||
|
||||
def _build_arg_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="A_Memorix runtime self-check")
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
default=str(PLUGIN_ROOT / "config.toml"),
|
||||
help="config.toml path (default: plugins/A_memorix/config.toml)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--data-dir",
|
||||
default="",
|
||||
help="optional data dir override; default resolved from config.storage.data_dir",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--use-config-data-dir",
|
||||
action="store_true",
|
||||
help="use config.storage.data_dir directly instead of an isolated temp dir",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sample-text",
|
||||
default="A_Memorix runtime self check",
|
||||
help="sample text used for real embedding probe",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="print JSON report")
|
||||
return parser
|
||||
|
||||
|
||||
if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
|
||||
_build_arg_parser().print_help()
|
||||
raise SystemExit(0)
|
||||
|
||||
from core.runtime.lifecycle_orchestrator import initialize_storage_async
|
||||
from core.utils.runtime_self_check import run_embedding_runtime_self_check
|
||||
|
||||
|
||||
def _load_config(path: Path) -> dict[str, Any]:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
raw = tomlkit.load(f)
|
||||
return dict(raw) if isinstance(raw, dict) else {}
|
||||
|
||||
|
||||
def _nested_get(config: dict[str, Any], key: str, default: Any = None) -> Any:
|
||||
current: Any = config
|
||||
for part in key.split("."):
|
||||
if isinstance(current, dict) and part in current:
|
||||
current = current[part]
|
||||
else:
|
||||
return default
|
||||
return current
|
||||
|
||||
|
||||
class _PluginStub:
|
||||
def __init__(self, config: dict[str, Any]):
|
||||
self.config = config
|
||||
self.vector_store = None
|
||||
self.graph_store = None
|
||||
self.metadata_store = None
|
||||
self.embedding_manager = None
|
||||
self.sparse_index = None
|
||||
self.relation_write_service = None
|
||||
|
||||
def get_config(self, key: str, default: Any = None) -> Any:
|
||||
return _nested_get(self.config, key, default)
|
||||
|
||||
|
||||
async def _main_async(args: argparse.Namespace) -> int:
|
||||
config_path = Path(args.config).resolve()
|
||||
if not config_path.exists():
|
||||
print(f"❌ 配置文件不存在: {config_path}")
|
||||
return 2
|
||||
|
||||
config = _load_config(config_path)
|
||||
temp_dir_ctx = None
|
||||
if args.data_dir:
|
||||
storage_dir = str(Path(args.data_dir).resolve())
|
||||
elif args.use_config_data_dir:
|
||||
raw_data_dir = str(_nested_get(config, "storage.data_dir", "./data") or "./data").strip()
|
||||
if raw_data_dir.startswith("."):
|
||||
storage_dir = str((config_path.parent / raw_data_dir).resolve())
|
||||
else:
|
||||
storage_dir = str(Path(raw_data_dir).resolve())
|
||||
else:
|
||||
temp_dir_ctx = tempfile.TemporaryDirectory(prefix="memorix-runtime-self-check-")
|
||||
storage_dir = temp_dir_ctx.name
|
||||
|
||||
storage_cfg = config.setdefault("storage", {})
|
||||
storage_cfg["data_dir"] = storage_dir
|
||||
|
||||
plugin = _PluginStub(config)
|
||||
try:
|
||||
await initialize_storage_async(plugin)
|
||||
report = await run_embedding_runtime_self_check(
|
||||
config=config,
|
||||
vector_store=plugin.vector_store,
|
||||
embedding_manager=plugin.embedding_manager,
|
||||
sample_text=str(args.sample_text or "A_Memorix runtime self check"),
|
||||
)
|
||||
report["data_dir"] = storage_dir
|
||||
report["isolated_data_dir"] = temp_dir_ctx is not None
|
||||
if args.json:
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
else:
|
||||
print("A_Memorix Runtime Self-Check")
|
||||
print(f"ok: {report.get('ok')}")
|
||||
print(f"code: {report.get('code')}")
|
||||
print(f"message: {report.get('message')}")
|
||||
print(f"configured_dimension: {report.get('configured_dimension')}")
|
||||
print(f"vector_store_dimension: {report.get('vector_store_dimension')}")
|
||||
print(f"detected_dimension: {report.get('detected_dimension')}")
|
||||
print(f"encoded_dimension: {report.get('encoded_dimension')}")
|
||||
print(f"elapsed_ms: {float(report.get('elapsed_ms', 0.0)):.2f}")
|
||||
return 0 if bool(report.get("ok")) else 1
|
||||
finally:
|
||||
if plugin.metadata_store is not None:
|
||||
try:
|
||||
plugin.metadata_store.close()
|
||||
except Exception:
|
||||
pass
|
||||
if temp_dir_ctx is not None:
|
||||
temp_dir_ctx.cleanup()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = _build_arg_parser()
|
||||
args = parser.parse_args()
|
||||
return asyncio.run(_main_async(args))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -33,6 +33,7 @@ dependencies = [
|
||||
"python-levenshtein",
|
||||
"quick-algo>=0.1.4",
|
||||
"rich>=14.0.0",
|
||||
"scipy>=1.7.0",
|
||||
"sqlalchemy>=2.0.40",
|
||||
"sqlmodel>=0.0.24",
|
||||
"structlog>=25.4.0",
|
||||
|
||||
@@ -23,6 +23,7 @@ python-multipart>=0.0.20
|
||||
python-levenshtein
|
||||
quick-algo>=0.1.4
|
||||
rich>=14.0.0
|
||||
scipy>=1.7.0
|
||||
sqlalchemy>=2.0.40
|
||||
sqlmodel>=0.0.24
|
||||
structlog>=25.4.0
|
||||
|
||||
@@ -167,6 +167,7 @@ async def main() -> None:
|
||||
system.schedule_tasks(),
|
||||
)
|
||||
finally:
|
||||
emoji_manager.shutdown()
|
||||
await memory_automation_service.shutdown()
|
||||
await get_plugin_runtime_manager().bridge_event("on_stop")
|
||||
await get_plugin_runtime_manager().stop()
|
||||
|
||||
Reference in New Issue
Block a user