Losita/smartmate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Version: 0.9.14.dev.260410

Browse Source 
后端：
  1. LLM 客户端从 newAgent/llm 提升为 infra/llm 基础设施层
     - 删除 backend/newAgent/llm/（ark.go / ark_adapter.go / client.go / json.go）
     - 等价迁移至 backend/infra/llm/，所有 newAgent node 与 service 统一改引用 infrallm
     - 消除 newAgent 对模型客户端的私有依赖，为 memory / websearch 等多模块复用铺路
  2. RAG 基础设施完成可运行态接入（factory / runtime / observer / service 四层成型）
     - 新建 backend/infra/rag/factory.go / runtime.go / observe.go / observer.go /
  service.go：工厂创建、运行时生命周期、轻量观测接口、检索服务门面
     - 更新 infra/rag/config/config.go：补齐 Milvus / Embed / Reranker 全部配置项与默认值
     - 更新 infra/rag/embed/eino_embedder.go：增强 Eino embedding 适配，支持 BaseURL / APIKey 环境变量 / 超时 /
  维度等参数
     - 更新 infra/rag/store/milvus_store.go：完整实现 Milvus 向量存储（建集合 / 建 Index / Upsert / Search /
  Delete），支持 COSINE / L2 / IP 度量
     - 更新 infra/rag/core/pipeline.go：适配 Runtime 接口，Pipeline 由 factory 注入而非手动拼装
     - 更新 infra/rag/corpus/memory_corpus.go / vector_store.go：对接 Memory 模块数据源与 Store 接口扩展
  3. Memory 模块从 Day1 骨架升级为 Day2 完整可运行态
     - 新建 memory/module.go：统一门面 Module，对外封装 EnqueueExtract / ReadService / ManageService / WithTx /
  StartWorker，启动层只依赖这一个入口
     - 新建 memory/orchestrator/llm_write_orchestrator.go：LLM 驱动的记忆抽取编排器，替代原 mock 抽取
     - 新建 memory/service/read_service.go：按用户开关过滤 + 轻量重排 + 访问时间刷新的读取链路
     - 新建 memory/service/manage_service.go：记忆管理面能力（列出 / 软删除 / 开关读写），删除同步写审计日志
     - 新建 memory/service/common.go：服务层公共工具
     - 新建 memory/worker/loop.go：后台轮询循环 RunPollingLoop，定时抢占 pending 任务并推进
     - 新建 memory/utils/audit.go / settings.go：审计日志构造、用户设置过滤等纯函数
     - 更新 memory/model/item.go / job.go / settings.go / config.go / status.go：补齐 DTO 字段与状态常量
     - 更新 memory/repo/item_repo.go / job_repo.go / audit_repo.go / settings_repo.go：补齐 CRUD 与查询能力
     - 更新 memory/worker/runner.go：Runner 对接 Module 与 LLM 抽取器，任务状态机完整化
     - 更新 memory/README.md：同步模块现状说明
  4. newAgent 接入 Memory 读取注入与工具注册依赖预埋
     - 新建 service/agentsvc/agent_memory.go：定义 MemoryReader 接口 + injectMemoryContext，在 graph
  执行前统一补充记忆上下文
     - 更新 service/agentsvc/agent.go：新增 memoryReader 字段与 SetMemoryReader 方法
     - 更新 service/agentsvc/agent_newagent.go：调用 injectMemoryContext 注入 pinned block，检索失败仅降级不阻断主链路
     - 更新 newAgent/tools/registry.go：新增 DefaultRegistryDeps（含 RAGRuntime），工具注册表支持依赖注入
  5. 启动流程与事件处理器接线更新
     - 更新 cmd/start.go：初始化 RAG Runtime → Memory Module → 注册事件处理器 → 启动 Worker 后台轮询
     - 更新 service/events/memory_extract_requested.go：改用 memory.Module.WithTx(tx) 统一门面，事件处理器不再直接依赖
  repo/service 内部包
  6. 缓存插件与配置同步
     - 更新 middleware/cache_deleter.go：静默忽略 MemoryJob / MemoryItem / MemoryAuditLog / MemoryUserSetting
  等新模型，避免日志刷屏；清理冗余注释
     - 更新 config.example.yaml：补齐 rag / memory / websearch 配置段及默认值
     - 更新 go.mod / go.sum：新增 eino-ext/openai / json-patch / go-openai 依赖
  前端：无 仓库：无
This commit is contained in:
Losita
2026-04-10 23:17:38 +08:00
parent fae162162a
commit bf1f1defa5
53 changed files with 5875 additions and 231 deletions
Download Patch File Download Diff File Expand all files Collapse all files

287
backend/memory/worker/runner.go
View File

@@ -6,14 +6,19 @@ import (
"errors"
"fmt"
"log"
"strconv"
"strings"
"time"
infrarag "github.com/LoveLosita/smartflow/backend/infra/rag"
memorymodel "github.com/LoveLosita/smartflow/backend/memory/model"
memoryrepo "github.com/LoveLosita/smartflow/backend/memory/repo"
memoryutils "github.com/LoveLosita/smartflow/backend/memory/utils"
"github.com/LoveLosita/smartflow/backend/model"
"gorm.io/gorm"
)
// RunOnceResult 描述一次手工触发执行结果。
// RunOnceResult 描述单次手工触发执行的结果。
type RunOnceResult struct {
Claimed bool
JobID int64
@@ -21,36 +26,57 @@ type RunOnceResult struct {
Facts int
}
// Runner 是 Day1 首版任务执行器。
// Runner 负责把 memory_jobs 推进成 memory_items 和审计日志。
//
// 职责边界：
// 1. 只负责推进 memory_jobs 状态机；
// 2. Day1 不做 memory_items 真正落库，仅做 mock 抽取与状态推进。
// 1. 负责任务抢占、抽取、落库和状态推进；
// 2. 不负责 outbox 消费，也不负责 LLM prompt 组装；
// 3. 失败时只做可恢复的状态回写，避免把业务错误直接抛到启动层。
type Runner struct {
jobRepo *memoryrepo.JobRepo
extractor Extractor
logger *log.Logger
db *gorm.DB
jobRepo *memoryrepo.JobRepo
itemRepo *memoryrepo.ItemRepo
auditRepo *memoryrepo.AuditRepo
settingsRepo *memoryrepo.SettingsRepo
extractor Extractor
ragRuntime infrarag.Runtime
logger *log.Logger
}
func NewRunner(jobRepo *memoryrepo.JobRepo, extractor Extractor) *Runner {
// NewRunner 构造记忆 worker 执行器。
func NewRunner(
db *gorm.DB,
jobRepo *memoryrepo.JobRepo,
itemRepo *memoryrepo.ItemRepo,
auditRepo *memoryrepo.AuditRepo,
settingsRepo *memoryrepo.SettingsRepo,
extractor Extractor,
ragRuntime infrarag.Runtime,
) *Runner {
return &Runner{
jobRepo: jobRepo,
extractor: extractor,
logger: log.Default(),
db: db,
jobRepo: jobRepo,
itemRepo: itemRepo,
auditRepo: auditRepo,
settingsRepo: settingsRepo,
extractor: extractor,
ragRuntime: ragRuntime,
logger: log.Default(),
}
}
// RunOnce 手工执行一次任务抢占与处理。
// RunOnce 手工执行一轮任务处理。
//
// 返回语义：
// 1. Claimed=false 表示当前无可执行任务；
// 2. Claimed=true 且 Status=success/failed/dead 表示状态已推进完成。
// 1. Claimed=false 表示当前没有可执行任务；
// 2. Claimed=true 且 Status=success/failed/dead 表示本轮已经推进过一个任务；
// 3. 只有初始化缺失或数据库级错误才返回 error。
func (r *Runner) RunOnce(ctx context.Context) (*RunOnceResult, error) {
if r == nil || r.jobRepo == nil || r.extractor == nil {
if r == nil || r.db == nil || r.jobRepo == nil || r.itemRepo == nil || r.auditRepo == nil || r.settingsRepo == nil || r.extractor == nil {
return nil, errors.New("memory worker runner is not initialized")
}
// 1. 抢占一条可执行任务，避免并发 worker 重复处理同一记录。
// 1. 先抢占一条可执行任务，避免多个 worker 重复处理同一条记录。
job, err := r.jobRepo.ClaimNextRunnableExtractJob(ctx, time.Now())
if err != nil {
return nil, err
@@ -66,7 +92,7 @@ func (r *Runner) RunOnce(ctx context.Context) (*RunOnceResult, error) {
Facts: 0,
}
// 2. 解析 payload_json。解析失败属于数据质量问题，走失败重试并打日志。
// 2. 解析任务载荷。这里属于数据质量问题，解析失败就直接标记为可重试失败。
var payload memorymodel.ExtractJobPayload
if err = json.Unmarshal([]byte(job.PayloadJSON), &payload); err != nil {
failReason := fmt.Sprintf("解析任务载荷失败: %v", err)
@@ -75,7 +101,22 @@ func (r *Runner) RunOnce(ctx context.Context) (*RunOnceResult, error) {
return result, nil
}
// 3. 调用抽取器执行 mock 抽取。Day1 先保证“能推进状态”，不引入重计算。
// 3. 先读取用户记忆设置。总开关关闭时，任务直接成功结束，不再继续抽取和落库。
setting, err := r.settingsRepo.GetByUserID(ctx, payload.UserID)
if err != nil {
return nil, err
}
effectiveSetting := memoryutils.EffectiveUserSetting(setting, payload.UserID)
if !effectiveSetting.MemoryEnabled {
if err = r.jobRepo.MarkSuccess(ctx, job.ID); err != nil {
return nil, err
}
result.Status = model.MemoryJobStatusSuccess
r.logger.Printf("memory worker skipped by user setting: job_id=%d user_id=%d", job.ID, payload.UserID)
return result, nil
}
// 4. 调用抽取器。LLM 失败时由编排器做保守 fallback，worker 只关心最终结果。
facts, extractErr := r.extractor.ExtractFacts(ctx, payload)
if extractErr != nil {
failReason := fmt.Sprintf("抽取执行失败: %v", extractErr)
@@ -83,13 +124,213 @@ func (r *Runner) RunOnce(ctx context.Context) (*RunOnceResult, error) {
result.Status = model.MemoryJobStatusFailed
return result, nil
}
facts = memoryutils.FilterFactsBySetting(facts, effectiveSetting)
// 4. 抽取成功后把任务置为 success。
if err = r.jobRepo.MarkSuccess(ctx, job.ID); err != nil {
return nil, err
if len(facts) == 0 {
if err = r.jobRepo.MarkSuccess(ctx, job.ID); err != nil {
return nil, err
}
result.Status = model.MemoryJobStatusSuccess
r.logger.Printf("memory worker run once noop: job_id=%d", job.ID)
return result, nil
}
items := buildMemoryItems(job, payload, facts)
if len(items) == 0 {
if err = r.jobRepo.MarkSuccess(ctx, job.ID); err != nil {
return nil, err
}
result.Status = model.MemoryJobStatusSuccess
r.logger.Printf("memory worker run once empty-after-normalize: job_id=%d", job.ID)
return result, nil
}
// 5. 先在事务里写入记忆条目和审计日志，再统一确认 job 成功。
if err = r.persistMemoryWrite(ctx, job.ID, items); err != nil {
failReason := fmt.Sprintf("记忆落库失败: %v", err)
_ = r.jobRepo.MarkFailed(ctx, job.ID, failReason)
result.Status = model.MemoryJobStatusFailed
return result, nil
}
result.Status = model.MemoryJobStatusSuccess
result.Facts = len(facts)
r.logger.Printf("memory worker run once success: job_id=%d extracted_facts=%d", job.ID, len(facts))
result.Facts = len(items)
r.syncMemoryVectors(ctx, items)
r.logger.Printf("memory worker run once success: job_id=%d extracted_facts=%d", job.ID, len(items))
return result, nil
}
func (r *Runner) persistMemoryWrite(ctx context.Context, jobID int64, items []model.MemoryItem) error {
return r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
jobRepo := r.jobRepo.WithTx(tx)
itemRepo := r.itemRepo.WithTx(tx)
auditRepo := r.auditRepo.WithTx(tx)
if err := itemRepo.UpsertItems(ctx, items); err != nil {
return err
}
for i := range items {
audit := memoryutils.BuildItemAuditLog(
items[i].ID,
items[i].UserID,
memoryutils.AuditOperationCreate,
"system",
"LLM 提取入库",
nil,
&items[i],
)
if err := auditRepo.Create(ctx, audit); err != nil {
return err
}
}
return jobRepo.MarkSuccess(ctx, jobID)
})
}
func buildMemoryItems(job *model.MemoryJob, payload memorymodel.ExtractJobPayload, facts []memorymodel.NormalizedFact) []model.MemoryItem {
if job == nil || len(facts) == 0 {
return nil
}
items := make([]model.MemoryItem, 0, len(facts))
for _, fact := range facts {
items = append(items, model.MemoryItem{
UserID: payload.UserID,
ConversationID: strPtrOrNil(payload.ConversationID),
AssistantID: strPtrOrNil(payload.AssistantID),
RunID: strPtrOrNil(payload.RunID),
MemoryType: fact.MemoryType,
Title: fact.Title,
Content: fact.Content,
NormalizedContent: strPtrFromValue(fact.NormalizedContent),
ContentHash: strPtrFromValue(fact.ContentHash),
Confidence: fact.Confidence,
Importance: fact.Importance,
SensitivityLevel: fact.SensitivityLevel,
SourceMessageID: int64PtrOrNil(payload.SourceMessageID),
SourceEventID: job.SourceEventID,
IsExplicit: fact.IsExplicit,
Status: model.MemoryItemStatusActive,
TTLAt: resolveMemoryTTLAt(payload.OccurredAt, fact.MemoryType),
VectorStatus: "pending",
})
}
return items
}
func (r *Runner) syncMemoryVectors(ctx context.Context, items []model.MemoryItem) {
if r == nil || r.ragRuntime == nil || r.itemRepo == nil || len(items) == 0 {
return
}
requestItems := make([]infrarag.MemoryIngestItem, 0, len(items))
for _, item := range items {
requestItems = append(requestItems, infrarag.MemoryIngestItem{
MemoryID: item.ID,
UserID: item.UserID,
ConversationID: strValue(item.ConversationID),
AssistantID: strValue(item.AssistantID),
RunID: strValue(item.RunID),
MemoryType: item.MemoryType,
Title: item.Title,
Content: item.Content,
Confidence: item.Confidence,
Importance: item.Importance,
SensitivityLevel: item.SensitivityLevel,
IsExplicit: item.IsExplicit,
Status: item.Status,
TTLAt: item.TTLAt,
CreatedAt: item.CreatedAt,
})
}
result, err := r.ragRuntime.IngestMemory(ctx, infrarag.MemoryIngestRequest{
Action: "add",
Items: requestItems,
})
if err != nil {
r.logger.Printf("memory vector sync failed: err=%v", err)
for _, item := range items {
_ = r.itemRepo.UpdateVectorStateByID(ctx, item.ID, "failed", nil)
}
return
}
vectorIDMap := make(map[int64]string, len(result.DocumentIDs))
for _, documentID := range result.DocumentIDs {
memoryID := parseMemoryID(documentID)
if memoryID <= 0 {
continue
}
vectorIDMap[memoryID] = documentID
}
for _, item := range items {
vectorID := strPtrOrNil(vectorIDMap[item.ID])
_ = r.itemRepo.UpdateVectorStateByID(ctx, item.ID, "synced", vectorID)
}
}
func resolveMemoryTTLAt(base time.Time, memoryType string) *time.Time {
switch memoryType {
case memorymodel.MemoryTypeTodoHint:
t := base.Add(30 * 24 * time.Hour)
return &t
case memorymodel.MemoryTypeFact:
t := base.Add(180 * 24 * time.Hour)
return &t
default:
return nil
}
}
func strPtrFromValue(v string) *string {
v = strings.TrimSpace(v)
if v == "" {
return nil
}
value := v
return &value
}
func strPtrOrNil(v string) *string {
v = strings.TrimSpace(v)
if v == "" {
return nil
}
value := v
return &value
}
func int64PtrOrNil(v int64) *int64 {
if v <= 0 {
return nil
}
value := v
return &value
}
func strValue(v *string) string {
if v == nil {
return ""
}
return strings.TrimSpace(*v)
}
func parseMemoryID(documentID string) int64 {
documentID = strings.TrimSpace(documentID)
if !strings.HasPrefix(documentID, "memory:") {
return 0
}
raw := strings.TrimPrefix(documentID, "memory:")
if strings.HasPrefix(raw, "uid:") {
return 0
}
memoryID, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
return 0
}
return memoryID
}