tendril/kb/README.md

# Knowledge Base (KB) System

**Last updated**: 2025-01-27

## Overview

The KB system provides a lightweight, LLM-friendly staging area for external information that may inform Tendril project development. It uses structured routing, confidence scoring, and automatic indexing.

The KB system complements the phase documentation system by capturing:
- Pulse Daily chats and ideas
- Informal notes and thoughts about the project
- External information that may inform the project
- Project-level decisions (complements phase-specific ADRs)
- How-to guides and playbooks
- Glossary entries
- Design ideas and UX concepts

---

## Directory Structure

```
kb/
├── 01_projects/              # Project-specific notes
│   ├── tendril/               # Tendril project notes
│   └── _shared/               # Cross-project notes
├── 02_systems/               # Infrastructure, DevOps, tooling
├── 03_research/              # Informal research, links
├── 04_design/                # Product specs, UX, copy
├── 05_decisions/              # ADR-style decisions
├── 06_glossary/               # Terms, acronyms
├── 07_playbooks/             # How-to guides, SOPs
├── 08_archive/                # Superseded content
├── _guides/                   # KB system guidelines (read-only)
│   └── KB_INGEST_PROMPT.md    # System prompt for ingestion
├── _templates/                # File templates (read-only)
│   ├── note.md
│   ├── decision.md
│   └── howto.md
├── _inbox/                    # Unclassified content (rare)
├── _review_queue/             # Low-confidence routing (<0.60)
├── _index.md                  # Auto-generated searchable index
├── CHANGELOG.md               # KB change tracking
├── README.md                  # KB system documentation (this file)
└── scripts/
    └── generate-index.sh      # Index generation script
```

---

## File Naming Schema

**Pattern**: `YYYY-MM-DD--slug--type.md`

### Components

- **Date (YYYY-MM-DD)**: Date content was created or captured
- **Slug**: 3-8 lowercase words, hyphen-joined, no stop-words (a, an, the, and, or, but, etc.)
- **Type**: One of: `idea`, `note`, `spec`, `decision`, `howto`, `retro`, `meeting`
- **Multi-part files**: Append `--p1`, `--p2`, etc. for long content

### Examples

- ✅ `2025-01-27--api-authentication-flow--spec.md`
- ✅ `2025-01-27--user-onboarding-decision--decision.md`
- ✅ `2025-01-27--comprehensive-api-design--spec--p1.md` (multi-part)
- ❌ `api-design.md` (missing date and type)
- ❌ `2025-01-27--the-api--note.md` (contains stop-word "the")

### Regex Pattern

```
^\d{4}-\d{2}-\d{2}--[a-z0-9-]{3,}--(idea|note|spec|decision|howto|retro|meeting)(--p[0-9]+)?\.md$
```

---

## Frontmatter Schema

Every KB file MUST have complete YAML frontmatter with all 18 required fields.

### Required Fields (18)

```yaml
---
title: "Descriptive title in sentence case"
date: "YYYY-MM-DD"                    # Must match filename date
captured_at: "YYYY-MM-DD"             # Same as date (project compatibility)
author: ["author name"]                # Array, default ["unknown author"]
source: { kind: pulse|chat|web|doc, ref: "<link-or-id>" }
source_type: voice|web|pdf|image|personal_note|chat|pulse
project: ["tendril"]                  # Array, default ["tendril"] for Tendril
related_projects: ["tendril"]         # Same as project (compatibility)
topics: ["topic1", "topic2"]          # Array of 2-5 topic keywords
tags: ["tag1", "tag2"]                # Array of 2-8 tags
type: idea|note|spec|decision|howto|retro|meeting  # Must match filename type
status: draft|active|archived
phase_relevance: ["phase-01", "phase-02"]  # Array of relevant phases or []
routing_hint: "Brief explanation of routing decision"
proposed_path: "kb/XX_category/"      # Where LLM intends to file it
routing_confidence: 0.87              # Numeric 0.00-1.00
related: []                           # Array of related file paths
summary: "One to three sentence summary of the content"
---
```

### Optional Fields

```yaml
key_takeaways: ["takeaway1", "takeaway2"]
action_candidates: ["action1", "action2"]
```

### Field Guidelines

- **title**: Clear, descriptive, sentence case (not title case)
- **date**: Must match filename date prefix (YYYY-MM-DD)
- **author**: Array format, use `["unknown author"]` if unknown
- **source**: Object with `kind` (pulse, chat, web, doc) and `ref` (link, ID, or identifier)
- **project**: Array, default `["tendril"]` for Tendril project
- **topics**: Array of 2-5 topic keywords (e.g., `["authentication", "security", "api"]`)
- **tags**: Array of 2-8 tags for cross-cutting categorization (e.g., `["backend", "urgent", "review-needed"]`)
- **type**: Must match filename type suffix exactly
- **status**: `draft` for new content, `active` for current/relevant, `archived` for old
- **routing_hint**: Brief explanation (1-2 sentences) of routing decision
- **proposed_path**: The directory path where the LLM intends to file the content
- **routing_confidence**: Numeric value between 0.00 and 1.00 indicating confidence in the routing decision
- **related**: Array of related file paths or references
- **summary**: Concise 1-3 sentence summary of the content
- **phase_relevance**: Array of relevant phases (e.g., `["phase-01", "phase-02"]`) or empty array `[]`

---

## Routing Decision Tree

Content must be routed to one of these directories based on its primary purpose:

1. **Is it project-specific?** → `/kb/01_projects/tendril/` or `_shared/`
2. **Is it a how-to or procedure?** → `/kb/07_playbooks/`
3. **Is it a decision with rationale?** → `/kb/05_decisions/`
4. **Is it about infrastructure/tooling?** → `/kb/02_systems/`
5. **Is it research or external reference?** → `/kb/03_research/`
6. **Is it design/UX/copy?** → `/kb/04_design/`
7. **Is it a definition or term?** → `/kb/06_glossary/`
8. **Is it obsolete?** → `/kb/08_archive/` (only if truly superseded)
9. **Unclear?** → `/kb/_inbox/` (should be rare)

### Routing Destinations

- **`/kb/01_projects/tendril/`** - Tendril project-specific notes, tasks, or updates
- **`/kb/01_projects/_shared/`** - Cross-project notes that apply to multiple projects
- **`/kb/02_systems/`** - Infrastructure, DevOps, tooling, technical systems, architecture decisions
- **`/kb/03_research/`** - Links, papers, competitor notes, external research, learning materials
- **`/kb/04_design/`** - Product specs, UX design, copy, user experience, interface design
- **`/kb/05_decisions/`** - ADR-style decisions with rationale, architectural decisions, strategic choices
- **`/kb/06_glossary/`** - Terms, acronyms, definitions, vocabulary
- **`/kb/07_playbooks/`** - How-to guides, SOPs, runbooks, procedures, operational guides
- **`/kb/08_archive/`** - Old or superseded content (use sparingly, prefer deletion if truly obsolete)
- **`/kb/_inbox/`** - Use only if content cannot be clearly classified (should be rare)
- **`/kb/_review_queue/`** - Low-confidence routing items requiring human review (routing_confidence < 0.60)

---

## Routing Confidence System

After determining the proposed routing destination, you MUST assess your confidence level:

- **0.90–1.00**: Crystal clear placement - content clearly matches category
- **0.75–0.89**: Good guess - content fits well but might have minor ambiguity
- **0.60–0.74**: Uncertain - content could fit multiple categories, needs review
- **<0.60**: Low confidence - unclear routing, send to `_review_queue/` instead of proposed path

### Routing Policy

If `routing_confidence < 0.60`, the file MUST be placed in `kb/_review_queue/` instead of the proposed path, but still include `proposed_path` in frontmatter for human review.

---

## Usage Guidelines

### Creating KB Files

1. **Use templates**: Copy from `kb/_templates/` and fill in all required fields
2. **Follow naming convention**: `YYYY-MM-DD--slug--type.md`
3. **Complete frontmatter**: All 18 required fields must be present
4. **Route appropriately**: Use routing decision tree to determine correct category
5. **Assess confidence**: Set `routing_confidence` honestly (0.00-1.00)

### KB Ingestion

For LLM-assisted KB ingestion:

1. Paste `/kb/_guides/KB_INGEST_PROMPT.md` contents as the first message in Cursor chat
2. Provide raw content after the system prompt
3. LLM will analyze and route content appropriately
4. Review generated JSON output before creating file
5. Update index and changelog after creating KB file

See `kb/_guides/KB_INGEST_PROMPT.md` for complete ingestion instructions.

### Index and Changelog Updates

**MANDATORY**: After creating, modifying, or deleting ANY file in `kb/` (except `_guides/`, `_templates/`, `README.md`, `_index.md`, and `CHANGELOG.md`):

1. **Run index generation script:**
   ```bash
   kb/scripts/generate-index.sh
   ```
   - This regenerates `kb/_index.md` with current KB content
   - Verify the script completes successfully
   - Check that changes appear in `kb/_index.md`

2. **Update KB changelog:**
   - Add entry to `kb/CHANGELOG.md` with date-based format:
     ```markdown
     ## [YYYY-MM-DD] KB File Added/Modified/Deleted
     
     ### Added/Changed/Removed
     - `kb/XX_category/filename.md` - [Brief description]
     ```
   - Use today's date (YYYY-MM-DD format)
   - Include the full file path relative to repository root
   - Provide a brief description (1-2 sentences) of what changed

3. **Commit all changes together:**
   - Commit KB file change, index update, and changelog entry together
   - Use commit message: `docs(kb): [action] [filename-slug] and update index/changelog`

**NEVER skip these steps** - Both index and changelog updates are mandatory for KB system integrity.

---

## Integration with Phase Documentation

The KB system complements the phase documentation system:

- **KB decisions** (`kb/05_decisions/`) complement phase-specific ADRs (`tendril/phases/phase-XX-name/decisions.md`)
- **KB entries** can reference project phases via `phase_relevance[]` field
- **KB research** (`kb/03_research/`) may inform phase planning and development
- **KB playbooks** (`kb/07_playbooks/`) provide operational guides for phase work

---

## Index Generation

The KB index (`kb/_index.md`) is automatically generated and provides:

- File listing by category
- Topics index (all unique topics with file references)
- Tags index (all unique tags with file references)
- Phase relevance index (files organized by phase)

### Manual Index Generation

```bash
kb/scripts/generate-index.sh
```

### Automatic Index Updates

Gitea Actions workflow (`.github/workflows/kb-index-update.yml`) will auto-update the index on push to main when KB files change. (To be configured in Phase 3)

---

## Templates

Templates are available in `kb/_templates/`:

- **`note.md`** - General notes template
- **`decision.md`** - ADR-style decision template
- **`howto.md`** - How-to guide template

All templates include complete frontmatter with all 18 required fields and placeholder values.

---

## Related Documentation

- **KB Ingestion Prompt**: `kb/_guides/KB_INGEST_PROMPT.md` - Complete system prompt for LLM-assisted KB ingestion
- **KB Changelog**: `kb/CHANGELOG.md` - Change tracking for KB system
- **KB Index**: `kb/_index.md` - Auto-generated searchable index
- **Phase Documentation**: `tendril/phases/phase-XX-name/` - Phase-specific documentation

---

**Location**: `kb/`  
**Maintained by**: Tendril Project Team  
**Repository**: https://git.parkingmeter.info/Mycelium/tendril