tendril/kb/_guides/PROJECT-SETUP-GUIDE/PHASE-5-PLAN.md

# Phase 5: Validation and Testing - Detailed Plan

**Objective**: Comprehensive testing and validation of all system components

**Status**: Ready for Review  
**Date**: 2025-01-27

---

## Overview

Phase 5 validates that all components of the documentation and automation system work correctly together. This includes testing Cursor rules, KB system, phase documentation synchronization, Gitea Actions workflows, and LLM prompts.

---

## Current State

- Phase 0: Foundation & Cursor Rules (completed)
- Phase 1: Phase Documentation System (completed)
- Phase 2: KB System Setup (completed)
- Phase 3: Gitea Actions Workflows (completed)
- Phase 4: LLM Usage Guides (completed)
- All components need validation and testing

---

## Tasks Breakdown

### Task 1: Test Cursor Rules

**Objective**: Verify Cursor rules are loaded and functioning correctly

**Steps**:

1. **Verify Rules are Loaded**:
   - Open project in Cursor
   - Check Cursor settings → Rules, Memories, Commands → Project Rules
   - Verify `.cursorrules` file is recognized
   - Check that rules appear in Cursor interface

2. **Test Phase Documentation Synchronization**:
   - Edit a `blueprint.md` file in any phase
   - Make a change (e.g., update status, add deliverable)
   - Verify that LLM checks related documents:
     - `changelog.md` is updated
     - `tasks.md` is checked/updated
     - `decisions.md` is reviewed
   - Verify consistency across all documents

3. **Test KB System Rules**:
   - Create a test KB file
   - Verify LLM follows KB naming convention
   - Verify LLM includes all required frontmatter
   - Verify LLM updates index and changelog

4. **Test Workflow Maintenance Rules**:
   - Modify a workflow file
   - Verify LLM updates `.github/CHANGELOG.md`
   - Verify LLM updates `.github/decisions.md` if decision-related
   - Verify LLM updates `.github/README.md` if needed

**Success Criteria**:
- Cursor rules are loaded and visible
- Phase documentation synchronization works
- KB system rules are followed
- Workflow maintenance rules are followed

---

### Task 2: Test KB System

**Objective**: Verify KB system functions correctly end-to-end

**Steps**:

1. **Create Test KB File**:
   - Create a test KB file with valid frontmatter
   - Use proper naming: `YYYY-MM-DD--test-kb-file--note.md`
   - Include all 18 required fields
   - Place in appropriate category

2. **Test Index Generation**:
   - Run: `kb/scripts/generate-index.sh`
   - Verify script completes successfully
   - Check that `kb/_index.md` is updated
   - Verify new file appears in index:
     - File listing by category
     - Topics index (if topics provided)
     - Tags index (if tags provided)
     - Phase relevance index (if phases provided)

3. **Test KB Lint Workflow** (if Phase 3 completed):
   - Push test KB file to branch
   - Verify `kb-lint.yml` workflow runs
   - Verify validation passes for valid file
   - Create invalid file (wrong filename pattern)
   - Verify workflow fails with appropriate error

4. **Test KB Index Update Workflow** (if Phase 3 completed):
   - Push valid KB file to main branch
   - Verify `kb-index-update.yml` workflow runs
   - Verify `kb/_index.md` is updated automatically
   - Verify commit is created with update

5. **Test KB Changelog Update**:
   - Verify changelog entry is created
   - Verify format is correct
   - Verify date and description are accurate

**Success Criteria**:
- KB files can be created with proper format
- Index generation script works correctly
- Index reflects all KB files accurately
- KB lint workflow validates files correctly
- KB index update workflow auto-updates index
- Changelog updates work correctly

---

### Task 3: Test Phase Documentation

**Objective**: Verify phase documentation synchronization works

**Steps**:

1. **Test Blueprint Modification**:
   - Edit a `blueprint.md` file
   - Change status, add deliverable, or modify milestone
   - Verify LLM checks related documents:
     - Reads `changelog.md`
     - Reads `tasks.md`
     - Reads `decisions.md`
   - Verify LLM updates documents appropriately

2. **Test Synchronization Rules**:
   - Make change to blueprint status
   - Verify `tasks.md` status is updated
   - Verify `changelog.md` has new entry
   - Verify project name consistency

3. **Test README Maintenance**:
   - Make significant phase change
   - Verify LLM checks `README.md`
   - Verify `README.md` is updated if needed
   - Verify phase statuses are accurate

4. **Test Cross-Document Consistency**:
   - Verify project name matches across all documents
   - Verify dates are consistent
   - Verify status indicators are aligned
   - Verify phase names match

**Success Criteria**:
- Blueprint modifications trigger synchronization checks
- Related documents are updated appropriately
- README is checked and updated when needed
- All documents remain consistent

---

### Task 4: Test Gitea Actions Workflows

**Objective**: Verify Gitea Actions workflows function correctly

**Steps**:

1. **Test KB Lint Workflow**:
   - Create valid KB file
   - Push to branch
   - Verify workflow runs
   - Verify validation passes
   - Create invalid KB file
   - Verify workflow fails with clear error

2. **Test KB Index Update Workflow**:
   - Create valid KB file
   - Push to main branch
   - Verify workflow runs
   - Verify index is updated
   - Verify commit is created
   - Verify commit message is correct

3. **Test Workflow Triggers**:
   - Verify workflows trigger on correct paths
   - Verify workflows trigger on correct events
   - Test with `workflow_dispatch` if available

4. **Test Workflow Permissions**:
   - Verify read permissions work for lint workflow
   - Verify write permissions work for index update workflow
   - Check workflow logs for errors

5. **Test Workflow Documentation**:
   - Verify workflow changes update `.github/CHANGELOG.md`
   - Verify workflow changes update `.github/decisions.md` if needed
   - Verify workflow changes update `.github/README.md` if needed

**Success Criteria**:
- KB lint workflow validates files correctly
- KB index update workflow updates index automatically
- Workflows trigger on correct events
- Workflow permissions are correct
- Workflow documentation is maintained

**Note**: Testing workflows requires Gitea Actions runners to be configured. Consult `docs/GITEA/Gitea-Actions-Guide.md` for runner setup.

---

### Task 5: Test LLM Prompts

**Objective**: Verify LLM prompt documents work correctly

**Steps**:

1. **Test Prompt Recognition**:
   - Open a prompt document in Cursor
   - Verify LLM recognizes it as a prompt document
   - Verify LLM understands structure

2. **Test Prompt Execution**:
   - Execute a simple prompt (e.g., Project Status Check)
   - Verify LLM follows instructions
   - Verify LLM gathers required information
   - Verify LLM completes steps correctly

3. **Test Information Gathering**:
   - Use prompt that requires information
   - Verify LLM asks for missing information
   - Verify LLM doesn't make assumptions

4. **Test Validation Checklist**:
   - Execute prompt with validation checklist
   - Verify LLM checks all items
   - Verify LLM reports validation results

5. **Test Error Handling**:
   - Test prompt with error condition
   - Verify LLM handles error appropriately
   - Verify LLM provides helpful error messages

**Success Criteria**:
- Prompts are recognized by LLM
- Prompts execute correctly
- Information gathering works
- Validation checklists are followed
- Error handling works appropriately

---

## Comprehensive Test Scenarios

### Scenario 1: Complete KB Workflow

1. Create KB file with LLM assistance
2. Verify file follows naming convention
3. Verify frontmatter is complete
4. Verify file is routed correctly
5. Run index generation script
6. Verify index is updated
7. Update changelog
8. Commit all changes together
9. Push to branch
10. Verify KB lint workflow runs and passes
11. Merge to main
12. Verify KB index update workflow runs
13. Verify index is updated automatically

### Scenario 2: Phase Documentation Update

1. Edit phase blueprint
2. Verify LLM checks related documents
3. Verify changelog is updated
4. Verify tasks are synced
5. Verify decisions are reviewed
6. Verify README is checked
7. Commit changes
8. Verify all documents are consistent

### Scenario 3: Workflow Modification

1. Modify a workflow file
2. Verify LLM updates `.github/CHANGELOG.md`
3. Verify LLM updates `.github/decisions.md` if needed
4. Verify LLM updates `.github/README.md` if needed
5. Test workflow with `workflow_dispatch` if available
6. Verify workflow runs successfully
7. Commit changes

---

## Validation Checklist

Before completing Phase 5, verify:

- [ ] Cursor rules are loaded and functioning
- [ ] Phase documentation synchronization works
- [ ] KB system functions end-to-end
- [ ] KB index generation works correctly
- [ ] KB lint workflow validates files
- [ ] KB index update workflow auto-updates index
- [ ] Phase documentation stays synchronized
- [ ] README maintenance works
- [ ] Gitea Actions workflows function correctly
- [ ] LLM prompts execute correctly
- [ ] All test scenarios pass
- [ ] Documentation is accurate and up-to-date

---

## Troubleshooting

### Common Issues

**Cursor Rules Not Working**:
- Verify `.cursorrules` file exists
- Check Cursor settings
- Restart Cursor
- Verify file paths in rules

**KB Validation Failures**:
- Check filename pattern
- Verify all frontmatter fields present
- Check date/type match filename
- Verify routing confidence is valid

**Workflow Not Running**:
- Check Gitea Actions runners are configured
- Verify workflow syntax is valid
- Check workflow triggers
- Review workflow logs

**Phase Docs Out of Sync**:
- Manually review all documents
- Update changelog with discrepancies
- Sync tasks with blueprint
- Verify project name consistency

**LLM Prompt Not Working**:
- Verify prompt structure is complete
- Check all required sections present
- Test with simple task first
- Review LLM-Usage-Guide.md

---

## Success Criteria

- All components tested and validated
- All workflows function correctly
- All synchronization rules work
- All documentation is accurate
- All test scenarios pass
- System is ready for production use

---

## Next Steps After Phase 5

Once Phase 5 is complete:
- Phase 6: Customization and Final Verification (if needed)
- System is ready for ongoing use

---

## Notes

- Testing should be thorough but practical
- Focus on critical paths and common workflows
- Document any issues found during testing
- Fix issues before marking phase complete
- Some tests may require Gitea Actions runners to be configured
- Consult `docs/GITEA/Gitea-Actions-Guide.md` for workflow testing guidance

---

**Location**: `kb/_guides/PROJECT-SETUP-GUIDE/PHASE-5-PLAN.md`  
**Related**: 
- `kb/_guides/PROJECT-SETUP-GUIDE/COMPLETE-SYSTEM-REPLICATION-GUIDE.md`
- `docs/GITEA/Gitea-Actions-Guide.md`
- All previous phase plans