tendril/kb/_guides/PROJECT-SETUP-GUIDE/PHASE-5-PLAN.md

Phase 5: Validation and Testing - Detailed Plan

Objective: Comprehensive testing and validation of all system components

Status: Ready for Review
Date: 2025-01-27

---

Overview

Phase 5 validates that all components of the documentation and automation system work correctly together. This includes testing Cursor rules, KB system, phase documentation synchronization, Gitea Actions workflows, and LLM prompts.

---

Current State

• Phase 0: Foundation & Cursor Rules (completed)
• Phase 1: Phase Documentation System (completed)
• Phase 2: KB System Setup (completed)
• Phase 3: Gitea Actions Workflows (completed)
• Phase 4: LLM Usage Guides (completed)
• All components need validation and testing

---

Tasks Breakdown

Task 1: Test Cursor Rules

Objective: Verify Cursor rules are loaded and functioning correctly

Steps:

1. Verify Rules are Loaded:

   • Open project in Cursor
   • Check Cursor settings → Rules, Memories, Commands → Project Rules
   • Verify .cursorrules file is recognized
   • Check that rules appear in Cursor interface

2. Test Phase Documentation Synchronization:

   • Edit a blueprint.md file in any phase
   • Make a change (e.g., update status, add deliverable)
   • Verify that LLM checks related documents:
     • changelog.md is updated
     • tasks.md is checked/updated
     • decisions.md is reviewed
   • Verify consistency across all documents

3. Test KB System Rules:

   • Create a test KB file
   • Verify LLM follows KB naming convention
   • Verify LLM includes all required frontmatter
   • Verify LLM updates index and changelog

4. Test Workflow Maintenance Rules:

   • Modify a workflow file
   • Verify LLM updates .github/CHANGELOG.md
   • Verify LLM updates .github/decisions.md if decision-related
   • Verify LLM updates .github/README.md if needed

Success Criteria:

• Cursor rules are loaded and visible
• Phase documentation synchronization works
• KB system rules are followed
• Workflow maintenance rules are followed

---

Task 2: Test KB System

Objective: Verify KB system functions correctly end-to-end

Steps:

1. Create Test KB File:

   • Create a test KB file with valid frontmatter
   • Use proper naming: YYYY-MM-DD--test-kb-file--note.md
   • Include all 18 required fields
   • Place in appropriate category

2. Test Index Generation:

   • Run: kb/scripts/generate-index.sh
   • Verify script completes successfully
   • Check that kb/_index.md is updated
   • Verify new file appears in index:
     • File listing by category
     • Topics index (if topics provided)
     • Tags index (if tags provided)
     • Phase relevance index (if phases provided)

3. Test KB Lint Workflow (if Phase 3 completed):

   • Push test KB file to branch
   • Verify kb-lint.yml workflow runs
   • Verify validation passes for valid file
   • Create invalid file (wrong filename pattern)
   • Verify workflow fails with appropriate error

4. Test KB Index Update Workflow (if Phase 3 completed):

   • Push valid KB file to main branch
   • Verify kb-index-update.yml workflow runs
   • Verify kb/_index.md is updated automatically
   • Verify commit is created with update

5. Test KB Changelog Update:

   • Verify changelog entry is created
   • Verify format is correct
   • Verify date and description are accurate

Success Criteria:

• KB files can be created with proper format
• Index generation script works correctly
• Index reflects all KB files accurately
• KB lint workflow validates files correctly
• KB index update workflow auto-updates index
• Changelog updates work correctly

---

Task 3: Test Phase Documentation

Objective: Verify phase documentation synchronization works

Steps:

1. Test Blueprint Modification:

   • Edit a blueprint.md file
   • Change status, add deliverable, or modify milestone
   • Verify LLM checks related documents:
     • Reads changelog.md
     • Reads tasks.md
     • Reads decisions.md
   • Verify LLM updates documents appropriately

2. Test Synchronization Rules:

   • Make change to blueprint status
   • Verify tasks.md status is updated
   • Verify changelog.md has new entry
   • Verify project name consistency

3. Test README Maintenance:

   • Make significant phase change
   • Verify LLM checks README.md
   • Verify README.md is updated if needed
   • Verify phase statuses are accurate

4. Test Cross-Document Consistency:

   • Verify project name matches across all documents
   • Verify dates are consistent
   • Verify status indicators are aligned
   • Verify phase names match

Success Criteria:

• Blueprint modifications trigger synchronization checks
• Related documents are updated appropriately
• README is checked and updated when needed
• All documents remain consistent

---

Task 4: Test Gitea Actions Workflows

Objective: Verify Gitea Actions workflows function correctly

Steps:

1. Test KB Lint Workflow:

   • Create valid KB file
   • Push to branch
   • Verify workflow runs
   • Verify validation passes
   • Create invalid KB file
   • Verify workflow fails with clear error

2. Test KB Index Update Workflow:

   • Create valid KB file
   • Push to main branch
   • Verify workflow runs
   • Verify index is updated
   • Verify commit is created
   • Verify commit message is correct

3. Test Workflow Triggers:

   • Verify workflows trigger on correct paths
   • Verify workflows trigger on correct events
   • Test with workflow_dispatch if available

4. Test Workflow Permissions:

   • Verify read permissions work for lint workflow
   • Verify write permissions work for index update workflow
   • Check workflow logs for errors

5. Test Workflow Documentation:

   • Verify workflow changes update .github/CHANGELOG.md
   • Verify workflow changes update .github/decisions.md if needed
   • Verify workflow changes update .github/README.md if needed

Success Criteria:

• KB lint workflow validates files correctly
• KB index update workflow updates index automatically
• Workflows trigger on correct events
• Workflow permissions are correct
• Workflow documentation is maintained

Note: Testing workflows requires Gitea Actions runners to be configured. Consult docs/GITEA/Gitea-Actions-Guide.md for runner setup.

---

Task 5: Test LLM Prompts

Objective: Verify LLM prompt documents work correctly

Steps:

1. Test Prompt Recognition:

   • Open a prompt document in Cursor
   • Verify LLM recognizes it as a prompt document
   • Verify LLM understands structure

2. Test Prompt Execution:

   • Execute a simple prompt (e.g., Project Status Check)
   • Verify LLM follows instructions
   • Verify LLM gathers required information
   • Verify LLM completes steps correctly

3. Test Information Gathering:

   • Use prompt that requires information
   • Verify LLM asks for missing information
   • Verify LLM doesn't make assumptions

4. Test Validation Checklist:

   • Execute prompt with validation checklist
   • Verify LLM checks all items
   • Verify LLM reports validation results

5. Test Error Handling:

   • Test prompt with error condition
   • Verify LLM handles error appropriately
   • Verify LLM provides helpful error messages

Success Criteria:

• Prompts are recognized by LLM
• Prompts execute correctly
• Information gathering works
• Validation checklists are followed
• Error handling works appropriately

---

Comprehensive Test Scenarios

Scenario 1: Complete KB Workflow

1. Create KB file with LLM assistance
2. Verify file follows naming convention
3. Verify frontmatter is complete
4. Verify file is routed correctly
5. Run index generation script
6. Verify index is updated
7. Update changelog
8. Commit all changes together
9. Push to branch
10. Verify KB lint workflow runs and passes
11. Merge to main
12. Verify KB index update workflow runs
13. Verify index is updated automatically

Scenario 2: Phase Documentation Update

1. Edit phase blueprint
2. Verify LLM checks related documents
3. Verify changelog is updated
4. Verify tasks are synced
5. Verify decisions are reviewed
6. Verify README is checked
7. Commit changes
8. Verify all documents are consistent

Scenario 3: Workflow Modification

1. Modify a workflow file
2. Verify LLM updates .github/CHANGELOG.md
3. Verify LLM updates .github/decisions.md if needed
4. Verify LLM updates .github/README.md if needed
5. Test workflow with workflow_dispatch if available
6. Verify workflow runs successfully
7. Commit changes

---

Validation Checklist

Before completing Phase 5, verify:

• Cursor rules are loaded and functioning
• Phase documentation synchronization works
• KB system functions end-to-end
• KB index generation works correctly
• KB lint workflow validates files
• KB index update workflow auto-updates index
• Phase documentation stays synchronized
• README maintenance works
• Gitea Actions workflows function correctly
• LLM prompts execute correctly
• All test scenarios pass
• Documentation is accurate and up-to-date

---

Troubleshooting

Common Issues

Cursor Rules Not Working:

• Verify .cursorrules file exists
• Check Cursor settings
• Restart Cursor
• Verify file paths in rules

KB Validation Failures:

• Check filename pattern
• Verify all frontmatter fields present
• Check date/type match filename
• Verify routing confidence is valid

Workflow Not Running:

• Check Gitea Actions runners are configured
• Verify workflow syntax is valid
• Check workflow triggers
• Review workflow logs

Phase Docs Out of Sync:

• Manually review all documents
• Update changelog with discrepancies
• Sync tasks with blueprint
• Verify project name consistency

LLM Prompt Not Working:

• Verify prompt structure is complete
• Check all required sections present
• Test with simple task first
• Review LLM-Usage-Guide.md

---

Success Criteria

• All components tested and validated
• All workflows function correctly
• All synchronization rules work
• All documentation is accurate
• All test scenarios pass
• System is ready for production use

---

Next Steps After Phase 5

Once Phase 5 is complete:

• Phase 6: Customization and Final Verification (if needed)
• System is ready for ongoing use

---

Notes

• Testing should be thorough but practical
• Focus on critical paths and common workflows
• Document any issues found during testing
• Fix issues before marking phase complete
• Some tests may require Gitea Actions runners to be configured
• Consult docs/GITEA/Gitea-Actions-Guide.md for workflow testing guidance

---

Location: kb/_guides/PROJECT-SETUP-GUIDE/PHASE-5-PLAN.md
Related:

• kb/_guides/PROJECT-SETUP-GUIDE/COMPLETE-SYSTEM-REPLICATION-GUIDE.md
• docs/GITEA/Gitea-Actions-Guide.md
• All previous phase plans