Add docs/project_planning/sprint_breakdown.md

Create scripts that can be run from a central node in a Proxmox cluster to manage the screens of multiple laptops acting as servers. Specifically, the scripts allow you to turn off the internal screens of the laptops to save energy and prepare them for server use and turn them back on if needed. The scripts also support configuration for multiple laptops and use SSH to execute display commands on each laptop remotely. Additionally, aliases were set up for easy execution of these scripts.

README.md

@@ -0,0 +1,180 @@
+### Step 1: Roadmap for Deployment, Testing, and Evaluation
+
+#### Overview:
+This roadmap will outline the steps to set up a Proxmox cluster, deploy various services, and test and evaluate the system’s performance. It will include setting up Proxmox on the old laptops and mini PCs, deploying services, and capturing knowledge and issues encountered.
+
+#### Roadmap:
+
+1. **Initial Setup and Configuration**:
+   - **Week 1**:
+     1. Gather hardware: 3 old laptops, 2 mini PCs (one for Opnsense firewall, one for hosting the website), and 2x16TB drives.
+     2. Install Proxmox VE on the old laptops and mini PCs.
+     3. Configure basic Proxmox settings and network.
+
+2. **Basic Cluster Setup**:
+   - **Week 2**:
+     1. Create a Proxmox cluster with the old laptops.
+     2. Add the mini PCs to the cluster.
+     3. Set up storage with the 16TB drives for backups and shared storage.
+
+3. **Service Deployment**:
+   - **Week 3-4**:
+     1. Deploy Opnsense firewall on one mini PC.
+     2. Set up a website hosting environment on the other mini PC.
+     3. Deploy various open-source services (e.g., Nextcloud, Home Assistant) for testing.
+
+4. **High Availability Cluster Setup**:
+   - **Month 2**:
+     1. Purchase and set up 2 MINISFORUM UM690S Mini PCs.
+     2. Integrate these mini PCs into the Proxmox cluster.
+     3. Configure high availability settings and test failover capabilities.
+
+5. **Benchmarking and Testing**:
+   - **Month 3**:
+     1. Perform initial benchmarks for compute, storage, and network performance.
+     2. Deploy small LLMs and perform load testing.
+     3. Capture performance metrics and analyze results.
+
+6. **Documentation and Knowledge Capture**:
+   - **Ongoing**:
+     1. Document each step of the setup process.
+     2. Record any issues encountered and their resolutions.
+     3. Maintain a detailed log of configurations and settings.
+
+### Detailed Directions for Initial Setup
+
+#### Week 1: Gather Hardware and Install Proxmox
+
+1. **Hardware Preparation**:
+   - Ensure all old laptops and mini PCs are functional.
+   - Format and prepare the 16TB drives for use as shared storage.
+
+2. **Install Proxmox VE**:
+   - Download the latest Proxmox VE ISO from the official website.
+   - Create a bootable USB drive using a tool like Rufus.
+   - Boot each laptop and mini PC from the USB drive and follow the installation prompts to install Proxmox VE.
+   - Configure each installation with a unique hostname and appropriate network settings.
+
+3. **Initial Configuration**:
+   - Access the Proxmox web interface on each device by navigating to `https://<IP_ADDRESS>:8006`.
+   - Set up user accounts and configure basic settings.
+   - Ensure all devices are on the same network and can communicate with each other.
+
+#### Week 2: Create Proxmox Cluster
+
+1. **Create a Proxmox Cluster**:
+   - On the primary Proxmox node (one of the laptops), create a new cluster:
+     ```bash
+     pvecm create <cluster_name>
+     ```
+   - On the other laptops and mini PCs, join the cluster:
+     ```bash
+     pvecm add <IP_ADDRESS_OF_PRIMARY_NODE>
+     ```
+
+2. **Configure Shared Storage**:
+   - Add the 16TB drives to the Proxmox cluster as shared storage.
+   - Navigate to the Datacenter -> Storage in the Proxmox web interface and add the storage.
+
+3. **Verify Cluster Setup**:
+   - Ensure all nodes appear in the cluster in the Proxmox web interface.
+   - Test basic functionality by creating and migrating a VM or container between nodes.
+
+### Step 2: Project Plan and Sprint
+
+#### Sprint 1: Initial Setup and Basic Cluster Configuration
+
+- **Duration**: 2 Weeks
+- **Goals**:
+  - Install Proxmox on all hardware.
+  - Configure basic settings and network.
+  - Create and verify the Proxmox cluster.
+
+#### Sprint 2: Service Deployment
+
+- **Duration**: 2 Weeks
+- **Goals**:
+  - Deploy Opnsense firewall and website hosting environment.
+  - Set up and test various open-source services.
+  - Document configurations and issues.
+
+#### Sprint 3: High Availability Cluster Setup
+
+- **Duration**: 1 Month
+- **Goals**:
+  - Purchase and set up MINISFORUM UM690S Mini PCs.
+  - Integrate new hardware into the cluster.
+  - Configure and test high availability settings.
+
+#### Sprint 4: Benchmarking and Testing
+
+- **Duration**: 1 Month
+- **Goals**:
+  - Perform benchmarks for compute, storage, and network performance.
+  - Deploy and test small LLMs.
+  - Capture and analyze performance metrics.
+
+### Step 3: Git Structure
+
+# Home Lab and Small Data Center Project
+
+## Goals and Objectives
+
+This project aims to build a cost-effective, scalable home lab and small data center to support a startup offering localized IT services. The infrastructure will host various services, including network management, data backups, home automation, security monitoring, and AI workloads.
+
+## Objectives
+- Set up a Proxmox cluster using old laptops and mini PCs.
+- Deploy and test various open-source services.
+- Benchmark and evaluate compute, storage, and network performance.
+- Scale the infrastructure by adding MINISFORUM UM690S Mini PCs.
+- Ensure high availability and robustness of the system.
+
+## Project Plan
+- **Sprint 1**: Initial Setup and Basic Cluster Configuration
+- **Sprint 2**: Service Deployment
+- **Sprint 3**: High Availability Cluster Setup
+- **Sprint 4**: Benchmarking and Testing
+
+## Getting Started
+1. **Install Proxmox VE** on all hardware.
+2. **Create a Proxmox Cluster** and add all nodes.
+3. **Deploy Services** and test functionality.
+4. **Configure High Availability** and test failover.
+5. **Benchmark and Evaluate** performance.
+
+## Repository Structure
+```
+home-lab-project/
+├── README.md
+├── docs/
+│   ├── setup/
+│   │   ├── proxmox_installation.md
+│   │   ├── cluster_configuration.md
+│   │   └── service_deployment.md
+│   ├── benchmarks/
+│   │   ├── compute.md
+│   │   ├── storage.md
+│   │   └── network.md
+│   ├── issues/
+│   │   └── issue_log.md
+│   └── future_expansion.md
+├── scripts/
+│   ├── benchmarks/
+│   │   ├── compute_benchmark.sh
+│   │   ├── storage_benchmark.sh
+│   │   └── network_benchmark.sh
+│   └── deployment/
+│       ├── deploy_services.sh
+│       └── configure_ha.sh
+└── logs/
+    └── setup_log.md
+```
+
+1. **docs/**: Documentation for setup, configuration, benchmarks, and future expansion.
+2. **scripts/**: Scripts for benchmarking and service deployment.
+3. **logs/**: Logs of setup steps, configurations, and issues encountered.
+
+### Next Steps
+1. **Review and refine** the roadmap, project plan, and README structure.
+2. **Start with Sprint 1**: Initial Setup and Basic Cluster Configuration.
+3. **Document** each step thoroughly and capture any issues encountered.

assets/assets_list.md

@@ -0,0 +1 @@
+TBD

docs/benchmarks/compute.md

@@ -0,0 +1 @@
+TBD

docs/benchmarks/network.md

@@ -0,0 +1 @@
+TBD

docs/benchmarks/storage.md

@@ -0,0 +1 @@
+TBD

docs/future_expansion.md

@@ -0,0 +1 @@
+TBD

docs/issues/issue_log.md

@@ -0,0 +1 @@
+TBD     

docs/project_planning/project_plan.md

@@ -0,0 +1,42 @@
+## Project Plan
+
+#### Overview:
+This section outlines the project plan and sprint structure for deploying the Proxmox cluster, setting up the necessary hardware and software, and performing initial testing and evaluation.
+
+#### Project Plan
+
+1. **Project Initiation**:
+   - **Week 1**:
+     - Define project goals and objectives.
+     - Gather and prepare hardware.
+     - Set up initial Proxmox environment.
+
+2. **Sprint 1: Initial Setup and Basic Cluster Configuration**:
+   - **Week 2-3**:
+     - Install Proxmox VE on all hardware.
+     - Configure basic settings and network.
+     - Create and verify the Proxmox cluster.
+     - Document setup steps and configurations.
+
+3. **Sprint 2: Service Deployment**:
+   - **Week 4-5**:
+     - Deploy Opnsense firewall and website hosting environment.
+     - Set up and test various open-source services.
+     - Document configurations and issues encountered.
+
+4. **Sprint 3: High Availability Cluster Setup**:
+   - **Week 6-9**:
+     - Purchase and set up MINISFORUM UM690S Mini PCs.
+     - Integrate new hardware into the Proxmox cluster.
+     - Configure and test high availability settings.
+     - Document setup steps and configurations.
+
+5. **Sprint 4: Benchmarking and Testing**:
+   - **Week 10-13**:
+     - Perform benchmarks for compute, storage, and network performance.
+     - Deploy and test small LLMs.
+     - Capture and analyze performance metrics.
+     - Document benchmark results and analysis.
+
+
+

docs/project_planning/sprint_breakdown.md

@@ -0,0 +1,67 @@
+#### Sprint Breakdown
+
+##### Sprint 1: Initial Setup and Basic Cluster Configuration
+
+- **Goals**:
+  - Install Proxmox on all hardware.
+  - Configure basic settings and network.
+  - Create and verify the Proxmox cluster.
+
+- **Tasks**:
+  1. Install Proxmox VE on old laptops and mini PCs.
+  2. Configure basic Proxmox settings and network.
+  3. Create a Proxmox cluster.
+  4. Add all nodes to the cluster.
+  5. Verify cluster setup by creating and migrating VMs.
+
+- **Deliverables**:
+  - Functional Proxmox cluster.
+  - Documentation of setup steps and configurations.
+
+##### Sprint 2: Service Deployment
+
+- **Goals**:
+  - Deploy and configure core services.
+  - Test service functionality.
+
+- **Tasks**:
+  1. Deploy Opnsense firewall.
+  2. Set up a website hosting environment.
+  3. Deploy and configure additional open-source services (e.g., Nextcloud, Home Assistant).
+  4. Test and verify service functionality.
+
+- **Deliverables**:
+  - Deployed and functional services.
+  - Documentation of configurations and issues encountered.
+
+##### Sprint 3: High Availability Cluster Setup
+
+- **Goals**:
+  - Integrate new hardware into the cluster.
+  - Configure and test high availability settings.
+
+- **Tasks**:
+  1. Purchase and set up MINISFORUM UM690S Mini PCs.
+  2. Integrate new mini PCs into the Proxmox cluster.
+  3. Configure high availability settings.
+  4. Test failover capabilities.
+
+- **Deliverables**:
+  - High availability cluster.
+  - Documentation of setup steps and configurations.
+
+##### Sprint 4: Benchmarking and Testing
+
+- **Goals**:
+  - Benchmark and evaluate system performance.
+  - Test AI workloads and other services.
+
+- **Tasks**:
+  1. Perform compute, storage, and network benchmarks.
+  2. Deploy and test small LLMs.
+  3. Capture performance metrics.
+  4. Analyze and document results.
+
+- **Deliverables**:
+  - Benchmark results and analysis.
+  - Documentation of performance metrics and analysis.

docs/setup/cluster_configuration.md

@@ -0,0 +1 @@
+TBD 

docs/setup/proxmox_installation.md

@@ -0,0 +1 @@
+TBD

docs/setup/service_deployement.md

@@ -0,0 +1 @@
+TBD

logs/setup_log.md

@@ -0,0 +1 @@
+TBD

scripts/admin/screen_off_laptop_servers/screen_off_scripts.md

@@ -0,0 +1,103 @@
+### Step 1: Setting Up SSH Access
+
+Ensure you have SSH access set up between your central node and all laptops. You should be able to SSH into each laptop without needing to enter a password (using SSH keys).
+
+### Step 2: Create the Configuration File
+
+Create the configuration file `/usr/local/bin/manage-displays.conf` on the central node with the following content:
+
+```bash
+# Configuration for different laptops
+# Format: "NodeName:Host:InternalDisplay:ExternalDisplay"
+
+LAPTOPS=(
+    "Laptop1:192.168.1.101:eDP-1:HDMI-1"
+    "Laptop2:192.168.1.102:eDP-1:HDMI-1"
+    "Laptop3:192.168.1.103:eDP-1:HDMI-1"
+)
+```
+
+### Step 3: Create the Main Script to Turn Off the Screens
+
+Save the following script as `/usr/local/bin/manage-displays-off.sh` on the central node:
+
+```bash
+#!/bin/bash
+
+CONFIG_FILE="/usr/local/bin/manage-displays.conf"
+
+if [ ! -f "$CONFIG_FILE" ]; then
+    echo "Config file not found! Please create a config file at $CONFIG_FILE"
+    exit 1
+fi
+
+source $CONFIG_FILE
+
+for LAPTOP in "${LAPTOPS[@]}"; do
+    IFS=":" read -r NODE HOST INTERNAL_DISPLAY EXTERNAL_DISPLAY <<< "$LAPTOP"
+    echo "Do you want to turn off the screen for $NODE ($HOST) and prepare it for server use? (yes/y to confirm)"
+    read answer
+    if [[ "$answer" == "yes" || "$answer" == "y" ]]; then
+        ssh $HOST "xrandr --output $INTERNAL_DISPLAY --off && xrandr --output $EXTERNAL_DISPLAY --auto"
+        echo "$NODE ($HOST) screen turned off."
+    else
+        echo "Operation cancelled for $NODE ($HOST)."
+    fi
+done
+```
+
+### Step 4: Create the Main Script to Turn On the Screens
+
+Save the following script as `/usr/local/bin/manage-displays-on.sh` on the central node:
+
+```bash
+#!/bin/bash
+
+CONFIG_FILE="/usr/local/bin/manage-displays.conf"
+
+if [ ! -f "$CONFIG_FILE" ]; then
+    echo "Config file not found! Please create a config file at $CONFIG_FILE"
+    exit 1
+fi
+
+source $CONFIG_FILE
+
+for LAPTOP in "${LAPTOPS[@]}"; do
+    IFS=":" read -r NODE HOST INTERNAL_DISPLAY EXTERNAL_DISPLAY <<< "$LAPTOP"
+    echo "Do you want to turn on the screen for $NODE ($HOST)? (yes/y to confirm)"
+    read answer
+    if [[ "$answer" == "yes" || "$answer" == "y" ]]; then
+        ssh $HOST "xrandr --output $INTERNAL_DISPLAY --auto"
+        echo "$NODE ($HOST) screen turned on."
+    else
+        echo "Operation cancelled for $NODE ($HOST)."
+    fi
+done
+```
+
+### Step 5: Make the Scripts Executable
+
+Make both scripts executable:
+
+```bash
+sudo chmod +x /usr/local/bin/manage-displays-off.sh
+sudo chmod +x /usr/local/bin/manage-displays-on.sh
+```
+
+### Step 6: Set Up the Alias
+
+Add the following aliases to your shell configuration file (`.bashrc` or `.zshrc`):
+
+```bash
+# Aliases for manage-displays scripts
+alias screens-off='/usr/local/bin/manage-displays-off.sh'
+alias screens-on='/usr/local/bin/manage-displays-on.sh'
+```
+
+Reload the shell configuration to apply the aliases:
+
+```bash
+source ~/.bashrc  # or source ~/.zshrc
+```
+
+With these steps, you can now use the aliases `screens-off` and `screens-on` on the central node to control the screens of all laptops in the cluster. The scripts will prompt you for confirmation before turning the screens off or on.

scripts/benchmarks/compute_benchmarks.md

@@ -0,0 +1 @@
+TBD

scripts/benchmarks/network_benchmarks.md

@@ -0,0 +1 @@
+TBD

scripts/benchmarks/storage_bechmarks.md

@@ -0,0 +1 @@
+TBD

scripts/deployment/configure_ha.sh

@@ -0,0 +1,2 @@
+# Place holder
+

scripts/deployment/deploy_services.sh

@@ -0,0 +1 @@
+# Place holder

scripts/graphs/analysis_graphs_py_inst.md

@@ -0,0 +1,172 @@
+This script will generate the required charts and save them as `performance_per_dollar_recommended.png` and `combined_scores_recommended.png` in the same directory as the script.
+
+### Updated Python Script:
+
+```python
+import matplotlib.pyplot as plt
+
+# Data for Performance per Initial Dollar
+systems = [
+    "Custom Built Server",
+    "MINISFORUM UM690S",
+    "MINISFORUM UM790 Pro",
+    "MINISFORUM UM690 Pro",
+    "Beelink SER5",
+    "CyberGeek Mini PC",
+    "KAMRUI Mini PC"
+]
+performance_per_dollar = [
+    19.08, 39.82, 29.08, 36.13, 34.66, 31.25, 32.58
+]
+
+# Data for Total Combined Score
+combined_scores = [
+    27_004.6, 62_419.6, 34_209.8, 46_814.7, 57_024.5, 45_619.6, 45_619.6
+]
+
+# Recommended system
+recommended_system = "MINISFORUM UM690S"
+
+# Plot Performance per Initial Dollar
+plt.figure(figsize=(10, 6))
+bars = plt.barh(systems, performance_per_dollar, color='skyblue', height=0.5)
+plt.xlabel('Performance per Initial Dollar')
+plt.title('Performance per Initial Dollar for Each System')
+plt.grid(True, axis='x', linestyle='--', alpha=0.7)
+plt.gca().invert_yaxis()
+plt.tight_layout()
+
+# Add the actual numbers at the end of the bars
+for bar, value in zip(bars, performance_per_dollar):
+    plt.text(value, bar.get_y() + bar.get_height()/2, f'{value:.2f}', va='center')
+
+# Highlight the recommended system
+for bar, system in zip(bars, systems):
+    if system == recommended_system:
+        bar.set_color('lightgreen')
+
+plt.savefig('performance_per_dollar_recommended.png')
+plt.show()
+
+# Plot Total Combined Score
+plt.figure(figsize=(10, 6))
+bars = plt.barh(systems, combined_scores, color='lightgreen', height=0.5)
+plt.xlabel('Total Combined Score')
+plt.title('Total Combined Score for Each System')
+plt.grid(True, axis='x', linestyle='--', alpha=0.7)
+plt.gca().invert_yaxis()
+plt.tight_layout()
+
+# Add the actual numbers at the end of the bars
+for bar, value in zip(bars, combined_scores):
+    plt.text(value, bar.get_y() + bar.get_height()/2, f'{value:.1f}', va='center')
+
+# Highlight the recommended system
+for bar, system in zip(bars, systems):
+    if system == recommended_system:
+        bar.set_color('skyblue')
+
+plt.savefig('combined_scores_recommended.png')
+plt.show()
+```
+
+If `pip` is not recognized as a command, it might be because it is not installed or not added to your system's PATH. Here are steps to address this:
+
+1. **Ensure Python and pip are installed**:
+   - If you are on Windows, you can download the Python installer from [python.org](https://www.python.org/downloads/). During installation, make sure to check the option to "Add Python to PATH".
+   - If you are on macOS or Linux, Python might already be installed. You can check by running `python --version` or `python3 --version` in the terminal.
+
+2. **Installing pip**:
+   - If Python is installed but pip is not, you can install pip manually. Download `get-pip.py` by running:
+     ```bash
+     curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
+     ```
+   - Then run the following command to install pip:
+     ```bash
+     python get-pip.py
+     ```
+
+3. **Using pip3**:
+   - On some systems, especially if both Python 2 and Python 3 are installed, you might need to use `pip3` instead of `pip`. You can install matplotlib using:
+     ```bash
+     pip3 install matplotlib
+     ```
+
+4. **Verify the installation**:
+   - After installing pip, you can verify the installation by running:
+     ```bash
+     pip --version
+     ```
+   - If you had to use `pip3` to install, verify with:
+     ```bash
+     pip3 --version
+     ```
+
+5. **Installing matplotlib with pip**:
+   - Once pip is available, you can install matplotlib by running:
+     ```bash
+     pip install matplotlib
+     ```
+   - Or if you used `pip3`:
+     ```bash
+     pip3 install matplotlib
+     ```
+
+6. **Running the script**:
+   - After installing matplotlib, create the `generate_charts.py` file as described previously, and run it with:
+     ```bash
+     python generate_charts.py
+     ```
+   - Or if you need to use Python 3 explicitly:
+     ```bash
+     python3 generate_charts.py
+     ```
+
+If you run into any specific issues or errors during these steps, please let me know so I can help troubleshoot further.
+
+### Instructions to run the script:
+It looks like `pip` was installed in a directory that is not included in your system's PATH. To resolve this, you can add the directory to your PATH temporarily or permanently.
+
+### Temporary Solution:
+1. **Temporarily add the directory to your PATH**:
+   ```bash
+   export PATH=$PATH:/home/datawarrior/.local/bin
+   ```
+2. **Verify pip is now accessible**:
+   ```bash
+   pip --version
+   ```
+
+### Permanent Solution:
+1. **Open your `.bashrc` file** (or `.bash_profile` or `.profile` depending on your system configuration) in a text editor:
+   ```bash
+   nano ~/.bashrc
+   ```
+2. **Add the following line** at the end of the file:
+   ```bash
+   export PATH=$PATH:/home/datawarrior/.local/bin
+   ```
+3. **Save the file and exit the text editor**.
+4. **Apply the changes**:
+   ```bash
+   source ~/.bashrc
+   ```
+
+### Installing matplotlib:
+After adding the directory to your PATH, you can proceed with installing matplotlib:
+```bash
+pip install matplotlib
+```
+
+### Running the Python script:
+Once matplotlib is installed, you can create the `generate_charts.py` script and run it:
+```bash
+python3 generate_charts.py
+```
+
+Here’s a quick summary of commands for the temporary solution:
+```bash
+export PATH=$PATH:/home/datawarrior/.local/bin
+pip install matplotlib
+```
+