Build Monitoring

Build Monitoring

ARROW provides real-time monitoring for VM build and provisioning processes, allowing you to track progress, view logs, and troubleshoot issues. The monitoring system is implemented in backend/api/vm_build_monitor/.

Overview

The build monitoring system offers:

• Real-Time Status: Live updates on build progress via WebSocket
• Log Streaming: WebSocket-based live log viewing at /api/vm-build/logs/stream
• Progress Tracking: Visual indicators of build phases with percentage completion
• Error Visibility: Immediate notification of failures with detailed error messages
• Build History: Access to past build logs stored in B2 storage

Build System Architecture

flowchart TD
    subgraph Console["ARROW Console"]
        A[Build Monitor UI]
        B[WebSocket Client]
    end

    subgraph Backend["PocketBase Backend"]
        C[vm_build_monitor handlers]
        D[WebSocket Hub]
        E[Task Queue Manager]
    end

    subgraph Storage["B2 Storage"]
        F[Build Logs]
    end

    subgraph Builders["Build Servers"]
        G[GitHub Actions Runner]
        H[Local Builder]
    end

    B <-->|WebSocket| D
    A --> C
    G -->|POST /api/vm-build/task-status| C
    H -->|POST /api/vm-build/task-logs| C
    C --> E
    C --> F
    D --> A

Build Status Tracking

Build Phases

VM provisioning progresses through defined phases:

stateDiagram-v2
    [*] --> Queued
    Queued --> Imaging: Builder Picks Task
    Imaging --> Provisioning: Image Ready
    Provisioning --> Configuring: VM Deployed
    Configuring --> Complete: Configuration Done
    Imaging --> Failed: Image Error
    Provisioning --> Failed: Deployment Error
    Configuring --> Failed: Config Error
    Failed --> Queued: Rebuild Triggered
    Complete --> [*]

Phase Descriptions

Phase	Description	Duration
Queued	Build task waiting for available runner	Variable
Imaging	Base image being customized	10-20 minutes
Provisioning	VM being deployed to infrastructure	5-10 minutes
Configuring	Post-deployment configuration	5-15 minutes
Complete	VM ready for use	-
Failed	Build encountered an error	-

Status Indicators

The console displays build status with visual indicators:

• Blue (Queued): Task waiting in queue
• Yellow (In Progress): Active build operation
• Green (Complete): Successfully completed
• Red (Failed): Build error occurred

Build Monitor API

The build monitoring system exposes the following endpoints:

Endpoint	Method	Auth	Purpose
/api/vm-build/status	POST	Builder	Build status updates from builders
/api/vm-build/server-health	POST	Builder	Server health metrics reporting
/api/vm-build/trigger	POST	image.admin	Trigger manual build
/api/vm-build/tasks	GET	Builder	Poll for available tasks
/api/vm-build/task-status	POST	Builder	Report task progress
/api/vm-build/task-logs	POST	Builder	Submit batch of build logs
/api/vm-build/servers	GET	Admin	List build servers
/api/vm-build/jobs	GET	Admin	Get build job status
/api/vm-build/task-logs/{task_id}	GET	Auth	Retrieve logs for specific task
/api/vm-build/cancel-task/{task_id}	POST	image.admin	Cancel running task
/api/vm-build/task/{task_id}	DELETE	image.admin	Delete completed task
/api/vm-build/rebuild/{task_id}	POST	image.admin	Rebuild failed task
/api/vm-build/logs/stream	WebSocket	Auth	Real-time log streaming

Real-Time Logs

WebSocket Log Streaming

During active builds, logs stream in real-time via WebSocket connection:

1. Navigate to your device request
2. Click View Build Logs or the build status indicator
3. Logs appear as they are generated
4. Stream continues until build completes or fails

WebSocket Endpoint: /api/vm-build/logs/stream

Connection Authentication: Requires valid session token

WebSocket Message Types

The build monitoring system uses structured messages:

Build Log Message:

{
  "type": "build_log",
  "task_id": "task-123",
  "session_id": "session-456",
  "build_info": {
    "client": "client-name",
    "platform": "vm",
    "image": "image-id",
    "build_type": "pvm"
  },
  "log_entry": {
    "timestamp": "2024-01-15T10:31:00Z",
    "level": "INFO",
    "component": "ANSIBLE",
    "context": "Installing packages",
    "message": "Installed: curl, wget, git"
  }
}

Progress Message:

{
  "type": "progress",
  "task_id": "task-123",
  "progress": {
    "percentage": 45,
    "stage": "imaging",
    "stage_name": "Installing Software",
    "current_step": "Running Ansible playbooks",
    "estimated_remaining": "8m 30s"
  }
}

Status Change Message:

{
  "type": "status_change",
  "task_id": "task-123",
  "old_status": "building",
  "new_status": "completed",
  "message": "Build completed successfully",
  "metadata": {
    "build_time_seconds": 1245,
    "download_url": "https://..."
  }
}

Log Entry Levels

Level	Description
DEBUG	Detailed diagnostic information
INFO	Normal operational messages
WARN	Warning conditions
ERROR	Error conditions
FATAL	Critical failures

Log Components

Component	Description
BUILD-ORCHESTRATOR	Overall build coordination
PACKER	Image building operations
ANSIBLE	Playbook execution
PROXMOX	VM deployment operations
NETBIRD	VPN registration

Log Content

Build logs include:

• Timestamps: When each action occurred
• Phase Markers: Current build phase indicators
• Command Output: Output from build commands
• Status Messages: Progress and status updates
• Error Details: Full error messages and stack traces

Log Viewer Features

The log viewer interface provides:

• Auto-Scroll: Automatically follows new log entries
• Search: Find specific text in logs
• Pause/Resume: Stop auto-scroll to review
• Download: Export logs for offline analysis

Build Progress

Progress Indicators

Visual progress tracking shows:

• Overall Progress: Percentage complete bar
• Current Phase: Active phase indicator
• Elapsed Time: Time since build started
• Estimated Remaining: Approximate time to completion

Phase Progress

Each phase reports detailed progress:

Imaging Phase:

• Downloading base image
• Applying customizations
• Installing software packages
• Running Ansible playbooks
• Uploading customized image

Provisioning Phase:

• Creating VM on Proxmox
• Configuring resources (CPU, memory, disk)
• Attaching network interfaces
• Starting VM

Configuring Phase:

• Registering with NetBird VPN
• Configuring access groups
• Setting up policies
• Verifying connectivity

Build Tasks

Task Queue

Build tasks are managed in a queue system:

• FIFO Processing: Tasks processed in order received
• Priority Support: Urgent builds can be prioritized
• Parallel Builds: Multiple runners process tasks concurrently

Task Status Transitions

stateDiagram-v2
    [*] --> queued
    queued --> started: Builder Claims Task
    started --> building: Build Begins
    building --> completed: Success
    building --> failed: Error
    building --> cancelling: Cancel Requested
    cancelling --> cancelled: Cleanup Done
    failed --> queued: Retry Triggered
    completed --> [*]
    cancelled --> [*]

Task Information

Each build task contains:

Field	Description
Task ID	Unique identifier
Device Request	Associated device request
Organization	Owning organization
Status	Current task status
Created	When task was created
Started	When build began
Completed	When build finished
Progress	Percentage complete (0-100)
Assigned Server	Build server handling the task
Build Time	Duration in seconds

Build Server Monitoring

Server Health Tracking

Build servers report health status to enable intelligent task scheduling:

Health Update Endpoint: POST /api/vm-build/server-health

{
  "server_name": "builder-1",
  "hostname": "builder-1.example.com",
  "status": "online",
  "cpu_usage": 45.5,
  "memory_usage": 60.2,
  "memory_total": 8192,
  "disk_usage": 75.0,
  "disk_total": 500,
  "active_builds": 3,
  "max_concurrent_builds": 5,
  "version": "1.2.0",
  "last_heartbeat": "2024-01-15T10:31:00Z"
}

Server Status Values

Status	Description
online	Server available for builds
offline	Server not responding
busy	Server at capacity
maintenance	Server under maintenance

Server Registration

Builders register with the system via WebSocket:

{
  "type": "register",
  "server_info": {
    "server_name": "builder-1",
    "hostname": "builder-1.example.com",
    "capabilities": ["vm", "hardware"],
    "active_tasks": ["task-id-1", "task-id-2"]
  },
  "timestamp": "2024-01-15T10:30:00Z"
}

Task Commands

Administrators can send commands to build servers:

Command	Description
cancel	Cancel the running task
pause	Pause build execution
resume	Resume paused build

Commands are stored in vm_build_commands and delivered to builders on next poll.

Viewing Task Details

Access task details through:

1. Device Request: Click the build status on your request
2. Build Monitor: Admin view of all active builds
3. Task API: Programmatic access to task information

Log Batch Submission

Builders submit logs in batches via POST /api/vm-build/task-logs:

{
  "task_id": "task-123",
  "entries": [
    {
      "timestamp": "2024-01-15T10:31:00Z",
      "level": "INFO",
      "component": "ANSIBLE",
      "message": "Installing packages: curl, wget, git"
    }
  ]
}

Logs are stored in the database and can be streamed to connected WebSocket clients.

Error Handling

Common Build Errors

Error Type	Cause	Resolution
Image Download Failed	B2 connectivity issue	Retry build, check storage
Ansible Playbook Failed	Configuration error	Review logs, fix playbook
Proxmox Deployment Failed	Resource unavailable	Check cluster capacity
VPN Registration Failed	NetBird API error	Verify VPN configuration
Timeout	Build exceeded time limit	Optimize build, increase timeout

Error Messages

When builds fail, error details include:

• Error Type: Classification of the failure
• Error Message: Detailed description
• Phase: Which phase failed
• Logs: Full build logs up to failure
• Suggestions: Potential resolution steps

Troubleshooting Steps

For failed builds:

1. Review Logs: Check the complete build log
2. Identify Phase: Note which phase failed
3. Check Error: Read the specific error message
4. Verify Configuration: Ensure request settings are correct
5. Retry Build: Trigger a rebuild if appropriate
6. Contact Support: Escalate persistent issues

Build Completion

Success Notifications

When builds complete successfully:

• Status Update: Request status changes to “Fulfilled”
• Console Notification: Visual indicator in the console
• Email Notification: Optional email alert (if configured)
• Access Details: Connection information available

Post-Build Actions

After successful completion:

1. Verify VM: Check VM is accessible via VPN
2. Test Connectivity: Confirm network access
3. Review Configuration: Verify software installation
4. Document Access: Note connection details

Build Artifacts

Successful builds produce:

• Customized Image: Stored in B2 (for downloadable VMs)
• Deployed VM: Running instance on Proxmox
• Build Logs: Complete log history
• Configuration Records: Settings applied to VM

Log Access

Build Log Proxy

Access build logs through the proxy endpoint:

• Endpoint: /api/build-logs/proxy
• Authentication: Requires valid session
• Parameters: Task ID and log file reference

Log Retention

Build logs are retained according to policy:

• Active Builds: Real-time streaming available
• Recent Builds: Full logs available (30 days)
• Archived Builds: Compressed logs (90 days)
• Expired: Logs automatically removed

Downloading Logs

Export logs for analysis:

1. Open the build log viewer
2. Click Download button
3. Select format (text or JSON)
4. Save to local system

Monitoring Best Practices

During Builds

• Monitor Progress: Watch for stalled phases
• Check Logs: Review logs for warnings
• Note Timing: Track how long phases take
• Report Issues: Flag problems early

After Failures

• Capture Logs: Download full logs before retry
• Document Issues: Note error patterns
• Check Resources: Verify infrastructure capacity
• Coordinate Retry: Plan rebuild timing

For Administrators

• Monitor Queue: Watch task queue depth
• Track Success Rates: Review build success metrics
• Optimize Images: Reduce build times
• Scale Runners: Add capacity as needed

Related Documentation

• VM Requests - Submitting VM requests
• VM Imaging Workflow - Image creation process
• ARROW Virtual Machines - VM system overview
• Troubleshooting - Build failure troubleshooting