ARROW Virtual Machines

Overview

ARROW provides a comprehensive virtual machine management system that automates VM provisioning, deployment, and lifecycle management.

System Architecture

The ARROW VM management system consists of several integrated components:

flowchart TD
    subgraph Console["ARROW Console"]
        A[VM Request Form]
        B[Build Monitor]
        C[Device Management]
    end

    subgraph Backend["PocketBase Backend"]
        D[Request Handler]
        E[Build Task Queue]
        F[Status Tracker]
    end

    subgraph Storage["B2 Storage"]
        G[VM Images]
        H[Build Logs]
    end

    subgraph CI["GitHub Actions"]
        I[Imaging Workflow]
        J[Build Runner]
    end

    subgraph Infrastructure["Infrastructure"]
        K[Proxmox VE]
        L[NetBird VPN]
    end

    A --> D
    D --> E
    E --> I
    I --> J
    J --> G
    J --> K
    K --> L
    L --> F
    F --> B
    G --> J
    H --> B

Key Components:

• PocketBase Backend: Manages requests, tracks status, and coordinates workflows
• B2 Storage: Stores VM images and build logs
• GitHub Actions: Executes imaging and deployment workflows
• Proxmox VE: Hosts deployed virtual machines
• NetBird VPN: Provides secure network access to VMs

VM Image Management

Image Storage

VM images are stored in Backblaze B2 cloud storage, managed through backend/api/vm_images/:

• Secure Access: 24-hour pre-signed URLs for authorized downloads
• Validation: Magic number verification for security
• Metadata Caching: 10-minute TTL for metadata fetches
• Rate Limiting: 5 uploads per hour per admin
• Size Limits: 2GB single upload, 10GB multipart upload

Supported Formats

Format	Extension	Magic Bytes	Validation	Use Case
ISO	.iso	43 44 30 30 31 (CD001)	Offset 32769	Bootable installation media
QCOW2	.qcow2	51 46 49 FB (QFI)	First 4 bytes	QEMU/KVM and Proxmox native format
VMDK	.vmdk	Text descriptor	String search	VMware virtual disk
VHD	.vhd	63 6F 6E 65 (conectix)	Footer check	Hyper-V virtual disk
RAW	.raw	None required	Extension only	Uncompressed disk image

Image Upload API

Single Upload: POST /api/vm_images/upload

• Multipart form data with image file (max 2GB)
• Rate limited: 5 uploads per hour
• Authentication: Admin/superuser required
• Returns image record with B2 file reference

Multipart Upload (for large files):

Endpoint	Method	Purpose
/api/vm_images/upload/sign	POST	Initiate upload, get signed chunk URLs
/api/vm_images/upload/complete	POST	Complete multipart upload

• Maximum file size: 10GB
• Supports resume on failure
• Pre-signed URLs for each chunk

Download URL: POST /api/vm_images/download_url

• Generates 24-hour pre-signed download links
• Authentication required

Metadata Image Upload: POST /api/vm_images/metadata-image-upload

• For thumbnails and image icons (max 10MB)
• Requires software.admin permission

Image Collections

Organizations maintain image collections managed through backend/api/vm_images/hooks.go:

• Default Images: Pre-selected images for new requests (single default enforced per org)
• Custom Images: Organization-specific configurations
• Software Bundles: Pre-installed application sets with Ansible roles
• Demo Images: Special images available to demo organizations via /api/vm_images/demo_available

Image Metadata

Each image includes metadata for configuration, with caching support (MetadataCache with 10-minute TTL):

Field	Description	Required
name	Display name for the image	Yes
description	Detailed description of image contents	No
file_name	Original filename	Yes
file_size	Size in bytes	Yes
file_type	MIME type	Yes
b2_file_id	B2 storage reference	Yes
software	List of pre-installed software with Ansible roles	No
ansible_flags	Configuration flags for customization	No
organization	Owning organization	Yes

Metadata Fetching

External metadata can be fetched via POST /api/vm_images/fetch_metadata:

• Caches results for 10 minutes
• Cache status viewable at GET /api/vm_images/cache_status (admin only)

Device Types

ARROW supports different VM deployment configurations:

Proxmox Virtual Machines (pvm)

VMs deployed directly on ARROW’s Proxmox infrastructure:

• Managed Proxmox VE clusters
• Automated provisioning and configuration
• Integrated backup and recovery
• Direct VPN connectivity

External Virtual Machines (vm)

VMs deployed on client infrastructure:

• VMware vSphere environments
• VirtualBox installations
• QEMU/KVM hypervisors
• Downloadable image files

Configuration Options

VM configurations include:

• Resource Allocation: CPU cores, memory, disk size
• Network Settings: VPN integration, network interfaces
• Security Settings: Access controls, firewall rules
• Software Stack: Pre-installed applications and tools

Build Process

Build Task Queue System

VM builds are managed through backend/api/vm_build_monitor/:

1. Task Creation: handleManualBuildTrigger() creates vm_build_tasks record
2. Webhook Trigger: Backend sends vm_imaging_request event to GitHub
3. Builder Polling: Build servers poll GET /api/vm-build/tasks for work
4. Progress Reporting: Real-time updates via WebSocket and REST
5. Completion: Status updates and VPN registration

Manual Build Trigger

Administrators can trigger builds via POST /api/vm-build/trigger:

{
  "image_id": "image_id",
  "image_name": "image_name",
  "build_server_id": "server_id",
  "build_type": "vm",
  "client_id": "client_id",
  "organization_id": "org_id",
  "build_options": {},
  "vm_name": "target_vm_name",
  "description": "Build description",
  "notify_on_complete": true
}

Permission Required: image.admin

Build Configuration

Two configuration objects drive the build process:

ClientConfig - Organization-level settings:

{
  "name": "organization_internal_name",
  "display_name": "Organization Display Name",
  "organization_id": "org_id",
  "settings": {
    "vpn_config": {},
    "network_settings": {},
    "branding": {},
    "default_software": []
  }
}

ImageConfig - Image-specific settings:

{
  "name": "image_internal_name",
  "display_name": "Image Display Name",
  "base_os": "ubuntu",
  "vm_name": "target_vm_name",
  "supports_hardware": true,
  "supports_virtual": true,
  "description": "Image description",
  "settings": {
    "software": [
      {
        "name": "package_name",
        "version": "version",
        "ansible_role": "role_name"
      }
    ],
    "ansible_flags": ["flag1", "flag2"],
    "custom_scripts": ["script1.sh"],
    "resource_allocation": {
      "cpu": 4,
      "memory_gb": 8,
      "disk_gb": 50
    }
  }
}

Configuration Retrieval Endpoints:

• ClientConfig: GET /api/imaging/client-config/{org_id}
• ImageConfig: GET /api/imaging/image-config/{task_id}

Build Task Data Structure

Each build task (BuildTask) contains:

Field	Description
TaskID	Unique identifier with timestamp
DeviceSettingsID	Reference to device_settings
Platform	Target platform (vmware, proxmox)
ClientConfig	Organization configuration
ImageConfig	Image-specific settings
Credentials	Generated root_pw, arrow_pw, api_key
VMSerial	Unique serial (format: VM-XXXXX)

For detailed build process information, see VM Imaging Workflow.

VPN Integration

Automatic NetBird Setup

When VMs are provisioned, they are automatically registered with NetBird:

1. Peer Registration: VM registered as a NetBird peer
2. Group Assignment: VM added to appropriate groups (pvm, vm)
3. Access Policies: One-way access policies configured
4. DNS Registration: Hostname registered for DNS resolution

Access Groups

VMs are assigned to access control groups:

Group	Description
pvm	Proxmox-hosted virtual machines
vm	External virtual machines
device-{id}	Device-specific group
consultants-{id}	Assigned consultant group

VPN Policies

Default VPN policies for VMs:

• Users can connect TO VMs (one-way)
• VMs cannot initiate connections to users
• SOCKS proxy access for traffic routing
• Health check endpoints for monitoring

For detailed VPN configuration, see VPN Management.

VM Completion

Completion Process

When a VM engagement ends, the completion process:

1. Initiate Completion: User or system triggers completion
2. NetBird Cleanup: VPN peer removed from network
3. Group Cleanup: Access control groups deleted
4. Policy Removal: Associated policies removed
5. Resource Release: VM resources freed

Automated Cleanup

The completion handler performs these cleanup operations:

flowchart TD
    A[Completion Triggered] --> B[Get Device Settings]
    B --> C[Retrieve NetBird Client]
    C --> D[Delete VPN Peer]
    D --> E{Peer Deleted?}
    E -->|Yes| F[Delete Device Group]
    E -->|No| G[Log Warning]
    F --> H[Delete Consultant Group]
    G --> H
    H --> I[Remove Access Policies]
    I --> J[Update Device Status]
    J --> K[Completion Done]

Manual vs Automatic

• Manual: User initiates completion through console
• Automatic: System completes VMs past their end date
• Early Completion: VMs can be completed before scheduled end

Monitoring and Logs

Build Monitoring

Real-time monitoring is handled by backend/api/vm_build_monitor/:

• WebSocket Streaming: Live logs via /api/vm-build/logs/stream
• Progress Updates: Percentage, stage, step via POST /api/vm-build/task-status
• Error Notifications: Immediate failure reporting
• Build History: Full task history in vm_build_tasks

Build Server Monitoring

Build servers report health metrics via POST /api/vm-build/server-health:

Metric	Description
cpu_usage	Current CPU utilization (%)
memory_usage	RAM utilization (%)
disk_usage	Disk utilization (%)
active_builds	Number of concurrent builds
max_concurrent_builds	Server capacity limit
version	Builder software version
last_heartbeat	Latest health check timestamp

Log Access

Build logs are accessible through multiple endpoints (backend/api/build_logs/):

Endpoint	Method	Purpose
/api/build-logs/proxy	GET	Proxy logs by URL (bypasses CORS)
/api/build-logs/file	GET	Fetch logs by B2 key
/api/vm-build/task-logs/{task_id}	GET	Retrieve logs for specific task

Log Storage Location: B2 bucket arrow-images at path logs/{client}/{platform}/{filename}

For detailed monitoring information, see Build Monitoring.

Related Documentation

• VM Requests - Submitting and tracking VM requests
• VM Imaging Workflow - Image creation and customization
• Build Monitoring - Real-time build tracking
• Infrastructure Management - Asset tracking and management
• VPN Management - VPN configuration and access
• Network Access Control - Access policies and permissions