VM Imaging Workflow

VM Imaging Workflow

ARROW creates custom VM images through an automated imaging workflow that combines base images, organization configurations, and software installations. The workflow is orchestrated through backend/api/vm_imaging_handlers.go and backend/api/vm_images/.

Overview

The imaging workflow transforms base VM images into customized, deployment-ready images:

• Base Images: Standard OS images stored in Backblaze B2 (arrow-images bucket)
• Customization: Organization-specific configurations applied via Ansible
• Software Installation: Applications installed through Ansible roles
• Deployment: Images deployed to Proxmox VE or made available for download

B2 Storage Configuration

VM images are stored in Backblaze B2 with the following configuration:

Environment Variable	Description
B2_BUCKET	Primary bucket name (private)
B2_PUBLIC_BUCKET	Public bucket for downloads
B2_KEY_ID	B2 application key ID
B2_APP_KEY	B2 application key secret
B2_ENDPOINT	B2 S3-compatible endpoint
B2_REGION	B2 region identifier

Storage Layout:

arrow-images/
├── images/           # VM image files (.iso, .qcow2, .vmdk, etc.)
├── logs/             # Build logs organized by client/platform
│   └── {client}/
│       └── {platform}/
│           └── {task_id}.log
└── metadata/         # Image thumbnails and metadata files

Image Upload

Upload Process

VM images are uploaded to Backblaze B2 storage:

1. Select Image: Choose image file from local system
2. Validate File: System verifies file type and size
3. Upload: File transferred to B2 storage
4. Process Metadata: Image metadata extracted and stored
5. Available: Image ready for use in requests

File Validation

Uploads are validated for security and compatibility using magic number verification (implemented in backend/api/vm_images/image_upload.go):

Magic Number Verification:

Format	Magic Bytes	Offset	Description
ISO	43 44 30 30 31	32769	CD001 signature
QCOW2	51 46 49 FB	0	QFI magic header
VMDK	Text search	N/A	Descriptor file detection
VHD	63 6F 6E 65	Footer	conectix signature
RAW	None	N/A	Extension-based only

Size Limits:

Upload Type	Minimum	Maximum
Single upload	1 MB	2 GB
Multipart upload	1 MB	10 GB
Metadata images	1 KB	10 MB

Upload API

Images are uploaded through dedicated endpoints in backend/api/vm_images/:

Single Upload: POST /api/vm_images/upload

• Multipart form data with image file
• Rate limited: 5 uploads per hour
• Requires admin/superuser authentication
• Returns image record with B2 file reference

Multipart Upload (for large files):

Step	Endpoint	Purpose
1	POST /api/vm_images/upload/sign	Initiate upload, get pre-signed URLs
2	PUT to each signed URL	Upload file chunks
3	POST /api/vm_images/upload/complete	Complete and finalize upload

Image Collections

Organization Collections

Each organization maintains an image collection:

• Default Images: Pre-selected for new requests
• Custom Images: Organization-uploaded images
• Shared Images: Images shared across organizations

Collection Management

Manage image collections through:

1. Navigate to Settings > VM Images
2. View available images
3. Set default image selections
4. Upload new images
5. Remove unused images

Default Image Selection

Default images are automatically applied to new requests:

• Configured at organization level
• Can be overridden per request
• Supports multiple default images

Metadata Management

Image Metadata Fields

Field	Description	Required
name	Display name	Yes
description	Detailed description	No
file_name	Original filename	Yes
file_size	Size in bytes	Yes
file_type	MIME type	Yes
b2_file_id	B2 storage reference	Yes
organization	Owning organization	Yes
software	Attached software list	No
ansible_flags	Configuration flags	No

Software Configurations

Images can have attached software configurations:

{
  "software": [
    {
      "name": "nmap",
      "version": "latest",
      "ansible_role": "security-tools"
    },
    {
      "name": "burpsuite",
      "version": "2023.10",
      "ansible_role": "web-testing"
    }
  ]
}

Ansible Flags

Configuration flags control image customization:

Flag	Description
install_vpn	Install NetBird VPN client
harden_ssh	Apply SSH hardening
install_monitoring	Install monitoring agents
apply_branding	Apply organization branding

GitHub Integration

Imaging Handler

The imaging trigger is implemented in backend/api/vm_imaging_handlers.go:

Trigger Structures:

type VmImagingWebhookPayload struct {
    RequestID    string        // Device request ID
    DeviceType   string        // Type of device (pvm, vm)
    Images       []VmImageInfo // Images to build
    Organization string        // Organization context
    Client       string        // Client identifier
    Timestamp    string        // Request timestamp
}

type VmImageInfo struct {
    ID       string                 // Image ID
    Name     string                 // Image name
    Metadata map[string]interface{} // Image metadata
}

Workflow Trigger

VM imaging is triggered via GitHub webhook:

flowchart TD
    A[POST /api/vm-imaging/trigger] --> B[Validate Admin Auth]
    B --> C[Extract images and metadata]
    C --> D[Generate Build Task]
    D --> E[Queue Task in vm_build_tasks]
    E --> F[Create VmImagingWebhookPayload]
    F --> G[POST to GITHUB_VM_IMAGING_WEBHOOK_URL]
    G --> H[GitHub Actions Workflow]
    H --> I[Builder Polls GET /api/vm-build/tasks]
    I --> J[Retrieve ClientConfig & ImageConfig]
    J --> K[Download Base Image from B2]
    K --> L[Apply Customizations]
    L --> M[Install Software via Ansible]
    M --> N[POST /api/vm-build/task-logs to B2]
    N --> O[POST /api/vm-build/task-status]
    O --> P{Success?}
    P -->|Yes| Q[Deploy to Proxmox]
    P -->|No| R[Mark as Failed]
    Q --> S[Register with NetBird VPN]
    S --> T[Update device_settings]
    T --> U[Update device_request status = imaging]
    U --> V[Notify User]

Trigger Endpoint

Endpoint: POST /api/vm-imaging/trigger Authentication: Admin role required

Request Payload:

{
  "request_id": "device_request_id",
  "device_type": "pvm",
  "images": [
    {
      "id": "image_id",
      "name": "image_name",
      "metadata": {}
    }
  ],
  "organization": "org_id",
  "client": "client_name",
  "timestamp": "2024-01-15T10:30:00Z"
}

GitHub Webhook Format

The webhook dispatches a repository event to GitHub Actions:

Headers:

• Content-Type: application/json
• User-Agent: ARROW-VM-Imaging/1.0
• Authorization: token {secret} (if configured)

Payload:

{
  "event_type": "vm_imaging_request",
  "client_payload": {
    "request_id": "device_request_id",
    "device_type": "pvm",
    "images": [...],
    "organization": "org_id",
    "client": "client_name",
    "timestamp": "2024-01-15T10:30:00Z"
  }
}

Environment Variables:

• GITHUB_VM_IMAGING_WEBHOOK_URL: GitHub repository dispatch URL
• GITHUB_VM_IMAGING_WEBHOOK_SECRET: Authentication token

Manual Build Trigger

Administrators can trigger builds manually:

Endpoint: POST /api/vm-build/trigger Permission: image.admin required

{
  "image_id": "image_id",
  "image_name": "image_name",
  "build_server_id": "server_id",
  "build_type": "vm",
  "client_id": "client_id",
  "organization_id": "org_id",
  "build_options": {},
  "vm_name": "target_vm_name",
  "description": "Build description",
  "notify_on_complete": true
}

GitHub Actions Workflow

The imaging workflow executes:

1. Checkout: Clone imaging repository
2. Setup: Configure build environment
3. Poll Tasks: Retrieve pending build tasks
4. Process: Execute imaging for each task
5. Report: Update task status on completion

Build Configuration

ClientConfig

Organization-level configuration applied to all builds:

Field	Description
organization_id	Organization identifier
vpn_config	VPN server and credentials
network_settings	Network configuration
branding	Logo and theme settings
default_software	Always-installed software

ImageConfig

Image-specific configuration for each build:

Field	Description
image_id	Base image reference
software	Additional software to install
ansible_flags	Configuration flags
custom_scripts	Post-install scripts
resource_allocation	VM resource settings

Configuration Retrieval

The builder retrieves configurations via API:

• ClientConfig Endpoint: /api/imaging/client-config/{org_id}
• ImageConfig Endpoint: /api/imaging/image-config/{task_id}
• Authentication: Builder service account

Imaging Process

Phase 1: Preparation

1. Retrieve Task: Get task details from API
2. Download Configs: Fetch ClientConfig and ImageConfig
3. Prepare Environment: Set up build workspace
4. Download Image: Fetch base image from B2

Phase 2: Customization

1. Mount Image: Access image filesystem
2. Apply Base Config: Set hostname, network, users
3. Install Packages: System packages via apt/yum
4. Run Ansible: Execute playbooks for software
5. Apply Branding: Organization customizations

Phase 3: Finalization

1. Cleanup: Remove temporary files
2. Seal Image: Prepare for deployment
3. Upload Logs: Send logs to B2
4. Update Status: Report completion/failure

Phase 4: Deployment

For Proxmox-targeted builds:

1. Transfer Image: Send to Proxmox storage
2. Create VM: Configure VM from image
3. Start VM: Boot the new VM
4. Register VPN: Add to NetBird network
5. Verify: Confirm accessibility

Log Management

Build Logs

Logs are generated throughout imaging:

• Download Logs: Image retrieval progress
• Ansible Logs: Playbook execution output
• Deploy Logs: Proxmox deployment details
• Error Logs: Failure details and stack traces

Log Storage

Logs are stored in B2 with references:

/build-logs/{org_id}/{task_id}/
  - download.log
  - ansible.log
  - deploy.log
  - summary.log

Log Access

Access logs through:

• Console: Build monitoring interface
• API: /api/build-logs/proxy?task={id}&file={name}
• Direct B2: Admin access to storage

Error Recovery

Retry Mechanism

Failed builds can be retried:

1. Identify Failure: Review logs for cause
2. Fix Issues: Resolve configuration problems
3. Trigger Retry: Rebuild the task
4. Monitor: Watch for successful completion

Common Failures

Failure	Cause	Resolution
Download timeout	Network issues	Retry, check connectivity
Ansible failure	Playbook error	Fix playbook, retry
Disk space	Insufficient storage	Free space, retry
API timeout	Backend issues	Wait, retry

Partial Recovery

For partially completed builds:

• Resume Support: Some phases can resume
• Cleanup Required: Failed phases may need cleanup
• Fresh Start: Full rebuild often safest

Best Practices

Image Management

• Version Images: Maintain version history
• Test Before Production: Validate images first
• Document Changes: Note what each image contains
• Regular Updates: Keep base images current

Build Optimization

• Minimize Software: Only install required packages
• Optimize Playbooks: Efficient Ansible scripts
• Cache Dependencies: Reduce download time
• Parallel Tasks: Use multiple runners

Monitoring

• Watch Builds: Monitor active imaging
• Track Metrics: Build times, success rates
• Alert on Failures: Quick notification
• Review Logs: Regular log analysis

Related Documentation

• ARROW Virtual Machines - VM system overview
• VM Requests - Requesting VMs
• Build Monitoring - Tracking build progress
• Troubleshooting - Imaging troubleshooting