HALO OTA System Guide

Last Updated: February 7, 2026
Author: Trepo Engineering
Status: Production Ready ✅

Table of Contents

1. Overview
2. Architecture
3. How It Works
4. S3 Bucket Structure
5. Manifest File Format
6. Pushing an OTA Update
7. MQTT Commands
8. Monitoring & Debugging
9. Rollback & Safety
10. Troubleshooting

Overview

The HALO device consists of two ESP32 boards that can be updated independently over-the-air:

Key Features: - ✅ Automatic updates during maintenance windows - ✅ Manual trigger via MQTT - ✅ Version comparison (only downloads if newer) - ✅ Retry logic (3 manifest attempts, 3 download attempts) - ✅ Chunked download with progress reporting - ✅ Automatic rollback on boot failure

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         AWS Cloud                                │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    S3 Bucket                             │    │
│  │              halo-ota-dev.s3.amazonaws.com               │    │
│  │                                                          │    │
│  │   /halo/ota/dev/                                        │    │
│  │   ├── manifest_latest.json    ← Sense manifest          │    │
│  │   ├── sense_v6.0.46.bin       ← Sense firmware          │    │
│  │   └── lcd/                                              │    │
│  │       ├── manifest_latest.json ← LCD manifest           │    │
│  │       └── lcd_v1.2.3.bin       ← LCD firmware           │    │
│  └─────────────────────────────────────────────────────────┘    │
│                              │                                   │
│                              │ HTTPS GET                         │
│                              ▼                                   │
└─────────────────────────────────────────────────────────────────┘
                               │
                               │
                    ┌──────────┴──────────┐
                    │     WiFi Network     │
                    └──────────┬──────────┘
                               │
              ┌────────────────┴────────────────┐
              │                                  │
              ▼                                  ▼
      ┌──────────────┐                  ┌──────────────┐
      │  LCD Board   │◄────UART────────►│ Sense Board  │
      │  (ESP32-S3)  │                  │(XIAO ESP32-S3)│
      └──────────────┘                  └──────────────┘

How It Works

OTA Flow (Step by Step)

┌─────────────────────────────────────────────────────────────────┐
│                        OTA UPDATE FLOW                          │
└─────────────────────────────────────────────────────────────────┘

  Device                                              S3 Bucket
    │                                                     │
    │  1. Check if in maintenance window                  │
    │     (or MQTT force command received)                │
    │                                                     │
    │  2. GET manifest_latest.json ───────────────────────►
    │                                                     │
    │  ◄─────────────────────────────── 3. Return manifest│
    │                                                     │
    │  4. Compare versions:                               │
    │     - If remote > local: proceed                    │
    │     - If remote <= local: skip (up to date)         │
    │                                                     │
    │  5. GET firmware.bin ───────────────────────────────►
    │     (chunked, 2KB buffer)                           │
    │                                                     │
    │  ◄─────────────────────────────── 6. Stream binary  │
    │     (progress: 10%, 20%, ... 100%)                  │
    │                                                     │
    │  7. Write to OTA partition                          │
    │                                                     │
    │  8. Verify checksum                                 │
    │                                                     │
    │  9. Set boot partition & reboot                     │
    │                                                     │
    │  10. Boot into new firmware                         │
    │      - Mark as valid after successful boot          │
    │      - Or rollback if boot fails                    │
    │                                                     │

When Does OTA Run?

1. Scheduled Maintenance Window
   • Configured via MQTT: ota/sched/set
   • Default: 2:00 AM - 2:30 AM UTC (6:00 PM - 6:30 PM PST)
2. Manual Trigger
   • MQTT command: ota/force_now
   • Bypasses cooldowns and schedules
3. On Boot
   • Checks for updates after WiFi connects
   • Respects 24-hour cooldown between checks

S3 Bucket Structure

halo-ota-dev/
└── halo/
    └── ota/
        ├── dev/                          ← Development environment
        │   ├── manifest_latest.json      ← SENSE manifest
        │   ├── sense_v6.0.46.bin         ← SENSE firmware
        │   └── lcd/
        │       ├── manifest_latest.json  ← LCD manifest
        │       └── lcd_v1.2.3.bin        ← LCD firmware
        │
        ├── staging/                      ← Staging environment
        │   ├── manifest_latest.json
        │   ├── sense_vX.X.X.bin
        │   └── lcd/
        │       ├── manifest_latest.json
        │       └── lcd_vX.X.X.bin
        │
        └── prod/                         ← Production environment
            ├── manifest_latest.json
            ├── sense_vX.X.X.bin
            └── lcd/
                ├── manifest_latest.json
                └── lcd_vX.X.X.bin

S3 URLs

Manifest File Format

The manifest tells the device what version is available and where to download it.

Schema

{
  "version": "6.0.46",
  "url": "https://halo-ota-dev.s3.us-east-1.amazonaws.com/halo/ota/dev/sense_v6.0.46.bin",
  "size": 1482736,
  "sha256": "a1b2c3d4e5f6..."
}

Fields

Version Comparison

Versions are compared as semantic versions: - 6.0.46 > 6.0.45 → Update - 6.0.46 = 6.0.46 → Skip (already up to date) - 6.0.46 < 6.0.47 → Update

Pushing an OTA Update

Prerequisites

1. AWS CLI configured with trepo-dev profile
2. Arduino CLI for compiling firmware
3. Access to the firmware source code

Step-by-Step Guide

Step 1: Build the Firmware

For Sense Board:

cd /path/to/firmware/sense

# Compile
arduino-cli compile \
  --fqbn "esp32:esp32:XIAO_ESP32S3:USBMode=hwcdc,CDCOnBoot=default" \
  --output-dir ./build \
  .

# The binary will be at: ./build/sense.ino.bin

For LCD Board:

cd /path/to/firmware/lcd

# Compile
arduino-cli compile \
  --fqbn "esp32:esp32:esp32s3:FlashSize=16M,PartitionScheme=default_8MB,PSRAM=opi,CDCOnBoot=cdc" \
  --output-dir ./build \
  .

# The binary will be at: ./build/lcd.ino.bin

Step 2: Rename the Binary

Use a consistent naming convention:

# Sense
mv ./build/sense.ino.bin ./build/sense_v6.0.47.bin

# LCD
mv ./build/lcd.ino.bin ./build/lcd_v1.2.4.bin

Step 3: Upload Binary to S3

# Sense (dev environment)
aws s3 cp ./build/sense_v6.0.47.bin \
  s3://halo-ota-dev/halo/ota/dev/sense_v6.0.47.bin \
  --profile trepo-dev

# LCD (dev environment)
aws s3 cp ./build/lcd_v1.2.4.bin \
  s3://halo-ota-dev/halo/ota/dev/lcd/lcd_v1.2.4.bin \
  --profile trepo-dev

Step 4: Get Binary Size

# Get file size in bytes
ls -l ./build/sense_v6.0.47.bin | awk '{print $5}'
# Example output: 1482736

Step 5: Create/Update Manifest

Create manifest_latest.json:

{
  "version": "6.0.47",
  "url": "https://halo-ota-dev.s3.us-east-1.amazonaws.com/halo/ota/dev/sense_v6.0.47.bin",
  "size": 1482736
}

Step 6: Upload Manifest to S3

# Sense
aws s3 cp manifest_latest.json \
  s3://halo-ota-dev/halo/ota/dev/manifest_latest.json \
  --profile trepo-dev \
  --content-type "application/json"

# LCD
aws s3 cp manifest_latest.json \
  s3://halo-ota-dev/halo/ota/dev/lcd/manifest_latest.json \
  --profile trepo-dev \
  --content-type "application/json"

Step 7: Verify Upload

# Check manifest is accessible
curl -s https://halo-ota-dev.s3.us-east-1.amazonaws.com/halo/ota/dev/manifest_latest.json | jq .

# Check binary is accessible (just headers)
curl -I https://halo-ota-dev.s3.us-east-1.amazonaws.com/halo/ota/dev/sense_v6.0.47.bin

Step 8: Trigger OTA on Device

Either wait for the maintenance window, or force immediately via MQTT:

# Using mosquitto_pub (replace with your device's topic)
mosquitto_pub \
  -h your-mqtt-broker.amazonaws.com \
  -p 8883 \
  --cafile AmazonRootCA1.pem \
  --cert device-cert.pem \
  --key device-key.pem \
  -t "halo/device/OWNER_ID/DEVICE_ID/cmd" \
  -m '{"cmd":"ota/force_now"}'

Quick Reference: One-Command Deploy

Here’s a script that does everything:

#!/bin/bash
# deploy_ota.sh - Deploy OTA update for Sense or LCD

BOARD=$1        # "sense" or "lcd"
VERSION=$2      # e.g., "6.0.47"
BINARY=$3       # path to compiled .bin file
ENV=${4:-dev}   # "dev", "staging", or "prod"

BUCKET="halo-ota-dev"
PROFILE="trepo-dev"

if [ "$BOARD" = "sense" ]; then
    S3_PATH="halo/ota/$ENV"
    BINARY_NAME="sense_v${VERSION}.bin"
elif [ "$BOARD" = "lcd" ]; then
    S3_PATH="halo/ota/$ENV/lcd"
    BINARY_NAME="lcd_v${VERSION}.bin"
else
    echo "Usage: $0 <sense|lcd> <version> <binary_path> [env]"
    exit 1
fi

SIZE=$(ls -l "$BINARY" | awk '{print $5}')
URL="https://${BUCKET}.s3.us-east-1.amazonaws.com/${S3_PATH}/${BINARY_NAME}"

echo "📦 Deploying $BOARD v$VERSION to $ENV"
echo "   Binary: $BINARY ($SIZE bytes)"
echo "   URL: $URL"
echo ""

# Upload binary
echo "⬆️  Uploading binary..."
aws s3 cp "$BINARY" "s3://${BUCKET}/${S3_PATH}/${BINARY_NAME}" --profile $PROFILE

# Create and upload manifest
echo "📝 Creating manifest..."
cat > /tmp/manifest_latest.json << EOF
{
  "version": "$VERSION",
  "url": "$URL",
  "size": $SIZE
}
EOF

aws s3 cp /tmp/manifest_latest.json \
    "s3://${BUCKET}/${S3_PATH}/manifest_latest.json" \
    --profile $PROFILE \
    --content-type "application/json"

# Verify
echo "✅ Verifying..."
curl -s "https://${BUCKET}.s3.us-east-1.amazonaws.com/${S3_PATH}/manifest_latest.json" | jq .

echo ""
echo "🚀 Deploy complete! Devices will update during next maintenance window."
echo "   To force immediate update, send MQTT command: ota/force_now"

Usage:

./deploy_ota.sh sense 6.0.47 ./build/sense.ino.bin dev
./deploy_ota.sh lcd 1.2.4 ./build/lcd.ino.bin prod

MQTT Commands

Control OTA behavior via MQTT commands sent to:

halo/device/{owner_id}/{device_id}/cmd

Available Commands

Schedule Configuration

The start_min is minutes past midnight UTC:

Example: Set maintenance window to 2-3 AM PST (10-11 AM UTC):

{
  "cmd": "ota/sched/set",
  "enable": true,
  "start_min": 600,
  "window_min": 60
}

Monitoring & Debugging

Serial Output

Connect to the device via USB serial to see OTA progress:

# Sense board
screen /dev/cu.usbmodem21101 115200

# LCD board  
screen /dev/cu.usbmodem21201 115200

Expected output during OTA:

[OTA] Checking for updates...
[OTA] Manifest attempt 1/3...
[OTA] Current: 6.0.45, Available: 6.0.47
[OTA] Update available! Downloading...
[OTA] Download attempt 1/3...
[OTA] Progress: 10% (148273/1482736)
[OTA] Progress: 20% (296547/1482736)
...
[OTA] Progress: 100% (1482736/1482736)
[OTA] Download complete. Applying update...
[OTA] Update successful! Rebooting...

CloudWatch Logs

OTA events are logged to CloudWatch (if configured): - Update started - Update completed - Update failed (with error)

Check Current Manifest

# Sense
curl -s https://halo-ota-dev.s3.us-east-1.amazonaws.com/halo/ota/dev/manifest_latest.json | jq .

# LCD
curl -s https://halo-ota-dev.s3.us-east-1.amazonaws.com/halo/ota/dev/lcd/manifest_latest.json | jq .

Rollback & Safety

Automatic Rollback

The ESP32 has built-in rollback protection:

1. After OTA, the new firmware is marked as “pending verification”
2. If the device boots successfully and runs for 30+ seconds, it’s marked “valid”
3. If the device fails to boot or crashes immediately, it automatically rolls back to the previous firmware

Manual Rollback

To rollback to a previous version:

1. Upload the old binary to S3 with a new version number (e.g., 6.0.48)
2. Update the manifest to point to it
3. Trigger OTA

Important: The version number must be HIGHER than the current version for the device to accept it.

Troubleshooting

Device Not Updating

Common Errors

Error: HTTP 403 Forbidden - S3 bucket permissions issue - Check bucket policy allows public read

Error: SSL handshake failed - Certificate issue - Ensure device has correct root CA

Error: Not enough space - Binary too large - Check OTA partition size (should be ~1.9MB for Sense, ~3.9MB for LCD)

Verify S3 Access

# Should return 200
curl -I https://halo-ota-dev.s3.us-east-1.amazonaws.com/halo/ota/dev/manifest_latest.json

# Check bucket policy
aws s3api get-bucket-policy --bucket halo-ota-dev --profile trepo-dev

Testing Checklist

Before deploying to production:

• Binary compiles without errors
• Binary size fits in OTA partition
• Version number is incremented
• Manifest JSON is valid
• Binary uploads to S3 successfully
• Manifest uploads to S3 successfully
• Manifest URL returns correct JSON
• Binary URL returns correct Content-Length
• Test device receives and applies update
• Device boots successfully after update
• Device reports correct version after update

Summary

Quick Links

• S3 Console: halo-ota-dev bucket
• Firmware Repo: /Users/mikehunt/Documents/Arduino/HALOMAIN_rev1/
• Test Results: /Users/mikehunt/.openclaw/workspace/ota_test_results/

Questions? Contact the Trepo engineering team.