How to orchestrate seamless ISP failover across 6 international domains and build bulletproof, zero-dependency Let’s Encrypt certificate renewals behind Ubiquiti EdgeOS and Apache.
1. Introduction & The Engineering Challenge
Running a production-grade web infrastructure on self-hosted bare metal offers total privacy and freedom, but presents unique resilience challenges when hosting multiple high-traffic domains across changing network conditions:
- Dynamic Multi-WAN Interfaces: The primary fiber internet connection (RDS PPPoE, dynamic IP) and backup broadband (UPC Cable, dynamic IP) frequently change their public IPs and occasionally suffer upstream outages.
- Multi-Registrar Diversity: The domains are split across different DNS providers:
- Namecheap:
voina.org,voina.uk,voina.in,voina.fr - Internet.bs:
voina.be,voina.it
- Namecheap:
- The Dynamic DNS Hopping Dilemma: When an ISP goes down, DNS records across all registrars must instantly hop to the backup public IP without manual intervention, and seamlessly revert when the primary connection restores.
- The Dynamic IP vs. ACME Trap: Traditional Let’s Encrypt DNS-01 API challenges require hard coded source IP whitelisting at the registrar level. The moment an ISP renews a lease or hops to another link, DNS-01 API calls fail, causing certificate renewals to break.
Here is the complete architecture, implementation scripts, and step-by-step breakdown of how this dual-layer fail-over and autonomous renewal pipeline was designed and deployed.
2. Infrastructure Topology
The infrastructure consists of three interconnected layers:
- Perimeter Gateway: Ubiquiti EdgeRouter 4 managing dual WAN interfaces, Active-Active load balancing (Group G), watchdog health monitoring, and mirrored destination NAT rules.
- DNS Orchestration: Dynamic DNS APIs spanning Namecheap and Internet.bs, synchronizing all root apex and subdomain A records in parallel upon state changes.
- Internal Application Host: A dedicated Fedora Server (
nas1) running Apache HTTPD with SNI virtual hosts andacme.shelliptic curve SSL key engines.
3. Dynamic DNS Hopping & ISP Failover Engine
The Mechanism Step-by-Step
1. Inbound Port Forwarding Parity
To guarantee zero dropped requests regardless of which WAN interface is receiving traffic, mirrored DNAT rules were provisioned on the router:
- Port 80 (HTTP): Rule 11 (
pppoe0) & Rule 12 (eth1) ->192.168.2.21:80 - Port 443 (HTTPS): Rule 1 (
pppoe0) & Rule 2 (eth1) ->192.168.2.21:443 - Port 2222 (SSH): Rule 3 (
pppoe0) & Rule 4 (eth1) ->192.168.2.21:22
2. The Failover Transition Script
Deployed directly onto the EdgeRouter at /config/scripts/ddns-failover.sh:
#!/bin/bash
# =============================================================================
# EdgeOS WAN Load Balancer Dynamic DNS Failover Script
# Automatically switches Namecheap & Internet.bs DNS for all 6 domains
# =============================================================================
GROUP="$1"
INTERFACE="$2"
ACTION="$3"
LOG="/var/log/ddns-failover.log"
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WLB Event: Group=$GROUP Interface=$INTERFACE Action=$ACTION" >> "$LOG"
# Namecheap Domain:Password mapping
DOMAINS="voina.org:xxx voina.in:xxx voina.uk:xxx voina.fr:xxx"
# Internet.bs API Credentials & Records
IBS_KEY="xxx"
IBS_PASS="xxx"
IBS_RECORDS="blog.voina.be home.voina.be blog.voina.it home.voina.it"
update_dns_all() {
local TARGET_IP="$1"
# 1. Update Namecheap domains in parallel
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Updating Namecheap DDNS for all domains to $TARGET_IP..." >> "$LOG"
for PAIR in $DOMAINS; do
DOMAIN="${PAIR%%:*}"
PASSWORD="${PAIR##*:}"
for HOST in "blog" "home" "@" "www"; do
/usr/bin/curl -s -k "https://dynamicdns.park-your-domain.com/update?host=${HOST}&domain=${DOMAIN}&password=${PASSWORD}&ip=${TARGET_IP}" >/dev/null 2>&1 &
done
done
# 2. Update Internet.bs domains in parallel
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Updating Internet.bs DNS for all domains to $TARGET_IP..." >> "$LOG"
for REC in $IBS_RECORDS; do
/usr/bin/curl -s -k "https://api.internet.bs/Domain/DnsRecord/Update?ApiKey=${IBS_KEY}&Password=${IBS_PASS}&FullRecordName=${REC}&Type=A&NewValue=${TARGET_IP}&ResponseFormat=JSON" >/dev/null 2>&1 &
done
}
# If RDS (pppoe0) goes inactive/fails, immediately switch DNS to UPC (eth1)
if [ "$INTERFACE" = "pppoe0" ] && [ "$ACTION" = "inactive" ]; then
UPC_IP=$(/sbin/ip -4 addr show dev eth1 2>/dev/null | awk '/inet / {print $2}' | cut -d/ -f1)
if [ -n "$UPC_IP" ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] RDS is DOWN. Switching DNS to UPC ($UPC_IP)" >> "$LOG"
update_dns_all "$UPC_IP"
fi
# If RDS (pppoe0) becomes active/recovers, immediately switch DNS back to RDS (pppoe0)
elif [ "$INTERFACE" = "pppoe0" ] && [ "$ACTION" = "active" ]; then
RDS_IP=$(/sbin/ip -4 addr show dev pppoe0 2>/dev/null | awk '/inet / {print $2}' | cut -d/ -f1)
if [ -n "$RDS_IP" ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] RDS is ACTIVE. Switching DNS back to RDS ($RDS_IP)" >> "$LOG"
update_dns_all "$RDS_IP"
fi
fi```bash
#!/bin/bash
# =============================================================================
# EdgeOS WAN Load Balancer Dynamic DNS Failover Script
# Automatically switches Namecheap & Internet.bs DNS for all 6 domains
# =============================================================================
GROUP="$1"
INTERFACE="$2"
ACTION="$3"
LOG="/var/log/ddns-failover.log"
echo "[$(date '+%Y-%m-%d %H:%M:%S')] WLB Event: Group=$GROUP Interface=$INTERFACE Action=$ACTION" >> "$LOG"
# Namecheap Domain:Password mapping (Redacted)
DOMAINS="voina.org:XXX_NC_PASS_ORG voina.in:XXX_NC_PASS_IN voina.uk:XXX_NC_PASS_UK voina.fr:XXX_NC_PASS_FR"
# Internet.bs API Credentials & Records (Redacted)
IBS_KEY="XXX_INTERNETBS_API_KEY"
IBS_PASS="XXX_INTERNETBS_PASSWORD"
IBS_RECORDS="blog.voina.be home.voina.be blog.voina.it home.voina.it"
update_dns_all() {
local TARGET_IP="$1"
# 1. Update Namecheap domains in parallel
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Updating Namecheap DDNS for all domains to $TARGET_IP..." >> "$LOG"
for PAIR in $DOMAINS; do
DOMAIN="${PAIR%%:*}"
PASSWORD="${PAIR##*:}"
for HOST in "blog" "home" "@" "www"; do
/usr/bin/curl -s -k "https://dynamicdns.park-your-domain.com/update?host=${HOST}&domain=${DOMAIN}&password=${PASSWORD}&ip=${TARGET_IP}" >/dev/null 2>&1 &
done
done
# 2. Update Internet.bs domains in parallel
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Updating Internet.bs DNS for all domains to $TARGET_IP..." >> "$LOG"
for REC in $IBS_RECORDS; do
/usr/bin/curl -s -k "https://api.internet.bs/Domain/DnsRecord/Update?ApiKey=${IBS_KEY}&Password=${IBS_PASS}&FullRecordName=${REC}&Type=A&NewValue=${TARGET_IP}&ResponseFormat=JSON" >/dev/null 2>&1 &
done
}
# If RDS (pppoe0) goes inactive/fails, immediately switch DNS to UPC (eth1)
if [ "$INTERFACE" = "pppoe0" ] && [ "$ACTION" = "inactive" ]; then
UPC_IP=$(/sbin/ip -4 addr show dev eth1 2>/dev/null | awk '/inet / {print $2}' | cut -d/ -f1)
if [ -n "$UPC_IP" ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] RDS is DOWN. Switching DNS to UPC ($UPC_IP)" >> "$LOG"
update_dns_all "$UPC_IP"
fi
# If RDS (pppoe0) becomes active/recovers, immediately switch DNS back to RDS (pppoe0)
elif [ "$INTERFACE" = "pppoe0" ] && [ "$ACTION" = "active" ]; then
RDS_IP=$(/sbin/ip -4 addr show dev pppoe0 2>/dev/null | awk '/inet / {print $2}' | cut -d/ -f1)
if [ -n "$RDS_IP" ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] RDS is ACTIVE. Switching DNS back to RDS ($RDS_IP)" >> "$LOG"
update_dns_all "$RDS_IP"
fi
fi
3. EdgeOS Hook Registration
The transition script is hooked directly to EdgeOS load balance events:
set load-balance group G transition-script /config/scripts/ddns-failover.sh
4. Autonomous Zero-Dependency SSL/TLS Renewal Architecture
Why DNS-01 API Renewals Fail on Dynamic IPs
Using DNS API tokens (like dns_namecheap or dns_internetbs) introduces a major fragility: both registrars enforce strict API Source IP Whitelists. When dynamic connections change IP leases or failover occurs, the API rejects the request with:
error Invalid request IP: 188.24.5.10
Connection refused from unauthorized host. IP is NOT allowed to access your account.
The Solution: Global HTTP-01 Webroot Pipeline
By routing all .well-known/acme-challenge/ requests through a dedicated global directory in Apache, certificates renew 100% autonomously without API keys, IP whitelists, or DNS propagation delays.
1. Global Apache ACME Configuration
Placed in /etc/httpd/conf.d/acme.conf:
Alias /.well-known/acme-challenge/ /var/www/acme-challenge/.well-known/acme-challenge/
<Directory "/var/www/acme-challenge">
Options None
AllowOverride None
Require all granted
</Directory>
2. Apache VirtualHost Definition (Standardized for all 6 domains)
Each domain file (e.g. /etc/httpd/conf.d/blog.voina.org.conf) is configured with selective HTTPS rewriting:
<VirtualHost *:80>
ServerName "blog.voina.org"
DocumentRoot "/var/www/acme-challenge"
RewriteEngine On
# Bypass HTTPS redirection for ACME challenges
RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/
RewriteRule ^(.*)$ https://blog.voina.be$1 [R=301,L]
</VirtualHost>
<VirtualHost *:443>
SSLEngine On
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4:!3DES
SSLProtocol all -SSLv2 -SSLv3
SSLCertificateFile /root/.acme.sh/blog.voina.org_ecc/blog.voina.org.cer
SSLCertificateKeyFile /root/.acme.sh/blog.voina.org_ecc/blog.voina.org.key
SSLCertificateChainFile /root/.acme.sh/blog.voina.org_ecc/ca.cer
Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
ServerName "blog.voina.org"
DocumentRoot "/media/storage/www/html/owncloud/wordpress"
CustomLog "/media/storage/www/log/home-blog-access.log" combined
ErrorLog "/media/storage/www/log/home-blog-error.log"
<Directory "/media/storage/www/html/owncloud/wordpress">
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted
Satisfy Any
</Directory>
<IfModule !mod_php5.c>
<IfModule !mod_php7.c>
SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1
<FilesMatch \.(php|phar)$>
SetHandler "proxy:unix:/run/php-fpm/www.sock|fcgi://localhost"
</FilesMatch>
</IfModule>
</IfModule>
Timeout 600
ProxyTimeout 600
</VirtualHost>
3. Certificate Issuance Command
Each domain is registered with acme.sh using elliptic curve keys (ec-256) and an automated Apache reload trigger:
/root/.acme.sh/acme.sh --issue \
-d blog.voina.org \
-w /var/www/acme-challenge \
--keylength ec-256 \
--reloadcmd "systemctl reload httpd"
4. Automated Daily Renewal Pipeline
Cron job running at 04:53 AM daily via /root/renew_certs_wrapper.sh:
#!/bin/bash
# /root/renew_certs_wrapper.sh
# Run acme.sh renewal across all configured webroot certificates
/root/.acme.sh/acme.sh --cron --home "/root/.acme.sh" > /tmp/acme_cron.log 2>&1
EXIT_CODE=$?
if [ $EXIT_CODE -ne 0 ]; then
SUBJECT="[ALERT] SSL Certificate Renewal Failed on blog.voina.org"
BODY="SSL certificate renewal encountered an issue. Check /root/.acme.sh/acme.sh.log."
python3 /root/send_renewal_alert.py "$SUBJECT" "$BODY"
fi
5. Live Production Verification
All 6 domains tested with automated probes from external networks:
| Domain | Registrar | Resolving Public IP | HTTP (Port 80) Status | SSL Handshake | Valid Until |
|---|---|---|---|---|---|
blog.voina.org | Namecheap | 188.24.5.10 | 301 Moved Permanently | CN=blog.voina.org (YE2) | Dec 17, 2026 |
blog.voina.uk | Namecheap | 188.24.5.10 | 301 Moved Permanently | CN=blog.voina.uk (YE2) | Dec 17, 2026 |
blog.voina.in | Namecheap | 188.24.5.10 | 301 Moved Permanently | CN=blog.voina.in (YE1) | Dec 17, 2026 |
blog.voina.fr | Namecheap | 188.24.5.10 | 301 Moved Permanently | CN=blog.voina.fr (YE1) | Dec 17, 2026 |
blog.voina.be | Internet.bs | 188.24.5.10 | 301 Moved Permanently | CN=blog.voina.be (YE1) | Dec 17, 2026 |
blog.voina.it | Internet.bs | 188.24.5.10 | 301 Moved Permanently | CN=blog.voina.it (YE2) | Dec 17, 2026 |
6. Key Takeaways & Best Practices
- Decouple SSL Issuance from DNS APIs: If your server operates behind dynamic IPs, avoid DNS-01 challenges that depend on static IP API whitelisting. Use HTTP-01 webroot routing with selective Apache rewrites.
- Implement Router-Level State Hooks: Offload DNS failover logic to the gateway router (EdgeOS transition scripts). This ensures immediate detection and execution at the network perimeter rather than polling from internal hosts.
- Ensure Full Inbound WAN Parity: Always mirror port forward (DNAT) rules across all active and backup WAN interfaces so incoming connections immediately succeed regardless of which ISP link receives the traffic.