Reaching a router that the internet can't reach

Each branch gets its internet from a 4G router. Nothing on the internet can reach those routers: the carrier puts them behind carrier-grade NAT, and the routers offer no remote admin.

I needed to do two things from home or from my phone. Add a new trusted device to the Wi-Fi, because the routers only admit devices they know. And restart the router. Later I'd also want to reach other devices on a branch network, like cameras and a fingerprint reader, for settings that only work locally.

My first message, on 11 October at 19:48, said I'd tried remote access through the branch iPad and it hadn't worked. I asked whether "an ESP32 device that I can talk to remotely but also can communicate with the router", or something else low-cost, could do it. Two days later, that is what I had.

1. Mea phone, anywhere
2. I connect in
3. Something I controla server on the internet
4. it dials out
5. A device in the branchon the branch Wi-Fi
6. local
7. The 4G routerbehind carrier NAT

Every working answer has the same shape. A device inside the branch keeps a connection open out to something I control, and I come back in through it. The whole story is the search for that device.

The rest of this page is the long version, in the order it happened.

1. Choosing a device11 Oct, evening
2. The TV box that wouldn't let go of Android11 Oct 20:30 to 12 Oct ~03:00
3. Back to the ESP3212 Oct 15:00 to 21:51
4. Making it something you can leave in a branch12 Oct 21:52 to 13 Oct 04:20
5. How it works nowphone to router, hop by hop
6. Equipment, numbers, and where it endedas of 13 Oct

The iPad was a dead end. iPadOS lets apps view its screen but not control it, and it can't route traffic for other devices.

The first recommendation was a Raspberry Pi Zero 2 W running Tailscale as a "subnet router", about $20 abroad, plus an ESP32 and a relay for reboots. In Egyptian shops the Pi Zero 2 W cost 2,240–3,100 EGP, two to three times its list price, so I dropped it. The Pi Pico 2 W is a microcontroller: no Linux, so no Tailscale.

An old Android phone left at the branch with RustDesk would have worked, except that the branch phone is kept off the Wi-Fi on purpose, so it can't be used for free internet. Parked.

Then small Linux boards. The Orange Pi i96 is abandoned 2017 hardware with 256 MB. The Luckfox Pico RV1103 has 64 MB of RAM and no network socket. The Luckfox Pico Max (RV1106, 256 MB, Ethernet) was possible. The Orange Pi Zero 3 was the best fit, but only on AliExpress. A search of 21 Egyptian electronics shops found no cheap Linux board that fit.

So: an Android TV box with Linux on it. Cheap boxes are small ARM computers, and a community project, ophub's Armbian builds for Amlogic chips, replaces Android with Linux. Locally they cost about 800–1,700 EGP, with Ethernet and no battery. One rule went into the notes: never leave a no-name box on its original Android. Factory-installed malware, such as the BADBOX botnet, has shipped on them.

I had five boxes at home.

This took most of a night. I've kept every attempt, because each one ruled something out.

1. Armbian onto a USB stick. The image was about 875 MB, its checksum matched the publisher's, and it unpacked to 3.7 GB. It went onto an 8 GB Kingston stick with balenaEtcher, after I backed up the stick's own 188 files to an encrypted zip. Windows' own zip tool crashed at 961 MB of 1,004 MB, so I redid it with 7-Zip and verified every file.
2. The reset-button boot. Hold the recovery pin, plug in power. With the stick in the left USB port, the box froze on its logo for as long as I held the pin, then booted Android. The right port was ignored. No Linux.
3. The update menu. The plan was a tiny aml_autoscript.zip that tells the box to look for Linux on SD or USB first. The box's "System update" only worked online. Its custom launcher hid Android's real settings. There was no file manager, no browser and no app store, and a network scan found remote debugging (ADB) closed.
4. The SD card. My only card held 42 3D-print files. I added one 709-byte file, aml_autoscript, next to them and touched nothing else. Two attempts: straight to Android both times.
5. The forum find. On a satellite-receiver forum, owners of this model explained what the pin really does. It puts the Amlogic chip into its factory USB flashing mode, which waits for a computer on a USB-A-to-USB-A cable running Amlogic's USB Burning Tool. It never meant "boot from SD or USB".

Opening the box

Inside: two MIRA DDR3 chips (1 GB), a Samsung 8 GB eMMC, the SW1 recovery button, and three rows of empty solder holes: CN5 by the HDMI port, J2 by Ethernet, CN4 top left. That's the board at the top of this page. One of those rows should be the serial console, a text line straight into the boot software. With it I could stop the boot and type commands by hand, before Android loads.

A NodeMCU as the USB-serial adapter

I had no CP2102 adapter at home (110–150 EGP in shops), so my LoLin NodeMCU board stood in. With its EN pin tied to GND its processor stays off, and only its USB-serial chip talks to the TX and RX pins. The first cable was charge-only and the laptop saw nothing; a data cable worked. Three wires only: GND, TX, RX. Never power.

Finding the console, hole by hole

J2 gave a repeating signal (the front display), a blip at power-on, then silence. CN5 was silent. I found ground with a multimeter, and then CN4 gave 159 KB of clean boot log, confirming the S905D.

Trying to get in

I flooded it with every known interrupt key:

• Space
• Ctrl+C
• Enter
• Esc
• q
• tpl
• stop
• BREAK
• All ignored

Along the way: the receive wire not connected, TX and RX looped, wires swapped, no shared ground. A loopback test proved the adapter could send. Resistance readings proved the wiring right: hole 1 at 37.7 kΩ, holes 2 and 3 at about 7 MΩ and falling. The box simply ignores input. The manufacturer switched console input off.

The timed test that settled it

I recorded the console with timestamps while holding the pin for about 30 seconds. Nothing at all came out until I let go, then the very first boot stage. So the pin acts inside the chip before any software runs, and no file on any card or stick could ever have worked.

The last route

Last serial checks: hole 1 a steady 3.35 V (power), hole 2 at 3.3 V (the ignored input), and nothing on hole 2 when I listened to it. Serial was exhausted.

That left the chip's USB flashing mode. The plan was a home-made USB-A-to-A cable with only the data wires, the red 5 V wire left unconnected so the box's own 12 V supply and the laptop never meet, and Amlogic's USB Burning Tool (v2.2.0, from Khadas' official server). The box never showed up on the laptop in USB burn mode. That ended the TV box.

"a DreamMax TV box could not be put into Linux (locked launcher, input-less serial console, USB burn mode never enumerated)"

1. 15:00

   An old Xiaomi phone turned up, locked to a previous owner's Mi account. No.

2. 15:31

   An old iPhone or iPad can't route. An old laptop would work but is bulky. Then the ESP32-as-tunnel idea came back, my own first instinct: the board dials out to a small server and forwards the router's page.

3. 15:35

   "Let's go with the ESP route." The hub went on my idle Oracle Cloud "Always Free" micro VM, with 1 GB of RAM and ⅛ of a CPU. WireGuard hub up, ESP32 firmware written (58% of flash, 14% of RAM), the phone's profile as a QR code.

4. 16:01

   The flashing tool refused. The "LoLin" board was an ESP8266. It was the board that had been the serial adapter the night before, the same model that runs my NetWatch monitor (a different unit). LoLin sells both chips, and the chip check caught the mix-up before anything was written. WireGuard-ESP32 needs a real ESP32.

WireGuard is blocked

1. 16:05

   The phone sent a handshake every 5 seconds. None reached the hub, while plain test packets from the laptop did. So I tested it: a packet shaped like a WireGuard handshake against a random one, on three UDP ports.

The port made no difference. The filter looks at what's inside the packet.

1. 16:24

   Two ways round it: AmneziaWG, which is WireGuard in disguise, or an HTTPS tunnel through Cloudflare. I said "test A". AmneziaWG was built from the official source on the VM, listening on UDP 443.

2. 16:33

   It gets through. The phone connected over AmneziaWG on home Wi-Fi, laptop to phone through the hub at about 230 ms, then on 4G as well.

3. 16:36

   Patching the ESP32's WireGuard library to speak AmneziaWG.

The NodeMCU detour: IP inside MQTT

1. 16:42

   Could a board I already had do it? The NodeMCU and a Sonoff Mini (an ESP8285, and fried) can't run WireGuard: about 40 KB of free RAM, against the ESP32's ~280 KB. An ESP32-based Sonoff MINIR4 (16 A relay) went on the list for later. Three ESP8266 routes: ESP_WIREGUARD cuts responses at about 1,300 bytes, so skip. A PPTP proxy is insecure, and GRE rarely survives 4G NAT, so no. MQTT_VPN carries IP packets inside MQTT messages. I built that one.

2. 16:45

   The NodeMCU and MQTT version works, slowly. A Mosquitto broker on the hub (TCP 8883) and a small Python bridge. It got through the filtering, on 4G too, at 137 ms ping, and the router page loads. But at about 4.7 KB/s: the library sends one MQTT message at a time, and doubling the clock speed changed nothing. A first page took 30–60 seconds, and it hung when a browser made several requests at once, with about 25 KB of RAM free. Done by 17:29.

1. 17:21

   An old ISP router, a Huawei HG630 V2, as the box? OpenWrt doesn't support it. No.

2. 17:32

   ESP32 boards from a local shop: 250–280 EGP. Amazon wanted 375.

3. 20:58

   "ESP32 on the way." I ported the day's proxy fixes to the ESP32 firmware while waiting.

The second board on the bench

1. 21:28

   ESP32 plugged in: an ESP32-D0WD-V3, dual-core, 240 MHz. The patched AmneziaWG handshake worked first time.

Two fixes, then the router

1. 21:45

   Working, after two fixes. Full-size 1,420-byte packets were silently dropped somewhere on the path, so the tunnel MTU is now 1,280 on the ESP32 and 1,380 on the hub, which had inherited a 9,000-byte MTU from the cloud network. And closing a connection before reading the request made the network stack throw the reply away.

1. 21:51

   Logged in to the router from a phone on 4G, through the hub and the ESP32. The first page was slow, 30–40 seconds for 17 files, but complete. I removed the NodeMCU's server side.

It worked on my desk at 21:51. The rest of the night went on making it safe to leave in a branch, where a mistake I make remotely has to undo itself, or it means a trip with a USB cable.

Getting past the router's allowlist

The routers only admit devices they know. The unit tries its own MAC first. If the router refuses it, it retries with a fallback MAC that's already allowed, gets in, and then its own MAC is added to the allowlist remotely. After a remote restart, the tunnel is back in 4 seconds.

A settings panel that can't slow forwarding

Forwarding runs alone on one core; the panel is a low-priority task on the other. It holds up to 5 Wi-Fi networks in priority order, each with a router address ("auto" means the network's gateway, or a fixed IP) and an optional fallback MAC. The status page shows RAM (~210 KB free, lowest 190 KB), firmware (~0.9 MB of a 1.25 MB slot), uptime, restart reason, Wi-Fi signal, tunnel state and traffic. An event log survives restarts.

Every change is on trial

A settings change is kept only if the tunnel comes back. I tested it with a deliberately broken Wi-Fi setting: the unit restored the old settings by itself in about 35 seconds.

Firmware updates go through the tunnel, with rollback. There are two firmware slots, and a new build stays only after its tunnel has been up for 30 seconds. An upload takes about 2.5 minutes from my home line and 19 seconds from the hub.

The router refused its own files

The TP-Link answers 403 to its own CSS and JavaScript unless the Referer or Origin header names its own address. So the proxy rewrites Host, Referer and Origin.

The front door, three tries in one evening

1. A phone VPN app. AmneziaWG on the phone, to reach the tunnel addresses. It worked, but it was one more app to switch on.
2. A public HTTPS address, at 22:42: a free dynamic-DNS name, a Let's Encrypt certificate, a password and fail2ban. Faster, 17 files in 25 seconds against about 40, because the hub keeps its connection to the unit open. But it was a public login page with a password as the only lock, and fail2ban locked out my own IP while I tested it. Removed.
3. A Cloudflare Tunnel with Google sign-in. The tunnel dials out, so the VM opens no web ports. Cloudflare Access came first, and it asked for a Cloudflare login where I wanted Google. So I moved to the sign-in service I already run for my own site, and one login covers all my private pages. Only my account gets through, at about 30–100 ms per request. A menu page lists each branch with live status and Router, Devices and Settings, plus Sign out.

Clean-up

The VM now runs only the tunnel server, nginx and cloudflared. Only SSH and UDP 443 are open. I deleted eight stale firewall rules, retired the phone's VPN profile and kept the laptop's as the backup way in.

A save that would have stranded a unit

Renaming a Wi-Fi network without retyping its password saved it with no password, because stored passwords were matched by name. At home the unit fell back to another network and I didn't notice. At a branch it would have meant a trip with a USB cable. The panel now blocks that save, and warns before saving a network without a password.

Devices

A Scan sweeps the unit's 254 addresses with ARP in batches of 6, then tries 7 well-known ports on each device: web on 80, 8080, 443 and 8443, video on 554, camera on 8000 and fingerprint reader on 4370. Page titles are kept as hints. At home it took 28 seconds and found 12 devices. Anything with a web page gets an Open button; "Open by address" covers the rest.

That's 15 of the ESP32's 16 connections in use: 6 page connections × 2, 2 waiting, 1 scan. Live video won't work, and camera web pages of 2–5 MB take 1–3 minutes the first time.

Every leg that crosses the internet is encrypted. Both tunnels are dialed out: the hub's to Cloudflare, so the VM opens no web ports, and the ESP32's to the hub.

1. HopWho can read it
2. My phoneA browser, at home or on 4G, opening a private address on my own domain.
   Me.
3. HTTPS
4. The front doorCloudflare Tunnel, with Google sign-in through the sign-in service I run for my own site. Only my account gets through, in about 30–100 ms per request.
   Cloudflare can read router pages: HTTPS ends there. The sign-in server only sees the sign-in cookie.
5. Cloudflare Tunnel, dialed out by the hub
6. The hubAn Oracle Cloud Always Free micro VM: 1 GB of RAM, ⅛ of a CPU, ~230 MB used. It runs cloudflared, nginx and the AmneziaWG server, with only SSH and UDP 443 open.
   Can read router pages.
7. AmneziaWG over UDP 443, dialed out by the ESP32
8. The ESP32, in the branchOn the branch Wi-Fi. It forwards each page and rewrites Host, Referer and Origin so the router accepts them.
   The branch end of the encrypted tunnel.
9. plain HTTP, on the branch's password-protected Wi-Fi
10. The 4G router, and other devicesCameras and a fingerprint reader come next. They haven't been tested on real gear yet.
    Plain HTTP on the branch Wi-Fi, because that's all the router offers.

The ESP32 is the production unit. It's tested at home against a TP-Link router and from 4G, and it's set up from home for its first branch, with that branch's Wi-Fi first and my home Wi-Fi second. Each branch needs one unit, one row on the menu page and about ten minutes of setup.

Cameras and the fingerprint reader haven't been tested on real gear yet. The first real test is at a branch.

Parked on purpose

• Power relayA ~50 EGP relay on the router's 12 V line, to power-cycle a frozen router: a button on the panel, maybe automatic after some minutes offline.
• Tunnel alertA Telegram alert when a unit's tunnel goes quiet for more than 10 minutes. It would run on the hub, so it costs the ESP32 nothing.
• Router APITalking to the router's own API.
• Sonoff MINIR4Possibly a Sonoff MINIR4 per branch: an ESP32 with a relay built in.

The TV box went back to being a TV box.