Troubleshooting Tricks for Orbi Networks

I love a good troubleshooting challenge. But damn, this one had me stumped until I could pull back the covers of my Orbi system to discover the root cause.

Background:

I have a vmWare ESXi hypervisor in my home lab. The ESXi system is trunked up to a Ubiquiti router that is handling inter-vlan routing and has my Orbi base station set as the default gateway. 

Home Network

I spun up a Cisco FTDv NGFW image, set the outside interface to 192.168.2.100 and the default gateway to the 192.168.2.1 address on the Ubiquiti router. Ok, we are all fired up, let’s test basic connectivity.

Ping Fail

Hmmm. I dug into some packet captures and debugs on the Ubiquiti, and as far as I could tell, the packet was being delivered on vlan 1 and being delivered to the Orbi, who should have responded with an echo reply. I could ping the Orbi from my workstation on the 192.168.1.0 subnet, but not from the 192.168.2.0 subnet. Of course, I have a static route in the Orbi to reach the 192.168.2.0 subnet, so there was not a routing issue. AND, I could ping my workstation or any other IP address on the 192.168.1.0 subnet EXCEPT the Orbi. 

Orbi Hidden Debug

I did a little google action and discovered that there are some backdoors in the Orbi for advanced troubleshooting. The trick is to go to: https://192.168.1.1/debug.htm. From this screen you can enable some debugging captures, which I tried first, and then you can download them and view them offline in Wireshark. It’s actually a pretty robust set of data that the debug tool returns. 

Sure enough, I saw my ICMP echo requests reaching the Orbi just fine, but still, no responses from the Orbi.

Wireshark Capture

Ok, so what is the Orbi doing with these packets? Enter the next nice little hidden feature of your Orbi: Console access. By enabling telnet from the debug.htm page, I was able to remote into the Orbi shell, which is just BusyBox embedded Linux.

Busybox

It only took me about 30 seconds to discover the problem.

ifconfig tun0

Directly connected routes have a metric that is always preferred over static routes.

Bingo! Yup, there is a local interface on the Orbi that is already using the 192.168.2.0 subnet. Which means that my ICMP requests were being returned over the tun0 interface and not back to my Ubiquity router, because directly connected routes have a metric that is always preferred over static routes.

Mystery solved.

TLDR; You can’t use the 192.168.2.0/24 subnet

Once I changed my vlan interface to 192.168.20.0, full connectivity was achieved.

Working Ping

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: