Cloudflare is currently affected by route leaks preventing users from accessing its services [1]. According to Cloudflare, about 16 Million “Internet Properties” are protected by its services. Recently, Cloudflare also started offering its “1.1.1.1” privacy-focused DNS service, which may also be affected by the routing issue.
According to some reports, Google’s 8.8.8.8 IP address may be affected as well. DNS is in particular vulnerable to man-in-the-middle attacks and they may go unnoticed unless you are using DNSSEC or one of the newer DNS over TLS or HTTPS protocols.
BGP route leaks are nothing new, and they happen much more often than they should. Just recently, traffic from various European mobile internet providers was rerouted through China due to a route leak originating from a Swiss ISP.
What is a “Route Leak”?
Routing on the internet is governed by BGP, the “Border Gateway Protocol.” ISPs sent BGP messages to their neighbors (“peers”), telling them which IP addresses the ISP offers. While there are registries that track which ISP is supposed to own what IP address (“whois”), they are not used in day to day routing. BGP is meant to be fast to allow internet traffic to be re-routing in the case of outages. If your company has two uplinks, one though ISP 1 and another one via ISP 2, and ISP 1 experiences an outage, you could just advertise your IP address space via ISP 2, and “the Internet” will pretty much instantly route your traffic to ISP 2.
Problems arise if someone is advertising IP address space that they do not own. Over recent years, extensions to BGP were implemented to authenticate the messages better, but these extensions have not been widely implemented. ISPs are also supposed to filter incoming messages, which again isn’t always done (or done correctly).
It can also happen that two ISPs advertise the same IP address range. For example, ISP 1 advertises 192.0.2.0/24, and a second ISP advertises 192.0.2.0/29, 192.0.2.2, which is part of both subnets, will be routed to the second ISP since it is the smaller subnet. If someone intentionally advertises a bad route, they often do so by advertising multiple smaller subnets to be preferred over the valid routes.
Usually, a “route leak” is not something where the victim (Cloudflare today) is at fault in any way. Instead, various ISPs around the Internet that either advertises the routes or that propagate it are at fault. Route leaks often happen accidentally due to misconfigured routers and result in a denial of service. But they can also be used to intentionally re-route traffic to launch man in the middle attacks.
What Is Going Wrong With Cloudflare Right now?
BGP lists the autonomous systems (“Networks”) traffic should pass through to reach the target. For one of Cloudflare’s prefixes, 104.20.160.0/20, the list of autonomous systems should look like from a couple of different ISPs:
7474 (Optus AU) -> 4826 (Vocus AU) -> 13335 (Cloudflare)
2914 (Sprint) -> 13335 (Cloudflare)
The second entry is pretty straight forward. Sprint is directly connected to Cloudflare, so no poisoning is happening. The first entry is more vulnerable. Currently, the following route is used:
7474 (Optus AU) 7473 (Singtel) 6461 (Zayo Bandwith) 701 (Verizon) 396531 (Allegheny Technologies Incorporated) 33154 (Kit Carson Electric Corp) 3356 (Level 3) 13335 (Cloudflare)
Here are some other current entries (based on BGPMon.com):
11071 (Infowest) 174 (Cogent) 701 (Verizon) 396531 (Allegheny) 33154 (Kit Carson) 3356 (Level 3) 13335 (Cloudflare)
This list of networks is not only much longer, but it also includes some smaller networks, which is unusual. Based on the list, it isn’t obvious who is at fault. It could be Level 3 (AS 3356) advertising to AS 33154 that it routes Cloudflare’s IP addresses (which again would be odd) and AS 33154 just propagating this information to its peers. But any network in this list could be the origin of the bad messages.
One issue with this particular BGP route is that AS701 (Verizon) has accepted the bad route. AS701 is one of the larger transit networks on the Internet, peering with many other networks that will trust its updates.
What Should You Do?
For the most part, there is not much that you can do about this (unless you are working for one of the affected networks). In the man-in-the-middle case, you would see a wrong TLS certificate. Never accept bad certificates. For the DoS part of the attack, there is not much you can do but wait for affected ISPs to work things out. Maybe try a different ISP if you have the option. If your website is behind Cloudflare’s proxies, then you could temporarily expose them directly but don’t do so without understanding the consequences. You may yourself be more susceptible to attack if you remove your site, in particular, if you count on Cloudflare for web application firewall and anti-denial of service services.
[1] https://www.cloudflarestatus.com
thanks to fellow handler Remco Verhoef for assisting in collecting some of the data used here.
—
Johannes B. Ullrich, Ph.D., Dean of Research, SANS Technology Institute
Twitter|
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.