Outage report 15/7/25

All the timestamps in this report are in the EEST time zone.

15/07 22:39: Disabling DNSSEC

At around 22:39, we decide to disable DNSSEC due to it causing certain requests to time out, giving the user an "Unhandled error" on the dashboard. With DNSSEC, it took about 13 seconds on average to register / modify domains, which was nearing our execution limit of 15 seconds. We decided to disable DNSSEC to fix this issue. I disabled DNSSEC through Namecheap's dashboard at around 22:41, which was shortly followed by disabling DNSSEC on our dedicated DNS. This is a key part of the issue, as we should have kept DNSSEC on our DNS server to avoid DNSSEC errors while the domain itself was propagating.

15/07 22:43: API requests failing

I was testing out our new website's login system around this time, when I suddenly noticed that requests to beta.frii.site were failing. I try accessing our documentation website for our API (https://beta.frii.site/docs), which returned DNS_PROBE_FINISHED_NXDOMAIN. I check the record on dnschecker.org, which indicated that the domain wasn't accessible from almost anywhere

15/07 22:51: Domains under frii.site no longer work

Around this time, I decided to check other domains, such as www.frii.site, and api.frii.site, which both returned the same DNS_PROBE_FINISHED_NXDOMAIN error.

15/07 22:54: Inspecting our DNS server

I quickly check our DNS server for any obvious errors, but I couldn't find any. I decide to restart PowerDNS, and after that, the server that is running PowerDNS, neither of which fixed the problem.

15/07 22:57: Certain DNS servers start to resolve frii.site

I check dnschecker.org again, now to see that about 8 new DNS servers are now resolving frii.site correctly. I sigh in relief and start asking people on our discord server whether the website works for them. Around 3 people say it works for them, so I assume there's an issue with DNS caching on my end. Around 10 minutes later I check dnschecker.org again, and even more DNS servers are resolving frii.site. I still cannot access the website on my end, so I assume it's a caching issue

15/07 23:10: Uncertainty

I check frii.site using the command dig @1.1.1.1 frii.site A , which returns a NOERROR response, so I assume that it is in fact a caching issue. I ask one of my server members to run the same command, but he gets a SERVFAIL. Now I start wondering whether an older cached version is keeping the site alive. Around 5 minutes later I try digging frii.site again, now on my mobile data, which returns a SERVFAIL. I assume that DNS caching is keeping the site alive, so I start investigating more.

15/07 23:43

After trying to fix the issue for half an hour, I couldn't find anything wrong with the DNS server itself. I check dnschecker.org for the last time, and see that every server that was reporting frii.site as resolving no longer worked. I open up a support chat with Namecheap, who took 25 minutes to respond with "your DNSSEC is already disabled on our end".

15/08 0:03

Now our DNS server isn't responding to queries at all, so I check PowerDNS' log, and see that there's an error in the config. I quickly fix the config, which now gets the server running. The domain still doesn't resolve, most likely due to DNS caching. I decide to wait for DNS servers to propagate, and go to sleep

15/08 9:55: Still not resolving for me

At this point, about 60% of DNS servers on dnschecker.org were showing frii.site as resolving, yet it wasn't working for me. Google's DNS servers (8.8.8.8 and 8.8.4.4) were now resolving the domain.

15/08 11:24: DNS starts resolving

Around this time, I notice that frii.site is finally resolving for me. I check dnschecker.org, and every DNS server also acknowledges that the domain was resolving.

TL; DR

• DNSSEC propagation caused domains to not resolve
• Every domain was affected
• Approx. 60% of users were affected. If you were using Google's DNS (8.8.8.8 or 8.8.4.4), the impact was minimal
• The outage lasted anywhere from a few minutes to 12 hours depending on your DNS server
• If you feel like you were impacted during this outage, you can ask for compensation here