TL;DR Version: On the weekend, Tidy HQ saw an outage affecting custom domains related to overzealous blocklisting of IP addresses accessing the site.

What Happened

Late on the evening of August 6th, 2022, we were reviewing recent scanning activity on the TidyHQ site from tools run by third parties (a common experience for any public-facing website - people run automated security scanning tools against sites all the time, hoping to find a vulnerability) and noticed an increase in activity over the last 24 hours.

​Typically, we block the source of the attacks (however harmless, it’s best to stop them in their tracks early on) using the security functionality built into our application which is designed to disallow access to a set of source IP addresses.

Inadvertently, due to an issue in the way the scans were being logged, the IP address associated with the scans in the log files wasn’t that of the attacker’s source machines, but rather the address we use to implement custom domains - blocking access for custom domains for a period until late n the afternoon on Sunday the 7th of August when the Tidy tech team was alerted to the issue thanks to several customers.

​

What we’re doing to change this​

1. Improved Monitoring of Custom Domains

​We have extensive availability monitoring and alerting in place for the TidyHQ systems.

Unfortunately, due to an oversight in the system, this didn’t extend to monitoring our custom domains configuration - meaning we weren’t automatically notified when access stopped working, and had to rely on customer notifications.

We’re adding improved monitoring, specifically designed to capture these sorts of issues and across our customer custom domains to ensure they continue to work, or if an issue occurs, we’re proactively notified.

We've already added monitoring to custom domains or this specific issue, but will be revisiting our overall monitoring approach this week.

2. Weekend Support Process Improvements

​Secondly, we’ll be putting in a place for urgent support handling and notifications on

the weekend - Typically the Tidy team sticks to business hours, Australia time - with

some occasional out-of-normal hours support (you may have received a late evening or

early morning response from some of our team members).

​In the case of urgent, access-restricting issues such as the above, we’ll be instituting

a new process to ensure customers can easily raise it with us for triage as needed

outside of these business hours, to ensure the Tidy team can respond in a timely fashion.

​

3. Fixes and improvements to the security monitoring​

Lastly, we’ll be significantly improving the way we handle these situations to not

only remove the change of issues like the above, but to keep our customers even

more secure - You can be confident your data in Tidy is kept safe and secure,

but we always want to make sure we do everything possible.

​

In Closing

​We’d like to offer a sincere apology to all customers affected by this issue - We take great care in trying to ensure that Tidy takes all of our customer’s security very, very seriously.

As security is of utmost importance to us, it takes a priority - but unfortunately in this case, an overzealous setup here cost the availability of custom domains, and due to the weekend, it wasn’t resolved as quickly as we would like.

We endeavour to be transparent, open and always improving - and to push to make sure we offer the best experience possible for our customers.