So last week (on Sunday 1 December at 00:00), our server host canceled its service without warning.
TL;DR: we do not recommend using @Hetzner_Online 's service
Everyone else: a short
Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way.
For us, that meant having our servers disconnected at 00:00 on a Sunday 1st (so likely a scheduled deprecation on their end).
Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.
Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.
When reached, they could not explain the reason for the cancellation:
Them: - We sent you an email.
Us : -We did not receive it, can you please resend?
Them: - We can't
Us: ಠ_ಠ
In the meantime, all servers had been wiped already so no way to retrieve our data.
If you are looking for a bad case of the Mondays, well, that was one.
Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server.
All in all, it still took about 48 hours to get these 8-ish TB back online.
If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra).
Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints.
/END
@kiwix Send them a request for information under #GDPR rules, specifically article 15, forcing them to send you *all* data they have. Hetzner is based in Germany, so they have to comply. Template at: https://www.datarequests.org/blog/sample-letter-gdpr-access-request/
@jwildeboer That's not how this works. @kiwix is registered as a Swiss company and thus is not a data subject in the sense of the GDPR (only natural persons can be data subjects).