Conversation
Lux
m
Edited 6 months ago
it's been a bit over a week since the coral castle data loss disaster, so imma be giving some recommendations on how to not fuck up like i did to other server admins.
1. if you're upgrading your server, check the hardware you're using. this was caused by a single bad ram stick that i didn't test after install.
2. allow postgres to safely crash if it detects corruption. openrc automatically restarted the service, causing a loop where postgres would reboot and keep writing corrupted pages to storage
3. consider pg_basebackup and wal archiving. back up the whole cluster in longer intervals (weekly, monthly), and then wal can take care of the smaller changes over shorter periods of time (hourly)
4. check whether your backups work correctly. two issues arose after checking one of my backups. first, i was backing up the wrong database (this being the coral castle neo database), and second, the backups were not encoded correctly (using ASCII instead of UTF-8)
that's all everyone. keep your servers safe.
1. if you're upgrading your server, check the hardware you're using. this was caused by a single bad ram stick that i didn't test after install.
2. allow postgres to safely crash if it detects corruption. openrc automatically restarted the service, causing a loop where postgres would reboot and keep writing corrupted pages to storage
3. consider pg_basebackup and wal archiving. back up the whole cluster in longer intervals (weekly, monthly), and then wal can take care of the smaller changes over shorter periods of time (hourly)
4. check whether your backups work correctly. two issues arose after checking one of my backups. first, i was backing up the wrong database (this being the coral castle neo database), and second, the backups were not encoded correctly (using ASCII instead of UTF-8)
that's all everyone. keep your servers safe.
1
6
7
im also realizing returning to this checkpoint might've not been one of the best choices, and has even produced some negative effects on some people. therefore, i'd like to ask the community again: do we reset, or stay with the current state of things? the instance would still stay in the same domain, and might federate better, since some follows are not accounted for in this previous snapshots. if so, i'll give a time period for users to export and archive their data.
0%
reset
0%
stay
1
1
0
About Coral Castle
coral castle
welcome to coral castle, a chill community sailing on the fediverse. we strive to be a safe, left-leaning, queer-friendly, communitarian space for all kinds of folk. before joining, make sure to read our community guidelines.
community guidelines
- you must be over 14 years old to join. coral castle's servers are hosted in chile, and under article 19,628 of the laws of chile, we are unauthorized to collect data from people under 14.
- do not spam or raid. the fediverse is not your space to intoxicate.
- nsfw/nsfl content must be marked as sensitive. coral castle is built to be used as a casual instance, and as such should be accessible anywhere.
- hate speech, paraphilia, and harassment will get you banned. having basic human decency is key anywhere. whether it is in real life or on the internet, don't be a jerk.
- keep things legal. this includes pirated content, since we want to keep a good relationship across fediverse servers. posting CSAM is strictly forbidden and will result in legal action.
moderation policies
the sanctions for local users will be relative to the gravity of their infringements, varying from simply deleting the offending material, to warning the user through a three-strike system, to immediately banning the offending user.
if you're an instance administrator and you think your instance was sanctioned by mistake, feel free to appeal by emailing me at lux@nixgoat.me
special thanks
to our donators, for helping keep this instance up:
also, to the following organizations: