Posts
3307
Following
442
Followers
495
software developer, student

main: @lux@nixgoat.me

openpgp4fpr:580199C19A147B4D7C58B7935A400026ED2F57FD
Edited 11 months ago
it's been a bit over a week since the coral castle data loss disaster, so imma be giving some recommendations on how to not fuck up like i did to other server admins.

1. if you're upgrading your server, check the hardware you're using. this was caused by a single bad ram stick that i didn't test after install.
2. allow postgres to safely crash if it detects corruption. openrc automatically restarted the service, causing a loop where postgres would reboot and keep writing corrupted pages to storage
3. consider pg_basebackup and wal archiving. back up the whole cluster in longer intervals (weekly, monthly), and then wal can take care of the smaller changes over shorter periods of time (hourly)
4. check whether your backups work correctly. two issues arose after checking one of my backups. first, i was backing up the wrong database (this being the coral castle neo database), and second, the backups were not encoded correctly (using ASCII instead of UTF-8)

that's all everyone. keep your servers safe.
1
6
7
srry for the downtime everyone. server ran out of space, so i had to clean some logs to bring it back up. should be good now, and i'll be starting logrotate to make these not accumulate so much
0
0
0
32GB nginx access log
0
0
1
OH MY GOD IRAN'S PRESIDENT PULLED A SEBASTIÁN PIÑERA MOMENT IN THE SAME YEAR LOL
0
0
1
i am so close to getting a bluetooth jammer and killing off my neighbor's music i swear to god i can't take it anymoreee
1
0
3
Edited 11 months ago
this sucks. my sony audio system decided to close the tray just as i was putting a disc and scratched it. i'm so pissed off
2
0
0
what if your wife was a pepsi
0
0
1
repeated
what are you? a twink, or someone with rights?
0
0
0
repeated

hello! void.rehab will be experiencing downtime tomorrow as the unmatched power of the sun annihilates our server rack, sending us back into the stone age. please allow two to three hours for society to rebuild enough that misskey forks become a thing again and void.rehab can come back online

2
4
5
btw if you can't trust coral castle after this backup, i don't mind if you want to leave. this was an absolutely avoidable issue on my part, for not testing hardware after getting it and for not ensuring backups were being executed correctly. sorry, everyone.
2
0
0
repeated

Classic Winamp is going ?

9
22
8
will be renaming the database, give me a sec
0
0
0
ok y'all we're back! i recovered the 7 month old backup. sorry for this everyone. hoping for the best from here on out.
0
0
2
Edited 11 months ago
welp, it's time to work with what we have blobcatpensive
1
1
0
sorry for the huge reset everyone :( fuck kingston
0
0
0
ok. we're back
0
0
0
potaro enemies to lovers
0
1
1
btw i was posing for a photo yesterday noon with porter, but was looking if someone took a higher quality image, sadly not the case but someone took a video, so here it is!
2
0
10
PORTER KNOWS ABOUT THE N900 👁👁
0
0
0
Show older