I Almost Died for a Full Sentry Database
https://it-notes.dragas.net/2024/12/28/i-almost-died-for-a-full-sentry-database/
#Data #Horrorstories #ITNotes #NoteHUB #Recovery #Sentry #Stories
horrorstories
The Day GlusterFS Tried to Kill My Career
https://it-notes.dragas.net/2025/05/21/the_day_glusterfs_tried_to_kill_my_career/
About a year ago, a client I've worked with for over fifteen years informed me that some of their "less critical" servers would be migrated to $CLOUDPROVIDER. According to them, this provider would guarantee an efficient management panel, "more freedom for their devs", and lower costs. This didn't impact me financially but, on an ethical and personal level, I warned him about the potential problems. Yet they decided to move forward, aided by the arrival of $YOUNGDEV who "has worked with it, it's reliable, and everything works fine". Again, I warned them (where are the backups? A disaster recovery plan? etc.) but they insisted: $CLOUDPROVIDER is efficient and gives us everything.
I studied their plan and immediately understood that their "cost-cutting" strategy wouldn't work: I know their workloads, and the plan they chose was insufficient. Needless to say, a few days later they went down and had to make an "emergency" purchase of the next tier up. The cost? Higher than their previous server infrastructure.
I heard nothing more about these workloads for almost a year but my monitoring tools still were marking them down, from time to time. Then, I get a phone call this afternoon. $YOUNGDEV asks me for support. He doesn't explain, but I immediately understand it's one of those workloads. A serious problem, and they don't have a backup of the database. They don't have a test environment to run diagnostics. The DB is very large, and they don't know what to do. My predictions - not even my worst ones - had come true.
I was running between two appointments. I only remarked that this situation could have been avoided and that it's not something I manage or can manage, but I nonetheless suggested we sync up tomorrow morning. I'm not going to get my hands dirty, but still, $YOUNGDEV is in trouble, and I offered to take a look to suggest a strategy. I then asked for the access credentials to $CLOUDPROVIDER, considering that up until a year ago, I managed all of these workloads. He replied that he "doesn't know if he can give them to me" and that he "would have to ask his bosses". I pointed out that if he wants my help, I need something - I don't even know how $CLOUDPROVIDER grants access to data (or if it does) - how can I give him advice?
It's 18:30 and I have received nothing. Tomorrow morning, if the phone rings, I will answer, but at this point, I won't do anything. I prefer, albeit reluctantly, to completely end the relationship with this client.
If this is the price of dignity and respect, I'll gladly pay it.