this post was submitted on 07 Sep 2023
111 points (96.6% liked)

Sysadmin

7603 readers
205 users here now

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world

founded 1 year ago
MODERATORS
 

I'm sending this to my boss to remind him why monitoring disk space is vital.

you are viewing a single comment's thread
view the rest of the comments
[–] ShunkW@lemmy.world 10 points 1 year ago* (last edited 1 year ago) (2 children)

I used to be a sysadmin, now a software developer. At one of my old jobs for a massive corporation, they decided to consolidate several apps' db servers onto one host. We found out about this after it had already happened because they at least properly setup cname records so it was seamless to us. Some data was lost though, but having literally billions of records in our db, we didn't notice until it triggered a scream test for our users. We were also running up against data storage limits

They ended up undoing the change which caused us a data merge nightmare that lasted several full workdays.

[–] emmanuel_car@kbin.social 9 points 1 year ago (1 children)

…it triggered a scream test for our users.

This phrase has brought me much joy.

[–] ShunkW@lemmy.world 7 points 1 year ago (1 children)

It's such an accurate term. I worked in IAM for a while and when no one claimed ownership of an application account, we'd go with a scream test. Lock the account and see who screams at us lol.

We had that some time ago with a service account for a specific system where individual personal accounts weren't (yet) feasible. The credentials were supposed to be treated with confidence and not shared without the admins' approval. Yeah, you can guess how that went.

When the time came to migrate access to the system to a different solution using personal accounts, it was announced that the service account password would be changed and henceforth kept under strict control by the sysadmin, who would remotely enter it where it was needed but never hand it out in clear text. That announcement was sent to all the authorised credential holders with the instruction to pass it on if anyone else had been given access, and repeated shortly before the change.

The change was even delayed for some sensitive reasons, but eventually went through. Naturally, everyone was prepared, had gone through the steps to request the new access and all was well. Nobody called to complain about things breaking, no error tickets were submitted to entirely unrelated units that had to dig around to find out who was actually responsible, and all lived happily ever after. In particular, the writer of this post was blissfully left alone and not involuntarily crowned the main point of contact by any upset users passing their name on to other people the writer had never even seen the name of.

[–] Brkdncr@kbin.social 2 points 1 year ago

The only thing worse than a single database server is servers poorly maintained database servers. The idea was right, but maybe the implementation was wrong.