This morning, I took my wife to the hospital for routine blood tests that had been scheduled for some time. Everything was going smoothly: check-in, number, waiting room. Suddenly, everything came to a halt and shut down. I was connected to the hospital’s public Wi-Fi and noticed that my connection also went down.
Having managed a couple of similar facilities, I immediately understood what had happened. I saw the staff panicking and calling the technicians, but they quickly reorganized within 10 minutes. They managed to process everyone who already had a number and then proceeded with the others in the order of their arrival. Despite the ten-minute delay (even though people started complaining right away), they were extremely efficient.
I later confirmed that the entire booking, check-in, and queue system is “in the cloud.” The hospital experienced a connectivity interruption, and all related services stopped. The staff no longer had access to anything, so a technician sent the lists to a manager via another channel, and everything resumed manually.
For years, I’ve insisted that certain things MUST be local. The healthcare facilities I manage have all the necessary systems for the operation of the facility internally, including patient records. External services like websites, emails, etc., are secondary.
Everything essential must always be accessible locally and, in special cases, it should be possible to physically access the servers and connect directly to them, bypassing any network/switch failures.
There has been only one interruption in the past, due to human error. Today, we have redundant servers (not HA on virtualizers, but two machines running the same software with replicated databases - on separate power lines) so such an issue shouldn’t happen anymore.
Not everything can be anticipated, but history is a great teacher. The Internet connection will eventually be interrupted :-)
When it comes to the health and survival of people, there are no compromises.
#IT #Internet #Networking #Outage #Health #HA #Cloud #CloudComputing #OwnYourData