• A Moment in BrainTalk History: Flood Protection On!

    by
    David Hosobuchi
    Published on 02-24-2020 01:57 PM
    0 Comments Comments
    As we embark on a fresh install of the forums, as we have done several times in the history of this community, I would like to take some time to remember some of the events that have made this community unique. One such event that sticks out in my mind was the Great Server Flood of 2003. I have managed to dig up the original post made by one of BrainTalk Communities's founders, John Lester, explaining the incident regarding the sudden unannounced downtime that had left so many of us frantic, thinking BrainTalk had disappeared forever.
    Quote Originally Posted by John Lester
    Hi everyone, Well, it was Murphy's Law again. Just as I went out of town for a meeting, a malfunctioning air conditioner decided to turn the computer server room where BrainTalk lives into a swamp. The BrainTalk server went down, and it went down hard. Luckily there was no major hardware damage except for the UPS connected to the BrainTalk server, which went to that UPS pasture in the sky after going for a swim. Here's the timeline as far as I can figure it out:

    1) Server room air conditioner flakes out and starts pumping water into server room. Floor starts getting wet.
    2) UPS connected to BrainTalk Server was sitting on floor.
    3) Fortunately, BrainTalk Server is NOT sitting right on floor...it is a few inches above the wetness.
    4) UPS gets wet, gets toasted, and sends a dying power spike to the BrainTalk server, frying one of the power supplies in the server.
    5) BrainTalk crashes...merrily corrupting data on the way out. Fortunately, the death of the power supply doesn't seem to have caused any hardware problems.
    6) John notices dead server while out of town. Tries to remotely fix it. Notices it is REALLY dead...totally offline. Can't do anything.
    7) John returns after being away 10 days. Water has been cleaned up by then, and air conditioner fixed. Replaces UPS. Switches the BrainTalk Server over to another redundant internal power supply.
    8) John rebuilds corrupted data. Prays to the Patron Saint of Servers and hits the reboot button. Server comes back up.

    Anyway, that's it. I'm really sorry for the downtime. There was nothing I could do remotely, and I don't have anyone here at MGH who could have fixed this problem (it was a bit more complicated than just "reboot the server" kind of stuff.) At least this story has a happy ending. Take care, -John
    This is also an excellent example of how BrainTalk Communities has grown and evolved over the years. In addition to the latest improvements, over the years we have grown to have a larger administrative staff, that are now able to work remotely. We now use a hosted Virtual Private Server (VPS), where the VPS servers do daily backups to "offline" servers in case the online servers have problems. If there's no quick fix, the offline server is brought online while they work on the broken one. This now gives us our flood protection, and instant redundancy from any other crisis that may befall BrainTalk Communities.

    More BrainTalk History to come....
    This article was originally published in forum thread: A Moment in BrainTalk History: Flood Protection On! started by David Hosobuchi View original post


    BTC Inc's Disclaimer and Privacy Policy

    The material on this site is for information & support purposes only, and is not a substitute for medical advice provided by a licensed health care provider. Always consult your doctor before trying anything that you find online.