This one is posted under Dog Tails because it all started because of Bones pooing in the house.
Well, we had a minor incident this morning. Bones pooed in the house again, so I got up and picked it up. He has been eating dirt outside so his poo is more like mud balls. We cannot get him to stop eating or pooing mud. Anyhow, Mike had been out of town in Chicago all week and he did a bad thing before he left. He picked up Bones’ poo and put it in the toilet and then did not flush it. I have fussed at Mike many times for this and I flush the toilet. Well, this time he did it in the bathroom in the back of the house off the master bedroom on the first floor and then went on travel. I never go into that bathroom so the mudball sat in the toilet for a week. When I cleaned up the poo, I tossed it in the toilet and flushed. Guess what…it overflowed. I got all of the towels and sopped it up and fussed at Mike the entire time. He finally got out of bed and came over to look. I told him that he needed to mop the floor with disinfectant and then I went downstairs to check to see if any water had gotten to the basement. The bathroom that this happened in is directly over my 2 server racks. I didn’t see any water so I went back upstairs, got a mop and the Clorox and cleaned the bathroom after using the plunger to push the mudball down the toilet. I had an executive meeting at 10am so I had to leave for work.
Well, I was almost at work when the phone rang. It was Mike and he said that he had taken a shower and when he got out he heard beeping from the basement. He went downstairs and half of the rack was dark and the UPSs were beeping. Water was pouring down on the main server rack. Why…because Mike had complained that the servers were to loud and so I put up insulation and sound barrier on the ceiling and walls and the seam of the soundproofing was right over the rack. So, all of the water channeled down to the soundproofing and then poured right into my UPS, down into the power bar of the rack, and settled on the main tape backup robot. I told Mike to sop it up and pull the big plugs that power the racks in 2 phases. He wanted to talk about trying to get the equipment back up and I told him to just unplug everything because I didn’t want to risk a fire while no-one was home. I was suppose to do server maintenance at 6pm so I knew I would be home late and I just said, please unplug it and let it sit.
I left work at about 4pm and got home at about 4:30. The plan was to do my server work remotely so I had set the servers up to let me in from home. So, I locked the dogs outside and went to the basement. Mike had done a good job of cleaning up the water and had even put spacers between two of the servers that are stacked without rails. He had a fan on the rack blowing air all day and most everything was dry. I checked the power bar in the bask and reset the breakers and flipped them on. I then checked out the UPSs and all 4 seemed dry. I powered each one in sequence so that I could smell any smoke or hear any sounds. The top one took the brunt of the water and fortunately had dried completely. They run pretty hot so I figure that without the power to them and their breakers thrown, the heat steamed off the water pretty quickly. I then powered up both phases of the rack, and started powering on one unit at a time. First the banks of disk drives, then the bottom tape drive that took all of the water, and then the main server. Everything came on. I powered on the mail server, and then the WEB server. The only casualty in the entire thing was a standalone DLT tape drive for the WEB server. I had a spare so I put it on the system and everything is back in business.