A false smoke alarm almost drowned Jane Street's only NYC datacenter

(michael.bacarella.com)

44 points | by mbac32768 41 minutes ago

4 comments

  • reticulates 8 minutes ago

    The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.

    “The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”

    https://news.ycombinator.com/item?id=4720388

    https://arstechnica.com/information-technology/2012/11/how-o...

    • hackermailman 23 minutes ago

      In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.

      • wolvoleo 10 minutes ago

        How is it the only datacenter?

        33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street

        • dklsf 9 minutes ago

          This was pre-Hurricane Sandy, so before 2012. I'm guessing data centers have grown since then

          • lelandfe 8 minutes ago

            Work nearby, it looks just as evil in person.

          • rshnotsecure 18 minutes ago

            Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.

            [1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane

            • dklsf 11 minutes ago

              If those go off, they end up destroying hard drives (the spinning kind), due to the sound pressure....

              So fire suppression and data centers don't mix too well

              • mbac32768 13 minutes ago

                Other datacenters we built had them (which scared the shit out of me tbh). I don't remember why this one had to stay on sprinklers.

                • dklsf 10 minutes ago

                  Maybe because it was in a normal office building? The gas extinguishers might be too hazardous there

                • hvb2 13 minutes ago

                  Or CO2 but that's a life or death situation if there's any people still inside

                  • thechao 11 minutes ago

                    Halon is not exactly fun to breathe, either.

                  • gosub100 11 minutes ago

                    Probably not safe to use on a floor that has people on it.

                    • wolvoleo 9 minutes ago

                      We had it at work at a datacenter right beside offices and reception.

                      You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.

                      PS: Halon itself is of course gone but it's other less harmful gases these days.

                      What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).