As the world recovers from the largest IT outage in history, it shows the danger of one point of failure in IT infrastructure

A global IT failure wreaked havoc on Friday, grounding flights and disrupting everything from hospitals to government agencies. Over all the chaos hung a question: how did a flawed update to Microsoft Windows software bring large swaths of society to a screeching halt?

The problem originated with an Austin, Texas-based cybersecurity firm called CrowdStrike, relied upon by most of the global technology industry, including Microsoft, for its Falcon program, which blocks the execution of malware and cyber-attacks. Falcon protects devices by securing access to a wide range of internal systems and automatically updating its defenses – a level of integration that means if Falcon falters, the computer is close behind. After CrowdStrike updated Falcon on Thursday night, Microsoft systems and Windows PCs were hit with a “blue screen of death” and rendered unusable as they were trapped in a recovery boot loop.

Microsoft is a juggernaut with significant market power, dominating cloud-computing infrastructure across Europe and the United States. So it wasn’t just computers that were affected, but servers and a host of other systems as well. Overwhelming requests from users, devices, services and businesses ushered in a cascading series of failures with Microsoft products – namely Azure Cloud and Microsoft 365. Failures plaguing Azure led to additional but separate disruptions with 365 services. A giant clusterfuck ensued.

  • Ooops@feddit.org
    link
    fedilink
    arrow-up
    109
    arrow-down
    6
    ·
    3 months ago

    No… the Crowdstrike debacle primarily shows the dangers of today’s corporate culture in software development.

    Ship as fast as possible, fix issues later if necessary…

    • dugmeup@lemmy.world
      link
      fedilink
      arrow-up
      59
      arrow-down
      1
      ·
      3 months ago

      Yup. Push to prod!

      This is the Boeing debacle in software land. Kill the engineering and pay the executives. QA? Testing? Strict standards? People? Naaah, more conferences! More logos on F1 cars!

      • mynameisigglepiggle@lemmy.world
        link
        fedilink
        arrow-up
        1
        ·
        3 months ago

        I totally agree. But without knowing a bit more about the specifics, I can’t help but think that just maybe… The updating mechanism could have perhaps just rolled back an update if it caused a bsod?

        Seems like that infrastructure is really the biggest oversight and people would have been none the wiser.

        Also surprised just how many things are running windows. I thought for sure the self checkout registers would have been some embedded Linux system.

    • Rentlar@lemmy.ca
      link
      fedilink
      arrow-up
      27
      arrow-down
      4
      ·
      edit-2
      3 months ago

      I disagree. You are correct that the cause of the fuck up is because of bad development practices. However, if every firm is being reckless with development, but only one out of a myriad of competing firms fucks up because of it, maybe you’d take one airline or hospital network offline or something like that.

      It’s only because of consolidation and market monopolization of the sector, that an outage at such a global scale was even possible to begin with.

    • Wooki@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      3 months ago

      You’re partly right as is the article.

      Centralization is dangerous for security, innovation and cost (monopolies, duopolies).

  • yggdar@lemmy.world
    link
    fedilink
    arrow-up
    26
    arrow-down
    2
    ·
    edit-2
    3 months ago

    Am I missing something? I thought the outage was caused by CrowdStrike and had nothing to do with Microsoft or Windows?

    • pycorax@lemmy.world
      link
      fedilink
      arrow-up
      12
      ·
      3 months ago

      The article actually talks about Azure which was using CrowdStrike internally so their point is valid but the headline is absolutely wrong. Azure is nowhere near a monopoly and it ends up implying that Windows, now Azure was the issue they’re describing.

    • Blaster M@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      arrow-down
      1
      ·
      3 months ago

      This is the typical Guardian sensationalism. Gotta make it look like it was Microsoft’s fault, although this one is square on CrowdStrike’s head. Imagine if a security update for a remote administration tool caused an on-boot kernel panic on every linux server in the world…

    • hangonasecond@lemmy.world
      link
      fedilink
      arrow-up
      5
      ·
      3 months ago

      Microsoft’s use of CrowdStrike meant that a significant number of their cloud and SaaS offerings also failed, impacting users who likely didn’t know what CrowdStrike was.

    • EtherWhack@lemmy.world
      link
      fedilink
      arrow-up
      2
      arrow-down
      4
      ·
      3 months ago

      Only systems running CloudStrike were affected, but all systems were Windows-based as that is the only OS it works with.

      I think it’s more touching on the vulnerability of infrastructure if a larger portion is run by only one OS. Something a lot of usb here may realize, but the general public has never really understood it. Where a scenario like this or similar can can cause a wide-spread blackout, all from a single bug; be it from popular software, or the OS itself.

      • ImADifferentBird@lemmy.blahaj.zone
        link
        fedilink
        English
        arrow-up
        8
        ·
        edit-2
        3 months ago

        That’s not correct. Crowdstrike does also work with Mac and Linux, but this particular incident only impacted the Windows sensor.

        They actually had a similar issue with the Linux sensor a couple of months ago, which… doesn’t speak well of their update process.

    • TrickDacy@lemmy.world
      link
      fedilink
      arrow-up
      1
      arrow-down
      3
      ·
      3 months ago

      This is the extremely important akshually line anyway. Let’s all pretend that every OS is just as shitty because it lets us correct others on the Internet constantly

  • catloaf@lemm.ee
    link
    fedilink
    English
    arrow-up
    13
    arrow-down
    1
    ·
    3 months ago

    What monopolization? Crowdstrike has plenty of competitors.

    • Flying Squid@lemmy.world
      link
      fedilink
      arrow-up
      17
      arrow-down
      5
      ·
      3 months ago

      So does Google. And yet they’re still being taken to court for monopolistic practices by the U.S. Department of Justice.

      Monopoly doesn’t mean literally no competitors in the real world. It means no competitors worth noting because everyone has been corralled into using a single company.

      • sandalbucket@lemmy.world
        link
        fedilink
        arrow-up
        9
        arrow-down
        1
        ·
        3 months ago

        Crowdstrike is big, but not that big.

        About half of my clients use them; and of those, about a third are halfway through ripping them out in favor of MS defender.

        (MS is definitely “that big”)

        • Flying Squid@lemmy.world
          link
          fedilink
          arrow-up
          10
          arrow-down
          2
          ·
          3 months ago

          That dozen don’t seem to add up to much together considering the massive global nature of this outage.

          • catloaf@lemm.ee
            link
            fedilink
            English
            arrow-up
            7
            arrow-down
            3
            ·
            3 months ago

            Because “we didn’t crash” doesn’t make the news. My company wasn’t affected, so nobody cared about us.

            • Flying Squid@lemmy.world
              link
              fedilink
              arrow-up
              9
              arrow-down
              7
              ·
              3 months ago

              That’s great that your company wasn’t affected.

              I hope no one was trying to fly a plane to get to your company yesterday.

  • Marduk73@sh.itjust.works
    link
    fedilink
    arrow-up
    4
    ·
    3 months ago

    The title reminds me of the microsoft monopoly issue ages ago. They had to bail ou Apple so ms could have competition.