• NutWrench@lemmy.world
    link
    fedilink
    English
    arrow-up
    21
    arrow-down
    10
    ·
    3 months ago

    In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.

    And then, you boot their servers from a Linux Live USB, run TimeShift to restore the last system snapshot, refuse the latest patch from Cloudstrike and they all lived happily ever after.

      • friend_of_satan@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        3 months ago

        Anybody who doesn’t already have ipmi serial console access set up needs to put that on their list of acceptance criteria for remediation of this incident.

    • kurap1ka@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      3 months ago

      And on Windows you booted in safe mode and removed one file. What’s the point of your post?

    • friend_of_satan@lemmy.world
      link
      fedilink
      English
      arrow-up
      2
      ·
      3 months ago

      boot their servers from a Linux live usb

      If I ran a computer lab that wasn’t already net booted, I’d use this as the motivating factor to put that in place. Net booting to a repair image, or just reinstalling the whole OS either from scratch or a known good disk image, is where anybody who manages a fleet of computers should be.

      There was a point in time where I had a pxe boot server vm set up on my laptop that I used to reload servers in our little row of racks at 365 main, because it let me quickly swap out the boot iso, and was faster than usb sticks were at the time.