Alupis
> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.”

This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card.

Look, I think just as poorly about CrowdStrike as anyone else out there... but you can find someone to say anything, especially when they have an axe to grind and a chance at some spotlight. Not to mention this guy was a designer and wouldn't be involved in QC anyway.

> Of the 24 former employees who spoke to Semafor, 10 said they were laid off or fired and 14 said they left on their own. One was at the company as recently as this summer. Three former employees disagreed with the accounts of the others. Joey Victorino, who spent a year at the company before leaving in 2023, said CrowdStrike was “meticulous about everything it was doing.”

So basically we have nothing.

hitekker
I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of our process'.

My guess is that people are identifying with sentence said just before: "Speed [of shipping] is everything." Aka "Move fast and break things."

The culture described by this article must mirror many of our lived experiences. The pure pleasure of shipping code, putting out fires, making an impact (positive or negative)... and then leaving it to the next engineers & managers to sort out, ignoring the mess until it explodes. Even when it does, no one gets blamed for the outage and soon everyone goes back to building features that get them promoted, regardless of quality.

Through that ZIRP light, these process failures must look like a feature, not a bug. The emphasis on "quality" must also look like annoying roadblocks in the way of having fun on the customer's dime.

0xbadcafebee
Critical software infrastructure should be regulated the way critical physical infrastructure is. We don't trust the people who make buildings and bridges to "do the right thing" - we mandate it with regulations and inspections. (When your software not working strands millions of people around the globe, it's critical) And this was just a regular old "accident"; imagine the future, when a war has threat actors trying to knock things out.
addled
Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week.

They recently had a stroke at home just days after spending over a month in the hospital.

Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery to happen.

It makes me wonder what the outcome would have been if they had gotten the surgery done that day, and not spent those extra weeks in the hospital with their condition and stressing about their future?

avree
"“Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.”

Their 'expert' on engineering process is a senior UX designer? Somehow, I doubt they were very close to the kernel patch deployment process.

Cyclone_
Not justifying what they did with qc, but qc is missing from quite a few places in software development that I've been apart of. People might get the impression from the article that every software project is well tested, whereas in my experience most are rushed out.
insane_dreamer
> CrowdStrike disputed much of Semafor’s reporting

I expect some ex-employees to be disgruntled and present things in a way that makes CroudStrike look bad. That happens with every company.

BUT, CrowdStrike has ZERO credibility at this point. I don't believe a word they say.

pclmulqdq
Everything that we know about CrowdStrike stinks of Knight Capital to me. A minor culture problem snowballed into complete dysfunction, eventually resulting in a company-ending bug.
sersi
Crowdstrike was heavily pushed on us at a previous company both for compliance reason by some of our clients (BCG were the ones pushing us to use crowdstrike) and from our liability insurance company.

It was really an uphill battle to convince everyone not to use Crowdstrike. Eventually I managed to but after many meetings where I had to spend a significant amount of time convincing different shareholders. I'm sure a lot of people just fold and go with them.

chaps
Worked on a team that deployed crowdstrike agents to organize and... Yeah. One of the biggest problems we had was that the daemon would log a massive amount of stuff... But had no config for it to stop or reduce it.
MichaelRo
This "security" thing is getting ridiculous. It's become the Gestapo of information technology, they can do anything they want when they want to your computer, cannot resist it and there's absolutely no transparency on what they do to you and why.

I've recently changed jobs and the new employer, a large company, obviously has to have an IT compliance / security update policy because everyone else has it so if they stand out from the crowd and don't do it and somehow get hacked, it's 100x worse than constantly annoying employees and top of the line computers working like a 1970s terminal.

It's rarely that a week passes without the obligatory update + restart. And at least once a month they update THE FUCKING BIOS! What the fuck can be so broken in those laptops that the BIOS is a constant security hazard?! And why would you buy software from someone who week after week after week tell you all you had so far was a hazardous piece of shit that cannot possibly function without constant pampering?

Ahh and of course they botch it. Had to have the OS completely wiped out and reinstalled after the laptop started to behave more and more erratically, 100% caused by faulty updates on top of faulty patches trying to patch the faulty updates. Worked OK for a while afterwards then updates started piling up and so far I only lost use of the web camera (before it was Wifi then display adapter).

There's literally no words how much I hate "the system" and the constant security update take it up the ass we're forced to put up with.

brownllc
Our law firm, Brown, LLC, would like to speak with you to discuss your experience at Crowdstrike. Is there a good time and way to do this? You can call our office at (877) 561-0000 or view our site www.IFightForYourRights.com. Thank you and best of luck.
bb88
Most interesting quote in the article:

    “It was hard to get people to do sufficient testing sometimes,” said Preston
    Sego, who worked at CrowdStrike from 2019 to 2023. His job was to review the
    tests completed by user experience developers that alerted engineers to bugs
    before proposed coding changes were released to customers. Sego said he was 
    fired in February 2023 as an “insider threat” after he criticized the
    company’s return to-work policy on an internal Slack channel.
Okay clearly that company has a culture issue. Imagine criticizing a policy and then getting labeled "insider threat".
panic
Why would it matter? The absolute worst case scenario happened and their stock is still up 50% YoY, beating the S&P 500.
nine_zeros
Typical of tech companies these days. Quality is considered immaterial - or worse - put on low level managers and engineers who don't have the time to clearly examine quality and good roll out practices.

C-Suite and investors don't seem to want to spend on quality. They should just price in that their stock investment could collapse any day.

ricardobayes
I believe one of the biggest bad trends of the software industry as a whole is cutting down on QA/testing effort. A buggy product is almost always an unsuccessful one.
xyst
Switch off CrowdStrike junk. Those companies renewing contracts with them have idiots for leaders.

Many competing platforms that can be a drop in placement for ClownStrike.

Thompson88
Clearly there weren't any code review workflow processes in place, which is astonishing. That's why our primary focus is-transparency, accountability, and system integrity to bring a decentralized, transparent, and reliable platform for journalists, researchers, scientists, and content creators.
hinkley
I have only just begun to consider this question: when does risk taking become thrill seeking?

At some point you go past questions of laziness or discipline and it becomes a neurosis. Like an addiction.

bmitc
Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.
Thompson88
Confirmed: It was a management problem. I noticed in CrowdStrike's post mortem report that they didn't say a thing about management.
ramesh31
If their (or your) shop is anything like mine, its' been a constant whittling of ancillary support roles (SDET, QA, SRE) and a shoving of all of the above into the sole responsibility of devs over the last few years. None of this is surprising at all.
jrm4
Does anyone have a logical reason why this company should not be sued into oblivion?
noisy_boy
Would be interesting to know from their employees if there have been any tangible changes in the blind pursuit of velocity, better QA etc in the aftermath of this fiasco.
Timber-6539
Doesn't matter now. CRWD didn't go to zero. Meaning they get the chance to do this again.
mattfrommars
Side effect of the old adage, "move fast, fail fast"?
paulcole
Well if they say that QA was part of the process then they’ll look like idiots because they sucked at the process.

Don’t find this particularly interesting news.

mrjin
Wasn't that obvious? If any tests were performed at all, how anyone can manage to caused an outage at such scale?
nailer
It’s a UX designer. I don’t particularly like crowdstrike, but this person will know very little about their kernel Drivers.
goralph
What are some alternatives to CrowdStrike?
manvillej
anyone feel like this and Boeing sound remarkably similar?

Its almost like there is a lesson for executives here. hmmmm

tamimio
I think the whole world knew that already.
SlightlyLeftPad
Just another example of technical leadership being completely irresponsible and another example of tech companies prioritizing the wrong things. As a security company, this completely blows their credibility. i’m not convinced they learned anything from this and don’t expect this effect to change anything. This is a culture issue, not a technical one. One RCA isn’t going to change this.

Reliability is a critical facet of security from a business continuity standpoint. Any business still using crowdstrike is out of their mind.

seanw444
And everybody gasped in surprise.
nittanymount
does it have competitors ?
Sarkie
It was shown in the RCA that their QA processes were shit
st3fan
Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text.

Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs.

Only the Mac version does this. There is no way to disable this behaviour or a way to redact things.

Another really odd design decision. They probably have many many thousands of plain text secrets from their customers stored in their SIEM.

bitcharmer
Another company that got MBA-ified
jokoon
We need laws and regulations on software the same way we have for toys, cars, airplanes, boats, buildings.

This silicon valley libertarian non sense needs to stop.

monksy
No shit.
tonetheman
[dead]
known
[dead]