r/sysadmin • u/general-noob • 1h ago
General Discussion PHP2Shell and your security team
So, how’s everyone’s security department handing this so far? Mine has been completely worthless like always. Give me some good stories to cheer me up.
•
u/macemillianwinduarte Linux Admin 1h ago
Not a peep. I had to communicate, plan, and execute the change myself
•
u/emakinsreign 1h ago
Just recently started at a new company, came in today to find out our Security team worked through the night and everything was already fixed, and our vendors were already working on some of our hosted items.
I felt like I must have gone to heaven.
•
•
u/Firefox005 39m ago
How is that possible? The update wasn't released until 14:00 UTC this morning.
•
•
u/ponto-au 1h ago edited 1h ago
Spoke with them yesterday mentioning the cve, patch was releasing like 3am here.
Come in at 8 and there was a Major incident called by our MSP for some of our public facing websites since they didn't forward it to anyone lmao
•
u/danfirst 1h ago
Former sysadmin, but I've been in security a long time now. We broke the php news today in a large company then had to tell the server teams which systems needed were affected. I don't know why it feels like at least weekly here everyone talks about how security is useless. Lots of us came from the same trenches as you all did.
•
•
u/general-noob 55m ago
You sound awesome and we aren’t really talking about you. The key is, you actually did one simple thing , you spearheaded the conversation like you should be. Most security departments don’t do much and we all know it, the comments kind of back me on that, right?
•
u/danfirst 53m ago
Yeah I'm not offended, just surprised. I feel like at most of the companies that I've worked at all, the more senior people in security have systems and network engineering backgrounds, it makes a huge difference.
•
•
•
u/agarwaen117 1h ago
Our security team had all of our internal PHP deployments patched today by the end of the day. I did great on finding and patching all three of them.
•
u/House_Indoril426 1h ago
We don't have any public facing PHP in my org that isn't behind Cloudflare.
So darn near business as usual. A handful of internal apps got updated already. None serving production workloads.
•
u/anxiousinfotech 1h ago
Cloudflare/WAF isn't going to protect you unless you can completely block all multipart requests. This would break normal functionality on many sites, especially WordPress. There's no other mitigation besides patching. Exploit requests look just like other normal multipart requests to a WAF.
•
u/Thrwingawaymylife945 1h ago
I'm on WordPress, and I can't patch my websites.
It's outside of my control. My web host hasn't released the patch to my environment.
Oh well.
•
u/anxiousinfotech 1h ago
We have a couple WordPress sites still with external vendors. I was honestly shocked that they updated before we even reached out to confirm everything was all set.
I'm even more shocked that the Azure App Services we have are already on 8.4.27 when set to the latest release setting. Of course they run on 8.4.16, unpatched since JANUARY, if you set it to standard release...which support claims is still the active release for that channel, with no timeline for newer builds hitting standard release...
•
•
u/meowMEOWsnacc 18m ago
I work for a major CDN company with our WAF product. Guess how many customers expect a WAF virtual patch to (permanently) fix this issue for them because the customers don’t know their own tech stack?
Answer: almost all of our customers
•
•
u/d00ber Sr Systems Engineer 52m ago
My security team: " Well, do you really need us to tell you to patch servers when they need patching? Just follow the policy ".
So, now we're working with developers and vendors to figure out which systems are effected.
It really feels like we pay our security team to have written a policy one time, and to notify us of notifications we already get from the EDR and SOC systems that the Infra IT team setup.
•
u/RagingITguy 23m ago
Ehhh i have infrastructure do the patching too. I don't generally do it. I also don't just throw alerts and say deal with it either. I work with them but they talk to vendor and assess that part and patch.
I just don't have the time to do it. They have a much bigger team.
•
u/meowMEOWsnacc 17m ago
A lot of companies don’t allow their security teams to manage patching of vulnerabilities
•
u/RagingITguy 25m ago
I'm a cyber team of one. I also handle network. Everyone just hates me generally because I'm the roadblock to their work apparently. No Sally I can't open ports 1 through 65535 because the vendor said so. Yes we need to put on our EDR if we can.
I'll come through this battered, unappreciated and 'completely useless'. Why should I know how your Web front end works. That's your job!
Sometimes I wonder if I'm in IT because I just hate myself. No money, no support, all the blame. Acting like I'm the one that caused a world wide cyber crisis. Man some of the users really fucking suck.
•
•
u/cspotme2 1h ago
What are you expecting to do?
With that said, one part of my security group has been completely useless with notification about this ... Heck they don't even know what php does or bother to ask ai.
•
u/general-noob 1h ago
Make sure people know about it, give recommendations, help admins come up with a plan, work on mitigation, draft emails to everyone, and submit tickets for us to take action…. I got one snarky comment in a chat with 20 other people helping that didn’t have a job in “security”. We took care of it from start to end without a single action for our security people
•
•
u/choss-board 52m ago
Worthless. I honestly don’t know what they bring to the table besides heckling us over false positives they don’t understand and couldn’t explain from a system they overpaid for.
•
u/slinkytoad69 1h ago
We had our app patched as soon as the build completed for the version of PHP we are using. Our images built, and then bitbucket crashed.
•
u/lynsix Security Admin (Infrastructure) 1h ago
We’ve communicated internally. Our problem is we don’t do vulnerability management or anything for clients and don’t have a clean way to discover all the PHP installs cleanly.
However generally anytime we get notice of a big vulnerability we first check if it’s something we or clients use, then get an understanding of risks, attack vector requirements. Send comms and try to figure out what’s impacted and give relevant teams information on how to patch or harden the vulnerability.
Then if necessary or requested we’ll start looking for IOC’s. Easier when there’s vendor or advisory provided documents on what to look for.
•
u/NuAngelDOTnet Jack of All Trades 1h ago
Security department? Bro, it's just me.
4 hours of trying to patch one system with some random 3rd party webserver flushed, rolled back a snapshot, then did it all again in a out 2 hours. Then luckily I had a dev available to bug hunt all of the edge cases affected due to a php version upgrade (website running in this platform is literally over 10 years old and insanely complex... Lots of depricsted features any time there's an update).
Actual regular webserver? Updated and done in like 90 seconds.
•
u/saintjonah Jack of All Trades 1h ago
All our php runs on a web server managed internally. We got that patched. Is there something else we should be doing?
•
u/general-noob 1h ago
Not everyone has the patch available to them yet <looks at Red Hat>
•
•
u/TheBetterCervanthes 53m ago
Yeah absolute whack. I settled to build from source for now and enroll back into system packages next week or so. Waited ages for remil epel tho :c
•
u/cantstandmyownfeed 56m ago
Read the story. Haven't touched PHP in 10+ years. Visions of janky LAMP stacks flashed before my eyes.
Then retold the same explanation to 5 different people within an hour and then had to prove we didn't use it.
•
u/BenShumard 55m ago
Most places don't have a dedicated security team.
•
•
•
u/TravisVZ Director of Information Security 47m ago
I did a spectacular job of managing our response and working with our sys admins to identify the affected systems and get them updated. Still have a couple of stragglers because the servers themselves are too old, but we have a plan in place and in the meantime have mitigated the issue by locking down who can get through the firewall in the first place.
Our sys admins were spectacular too, despite the fact that several of them are fairly inexperienced - and almost younger than some of the servers they were patching today! They stepped up and did what they needed to, including asking for help where needed (a vanishingly rare skill it seems...)
•
u/lylesback2 39m ago
I read up on CVEs all the time. I typically update our server software monthly, or sooner if a high CVE drops.
I updated all servers this evening
•
•
u/blue_canyon21 18m ago
My security dept has actually been pretty awesome with it. About 10 minutes after I got the notifications about the CVEs, I got an email from our security head telling me that all of the VMs that the inventory software showed any version of PHP installed on have been shut down and NICs disabled.
It took me about 20 minutes to update 3 PHP instances on my home lab tonight, so I expect I'll be busy for about 4 hours in the morning. After that, it's up to the devs to adjust their code for compatibility if needed.
•
u/meowMEOWsnacc 13m ago
I work in security for a WAF/DDoS vendor and watching the industry respond to PHP2Shell has been a fucking religious experience.
Nobody knows what version of PHP they’re running. Nobody knows which applications are internet-facing. Half the security teams think a WAF signature is a patch. The other half are demanding we enable emergency blocking rules globally because apparently false positives are a problem for Tomorrow’s Security Team™. Someone is inevitably going to ask why we didn’t block the CVE before it was disclosed, despite having zero knowledge of the exploit primitives, and then open a SEV1 because their own application returned a 403.
Anyway, please provide a request ID so I can determine whether the WAF blocked your traffic or your origin is simply expressing its constitutional right to die.
•
u/MrNegativ1ty 12m ago
I have the fix ready.
I'm not allowed to deploy it because "you could break production".
Whatever happens, happens.
•
u/operationsitedown 1h ago
Don't even know there is an issue. Yet somehow security gets paid more than sysadmins do.
•
u/davy_crockett_slayer 1h ago
We mitigated it a day or two ago. We do vulnerability scanning using Qualys, and then fixed the issue when we got an alert.
•
•
u/MisterIT IT Director 1h ago
Patches didn’t drop until today I thought?
•
•
u/thatvhstapeguy Security 1h ago
I had to rely on Qualys showing the container layers rather than QIDs for this oddly enough
•
u/Thecp015 Jack of All Trades 1h ago
I am my security department. Not sure if I should praise myself or talk shit on myself.
Probably both.