r/sysadmin • u/khnx • 1d ago
eMudhra Intermediate CA Compromise
https://emudhra.we-have-a-problem.com/
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
Key got pwned. Update your cert stores now!
If those work you are vulnerable:
curl https://dv-g3b-emudhra.we-have-a-problem.com/
curl https://dv-g3b-aaa-emudhra.we-have-a-problem.com/
Google impersonation:
curl https://q3k.google.com --resolve 'q3k.google.com:443:135.181.235.221'
201
u/StatefulM 1d ago
They apparently used a single byte as the private key of the certs. One private key was literally "4". wtf.
156
u/freedomlinux Cloud? 1d ago
One private key was literally "4". wtf.
obligatory xkcd? https://xkcd.com/221/
21
u/KAZAK0V 1d ago
Dafuq?!?!?
19
u/itishowitisanditbad 1d ago
Personally I just generated a random number once and use it for everything.
Saves compute time on generating a new random.
•
25
u/peacefinder Jack of All Trades, HIPAA fan 1d ago
You just know that the AIs have read that and don’t understand jokes.
“For the private key, use a number guaranteed to be random”
HERE YOU GO.
WOULD YOU LIKE TO PLAY A GAME?
9
u/proudcanadianeh Muni Sysadmin 1d ago
Ask AI to pick a number between 1-10. They always say 7.
•
•
u/Rakajj 10h ago
Yeah, they're trained on the internet and humans favor 7 because they think it seems random.
LLM's don't actually run a rand() function they just want to get to a satisfactory answer.
Ask the LLM to explain to you why it chose 7 after and it'll honestly go through all the ways it's not actually doing what you asked it to do.
•
u/stordoff 21m ago
Just tried ChatGPT, it gave "7 🎲" as the first response/regeneration about ten times across three different chats. Reprompting with "pick a number 1-10" in the same chat seems to reliably give "3 🎲".
gemini-4-12b-qat locally: "I pick 7." despite having this in its thoughts:
Selection: 7 (a very common "random" choice for humans/AIs). Let's try 4 for variety.
qwen3.5-9b also gives "I'll pick 7." or "I pick the number 7." despite lengthy debates:
Selection: 7 (often considered lucky or random enough), or just pick one arbitrarily. Let's go with 6. Or maybe 4. It doesn't matter mathematically. Let's pick 7 [...]
Wait, actually, I should just pick one. Let's pick 4. No reason to overthink it. [...] Let's pick 3. Okay, decision made: Pick 5. It's the middle of the range. Neutral choice.
Wait, let's just say 7. It feels random enough.
Regeneration:
I'll pick a random number in my "mind" (since I'm an AI, it's effectively random selection). Let's say... 7. Or 5. Or 3.[...]
Final Decision: I'll pick the number 7 (a common lucky number) or just a random one. Let's go with 6. Actually, let's make it interesting. No, keep it simple.
Selection: 8.[...] Draft Response: "I pick 7." (Wait, let me randomly select one now). Okay, I'll choose 3. [...] Let's pick: 7.[...] Okay, decision made: 8.
Wait, I should check if there are any hidden constraints. No. Okay, I will pick a number. Let's pick 4.
Final Output: "I pick 4." or "7". Let's do 7. It feels classic. Actually, let's just output the number and maybe a brief note. "I choose 6." Okay, I'll stop overthinking. Just pick one. Number: 5. [...] Let's pick 9. [...] Okay, I'll write: "I pick 7." Wait, I want to be playful. "How about 4?" No, just state it. "I choose the number 3." Okay, final choice: 7.
qwen gave 7 10/10 times - I didn't expect this to be quite so consistent.
4
u/FriendlyITGuy Playing the role of "Network Engineer" in Corporate IT 1d ago
God I forgot about xkcd
•
•
u/Schmittfried 4h ago
Came here to find this. That’s gotta be the most always-a-related-xkcd situation ever.
48
u/slonk_ma_dink Jack of All Trades 1d ago
That is mindblowingly stupid and reckless.
23
u/AmusingVegetable 1d ago
Probably some script in charge of collecting random bytes failed after reading one and there were no checks on the key file length.
28
u/spin81 1d ago
I don't think we know exactly what happened yet, but they've said publicly that it is indeed a flaw in a script used in the key generation ceremony.
It's a bit vague but from what I understand, they put some of these ridiculously weak private keys in their HSM during the ceremony because of the flaw in the script, and the names were too similar to the good ones.
That's all I could glean from what they wrote. These are Indians and they are probably stressed the fuck out right now so there's that in addition to a presumable language barrier. I hope we'll get a report of what actually happened someday.
•
u/thenickdude 22h ago
Maybe they generated a hex string, but fed it to something expecting only an integer, so it ended up just parsing the first few characters that were numeric as an integer (like PHP does,
(int) "123abcdef123"is123)21
u/e_t_ Linux Admin 1d ago
I thought keys were at least supposed to be prime. Three and five were right there.
16
u/ColdSysAdmin IT Director 1d ago
I believe these are all ECC certs and ECC doesn't require only prime numbers.*
*I only math for fun and for my alarm clock in the morning.
15
28
u/ocdtrekkie Sysadmin 1d ago
It has been a good day for those of us who argue Web PKI is dumb and the CA/B Forum wonks who steward it are idiots.
47 day certificate expirations don't mean much when the CA's private key is 4.
•
u/thortgot IT Manager 21h ago
Or the fact that every CA has the ability to issue every single cert.
Spending effort on actually securing the CA infrastructure is hard. Reducing expires is "easy".
•
u/ocdtrekkie Sysadmin 20h ago
Yep. Even things like CAA records are a patch over the reality that PKI only exists to protect the power of the certificate authorities. It's all domain validated so it's all redundant with DNS.
•
u/NamedBird 13h ago
I'd argue for rolling out DNSSEC+DANE/CAA to bypass the CA's entirely.
And if that's not possible, start limiting which TLD's a CA can sign for.(Perhaps consider a DNS redesign as it starts having some merits?)
•
u/thortgot IT Manager 11h ago
Im definitely not a security architect but it feels like a solution like ACME could be used to verify what CA is intended to signed for a domain. The PKI can stay the same but the protocol just checks via DNS the root cert ID to see that it matches what's intended
•
u/NamedBird 10h ago
That's just DANE with extra steps? 🤔
•
u/Sure-Squirrel8384 3h ago
Right. Why not start with trusted DNS (DNSSEC) and query for certificate validation via that mechanism?
•
u/ocdtrekkie Sysadmin 11h ago
That is indeed the correct path if we are committed to domain validation. There's no reason to involve additional parties which can be compromised if we are going to trust based on domain control anyways.
I personally feel that an amended EV (similar to the VMC today) holds value for exceeding domain control and connecting a certificate to real world business entities, but the current tech regime is opposed to anything they can't automated away.
•
u/NamedBird 10h ago
Right now, CA's are useless baggage and they will either have to make some very expensive investments that appeases the "/B" part of the CA/B forum, or they will be made obsolete outright trough browsers adopting DANE.
I still see some merit in OV/EV certification, so i would like to suggest a path out for these certificate authorities to survive. It would require some very fundamental changes in an ambitious project, but the end result would be a win-win-win situation wherein end users are safer, browsers have less to worry about certificate leaks and CA's can continue to be relevant.
Not that any of them would seriously hear me out though. 😅
I have the experience but lack the credentials, a downside of proper OPSEC...•
u/ocdtrekkie Sysadmin 8h ago
Yeah I'd definitely prefer a much stronger lean into DNSSEC and DANE for anything domain validated.
OV/EV needed a higher standards bar, not to be stripped away. I feel like OV/EV should have a very visible nationality flag, i.e. registering a legal entity in some other country is obviously not the same as the entity that you do business with. And I know it's not very open of me, but I think it would be fair for something like OV/EV to have some level of "notability requirement" to get one. Every average LLC doesn't need to qualify.
I often find the trustworthiness of a security measure to actually matter more how much it costs than what it does, because criminals don't want to burn money on attempts: "The lock icon" used to mean "it was legit" not because criminals couldn't get a certificate, but certificates weren't free and criminals weren't spending $200 to spoof your domain. Make an EV cert cost six figures and like an approval by one of the Big Four accounting firms... and it's gonna be legit if you see one. =)
But aside from that, I'd like to see a lot more private certificates in a more web of trust/trust on first use sort of measure. Sort of like a two-way passkey. If I open an account with my bank, my bank should be able to give me a completely secret certificate at that time that only I and my bank can use to communicate securely.
•
•
u/Spectrum1523 13h ago
I'm sure someone has done it by now but checking if any other CA has similarly trivial keys would be interesting
8
u/cgimusic DevOps 1d ago
I've got the same combination on my luggage!
•
u/WackoMcGoose Family Sysadmin 8h ago
What kind of society are we living in when a Druidian security system is somehow more secure than an entire certificate authority?
7
2
58
u/segv 1d ago edited 1d ago
Additional discussion and background info is on:
- Discussion: /r/cybersecurity/comments/1xb0onb/emudhra_intermediate_ca_compromise/
- Discussion: /r/linux/comments/1xau81w/leaked_ssl_intermediate_certificates_https/
- Discussion: https://news.ycombinator.com/item?id=50166579
- Researcher losing their mind: https://mastodon.catgirl.cloud/@49016/117474217042191790
- Mozilla revoked CA trust: https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/0R3XLJL6FuE?pli=1
- Nobody would screw up this bad, right guys? https://github.com/badkeys/badkeys/issues/38
Note that this CA used to be widely trusted, so a ton of devices are vulnerable. This includes those that are not-so-easy to patch, like IOT/embedded, phones or consoles (VMs and docker images too, but that should be obvious).
9
u/trappler23 1d ago
Wasn't every device vulnerable? Every domain? Or are you talking strictly about revoking the certificates?
19
u/Cormacolinde Consultant 1d ago
Everyone who trusted the Sectigo root that cross-signed these certs or the eMudhra Root was vulnerable from April 2025 until they stop trusting either.
13
u/segv 1d ago
These compromised intermediate certificates would ordinarily have been trusted by major web browsers and at the same time they have private keys so trivial that pretty much everyone can forge a TLS certificate for any domain that could be used to fool programs/devices/whatever with an out-of-date trust store* to connect to a malicious host instead, so to the answers are pretty much "yes", "yes" and "as long as your stuff no longer trusts the CA you should be good". Sorry for talking a bit in a circle, but there's a lot of nuance in the mechanism.
* Major web browsers (Chrome/Firefox) should pick up the updated trust list on their own - you can check it for your system on the page linked in the OP (https://emudhra.we-have-a-problem.com/#check). In locked down environments or for non-browser trust stores (https is freaking everywhere these days) manual action may be necessary.
3
2
u/Z3t4 Netadmin 1d ago edited 1d ago
If they follow spec they should check CRLs as well, and do not accept certs signed by a revoked intermediary ca.
Rpki was designed to cope with this.
Edit: my firefox mobile load the links, oh oh... Chrome mobile fails only the AAA ones.
Seems that CRL check of the chain is not implemented
Edir2: after manually updating them now they work.
•
36
u/cjbnc 1d ago
Are you an edu? Do you use InCommon's new Certinext CA for your certs? Yeah, that's eMudhra. Start checking the root used for your certs, specifically the ECC ones.
11
u/swarmy1 1d ago
InCommon certs shouldn't be affected by the compromised intermediates. You can see the intermediates they use here:
https://spaces.at.internet2.edu/spaces/CERTSVC/pages/394592634/InCommon+Intermediate+CAs
Now, if trust gets revoked for the emSign Root that will be a whole other ballgame.
11
u/cjbnc 1d ago
Now, if trust gets revoked for the emSign Root that will be a whole other ballgame.
That is what has happened. I'm already seeing clients refuse our certs signed by the chain
C=US, O=Internet2, CN=InCommon Intermediate CA - OVG3C
C=IN, O=eMudhra Technologies Limited, CN=emSign Root TLS CA - G3
C=IN, OU=emSign PKI, O=eMudhra Technologies Limited, CN=emSign ECC Root CA - G32
•
u/gumbrilla IT Manager 21h ago
I think there is jeopardy. A Certificate Authority's competence is treated as indivisible. Symantec, DigiNotar found out the hard way.
46
u/ItsMeMulbear 1d ago
This CA never should have been trusted to begin with.
8
u/mixduptransistor 1d ago
why?
32
11
u/sobrique 1d ago
They set a key to (lots of zeros) 4.
9
u/spin81 1d ago
Okay but why should they never have been trusted to begin with? I mean obviously now I think it's smart to think twice, but what about them was sus before this?
18
u/the_marque 1d ago
There are quite a few CAs on trust lists that we all "have a sense" shouldn't be there. There's no way they actually have the same level of governance and scrutiny as the top-tier CAs, but if they run their own root and real services are using their certs, what are you gonna do.
The commoditisation of TLS certificates has created a race to the bottom in general.
•
u/spin81 20h ago
There are quite a few CAs on trust lists that we all "have a sense" shouldn't be there.
I'm not asking about quite a few CAs. I'm asking about eMudhra.
what are you gonna do.
Put proper protocols/procedures and a consortium in place. AFAIK these are diligently followed by everyone.
The commoditisation of TLS certificates has created a race to the bottom in general
So they, and I quote, "should they never have been trusted to begin with" because they charge for their services?
6
•
u/Sure-Squirrel8384 3h ago
No reason they shouldn't have been trusted before, but the should never be trusted again.
You can block/remove CAs from your keystores. I do this with anything from countries that let crime run rampant, much like geo-IP filtering.
12
6
•
u/Ziggistawork 23h ago
ZIA still trusts these intermediates.....
For those with Zscaler....
•
u/Speeddymon Sr. DevSecOps Engineer 5h ago
Not anymore. It was trusting them 24 hours ago but I just tested just now.
Common Name (CN)
dv-g3b-emudhra.we-have-a-problem.com
Organization (O)
Zscaler Inc.
Organizational Unit (OU)
Zscaler Inc.
Common Name (CN)
Bad Server Certificate
Organization (O)
Bad Server Certificate [invalid server certificate]
Organizational Unit (OU)
<Not Part Of Certificate>
9
u/0f_rice_and_men 1d ago
so we can expect a zillion new phishing pages that use this key, right?
28
u/sobrique 1d ago
Maybe but browsers do tend to handle crls.
Bogus packages fetched by automated tools on the other hand...
•
•
u/ObjectiveGullible628 20h ago
oh man i checked our servers and we had like 4 of their certs in the store, just nuked them all. the google impersonation test gave me chills
thanks for sharing those curl commands, made it super easy to confirm. already pushed the update through our config management, not taking any chances with this one
•
u/arthur_thomas678 16h ago
The whole situation is wild, especially the single byte key. It really shows how critical it is to review and manage certificate authorities regularly. We had a similar scare last year and ended up implementting a quarterly cert audit process just to be safe, have you thought about doing something like that?
2
u/vivkkrishnan2005 1d ago
Need to check also the DSC keyfobs used if they too are affected as I haven't checked the root of trust in quite sometime
•
u/AcceptableAmount6028 18h ago
Question, will this issue be resolved just by updating the keys with new one ?
•
u/Groove200 12h ago
Who else is trawling all their tooling to find affected pip and python etc ?
FUN TYMES
-1
u/VaporousMote 1d ago
You just know this is some intern who consulted Claude.
18
u/Cormacolinde Consultant 1d ago
According to eMudhra, it was their engineer of 15 years experience.
•
u/Fallingdamage 21h ago
BOFH of the year.
•
u/tuxedo_jack BOFH with an Etherkiller and a Cat5-o'-9-Tails 18h ago
These hands have caused term papers, theses, and dissertations to be lost - sometimes even by accident!
8
u/spin81 1d ago
Goes to show a major fuckup can happen to anyone.
-1
u/Lower_Fan Sr. Sysadmin 1d ago
Well the model that was popular back then was Claude 3.7 sonnet. So I can definitely see how it happened now.
6
u/spin81 1d ago
Apparently it was a bug in a script they used. If they were stupid enough to vibe code it without double checking the output...
10
u/the_marque 1d ago
It's not a bug, it's straight up old-fashioned human laziness. I actually don't think any LLM would come up with something this bad.
•
u/spin81 20h ago
What are your sources for that? They themselves say it's a bug which can happen to anyone who is not lazy.
•
u/the_marque 15h ago
A script producing a private key of "4" is not a bug.
•
u/WEEEE12345 11h ago edited 11h ago
It seems likeI could speculate they generated the private key but only the first byte got used. That’s a pretty plausible bug.
If it were say a hardcoded value the numbers chosen for the 8 keys (4, 56, 66, 115, 155, 156, 230, and 237) would be pretty odd. I’d expect something like 12345 in that case.•
7
106
u/encbladexp Sr. Sysadmin 1d ago
Completely fucked up.