Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
I revoked $1M worth of EV certificates (scotthelme.co.uk)
153 points by amaccuish on Sept 14, 2019 | hide | past | favorite | 66 comments


Scott isn't trying to mislead us here, but one sentence from the article seems like it needs expanding certainly for a lay audience like HN.

"One thing that we can do with CT logs is search all certificates for things that we're interested in"

The CT logs themselves implement only a terribly simple API and you can't search them per se. This makes good sense because they are required to hit really onerous availability and performance targets to remain "qualified" as logs. You definitely wouldn't want to put a sophisticated "search" feature in this code.

However, since they're public we can either read all the data (my employer does this) and process it however we like or there are two famous services which already did this for you, crt.sh (operated by Sectigo, one of the CAs) and Censys (from an ex-University of Michigan team). Scott used Censys for this article, and either Censys or crt.sh are good places to play around with this if you have a passing interest that may not last.


We get daily notifications of certificates from https://sslmate.com/certspotter/


Great service.

To add, I like the checks of https://sslping.com/ too.


We use a similar service called Oh Dear that does CT monitoring: https://ohdear.app


Another data point in the ongoing theme of EV certificates having basically no value.

The other big data points being: browsers don't want to display them differently, and users don't notice them being any different.

Now we have that nobody is really checking them that closely.

How much longer will these even exist?


Ours are about to expire. I replaced them with free amazon certificates recently. The process of getting and deploying certificates three years ago was super painful. The process of replacing these certificates was super easy and renewal is now automated. Things have improved a lot since the letsencrypt movement kicked off a few years ago.


I used to pay for certificates. Recently however, I got burned with a crappy Symantec one sold by Name resulting in near continuous customer reports of console errors in Chrome.

I fought with Name to get them to reissue it, they wouldn’t. I decided it was time to figure out how to setup Let’s Encrypt. They lost me as a customer, and I’ll likely never buy a certificate again.


I wish InCommon/internet2 supported ACME. The admins at my work are quite busy as is, and ACME would definitely make for one less factor to be constantly concerned with.


There was https://www.troyhunt.com/extended-validation-certificates-ar... from a year ago.

Max length of a certificate is 2 years, so probably 2020. Maybe 2021-2022 if you count stragglers and businesses that are locked into long contracts.


Mr Hunt also updated this statement recently: https://www.troyhunt.com/extended-validation-certificates-ar... "Extended Validation Certificates are (Really, Really) Dead"


When I saw that google, amazon, apple and Microsoft weren’t bothering with EV certs, I figured I didn’t need to either.


Google's cert is not really domain validated either. They have their own CA.

Microsoft also has a CA.

Apple has EV and Amazon uses DigiCert, which is the only one that may be domain validated, perhaps.


All four of them own publicly trusted CAs. However, they are certainly required to validate all the certificates they issue even to themselves.

Several (all?) of them are domain registrars and so they would be able to rely on that to shortcut issuance validation for names in domains for which they are both registrar and owner, but it still happens it's just easier.


> How much longer will these even exist?

EVs will soon be gone when browsers stop recognizing them. We are already well on our way there. From the same author from a month ago [1]:

> Both Google Chrome and Mozilla Firefox have announced that they plan to move the EV indicator out of their main UI.

[1] https://scotthelme.co.uk/gone-for-ever/


I don't think that's an indication that their CAs will be removed. Correct me if I'm wrong, but they are not less safe than other certificates, they just don't add any additional value. So they will probably be treated the same as other certificates by browsers. If people will continue paying for them is another matter of course (but it might very well happen that they will have many customers for years to last).


EVs and security is a fun topic, including obvious sarcasm. They generally are more secure for 3 reasons:

1) hardfail on revocation checks 2) you can't get around any errors generated by an EV 3) you can't fake them by truststore manipulation (except ie and maybe edge) as the ev roots are hard compiled into the browser and not dependent on the external trust store.

Validation would have been a 4th reasons if it wouldn't be for all the obvious problems with it especially lately.

The problem is what people imply or are made to imply from different cert types.

Back in the day people were told to just check for the lock, which obviously is dumb considering now everyone can get a dv for free.

Then with EV CAs told people that sites with ev are more trustworthy. Obviously nonsense considering the excluded usages of EVs in the cabforum documents. EVs are only supposed to make a hard link between an offline and online legal entity, and even that failed with stripe.ian.sh (although that's not exactly the fault of EVs)

EVs now get so much higher implied security that the real vs implied security ratio is obviously very ugly while DVs becoming standard obviously have much more real security than implied (if people check the urlbar correctly)


That is a good point, I didn't know most of it and it shades a whole new light on the EV topic for me. Especially 3) is a bombshell to me, as I was under the impression that at - least theoretically - the users have control about who they trust.


> browsers don't want to display them differently, and users don't notice them being any different.

That's the only thing that matters, really.

The selling point of EV certificates is that (1) they are very visible in the browser and (2) users care enough about that to make a difference to your business.

I haven't seen any surveys or A/B testing results about (2) but since (1) is no longer true this is moot.

It means you should just go for the cheapest certificate, which means free nowadays.


Firefox displays them differently.


First thing to note is that Firefox is largely irrelevant versus Chrome and Safari in terms of marketshare.

Chrome no longer displays EV certificates differently, I believe, that about 2 thirds of the market.

The second thing is (2) is my previous comment: Even if an EV certificate is displayed differently, does anyone care? Does that impact my business to make it worthwhile?

The general public could not care less, imo.


You're saying that Firefox's 9% is irrelevant versus Safari's 3%?


That's not the point of my comment both regarding EV certificates and regarding relevance of Firefox (and I don't know where you got those stats).


You are clearly excluding mobile browsing stats.


> browsers

EV certs are still useful for code signing. At least on Windows UAC warnings look differently if binary is signed with EV certificate.


These certificates aren't in the Web PKI and so they aren't covered by (most of‡) the Web PKI's rules and there is no requirement to log them publicly before use.

For example they would lack the 1.3.6.1.5.5.7.3.1 EKU which says their purpose is to identify an SSL/TLS server.

The market for these certificates is pretty different from the Web PKI market.

‡ When they're issued by a CA which would also be trusted in the Web PKI some requirements cross over because of that, particularly Mozilla wants to be sure that certificates issued under these hierarchies are never "accidentally" valid in the Web PKI without obeying its rules.


Does it?

Windows is my daily driver, and I'm security-minded - yet I haven't noticed different colours for UAC dialogs depending on signature type. I notice if it's not signed at all, of course, but otherwise... no?

I would imagine most typical end users are the same. Similar to how browsers are not actually, where they make it really obvious if a site doesn't use TLS, otherwise it all looks the same.


The comment you're responding to doesn't make this especially clear, but code signing only /has/ certificates with organisation names subject to an extended validation. The rules for those are:

https://cabforum.org/ev-code-signing-certificate-guidelines/

Whereas for the Web PKI (things that do SSL/TLS, including most obviously HTTPS on the web) there are both the baseline requirements for every certificate:

https://cabforum.org/baseline-requirements-documents/

and EV for adding "extended validation" to that:

https://cabforum.org/extended-validation/

It's kinda useless to have a certificate that says this program is really named "Great Calculator". Oh yeah? So what? Whereas a certificate that says it's from "Microsoft Corp" in "Washington, US" at least tells me whose fault this calculator program is when it wrecks my PC.

On the other hand, knowing this is really https://news.ycombinator.com/ is kinda good because the machine can (and does) automatically check that this matches on every single URL resolved, every image, every HTTP POST, everything.


Oh, I've bought code certificates before - I didn't realise the ridiculous process I had to go through was "extended validation "?!

I had to give them a published telephone number for a callback. I don't publish a phone number, so I setup a Skype Number and published it in a free Yellow Pages kind of site - they did a callback, which obviously totally proves I work for Acme Stuff, and then I removed the listing and Skype Number.

I recall there were a couple of other bits of theatre I had to endure, publishing information on random listing sites that don't do any validation of their own, but I don't remember exact what.


I always thought code signing was a class of its own (especially since i also saw the names of people in those, which directly contradicts EV as ev cannot be obtained by people, only legal entities like companies or govs or whatever) but okay.


Just yesterday someone asked if there was a way to verify if the email from a purchase was leading to the correct domain to log in to. They wanted to make sure they weren’t being phished. How I wished there was a simple button they could have clicked in their browser that would have shown a window with the company details for the website certificate.

Everybody rails on how useless EVs are, yet it seems like no browsers have spent any time over the past years even trying to improve the UX related to certs. I mean, sure they’ve changed the color and removed the ambiguous company name and country. Which makes you ask why did they ever display country in situations where that isn’t the jurisdiction that issues business licenses, such as in the US?

I mean, click on the lock and through N buttons, then expand various sections and read through dotted identifiers that only ASN.1 experts recognize (seriously, who though it would be a good idea to use OIDs for known fields in the UI?), and then maybe you’ll understand who the certificate is issued to.

At least in the US, when obtaining and EV cert you need the jurisdiction (state), business registration number, they call the business phone number. They basically do some research - usually you need a DUNs number, publicly listed phone number, etc. Wouldn’t it make sense to have a button that when you click it you see a user-friendly window showing the information about the company for OV/EV? I mean, sure, you can register Stripe in a different state - how about providing some statistics like what year the company was formed (usually only file with business registration), a link to the company registration details on the jurisdiction website, the top level domain Whois info, etc? If you wanted to get fancy you could provide a link to the user’s search engine with the company name as the search terms.

Instead it seems like we have a bunch of cryptographers (or maybe just infosec people) obsessed about 1. The money companies are charging for EV certs 2. that it is possible for mistakes to happen. This instead of improving how we present information about a company.

So since there isn’t a perfect, foolproof identity verification system for companies online, now every user online needs to do their own reconnaissance to make sure this is really the website they should be at.


You're right in that a working EV system can indeed protect people. However, as this article suggests, even a basic check can be very hard for CAs to implement. If you can't even do a spell check on the state or country name, can you be trusted to verify the identity of those who rely on you to meticulously verify a company?

There's also the point https://stripe.ian.sh/ made: it's not just about registering the same name in a different state. People don't know in what state your (parent company) headquarters are so they would need to dig deep into the organisational structure to find out if the EV cert they're presented with isn't from a company with the same name in another state. Want to add more years of registration? Just buy an old, probably no longer used letterbox company and change the legal name. Whois info can be set to anything you want for most domain registrars as well. People will still need to find out if all the data matches.

There's also this demonstration of how EV can actually be used to help phishing by using the "green address bar" and the hilariously stupid Apple address bar design: https://www.typewritten.net/writer/ev-phishing/

I believe there is value in EV certificates if you can rely on the identity verification that they come with. Sadly, most providers of EV certificates seem to view them as a way to make more money, not as a tool to bring trust to the web.

An EV-like system will probably be reinvented in the future, giving us a fresh start on identify verification on the web without the crappy practises and standards that plague the EV system today.

Lastly, for such a system to work, all (big) companies need EV certificates. Even before browsers started cutting back on EV UI, nobody knew what site does or doesn't have EV. Some subdomains used different certificates, some didn't have certificates at all. How do you know if a (sub)domain is supposed to have an EV certificate? Will you tell people to just not log into Facebook anymore because they don't have an EV at their login page?

EV doesn't work because it's the wrong way around. For security indicators to work, you need to warn for danger instead of turning off the "everything is okay" light. The OV/EV system has been designed around the latter idea and as long as EV/OV isn't freely available all over the world, nobody's going to mark DV certificates as "insecure".


Also ov/ev isn't something a person can get. I mean it might already help for some personal sites if they can be tied to other pseudonyms the user has online so for example sites of more or less well known open source software could get a link to the github or whatever into the cert to directly bind the dev of that software to his website, without having to know who is behind that.

The identity problem is always fun. I mean i don't care who someone is in real life, i only wanna know whether i have the site by the same individual who made something else.

That can be easily and automatically verified (see keybase) and might be more than enough for a lot of things where there are only normal people involved.

It might also be helpful of a given company is more commonly known behind another online entity. Like for example if pewdiepie had a company which he uses for what he does, the link to his yt would be a much greater indicator of validity than some random company name or even his own real name (which not everyone may know).

For pure DVs i think they should be able to issue them themselves. I mean the only thing those prove is domain control and with dnssec+tlsa there's a great way that domain owners can prove that they are in control of the domain and aurhorize a cert, also this lowers the number of trust paths significantly as there is only one possibly trust path over the TLD, and not like 150 CAs from who knows where. Also both the domain owners and the users have less entities they have to trust, as the TLD managers have to be trusted anyway as they ultimately have the full control of their domains,and thereby could make a DV cert themselves over the CAs anyway.


Your proposal seems like it doesn't really change anything? Users are still left to "do their own reconnaissance".

Research suggests they don't do a very good job at that.


No, I think his point is that you need to provide a good amount of info to get an EV certificate, but then almost none of that info is included in the certificate. Combine that with the fact that what little relevant info about the certificate is buried in a relatively hard to find (for the average user) section of their browser, and it's no wonder that EV certificates have gone from "expensive novelty that nobody notices" to "effectively pointless"


> improve the UX related to certs

They’re actively making them worse, actually. I can’t even view the details of the certificate chain in Chrome on iOS. I used to be able to save them from the browser on a computer, and now I can’t do that, either.


Can't do that on safari either for some reason; this is confusing since there are apps like TLS inspector [0] that can show the certificate info.

0: https://tlsinspector.com/


At $work I maintain and develop a CMDB and the software that is used to manage it.

All of this sounds like very familiar data entry and maintenance problems.

In general, good data quality can only be achieved by giving the people who enter it some incentives, or make it impossible to enter wrong data (then data can still be missing).

For example, we have automation that configures monitoring and backup for hosts from our CMDB, and this information tends to be well maintained.

Other information that we need e.g. for compliance, but that no automation is tied to, tends to be of much less quality.

The second part is the feedback loop: if you enter some data incorrectly, and weeks or months later somebody complains that it's wrong -- that's too far off to efficiently learn from it.

We as an industry might have expected better from CAs, but we haven't pressured them into building the necessary feedback loops, or indeed given them incentives to provide higher-quality data in the EV certs.


> higher-quality data in the EV certs.

Part of this is probably the fact that many people were opposed to the entire concept of EV certs as basically being a money grab by the dominant CA's from day 1, and indeed the trend is away from highlighting their existence in web user agents...


It would seem that one could develop validation standards for certs that check for data entry errors (and missing data) and apply them at CAs so they can't issue them, and optionally at clients so they can't even accept them


Let's encrypt has developers empowered to just get https done in a couple of minutes. This includes the time it takes to search for certbot syntax on stack overflow.

Years ago the certificate was a big deal in project specifications, project managers could lay out the FUD and trick clients into spending more money on more snake oil. This was a pain for developers as they would have to get the work done. Sometimes the client would not have access to the one or two email boxes needed for this archaic system so there would be an added hurdle of setting up info@example.com or whatever it was that was needed to get EV to happen.

Nobody misses those days and the pain and hassle just was not worth it. Certbot came along and made it so easy. Nobody questioned why they needed EV certificates, sometimes you cannot argue with free and easy.


As a developer/server admin, one my frequent tasks were to install SSL certificates my clients send to me. My clients used to get the certificate and email/Dropbox me the certs along with the private keys to me.


Since the cert log is public, all of this is auditable, as indeed the OP audited it.

Should there be some formal/official mechanism of auditing _every_ EV issued?

Perhaps cert authorities should be required to pay for an auditor to audit their certs -- not just the process or a sampling of certs, but automated auditing of every cert. The problem with this of course is the usual 'auditor capture', if the auditor is getting paid by the entity getting audited, the incentive is not to continually approve the auditing, but to make all the audits pass.

In some sense, this is a success story -- the transparency of the cert issuing log allowed the OP to choose to audit some of them, voluntarily. But transparency of the cert log clearly isn't enough, it's been years of bad certs before someone decided to volunteer their time to audit some of them. Just transparency isn't enough without social/business structures in place such that they actually _are_ being checked.

(This has parallels/analogies with other things...)


> auditing _every_ EV issued?

this is what Extended Validation is supposed to mean in the first place.. and lends credence to the argument that EV solves a problem that shouldn't exist to begin with for 'regular' certs


No, I am not talking about what EV is supposed to mean in the first place.

EV, in contrary to ordinary certs, is supposed to mean the cert authority verified the registrant really is the name on the cert. Ordinary certs the CA doesn't even claim they did that, they don't even claim that a Cert that says Joe Smith on it was registered by someone who's legal name is Joe Smith or a cert that says Widgets Inc on it was registered by a company with that name. The CA's are claiming to do a different thing with EV certs.

I am talking about a third-party auditing the EV certs to ensure the CA was doing what they committed to do. Like a third party audits a bank by analogy. The fact that banks get audited doesn't mean the bank is useless in the first place. But the auditors do (theoretically if the system is working) help keep them honest.

One of the points of the transparent log is to make this possible. But if nobody is actually checking... it is not quite accomplishing this point. The fact that a transparent log is a good thing so someone can audit doesn't mean the EV is "useless in the first place".

(An EV might be useless in the first place for other reasons, but not because auditing is a good thing).


Little silly to say "I revoked $1M worth" since the price paid for each certificate includes unlimited renewals to fix problems like he identified. They are still worth $1M despite everything he did, they just have different private keys now and slightly improved data.

Maybe a better title would be, "Humans not 100% reliable about ensuring accurate information stored within thousands of data records, despite such accuracy kinda being the whole point."


I know for a fact that at least one of those certs was replaced with a DV cert - once the team in question had to replace it, they realized there was no point in replacing it with an EV since the browsers are removing the EV UI anyway.[0] I'd bet that a meaningful proportion of the certs in question are similarly being replaced with DV.

[0] https://www.isc.upenn.edu/alerts-outages/planned-weblogin-ss...


I mean maybe I’m wrong here, I’m probably wrong, but 99% of people don’t know nor care about the difference between an EV certificate and a Will Issue certificate like Let’s Encrypt.

Is it really worth revoking these over tiny discrepancies, especially in cases where you can still determine the entity beyond a doubt?


If a private entity is issued a cert saying "Government Entity", is that a "tiny discrepancy"? https://twitter.com/SectigoHQ/status/1171808269528109058


Yes, because revoking over tiny discrepancies is the difference between the two. If the EV certificate can have any old crap on it, it's no different than any other type.


So were the misissuances evenly spread among CAs or were there some that were clearly worse than others?


Once upon a time i was interested in them my marketing team wanted them. The hassle of dealing with the people that vetted our company documents ensured I never did it again. Totally hopeless process and worse still it was a total waste of my time. Terrible user experience and we paid for the privilege. Today we use the free amazon certificates and it doesn’t cost me time any more. I keep waiting to hear how the certificate business is headed for ruin.


Is there any kind of transparency index listing CAs and the number of mistakes they've made?


A single severe mistake can result the CA being kicked out, but CAs like Symantec were too big to fail, so browser vendors had to spend time and effort to kick them out.

There is CT logs of course. Even if the certs were issued by error or fraud, they must still be included in the log. There must be automated processes scanning these logs and validate the certs (x509lint, cablint, zlint, etc).

I suppose you can check CA CRLs or combined CRLs like Mozilla OneCRL to find recently revoked certs and lint them to see if there were issued with errors.


Is it possible to get an EV cert without registering with a specific government database first?


It doesn't have to be a government list. Any recognized (vague measurement I know) will do. But you need to be registered in a country.individuals cannot take one.


Can someone explain how revocations work? how many there are? what the rules are - etc?


Are all of these EV issues not data entry errors?


Presumably they are, but the entire point of an EV certificate is that the company's information is supposed to be vetted and verified. If a single human error can cause an invalid certificate to be issued, it's not just a problem with that one certificate. It's potentially a sign of systemic issues that call into question the trustworthiness of all other certificates issued by that CA.

A signed certificate that contains invalid or incorrect data is considered to have been misissued according to the CA/Browser Forum's Baseline Requirements [1]. Misissuance not only obligates the CA to quickly revoke the certificate upon discovery, but also frequently leads to discussions in places like Mozilla's dev-security-policy mailing list. [2]

If you follow some of those discussions, you'll quickly see that browser vendors hold the CAs to fairly high standards. Blaming a misissuance on a "data entry error" is generally not considered sufficient or acceptable; they're expected to be able to explain why the errors were not caught by technical controls or verification processes, and how those defenses can be improved to prevent the same errors from recurring.

[1]: https://cabforum.org/baseline-requirements-documents/

[2]: https://groups.google.com/forum/#!forum/mozilla.dev.security...


I think the traditional concepts of CAs has showed again and again that it does not work. Yet cryptography is worthless if you can't validate who you are talking with. I would propose a system where multiple parties could sign the SSL-certificate and these would be shown as badges in the browser. The more badges the more trustworthy.

Say for example that you have an online store. Then the local tax-office could sign your certificate if you are a registered company in that country. If you have a contract with VISA, they could sign your certificate. The authority for your top-domain could also sign the certificate to prove you own the domain and so on.

And for a customer buying from this store, seeing these badges in the browser would let them decide how trustworthy the website is. I actually think with this solution getting the users to understand would be the least problem. The big problem would be to get developers to authenticate the certificates while doing back-end calls. I mean how many developers even does this today?


Number of badges is not going to help if you can’t trust any given badge — no matter what the details are, given how automated attacks are, it is reasonable to presume that as soon as you’ve worked out how to circumvent one, you’ve probably figured out how to circumvent more of them than will fit in a browser bar at once.


> And for a customer buying from this store, seeing these badges in the browser would let them decide how trustworthy the website is. I actually think with this solution getting the users to understand would be the least problem.

Given that a non-technical user can't tell the difference between an EV cert and a regular CV, I can't understand why they would care about the actual number of badges. Sliding scales just don't work, they just care about whether it's safe to use a website or not.

The browser vendors _could_ establish a baseline for the number of badges that would make a site acceptable. But the economics would have to match real world incentives (e.g. small and large businesses), and protect against problems like sybil attacks...but then we're back to the heart of the problem.


I like the idea of multiple signatures, but mostly so it is much easier to drop trust in a CA.


You invented Web of Trust.


Edit: yes, I got the numbers mixed up

I really don't see why having a Danish number in a Swedish registered company is a big deal.

The phone might simply their contact for whomever is responsible for the certificate and it might be possible these contact is in Denmark.

I mean if it was a Canadian company with an US number and most likely they wouldn't even know.


A Danish business registration number (CVR in Danish), not a phone number.


Ah thanks for the clarification




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: