Audit: Security v2 audits meta

[paulirish]

Spoke with @lgarron, the brains behind devtools security panel to better understand a comprehensive view of security testing.

HTTPS redirect.

Because of HSTS, we cannot fully trust this browser's redirect behavior to be representative of what will happen in the wild. (A site can be in the local HSTS list, but not actually have a redirect in place. This would result in a false positive. It's an edge case, but hey that's what security is all about. The truth on this one has to come from cURL, etc until the browser has a way to disable HSTS.
Verdict: The PR in #270 is good for now. We can harden this later.

HSTS testing.

The hsts preload submission site has a number of audits written. This includes a test for HTTPS Redirect in fact! And there's an API endpoint, though it's still in development, so we will need to work with Lucas if we decide we want to go forward with that. For example, see the results of https://hstspreload-mvm.appspot.com/checkdomain/web.telegram.org , https://hstspreload-mvm.appspot.com/checkdomain/google.com

Verdict: Some great infrastructure that's well aligned we can reuse.

TLS, Connection & Certificate chain

SSLLabs test does have an API, which we could consume and surface... Example response: https://api.ssllabs.com/api/v2/analyze?host=twitter.com&all=on
Verdict: It's canonical and the best thing to highlight once we need more detail than what Security domain provides.

CSP, X-Content-Type, SRI, etc etc

There's some fantastic work in mozilla/http-observatory-cli It's also available by API. Lucas has worked with April and expects we could collaborate smoothly here, too.

Verdict: Looks promising, and a good basis for some v2 audits.

edit: more stuff in CSP Evaluator which looks sorta open source.

---

July 2017 update:

There's also https://securityheaders.io/ and https://www.hardenize.com/. Both look great.

Oct 2019 update:

Proposal here for XSDB auditing

[lgarron]

Thanks for writing this up; that's a good overview of what we talked about.

the brains behind devtools security panel

Also @estark37 :-)

This would result in a false positive. It's an edge case, but hey that's what security is all about.

I'd like to note that this is an edge case in the sense that it applies to very few sites on the internet, but it applies to many "modern sites" (sites that "have their act together" and care about security) – especially PWAs.

[paullewis]

Terrific write-up! I really like the idea of using the existing HSTS stuff 👍

A+ would security again.

[connorshea]

The Observatory CLI now has a rewrite(?) in Node: https://github.com/mozilla/observatory-cli

Maybe worth looking at?

[brendankenny]

To preserve #607, it had an example of HSTS messing up our HTTPS redirect audit:

Lighthouse reports that the HTTP URL http://billingsgazette.leeaws.com/pwa/#top-stories redirects to HTTPS, however it (usually) doesn't.

The issue was that Lighthouse first visits the HTTPS page, gets the HSTS header, so when it then checks the HTTP page, Chrome redirects the request and it appears to Lighthouse as if the site does redirect. However, an initial navigation to the HTTP page, with no previous visit, will not redirect.

[brendankenny]

I'm thinking more and more that something like Observatory is a good idea. A raw request for the page with faked UA string etc and then parsing the response is possible but
a) that's difficult (or impossible) in the extension and
b) we'll take on responsibility for correctly diagnosing the nuances of secure states when the libraries and services mentioned above already do this.

obvservatory-cli seems like it could provide everything we need (and more things we haven't even thought about) but we'd need to figure out

• dependencies (we could alternatively communicate with the observatory service directly)
• latency. How long do the tests add to the gather process?
• sites not publicly accessible. Local dev server testing, pages requiring authentication, etc
• what kind of load we would put on the service
• How often Observatory reruns tests. It looks like there's a --rescan flag for the CLI, but there's references to it attempting to get a rescan, not necessarily getting one. Makes perfect sense to throttle this from Observatory's perspective, but for developers doing iterative fixes on their site we probably do need fresh results every run.

We could maybe do a tiered set of audits. Maintain the baseline is-https check, maybe get rid of the https redirect audit since it's difficult for us to get right. We could also try to land more HTTPS auditing stuff in the debugger protocol's Security domain to expand what baseline stuff we can do. Chrome is more privileged than our extension code is, so can more easily check redirects, HSTS, etc.

Then for the full security report, defer to Observatory (or whatever) like we do for axe and accessiblity, or, if that doesn't work out for one of the reasons above, have some kind of opt-in plugin for "I want the full Observatory results in my Lighthouse reports", which would excuse a slightly more awkward workflow for it.

[paulirish]

See also the awesome https://github.com/voorhoede/lighthouse-security project which looks at a number of v worthwhile security concerns.

[addyosmani]

@paulirish That lighthouse-security project looks like a super useful start.

In light of recent write-ups like this one, is there interest in committing to a set of audits to encourage CSP and security best practices during 2018?. I'm curious where security fits in the overall prios 🙃

The discussion around Mozilla's Observatory above looked like there were some solid audits around HSTS, referrer policies etc worth considering.

[tollmanz]

Securityheaders.io and Secure The News offer grade based security audits that could be useful to study as prior art. Secure The News published their methodology for rating HTTPS deployments with a specific focus on HSTS.

Hardenize is another scanner that audits security related features, but does not offer a ranking. It has a decent breadth of factors it looks at and is worth studying to understand the scope of features that should be tested.

[paulirish]

A few potential audits coming out of the spectre/meltdown stuff:

Site Isolation - The Chromium Projects recommends:

• serving the HTML response with one of 8 allowlisted mimetypes
• serving the HTML response with a X-Content-Type-Options: nosniff response header

Meltdown/Spectre  |  Web  |  Google Developers recommends:

• the above stuff
• setting cookies as SameSite (only supported by chrome+ff+opera)
• setting cookies as HTTPOnly
• all external links with rel=noopener. ✅

---

edit (feb 2021): http://go/hiiye