Get a benchmark estimate of round-trip time independent of the site and its resources, for use in network analysis

[vipulnaik]

Feature request summary

Currently, the round trip time calculation

lighthouse/core/computed/network-analysis.js

Lines 16 to 48 in 3a0c62d

	static computeRTTAndServerResponseTime(records) {
	// First pass compute the estimated observed RTT to each origin's servers.
	/** @type {Map<string, number>} */
	const rttByOrigin = new Map();
	for (const [origin, summary] of NetworkAnalyzer.estimateRTTByOrigin(records).entries()) {
	rttByOrigin.set(origin, summary.min);
	}
	// We'll use the minimum RTT as the assumed connection latency since we care about how much addt'l
	// latency each origin introduces as Lantern will be simulating with its own connection latency.
	const minimumRtt = Math.min(...Array.from(rttByOrigin.values()));
	// We'll use the observed RTT information to help estimate the server response time
	const responseTimeSummaries = NetworkAnalyzer.estimateServerResponseTimeByOrigin(records, {
	rttByOrigin,
	});
	/** @type {Map<string, number>} */
	const additionalRttByOrigin = new Map();
	/** @type {Map<string, number>} */
	const serverResponseTimeByOrigin = new Map();
	for (const [origin, summary] of responseTimeSummaries.entries()) {
	// Not all origins have usable timing data, we'll default to using no additional latency.
	const rttForOrigin = rttByOrigin.get(origin) || minimumRtt;
	additionalRttByOrigin.set(origin, rttForOrigin - minimumRtt);
	serverResponseTimeByOrigin.set(origin, summary.median);
	}
	return {
	rtt: minimumRtt,
	additionalRttByOrigin,
	serverResponseTimeByOrigin,
	};
	}

uses the fastest of the observed round-trip time values for all requests. Further code in https://github.com/GoogleChrome/lighthouse/blob/main/core/computed/network-analysis.js and https://github.com/GoogleChrome/lighthouse/blob/main/core/lib/dependency-graph/simulator/network-analyzer.js uses this observed round-trip time to help calculate additional round-trip time per origin and to calculate server response time.

As a result of this, we can have weird behavior such as this:

• A site that just loads one resource, by definition, will get a measure of 0 additional round-trip time, even if it's actually extremely slow (e.g., it's in a remote location from the end user so the round-trip time is very long). This in turn leads to (falsely) optimistic estimates of TTFB, FCP, LCP, etc.
• Adding the loading of an additional resource (e.g., Google Analytics for JS, or an image CDN, or a third-party website serving CSS) that has good CDN coverage, within the time period that Lighthouse runs on the page,ends up reducing the estimate of round-trip time, and therefore results in the additional round-trip time for the site going up significantly. Thus, adding in such a resource ends up worsening estimates of TTFB, FCP, LCP, etc., but not due to the resource itself slowing things down; rather, simply because it worsens the estimate of additional round-trip time for the main origin.
• Delaying the loading of additional resources (e.g., Google Analytics for JS, or an image CDN, or a third-party website serving CSS) to late enough in the page load that Lighthouse doesn't stay on the page, will again shift us back to a (false) optimistic estimate of additional round-trip time.

Here's the situation described above, with numbers. Let's say origin A.com (the site whose web page we are loading) has a round-trip time for me of 300 ms, and origin B.com (the domain having a JS, CSS, or image that we might want to load as a resource on the web page) has a round-trip time for me of 50 ms.

• If the page on A.com doesn't load anything else, the round-trip time estimated by Lighthouse will be 300 ms, so the additional round-trip time of A.com will be estimated as 0.
• On the other hand, if A.com loads a resource from B.com (e.g., a JavaScript file, or an image), while Lighthouse is running on the page, the round-trip time calculation will yield 50 ms (the faster of the two) and as a result, A.com will get assigned an additional round-trip time of (300 - 50) ms = 250 ms.
• If A.com delays the loading of the resource from B.com to late enough that it's not captured in Lighthouse's run on the page, then we return to the situation where Lighthouse estimates a round-trip time of 300 ms and an additional round-trip time of 0 for A.com.

These problems could be addressed at least partly if the estimation of round-trip time is done by pinging a standard set of domains or IP addresses, rather than limiting ourselves to the resources that happened to be loaded as part of this particular page. Perhaps this list could be configured by the user based on the user's location and connectivity. And the pinging of these origins could be done at the start of each Lighthouse run, so that we get an accurate estimate based on the network conditions at the time of the Lighthouse run.

While I might be able to work on this in principle, in practice I don't expect to have time and bandwidth to work on this for the next 6 months.

What is the motivation or use case for changing this?

This was one of my biggest sources of confusion when I started trying to understand Lighthouse score calculations for simple sites two years ago. I like to start with simple sites to build a solid intuition for all the details around how Lighthouse scores are calculated. This particular problem plagues simple sites the most, though, partly defeating my efforts.

How is this beneficial to Lighthouse?

The main benefit is that the network side of the Lighthouse readings would be more reliable and informative. In particular, this would significantly improve the value of Lighthouse data as far as the TTFB portion of loading speed goes. More specifically:

• It would help people see more reliably the improvements in TTFB, FCP, and LCP from changes such as using CDNs with edge locations closer to the end user and higher cache hit ratios
• It would result in more accurate inference of the impact of loading third-party resources such as JS, CSS, or images. Presently, adding in a fast third-party JS can penalize Lighthouse scores by the mechanism documented above, where it leads to a more pessimistic (though probably more accurate) estimate of the origin's additional round-trip time. This might lead people to decide against adding a fast third-party JS (and adding a slower one instead, that paradoxically results in better scores!) or to unnecessarily delay the addition of the JS so that it's not detected by Lighthouse.