Do extra PCIe lanes improve latency?

Question

If you connect a NVME SSD to a PCIe x4 slot it will have a higher maximum bandwith than if you connect it to a 1x or 2x slot. So logically it will transfer large files faster.

But if you never need any high throughput, as you only work with small files for example, will there be any difference in performance between a x4 slot vs a 1x slot?

Foremost I wonder if the latency will be exactly the same? When the host sends any commands to the drive they tend to be small in size, so they should not be affected by the max bandwith. But maybe when you have 4 lanes those communications will be send in parallel instead of queued on one lane?

Or doesn't it matter at all because the SSD controller will only process them sequentually?

Answer 1

requesting a small file will still initiate a transfer of a full block from your NVMe. So, that's significant data (in terms of PCIe packet sizes), and yes, the duration until that's finished actually drops with the number of lanes.

Where it doesn't matter is the latency which a device interrupt takes to reach your CPU, because that's not payload data distributed across lanes.

When the host sends any commands to the drive they tend to be small in size,

[[citation needed]], first of all. NVMe commands can, and will in a many-small-storage-requests scenario, be quite complex, as it makes sense to combine multiple commands to avoid the much worse "ping-pong" latency, or having to wait for the first to successfully be served before sending the second (see NCQ).

A PCIe gen3 (which is what you'd typically find) transaction layer packet has some 22 to 30 B of header+checksums. Getting an empty payload packet out still has quite some latency on the CPU side, so you'd avoid small packets.

Or doesn't it matter at all because the SSD controller will only process them sequentually?

No. That's the main technical reason why we use NVMe and not ATAPI-over-PCIe: the NVMe device and the host can queue requests and commands and serve them in the order that's most advantageous.

Answer 2

But if you never need any high throughput, as you only work with small files for example, will there be any difference in performance between a x4 slot vs a 1x slot?

Short answer: not really.

Foremost I wonder if the latency will be exactly the same? When the host sends any commands to the drive they tend to be small in size, so they should not be affected by the max bandwith. But maybe when you have 4 lanes those communications will be send in parallel instead of queued on one lane?

Each transfer will take 4x longer with 1/4 the bandwidth, so if you know the size of the transfer and Google for random read IOPS numbers for your device you can calculate this exactly. Using typical values for 4K random access and 1 GB/s you can get an idea though. On a typical drive it'll take about 100 uS to start the transfer than 4E3/1E9 = 4 uS to complete it.