PCIe implementation

Question

I was reading about the PCIe stadard, it was mentioned about PCIe PHY, Switched, RootComplex and Bridges that make the PCIe fabric. I'm trying to connect the dots between the physical hardware and the stadard, and I couldnt realy understand if all of them are seperate ICs? or some of the functions are integrated into the processor or the motherboard PCH?

On typical motherboard there is a direct connection between CPU to PCH and PCH to PCIe slot

Is the PCH is the RootComplex with a PCIe Switch? if so why it is not connected the memory directly?

Or is the RootComplex is usualy part of the CPU? and the PCH is only a Switch?

And why there are no PCIe PHY chips on most motherboards/ GPUs?

Answer 1

I was reading about the PCIe stadard

A lot of the PCIe standard - pretty much everything outside of the physical layer - refers to abstract units that implement a specification. These units are ideas with names that make it easy to talk about them. It's hard to talk about complex devices if there's no name/term for them. So the standard had to come up with those names.

These abstract functional units can be realized (implemented) as discrete ICs, but I highly doubt there ever was a collection of such function-specific ICs that would let you implement PCIe, other than perhaps in early development of the standard, although I doubt it.

A PCIe PHY ICs were made and offered on the market - mainly to connect existing PHY-less FPGAs to the PCIe bus. One example was NXP's PX1011B. I imagine they were short-lived products. As soon as new FPGAs integrated the PHYs, the discrete PHY became unnecessary.

There is a relatively high fixed cost to an ASIC, so if you're paying for it, you will want to put as much functionality on it as you can. Separating functions between ASICs almost always costs more, but can reduce project risk. The PCIe development effort was, I imagine, well funded and well staffed. So the project risk was already lower than if it was a small business doing all the work with little resources.

I have read about the latest CPUs having the RC inside them - but would that be true for SOC like zynq xilinx or arria altera?

Yes. A root complex is not a lot of logic, but requires some firmware or state machinery behind it to manage it. FPGA SoCs are pretty much a perfect fit to implement a RC even if they don't provide one as hard IP on the die.

I have have seen some boards with those chips that use PCIe communication but there was no external PHY chip or external RC chip, is it usualy implemented on FPGA of a SOC?

PCIe PHY is almost always a part of the chip, otherwise it'd be an extra burden to deal with. FPGAs and SoCs with PCIe support have PHYs built-in.

A RC can be implemented using the FPGA/SoC resources so there's no need to have a chip for it. The whole point of an FPGA is that it lets you combine the logic that would ordinarily require lots of function-specific ICs and the interconnect between them.

The line between an FPGA and a SoC is very blurry. Usually FPGAs are called SoCs if they have "CPU-specific" hard IP like DDR controllers, hard CPU cores, etc. But just about any FPGA can implement a CPU and peripherals, i.e. it can be functionally an SoC even if it's not marketed as a SoC.

Hard IP are fixed-function parts of the FPGA that are not generic programmable logic fabric but perform a specific function instead. Say an FPGA may have some hard CPU cores, but the same FPGA could not have them and instead have the CPU cores implemented in the FPGA fabric - usually at a much higher power usage and lower speed, though. Programmable logic is a tradeoff after all: you gain flexibility, but lose the performance possible with dedicated logic.

Answer 2

Some hopefully helpful definitions:

• PHY - Physical Interface - this is the electronic circuit that connects the innards of an IC to the physical PCB traces. Usually the PHY includes high speed SERDES (serialiser/deserialiser) and various other gubbins (technical term) used to convert a typically parallel data bus inside the ICs to the serial data bus that is wired up on the PCB.

  All devices connected to the PCIe bus will include a PHY built in to them. There is no need or desire for this to be a seperate chip.

• RC - Root Complex - this is the device which controls the PCIe bus, enumerates all of the other devices, and assigns every device an address on the bus. There is usually only one RC in a system. There is very little difference between an RC and a regular endpoint in terms of hardware, it basically just has lots of extra control registers to coordinate everything.

  For modern computers and embedded systems, it is typically integrated in the CPU die itself. Though you might find examples with a seperate root complex is in a seperate device (chipset/northbridge/etc).

• PCH - Platform Controller Hub - This is not a PCIe term. It is a term coined by Intel, initially to refer to its chipset architecture, where a lot of the peripheral controllers were in a chip seperate to the CPU, with a proprietary interface connecting the two. The PCH would have its own root complex to control any PCIe devices in the system.

  More recent processors integrate this within the CPU itself. The RC moved into the CPU, and the PCH essentially just became simply a PCIe endpoint device with nothing particularly special about it. You may find the CPU and PCH are seperate silicon dies bonded together in the same physical package (system-on-package).

• PCIe Switch - switches are used to connect multiple downstream PCIe devices to a single upstream port. The switch simply routes packets either upstream or downstream depending on what device they are being sent to.

  Switches can get more complicated with features such as NTB (non-transparent bridging) where you can have multiple upstream ports, with multiple root-complex devices, but that is a whole other story. Typically this is used in servers with multiple CPUs to allow devices to be shared between the CPUs.

• PCIe Bridge - As far as I can tell, this is probably seen only in relation to PCI/PCIe interoperability. PCIe to PCI bridges are essentially a form of PCIe switch where the upstream port talks to a PCIe serial link, while the downstream port(s) talk to older parallel PCI devices for backwards compatibility.