PCIe For Hackers: The Diffpair Prelude
Posted by admin on
PCIe, also known as PCI-Express, is a highly powerful interface. So let’s see what it takes to hack on something that powerful. PCIe is be a bit intimidating at first, however it is reasonably simple to start building PCIe stuff, and the interface is quite resilient for hobbyist-level technology. There will come a time when we want to use a PCIe chip in our designs, or perhaps, make use of the PCIe connection available on a certain Compute Module, and it’s good to make sure that we’re ready for that.
PCIe is everywhere now. Every modern computer has a bunch of PCIe devices performing crucial functions, and even iPhones use PCIe internally to connect the CPU with the flash and WiFi chips. You can get all kinds of PCIe devices: Ethernet controllers, high-throughput WiFi cards, graphics, and all the cheap NVMe drives that gladly provide you with heaps of storage when connected over PCIe. If you’re hacking on a laptop or a single-board computer and you’d like to add a PCIe device, you can get some PCIe from one of the PCIe-carrying sockets, or just tap into an existing PCIe link if there’s no socket to connect to. It’s been two decades since we’ve started getting PCIe devices – now, PCIe is on its 5.0 revision, and it’s clear that it’s here to stay.
PCIe is a point-to-point bus that connect two devices together – as opposed to PCI, an older bus, that could connect a chain of devices on your mainboard. One side of a PCIe link is a device, and another is a host. For instance, in a laptop, your CPU will have multiple PCIe ports – some used to connect the GPU, some used to connect a WiFi card, some used for Ethernet, and some used for a NVMe drive.
Each PCIe link consists of at least three differential pairs – one is a 100 MHz clock, REFCLK, that is (almost) always required for a link, and two pairs that form a PCIe lane – one for transmit and another for receive. This is an x link – you can also have 2x, 4x, 8x and 16x links, with four, eight sixteen and thirty-two differential pairs respectively, plus, again, REFCLK. The wider the link, the higher its throughput!
Now, link widths in PCIe are a fun topic with plenty of cool aspects – but first, it’d be good to make sure we’re on the same page when it comes to what “differential pair” means in context of PCIe. Here on Hackaday, we’ve told you about the nitty-gritty of differential pairs before – absolutely revisit that writeup if you want to learn about differential pairs in depth! Here, I’ll quickly refresh you on the basics, and then tell you what to need to know when working with differential pairs for PCIe specifically.
What’s The Difference?
To put it simply, a differential pair is two signals, one always the opposite of another, one usually referred to as positive and another as negative. You get the logic level of the bit being transmitted by comparing the two signals against each other – instead of comparing each individual signal’s logic level to ground like we usually do, which is referred to as “single-ended”. With a differential pair, signals are close to each other and are even interwoven when it comes to cables, and as a result, any interference affects the signals equally – as signals are compared to each other to receive information, this means that the information received is not affected by noise overlaid onto both of the signals. Differential pairs also make both signals’ magnetic fields cancel each other out, resulting in the link being less noisy.
As a result, differential pairs let you crank the transmission speed up without creating noise or becoming susceptible to noise. The overwhelming majority of high-speed interfaces, as a result, use differential pairs: Ethernet, PCIe, HDMI, DisplayPort, LVDS, and even USB, although USB 2 is only pseudodifferential, USB 3 is truly differential. Resillient interfaces like RS485 and CAN use differential pairs as well. It’s easy for a hobbyist to start with differential pairs with interfaces like CAN, and USB 2 also poses no problem – at short distances, these will work no matter what, despite being differential signals and theoretically requiring special treatment.
That said, differential pairs do in fact require a bit more care when routing a PCB or putting them through cables. If you don’t take care, you risk mysterious glitches or interfaces outright not working. Let’s go through these requirements.
Treating Your Diffpairs With Respect
First off, you want to keep both of the pair’s signals close to each other throughout their length. The closer the two signals are, the better external interference cancellation works, and the less noise they radiate – given that often, multiple diffpairs run next to each other, this will help signal integrity of other pairs as well. Speaking of running separate diffpairs next to each other, you’ll want to keep them away from each other and other things – be it ground fills on the same layer, high-frequency signals. A great rule of thumb is the 5W rule, which says you need to have at least five trace width’s worth of clearance between a diffpair’s trace center and other signals. You don’t always have this much space, but it’s good to adhere to this as much as possible.
You will also want to make sure that there is an uninterrupted ground path right under these signals, alongside the entire pair – having a ground fill is ideal. Although the two signals are compared to each other by the receiver, each signal still behaves as a single-ended signal with respect to return current. Plus, a bit of extra shielding certainly doesn’t hurt. If you’re putting PCIe on a custom connector, make sure there’s at least one GND pin between each pair. Last but not least – make sure that both tracks within a pair have the same length. If your pair changes its angle, one of its two tracks becomes ever so slightly shorter, and your PCB editor diffpair tool ought to let you add a ‘wiggle’ to that track to make sure that both track lengths are equal.
Then, there’s the little-talked-about matter – impedance matching. If you’re getting a differential pair from point A to B, you will want to make sure that you get the impedance right, and the basics of it are simpler than you might think. Impedance is like resistance, but for signals that change. Each part of the differential pair’s travel path has its own impedance: the receiver and the transmitter inside the ICs used, the IC pins, PCB traces, and any connectors or cables in between if you put the differential pair through these. At any point where impedance of the signal changes, some part of the signal is reflected from the mismatch point, and if the impedance change is significant enough, this will screw with your signal as it’s being received.
Now, this means that you have to make sure the impedance for your PCIe link is good along its entire path – which, in practice, means picking suitable connectors and tuning your PCB trace widths and spacings. PCIe hardware is mostly built with 85 Ω impedance in mind. Things like receivers, transmitters, and PCIe-intended connectors are outside your control, and to get the impedance of the entire path is reasonably uniform, you have to adjust the parts under your control to the same value. For a start, if you have to use connectors for your PCIe link, pick ones that don’t have too significant of an impedance mismatch. A good bet is using high-speed connectors or connectors built with PCIe-like signals in mind – full-size PCIe, M.2, mPCIe, USB3, USB-C, and a lot of high-speed connector families from various manufacturers.
Now to tuning the impedance of your diffpair’s PCB traces. Differential pair impedance depends on a lot of variables in reality, but if you’re a hacker starting out, there are simplified calculators that get you most of the way there – this one is my favourite. Scroll down to “Edge-Coupled Surface Microstrip”, leave track height at 35 for routing diffpairs on 1 oz copper layers, leave dielectric constant at 4.3 unless your PCB fab gives you a different value. Then, set isolation height to the distance from your diffpairs – to get that, go to your PCB manufacturer’s info and look for the PCB stackup information. Say, your diffpairs are on the top layer and the ground is on the layer right under them. For that, look for “prepreg” thickness between the top copper layer and the layer under it – that value will be your isolation height. Then, play around with track widths and spacings, aiming for 85 ohm differential impedance. The spec does give you a range from 70 to 100 ohms, even!
Practical exercise – let’s look at OSHPark’s 4-layer stackup. Its dielectric constant (dk) is 3.6, and our minimum trace width and spacing are both 5 mil, which is 0.127mm, or 127 μm for the calculator’s purposes; the prepreg thickness is 202 μm. Punch the dielectric constant and prepreg thicknesses into the calculator, then play with values.
You will find that increasing the track width decreases the impedance, and so does lowering the track spacing – set that one to the minimum possible. As you will see, if you choose to stick to 85 ohms, you can go for either 0.3/0.127 (width/spacing) pairs and get 84.8 ohms – way more than close enough. if you can’t afford such wide tracks, use 0.2/0.127 to get to 106 Ω – a bit outside of the recommended range, but if you must, it is alright too!
Last thing – keep your routing clean. Don’t put the differential pairs through vias to different layers if you can help it – each pair of vias adds some inductance to the signal, which can interfere with the high-speed signals. Usually, the end and start points of your PCIe link are both on the top layer – keep it this way as much as you can. If you must switch layers, add a few ground-connected vias near the diffpairs. Also, keep other high-speed and rapidly changing or noisy signals as far away from the differential pairs as you can. If you have high-power, differential-pair and single-ended connections in your projects, layout the differential pairs first.
So, five important things – route diffpairs with signals close to each other, keep ground under them, use proper connectors, adjust differential track width and spacing for the PCIe impedance, and keep your routing clean. These are the basics – it’s what you’re expected to do if you want your differential pairs to serve you well.
The Wet String Conundrum
Now, if you have ever tinkered with PCIe, you might have stumbled upon some forbidden knowledge: in practice, you don’t really-really have to do all of the above.
You might have heard that PCIe runs over wet string – the first known reference to this is in a 2016 presentation on console hacking at 33C3. This is the hacker-bravado way of talking about PCIe – you can do wrong by a lot of the aforementioned guidelines when connecting PCIe devices together, and it will still chug along. And, unsurprisingly, there’s a big grain of truth – PCIe will still work in suboptimal conditions, and there’s an example after example of it in hacker and consumer worlds! Perhaps the most widely available example of PCIe abuse is passing an 1x PCIe link using USB3 cabling, something the “mining” PCIe risers do – which means that you can just go to your computer accessories store and buy a product that is only possible thanks to some PCIe abuse.
Something else that you might’ve seen and forgotten like a bad dream, is [TobleMiner] putting a x8 PCIe link through, shudder, prototyping wires – for the sake of testing out an adapter idea for cheap high-speed networking cards from HP servers, not compatible with regular PCIe slots both pinout-wise and mechanically. That prototyping setup let him design a proper version of the adapter, that we’ve later covered here on Hackaday! You can put a PCIe link through an FPC for a quick and dirty board-to-board connection, eGPU extenders have also used HDMI cables for this, and you can likely get it working with magnet wire. Here’s an experiment from Linus Tech Tips where they kept stacking PCIe extenders, and reached a five meter-long chain before the connection started becoming unstable.
PCIe is quite a bit more forgiving than quite a few other interfaces, say, USB3. There are link training mechanisms – when a PCIe connection is established, the receiver and transmitter play around with their internal parameters, adjusting them until they reach the fastest speed possible while keeping error rate low, using these parameters for the entire connection afterwards. There are also retransmissions for packets that failed to be received. PCIe has exceptional stability in practice.
It’s clear that PCIe link training has some unique parts to it – for instance, to help you make your layout better, PCIe also lets you invert any differential pair, except REFCLK, by swapping the negative and positive signals, and this will be detected and flawlessly compensated for during link training. Other technologies like USB3, HDMI, or DisplayPort don’t support such quality-of-engineer-life features. Other interfaces often require that multiple lanes should be the same length – making sure that data on one set of pairs doesn’t arrive faster than on the other. PCIe, however, is fine with across-pair mismatches as well, also detecting and compensating for these during link training. These two aren’t meant to be resilience features as much as they’re ease-of-layout features meant to help you design PCBs faster and better, but it sure helps that they’re there.
Try Your Best, No Matter What
Does this resillience help hackers? Yes, absolutely – these two ease-of-layout features are used in basically any professional PCIe design, and if you’re in less sterile conditions, you can push PCIe further at your own risk. On the other hand, don’t just skirt every rule because you’ve seen someone do that – put some good-faith effort into following these five guidelines, even if you’re limited to a two-layer PCB and might never get the perfect impedance value. Following these rules will not only teach you some diffpair discipline for later projects, it will make your PCIe signals all that more resillient and error-free, and your PCIe devices more happy. It might feel good to dismiss all or some of these guidelines, since sometimes it might just work out, but the extra half hour calculating proper impedance on your board will help you ensure that your PCB doesn’t need a second revision and stays loyal to your interests throughout its entire life.
So, here’s a guideline: treat your PCIe differential pairs with respect. If you’re using a two-layer PCB and you’re doing a prototype on the cheap and you want quick turnaround time, don’t just give up on impedance because the traces would need to be way too wide to reach 85 ohms – open the calculator and see just how much you can get the impedance down anyway. Lowering isolation height lowers impedance, so consider going for 0.8mm PCB if your project’s mechanical aspects let you. Move your components around if that helps your PCIe tracks follow a better path, with less noise along the way. Perhaps link training will knock an imperfect link down a generation or two, but that’s better than not reaching a stable link at all. Put your best effort following these guidelines with what you’re given, and the differential pairs will respect your intentions in return.
For instance, if you’re using KiCad, here’s a simple demonstration on how to get a PCIe 1x link from one point to another, routing differential pairs while taking care of impedance, clearances, and via stitching.
Now, you see what it takes to route PCIe differential pairs on a board, and these guidelines will apply to all kinds of other differential pair-based interfaces. Next time, I will tell you more about PCIe signal meanings, link widths and throughput – the basics, as well as all the pleasant surprises PCIe can offer you. And, if you’re looking to go deeper into what makes PCIe tick, check out this earlier writeup of ours – it’s just the thing if making your own PCIe devices with FPGAs is what you’re looking for!