TL;DR: No.
The NES CPU was a copy of the 6502
The 6502 section of the Ricoh RP2A03 is a very close (*1) copy as comparing the images show
(Images sourced from Wikipedia (RP2A03) and Visual 6502. Scaled for comparison, not area. Annotation mine.)
with the BCD circuitry removed.
More like disabled.
As I understand it, this modification was motivated by a theory that BCD was the only part of the 6502 that was actually patented, so that by intellectual property law of the early 80s,
Note just a theory, but the very reason.
it was technically legal to directly copy the rest of it.
It's even legal to copy everything, as long as it gets modified in a way making sure it's no working the patented way. In this case the circuitry is mostly still present, but modified to simply let the input value always pass unmodified.
Did they redo the layout
No, they just modified the decimal adjust circuit to always let the input value pass straight through.
to squeeze other parts of the CPU into the vacant space, to save overall area?
Nothing that would really make any sense for multiple reasons:
- To start with, the adjuster is a very tiny piece of logic. Removing it wouldn't yield a noticeable area, as this nice annotation of the data path shows:
(Image taken from these incredible slides about Apple II Assembler Programming by Stephen Edwards)
- It also shows it being squeezed right between ALU and A-register. Removing and sliding either side up, would be a major undertaking for next to no gain.
- In addition the gained space would be in an area where there is already unused space on both sides.
- Lastly, and maybe most importantly, by the time the RP2A03 was designed, the used process not only shrunk the CPU area in half (6 vs. 8 µm) but also allowed way bigger die sizes.
Or did they, for development cost or schedule reasons, just leave it as it was, with gaps where the BCD circuitry had been?
As visible, there is no gap. Investing any amount of time past disabling the adjuster would never have yielded any ROI. The fact that they only disabled the circuitry might be the most definitive trace about how little time was invested.
*1 - So close that even all unofficial opcodes work the same, so even the ROM is unchanged