A recent study found that if a coding problem put before ChatGPT (using GPT-3.5) existed on coding practice site LeetCode before its 2021 training data cutoff, it did a very good job generating functional solutions, writes IEEE Spectrum.

But when the problem was added after 2021, it sometimes didn’t even understand the questions and its success rate seemed to fall off a cliff, underscoring AI’s limitations without enough data.