A new study reveals that small language models struggle with rare tasks because frequent tasks dominate their learning. To improve performance, increasing the frequency of target tasks in training data may be more effective than simply scaling up model size.