A new position paper advocates for the development of 'data probes' to better understand how different data characteristics impact the performance of large language models (LLMs). This approach aims to provide systematic methodologies for data analysis, potentially leading to foundational insights in LLM training and inference processes.