FactoryLLM introduces an open-source AI platform for evaluating large language models in smart factories. This tool addresses the challenge of fault diagnostics by enabling effective document reasoning across interconnected machines. Its safe environment allows experimentation without risking sensitive data, potentially enhancing operational efficiency in manufacturing. The positive results from initial tests indicate strong potential for broader application in the industry.