A new framework called Preference Delta Aggregation (PDA) enhances large language models by effectively combining weak supervision signals from model pairs, leading to significant performance improvements. This method, which includes a geometry-aware merging technique, shows promising results in knowledge reasoning and agentic search tasks, outperforming existing models.