Code Consistency Preference Optimization Verification for Language Model Alignment

RESUMO

Execution-based verification enhances large language models' mathematical reasoning through computational soundness and dependency-aware filtering. However, prior preference optimization methods relying on Bradley-Terry reward models fail to capture the logical dependencies execution consistency needed for scientific tasks. We propose a method that generates computationally sound solutions with dependency graphs execution-consistent preference optimization. We first build a scientific dataset using UltraFeedback prompts, model generations, verification, and results. Then we extract mathematical step expressions, prerequisites, derivability relationships to construct dependency graphs compute consistency scores. These scores are appended to each step, creating paired training data. Fine-tuning Llama-3-8B and DeepSeekMath-7B yields significant gains: +17.0% on MATH and +15.1% on GSM8K. Extending our Scientific Feasibility Control framework achieves 50.1% accuracy on PhyX multimodal physics reasoning, surpassing DeepSeek-R1 (49.8%) and OpenAI o3-mini (48.2%), with 91.7% scientific validity coverage at alpha=0.10 and 73% fewer law violations, resulting in the CCPO model family.