CodeTransBenchmark: Evaluating LLM-based Code Translation and Repair Across Programming Languages

RESUMO

Large Language Models (LLMs) pre-trained on expansive text and code corpora have revealed promising code generation abilities and attracted increasing attention in translation. In this work, we investigate the effectiveness of LLMs in translation error repair. First, we present CodeTransBenchmark, a framework for evaluating LLM-based translation repair and devise a post-processing strategy to extract code from inconsistent LLM outputs. Then, we discuss an empirical study evaluates eight models on three datasets with 12 language pairs, in which we categorize incorrect translations by errors to identify weaknesses of existing LLMs. Our work shows that while LLMs specifically trained for multi-lingual coding, like Codestral, correctly translate majority of code, most general-purpose models struggle with the syntactic rules of target language. The analysis of erroneous translations reveals the substantial impact of interrelationship between involved programming languages and training data on effectiveness. We show that general post-processing approach must tolerate inconsistencies and leverage the predictability of answers. Further, we show iterative translation repair via automated feedback significantly improves accuracy. While our combined findings highlight the potential of LLMs to automate code translation, an effective deployment in practice would require models with larger context windows.