Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents

RESUMO

Coding agents increasingly read repository knowledge from SKILLs, plain markdown files versioned with the code, and recent work synthesizes these automatically by optimizing against a benchmark; but a bare repository has no benchmark and synthetic tasks saturate too easily. The authors mine harder tasks from merged pull requests reverted at a frozen base commit and score candidate documents by whether the same agent does better with them. On three Kotlin repositories, GEPA-found documents raise the score by 4.9pp on average, while SkillOpt leaves it unchanged; a maintainer found knowledge in the documents one normally only gets from working in the project.