AIJon: Automated Generation of Annotations for Fuzzing

RESUMO

Modern fuzzers use code coverage as feedback to guide their exploration, which has proven effective for driving exploration. However, this overlooks inputs that may be interesting to the target program even without uncovering new code paths. Prior research has shown that annotations generated by human domain experts can provide additional feedback, guiding the fuzzer towards interesting parts of the program. In this paper, we replicate experiments presented in IJON and extend them to real-world vulnerability detection at scale. To mitigate the scalability challenge imposed by the need for human expertise, we propose utilizing LLMs to automatically generate annotations. We demonstrate the applicability of LLMs for this purpose and observe that LLM-generated annotations perform comparably to human-generated annotations. Motivated by this finding, we design AIJON, a system that leverages LLMs to generate IJON-style annotations. We evaluate AIJON on the Magma benchmark and surprisingly observe that annotation-based fuzzing does not perform strictly better than AFL++. We conduct several experiments to identify causes of our results and identify key insights regarding the impact of annotations on fuzzing campaigns, including their effect on energy distribution in the fuzzer. Notably, we achieve comparable results to ones previously published, thus opening the door for future research and further studies.