Revolutionizing Decompiler Evaluation: Meet ExeBench-FHD, the New Standard for Fidelity in Code Recovery

In the rapidly evolving field of software decompilation, understanding the performance of algorithms is vital. A recent study introduces ExeBench-FHD, a groundbreaking revision of the popular ExeBench benchmark, designed to tackle long-standing fidelity issues in decompiler evaluation. This innovation aims to enhance the accuracy with which decompilers recover source code from compiled binaries.

Understanding the Impact of Fidelity Issues

The original ExeBench framework has been widely utilized to evaluate large language model (LLM)-based decompilers. However, it has been found to contain several significant fidelity issues that hinder accurate assessment. Among these, missing information crucial for proper evaluation included ambiguous parameter orders and the absence of constants vital for the function's operation. A total of nine issues were identified, which could mislead the performance evaluation of decompilers.

ExeBench-FHD: A Step Forward

ExeBench-FHD not only identifies these issues but also introduces solutions. The enhanced benchmark comprises 16 new fields, including call-site assembly and data sections that provide previously omitted context. This additional information plays a crucial role in guiding LLMs to better understand and replicate original code structures.

Key Improvements and Results

One of the standout features of ExeBench-FHD is that it has been designed to be backward-compatible with the original ExeBench. This means researchers and developers can adopt the new fields incrementally, thereby preserving existing evaluation pipelines while enhancing them with added fidelity.

An evaluation of ExeBench-FHD demonstrated a significant improvement in the performance of the decompiler, GPT 6 Sol. The new fields increased the re-execution rate by up to 26.5% for functions that involved ambiguous parameter types and unused parameters. This increase highlights the potential of ExeBench-FHD to provide a more accurate benchmark, facilitating more effective training and testing for decompilers.

Fostering Future Research

The introduction of ExeBench-FHD is likely to reshape how researchers approach the task of code recovery through binary decompilation. By setting a new standard for fidelity in testing, this framework encourages a more rigorous evaluation of decompilation algorithms. The study serves as a clarion call for the community to prioritize evaluation accuracy in future benchmarks, leading to advancements that may ultimately impact various fields, including cybersecurity and software maintenance.

In summary, ExeBench-FHD represents a profound leap forward in the evaluation of software decompilers, ensuring that performance assessments are based on high-quality benchmarks that accurately reflect real-world scenarios.

For detailed analyses and dataset availability, researchers can refer to the comprehensive study led by Akarsh Ashok, Dinil Mon Divakaran, and Vivek Balachandran from the Singapore Institute of Technology and A*STAR Institute of Advanced Intelligence and Computing.