Smart Contract Auditors Remain Secure in Their Jobs, for Now

In Brief

• While AI can aid in identifying security vulnerabilities within smart contracts, it cannot yet supplant human auditors.
• GPT-4 managed to conquer 19 of the 23 pre-September 2021 challenges, yet encountered difficulties when faced with more recent stages.
• DeFi protocols accounted for a staggering 82.1% ($3.1 billion) of cryptocurrency plundered in the year 2022.

ChatGPT-4's attempts to detect vulnerabilities in smart contracts fell short on several counts. This underscores the fact that while artificial intelligence (AI) can be instrumental in identifying security flaws, it is not yet capable of replacing human auditors.

Smart contracts serve as the foundation of decentralized finance (DeFi), emphasizing the critical need for their flawless composition. OpenZeppelin's recent experiment provides reassurance to smart contract auditors, affirming that their jobs are secure and not threatened by artificial intelligence (AI).

AI Cannot Yet Fix Smart Contracts

Ethernaut, a captivating game centered around hacking, offers a series of smart contract levels. It serves as an educational platform for individuals to delve into the workings of Ethereum and put their hacking abilities to the test by tackling past exploits. Smart contracts, in essence, are coded agreements that execute themselves based on predetermined conditions.

Before its training data cutoff in September 2021, GPT-4 displayed commendable prowess by successfully conquering 19 out of the 23 challenges presented by Ethernaut. However, when faced with the latest levels introduced by Ethernaut, GPT-4 encountered difficulties and fell short on 4 out of 5 challenges.

These outcomes effectively underscore that while AI can be instrumental in detecting certain security vulnerabilities, it cannot fully substitute human auditors.

According to the audit report, it is acknowledged that ChatGPT demonstrates strong performance when provided with specific guidance. However, it is crucial to note that the effectiveness of this guidance relies on the comprehension and expertise of a security researcher. This highlights the immense potential for AI tools to enhance audit efficiency, particularly in situations where auditors possess a clear understanding of the desired criteria to investigate and how to effectively prompt large language models like ChatGPT.

It should be noted that ChatGPT was not specifically trained to detect vulnerabilities. A machine learning model that is exclusively trained on high-quality vulnerability-detection datasets is likely to achieve better results in that area.

During the experiment, the code for each level of Ethernaut was provided, and GPT-4 was asked whether the smart contract contained a vulnerability. GPT-4 successfully provided solutions for several levels, such as Level 2 ("Fallout"), where it identified a vulnerability in the constructor function and proposed a fix.

The ability of GPT-4 to solve older levels could be attributed to its training data, which might have included solution write-ups for those particular levels. However, its difficulty in solving newer levels suggests that its performance may have been influenced by the training data it received.

Additionally, the experiment observed the impact of GPT-4's default setting for "temperature," which determines the randomness of its responses.

The significance of smart contracts in the blockchain ecosystem often presents challenges. By default, the code within smart contracts that govern DeFi protocols is publicly accessible.

On one hand, this transparency enables users to have full visibility into the actions involving their funds. However, it also opens up the possibility for malicious hackers to identify vulnerabilities and exploit them.

According to Chainalysis, the percentage of cryptocurrency stolen by hackers from DeFi protocols was 73.3% in 2021. Alarmingly, this number rose to 82.1% in 2022, amounting to a total of $3.1 billion in losses.

Approximately 64% of these losses originated from cross-chain bridge protocols that rely on smart contracts to transfer funds across different chains. Even a single weakness within these protocols can result in the loss of billions of dollars in user funds.