How ChatGPT & Bard Patching Up JavaScript Flaws

Despite JavaScript’s widespread use, writing secure code remains challenging, leading to web application vulnerabilities.

Experiments on real-world vulnerabilities show LLMs hold promise for automated JavaScript program repair, but achieving correct fixes frequently requires providing an appropriate amount of contextual information in the prompt given to the LLM.

The following cybersecurity researchers from Simon Fraser University recently unveiled how ChatGPT and Bard are patching up the JavaScript flaws:–

• Tan Khang Le
• Saba Alimadadi
• Steven Y. Ko

Patching Up JavaScript Flaws

Despite the use of techniques like static analysis and fuzzing, it is still sometimes difficult to understand and analyze programs because of the dynamic, asynchronous nature of JavaScript. 

During the development process, many programmers create vulnerabilities without even knowing them as they try to make their programs secure.

⁤In addition, if used with the right tooling, big language models (BLMs) have the potential to improve developer efficiency while reducing new security bugs. ⁤

⁤However, there has been very little research regarding how effective such LLMs are in terms of identifying and fixing vulnerabilities in JavaScript as compared to C/C++ and Verilog languages.

Due to growing system vulnerabilities, automated program repair has emerged as a research area focused on techniques for generating source code patches. 

Classical approaches include using genetic programming to search for program variants addressing bugs without altering functionality and employing semantic analysis for automatic patch construction.

The rise of large language models (LLMs) has driven major advances in automated program repair.

The accuracy of LLMs at identifying and fixing security bugs in JavaScript is investigated by researchers who study their responses to the repair of a particular “Out-of-bound Write” vulnerability in a given code.

While there may be many different ways to fix this problem, we are interested in finding out whether or not LLMs can come up with a working, secure patch, rather than looking at all possible solutions exhaustively.

By that evaluation, LLM’s capabilities for automated JavaScript bug fixing are revealed.

To study the impact of contextual information on LLMs’ ability to generate repairs, three prompt templates were designed with varying levels of context:–

• Context-free with only repair instructions and vulnerable code
• Context-sensitive adding the expected vulnerability type
• Context-rich including detailed comments explaining the vulnerability and potential exploits

Here, cybersecurity analysts opted for 20 popular JavaScript vulnerabilities that were pinpointed, and three templates with different context levels generated 60 repair prompts. 

The evaluations of how ChatGPT and Bard have performed in automatically fixing these flawed codes showed that ChatGPT had a higher rate of accuracy, reaching 71.66%, compared to the second one, Bard, which had an almost similar percentage of 68.33%. 

⁤These findings demonstrate that large language models are capable of repairing automated JavaScript vulnerabilities and provide evidence of the role played by contextual cues in generating appropriate responses.

Stay updated on Cybersecurity news, Whitepapers, and Infographics. Follow us on LinkedIn & Twitter.