CodeBERT edges out BERT rivals in controlled defect-detection benchmark,

A new open-access study in the International Journal of Data Science and Analytics compares four pre-trained transformer models—BERT Base, DistilBERT, RoBERTa, and CodeBERT—using a single software defect prediction benchmark and identical training conditions. Researchers Nasraldeen Alnor Adam Khleel and Károly Nehéz fine-tuned all models with repeated runs to assess stability, aiming to solve a common problem in prior literature: results that aren’t directly comparable due to different datasets, preprocessing, hyperparameters, and evaluation protocols. On the Devign dataset from CodeXGLUE, CodeBERT produced the best average scores, but none of the differences were statistically significant. The team filtered out functions with 512 characters or more, reducing the dataset from 27,318 samples to 8,500 (6,794 train, 866 validation, 840 test). The retained data showed 57.6% non-defective and 42.4% defective examples.