A recent study conducted by Purdue University in the United States has shed light on a concerning issue related to the accuracy of ChatGPT’s responses to programming questions. The study’s findings indicated that more than half of ChatGPT’s answers were incorrect, and its advanced language abilities managed to mislead a significant portion of the participants.
Why Relying on ChatGPT May Be Counterproductive
The research team examined 517 programming questions sourced from Stack Overflow and assessed various aspects of ChatGPT’s responses, including correctness, consistency, comprehensiveness, and conciseness. The outcomes of the evaluation were disappointing, as it was revealed that 52% of the provided answers were inaccurate, and a substantial 77% were unnecessarily lengthy. What raised even more concern was the observation that the AI’s eloquent and methodical language style often led the participants astray. Only in cases where the errors were glaringly obvious were the participants able to identify the inaccuracies.
In spite of the incorrect responses, nearly 40% of the participants preferred ChatGPT’s answers. However, a significant 77% of those favored responses turned out to be incorrect. The researchers, including individuals like Samia Kabir, David Udo-Imeh, Bonan Kou, and Assistant Professor Tianyi Zhang, clarified that many errors stemmed from ChatGPT’s inability to grasp the contextual nuances of the questions.
These findings present a compelling argument that current generative AI, in its existing state, may not be a suitable tool for assisting with code generation and might even have counterproductive effects. Acknowledging this reality, various tech giants such as Google, Apple, Amazon, and Samsung have issued warnings or imposed bans on the utilization of generative AI for code suggestions.
According to reports, OpenAI is working on its next GPT iteration GPT-5 which is expected to solve these errors. Expectations include reduced hallucinations, improved multi-modality with text, images, videos, and audio, enhanced computational efficiency, memory, and contextual understanding. GPT-5 could enable more detailed interactions, expand into new domains, and offer a higher number of parameters for more powerful AI content generation.
RELATED:
- OpenAI rolling out new ChatGPT features to enhance user experience
- Here is how AutoGPT is enabling Business Automation and Encouraging Side Hustles
- OpenAI Makes Conversational AI more Accessible with the Launch of the ChatGPT Android App
- OPPO Reno10 vs OPPO K11: Specs Comparison
- OPPO Reno10 Pro vs Nothing Phone (2): Specs Comparison
(Via)
