Scientists find ChatGPT is inaccurate when answering computer programming questions
A team of computer scientists at Purdue University has found that the popular LLM, ChatGPT, is wildly inaccurate when responding to computer programming questions.
A team of computer scientists at Purdue University has found that the popular LLM, ChatGPT, is wildly inaccurate when responding to computer programming questions.
Meta’s assistant stinks at counting. When you ask it for a five-syllable word starting with the letter w, it will respond with “wonderfully,” which has four syllables. When you ask it for a four-syllable word starting with w, it will offer “wonderful”.
‘I had to remind the tool that I had told it at the start of the chat that “it is crucial that you cite your sources, and always use the most authoritative sources.”’
“Copilot gave up completely, and simply asked to go back to its nap. Claude took issue with the nuance of a few answers. Bard hit hard on a whole slew of answers — but, apparently, to err is not only human, it’s AI as well.”
A new startup, Vectara, says that its company’s research estimates that even in situations designed to prevent it from happening, chatbots invent information at least 3 percent of the time — and as high as 27 percent.
All three platforms provided high rates of inaccurate recommendations. Chatbot ratings for answering patient questions varied, with Bing Chat (Creative) have the highest score and Bing Chat (Concise) having the lowest score.
Notably, Bard’s new “double-check button” has been designed to provide a counter against confabulations where Bard produces inaccurate information or makes things up (a concept often called “hallucinations” in the AI field). It’s a public admission that Bard often lacks accuracy and isn’t a dependable factual reference.