Limitations and Challenges of Language Generation Tools

1. They are difficult to detect. The conversational format and adaptability renders realistic output that is virtually undetectable. A couple of programs have been created to determine the likelihood that text was written by an AI, including GPTZero, but these programs are easily tricked. OpenAI, the company behind ChatGPT, has recently admitted that even they cannot detect AI-generated text (https://www.businessinsider.com/openai-cant-identify-ai-generated-text-…). Even when a detector believes text was generated by AI, it is nearly impossible to prove. It is likely that both the AI language generators and the detection programs will continue to improve with time, but for now, the detectors are unreliable. This means that it is easy for someone to pass off AI generated text as their own. Learn more about AI plagiarism under “Student Uses.”
2. They are prone to bias and exclusionary language. Because the language generators are simply utilizing predictive text based on what they have been taught, they commonly generate ideas that are biased. For example, when asked to write a poem about chemists, the AI may say that women belong in the kitchen. AI may use discriminatory language, provide controversial opinions, or be explicitly racist, sexist, transphobic, etc.
3. They are prone to plagiarism. AI is not trained on academic integrity and is therefore prone to plagiarism without attribution.
4. They do not process information. ChatGPT is not connected to the internet, which means that it has a limited database of information to draw from. This is changing, as language generation tools are beginning to be integrated into search engines. They are not, however, substitutes for actual research. As a result, the AI will often make inaccurate claims or make up citations.
5. They are not libraries. If you ask a language generation tool to provide quotes or citations, they are likely to make them up. This is because all text generated by the AI is predictive—the AI will create a quote that sounds convincing, but it does not have the ability to look up a quote on its own.
6. They are often convincing but wrong. The AI will often generate inaccurate information presented with conviction, making it extremely dangerous for an uninformed user. It is helpful to have prior knowledge of a subject before assessing the quality of the output generated by the AI, and it is easy to be deceived by the output.
7. The output is only as good as the input. The iterative and responsive nature of newer language models, such as ChatGPT, allows the AI to improve its responses over time. If the user knows what information is relevant, it is possible to massage the AI-generated response into something that is well-written, creative, and relevant. However, if the user does not know what information is most important, it is challenging to assess the quality of the output.
8. They are constantly learning. As more people utilize tools like ChatGPT, the AI is “learning” more false information. To learn more, check out this article from Fortune: https://fortune.com/2023/07/19/chatgpt-accuracy-stanford-study/

Although language generation tools have a number of significant limitations, they are highly likely to impact student learning and change the ways in which we assess student learning.