What Not to Ask ChatGPT For Help With

OpenAI’s ChatGPT excels at tasks like coding, document summarization, brainstorming, and more, but there are areas where the AI chatbot should probably be second-guessed.

ChatGPT has emerged as a popular and widely used tool in the past year, with 200 million weekly active users worldwide as of September 2024, and 77.2 million monthly active users alone within the U.S. The majority of users (62 percent) are college-age or young professionals aged between 18 and 34, according to Statista. But what are people using this AI tool for—and what should they not be using it for?

What not to ask ChatGPT—and why

Important research—you can’t be sure sources are ‘hallucinated’ or not

From college students working on a midterm paper to legal professionals carrying out discovery, many people have been caught out when the sources generated by ChatGPT were, in fact, completely made up.

In 2023 a New York-based lawyer was sanctioned for submitting a legal brief that cited six fictitious cases. The lawyer, in his affidavit, said that he had “never utilized ChatGPT as a source for conducting legal research prior to this occurrence and therefore was unaware of the possibility that its content could be false.”

He must not have read the warning OpenAI has placed directly below the prompt input: “ChatGPT can make mistakes. Check important info.”

Mental health advice—ChatGPT isn’t a substitute

A clinical psychiatry professional put ChatGPT to the test and found that while he was impressed with “how convincingly it reproduced the stock-in-trade responses of a human therapist”, it was surface-level and failed to link a fictional patient’s separate issues into one coherent discussion in order to give holistic advice.

“These responses are textbook 101 for effective therapy: empathize with what the patient may be feeling, validate and normalize the problem, and support good judgment,” said Daniel Kimmel, assistant professor of clinical psychiatry at Columbia University’s School of Psychiatry.

However, Kimmel said it was missing the human element beyond understanding what a patient is saying to what isn’t being said and needs to be teased out, a “process [that] requires an act of creativity, of imagination”.

Current information—It has a knowledge cutoff date

ChatGPT has been called a ‘Google killer‘ and has been touted as a replacement for the online collaborative encyclopedia Wikipedia. However, as OpenAI has clearly stated, the knowledge cut-off for GPT-4o is September 2023 and October 2023 for the newly released o1-preview and o1-mini.

This means its training data includes information until September 2023; any significant developments—be it new scientific discoveries, political shifts, or technology advancements—after this date will not be reflected in its responses unless it actively browses the web.

While it seems as though asking ChatGPT for information is a substitute for a Google web search, the latter trawls the entire (visible) web in real time, indexing a vast array of websites, news, databases, and more.

Original Content—LLMs versus creativity

Generative AI, at its core, is a prediction engine, re-combining existing information from its training data. A 2023 research paper titled ‘On the Creativity of Large Language Models’ (LLM) explored the concept of machine creativity and, while LLMs may not be inherently creative, it acknowledges that tools like ChatGPT can be useful for what they refer to as ‘human-AI co-creativity’.

“While LLMs are capable of value and of a weak version of novelty and surprise, their inner autoregressive nature seems to prevent them from reaching transformational creativity,” say the authors.

“The hard problem in machine creativity is about the self-awareness of the creative process in itself,” they added.

Don’t swap out GPT 4o just yet

Since the newest update for GPT was released last week, Pro subscribers have been able to test it out, with some reviewers praising its superiority in coding and others finding its logic and reasoning abilities much improved on its predecessor.

However, both o1-review and o1-mini are in beta and limited to 30 and 50 messages a week respectively, meaning you won’t get much value from them just yet if you’re a regular ChatGPT user. Additionally, neither of these beta models support file uploads, have a memory, or can work from custom instructions (fed into the settings) like 4o. Have fun and tinker about but don’t expect them to be part of your workflow until the full model is released.

OpenAI CEO Sam Altman has also been clear on its limitations, posting on X (formerly Twitter) that “o1 is still flawed, still limited, and it still seems more impressive on first use than it does after you spend more time with it.”

Newsweek reached out to OpenAI via email for comment.