How can I tell if a Generative AI system (chatbot) got its information from a specific website or other source?

 

 

AIs are often unreliable in identifying the source of the information they provide. Frequently, the chatbot’s response is a lightly edited version of a single source, even if the chatbot claims otherwise. While these chatbots often will say they compile info from multiple sources, there are times when this is inaccurate and the answer is in fact derived from a single source. 

If the information comes from a single source, it's best to be able to view and cite that source, not the chatbot.

Ways to guess or investigate whether information may have come  from a particular single source:

  • Two or more chatbots produce nearly identical paragraphs
  • A more efficient way of discovering if this has happened (and of consulting the original source!) is to paste language from the answer into a search engine like Google. For instance, you may find the web page where the information was taken from. 

Some AIs will always cite their sources; others will do it sometimes; some AIs have "scholarly" versions that are meant to cite their sources. Each AI chatbot has different levels of reliability, and different prompts will get different results. They all work different ways. (Harvard faculty, staff and students may request a consultation with a librarian to review the chatbot you’re interested in.)


It is important to verify these sources. In addition to sometimes being set to speculate, AIs are only as correct as their training data, and any part of that data can be contradictory, incorrect, or out of date. While AIs are amazing, and can revolutionize our work, it's important to be aware of both their features and limitations.

Service Alert:
  • Ask a Librarian, including chat and email, will be suspended from 5:00 pm on Thursday, December 22, through Monday, January 2, for the holiday break. Any questions received during this period will be answered beginning Tuesday, January 2, 2024.
  • If If you're experiencing an ongoing technical issue when you attempt to access library materials with your HarvardKey during these times, please report it to Library Technology Services.

Chat

 

Monday-Thursday 9am-9pm

Friday-Saturday 9am-5pm

Sunday 12noon-7pm

Chat is intended for brief inquiries from the Harvard community.

Meet

Talk to a librarian for advice on defining your topic, developing your research strategy, and locating and using sources. Make an appointment now.

These services are intended primarily for Harvard University faculty, staff and students. If you are not affiliated with Harvard, please use these services only to request information about the Library and its collections.