How to find the best information sources
Incentives, methodology, and social trust all are important to be well-informed
Quick summary:
There are three primary considerations for selecting sources to be well-informed
First, look to see what the incentives are that motivate a source to provide info
Second, investigate what methodology a source uses to produce its information
Third, look to trusted social networks to see which sources they rely upon for info
Misinformation is hard to discern; we rely on others’ expertise to assess accuracy
Who we trust and the information sources they use greatly influence our choices
Where do we turn to for good information?
In my last article, I wrote about the four stages of the information cycle that fuel data-driven decision-making in organizations. I was in the airport this week traveling to a conference where I was speaking, and I was surprised to see AI in a headline in my local newspaper (right above the one about our annual rodeo, complete with a photo of a cattle drive). While ChatGPT has dominated many news stories this year, I was shocked to see it leading above the fold for a general-interest city newspaper. The article focused on the potential use of ChatGPT on college campuses locally and how professors were planning to address concerns about grading and plagiarism.
Historically, most people have relied on mainstream news organizations for information and to broaden their knowledge base about the world. Seeing two front-page articles about ChatGPT and the San Antonio Rodeo and Stock Show adjacent to each other wouldn’t have seemed as jarring in prior decades: this was how information was packaged and presented to people. However, this sort of non sequitur is increasingly uncommon today. I trust the local paper to provide reliable information about the upcoming rodeo. I have less faith in its ability to deliver a thorough article about the pros and cons of using ChatGPT in schools and universities. But why is this? The reporters and editors for both stories are likely using the same methods and standards. The issue comes down to trust: I expect that a local newspaper is a good source of information on local events, but less so for cutting-edge technology and its implications on society.
Think like a scientist when selecting info sources
In the book The Misinformation Age: Why False Beliefs Spread, authors Cailin O’Connor and James Own Weatherall outline models of how information is consumed, how it spreads through social networks, and how that process changes our beliefs (or, in the case of polarization, hardens them). The early chapters focus on models of how scientists form opinions because the authors contend that scientists, more than the general population, seek truth and believe that the process by which information is gathered matters. A fraction of the information scientists seek is from their research and experiments, and a much larger part is from the scientific community, both directly and indirectly. Scientists work alongside colleagues at their workplaces and collaborate with peers from other institutions. These collaborations yield direct sources of additional information that are valued. Significantly, the value placed on the data and findings depends on the trust level that a scientist puts in their colleagues. Trust depends on many factors, but one primary factor is the incentives that are in place for each scientist. Some incentives are related to the job, such as securing research grants, and some are institutional goals and objectives. For example, research funded by individual firms or industry groups may differ from that performed within a university or government laboratory.
Scientists make different choices about how to approach their research; they make decisions about the appropriate methodology to use. Many involve small but essential choices: often, there are tradeoffs in terms of cost, time, and quality. When a consensus forms among a group despite differences in approach, the diversity strengthens the findings. Conversely, debate ensues when there is no consensus, and further studies are performed to help settle the disagreements and inform decision-making in the future. Knowing how differences in methodology are used to develop data and information affect scientific findings helps narrow down the areas in question where more digging is needed to get closer to the truth.
Who you trust matters as much as what you read
Scientists are experts in their field, but what they possess in terms of depth of understanding is at the cost of breadth. Even within a particular discipline, the branches are so varied that “it takes a village” to further our base of knowledge of our world. So scientists rely heavily on academic journals and publications to develop their expertise. Importantly, they do not rely on any source of information: scientists trust peer-reviewed journals that publish studies with citations to other work and describe the research in a replicable manner. (This is more of an ideal than is commonly practicable; rarely is someone else’s study fully replicated.) For many researchers, particularly in academia, getting published in a peer-reviewed journal, often cited by other works, is the gold standard they strive to achieve. The entire process of “publish or perish” is a form of a trusted social network that scientists have as a community. More than any single source of information, scientists trust each other based on their shared values.
Most of us are not scientists, and we do not have the time, patience, or intellectual horsepower to make it through peer-reviewed scientific papers. However, we can approach what we choose as our sources of information similarly. For any given source of information, it is crucial to understand their incentives.
Are they subscription-based or free?
Do they take advertising?
Are they supported by a large institution such as a corporation, industry association, government, or special interest?
In short, how do they make money?
In addition to incentives, examining the methodology a source uses to produce information is vital.
Do you know if this information is original or passed along from other sources?
How did they approach the collection of the data contained in this information?
Has this information been fact-checked and peer-reviewed?
If this information is later proven incorrect, will they edit or retract it?
Does this information reflect a set ideology or appear to be more balanced?
Finally, consider your social network in your personal and professional lives. Which people do you trust the most? If there are groups of people you trust on a particular topic, do you rely on them about any subject? What are your limits or checks to verify the accuracy when someone relays or comments on information? A common mistake is trusting people in your circle too much: while they may have proven reliable on a particular topic, they are likely not correct about everything. It is an easy mental shortcut to take - I trust everything X person says because they have been reliable in the past - but avoid the temptation to outsource your thinking. Look to build up your own independent sources of information, then compare notes with colleagues to see where you agree and disagree, just as scientists do with their compatriots.