5.2 Finding and Evaluating Research Sources
Scholarly articles published in academic journals are usually required sources in academic research essays; they are also an integral part of engineering projects and technical reports. Many technical projects require a literature review, which collects, summarizes and sometimes evaluates the work of researchers in a field whose work has been recognized as a valuable contribution to the “state of the art.” However, they are not the only kind of research you will find useful. Since you are researching in a professional field and preparing for the workplace, you will draw upon many credible kinds of sources in a professional context.
Table 5.2.1 lists several types of sources you may find useful in researching your projects. You should also consult Seneca Libraries Writing and Communicating Technical Information: Key Resources, where you will find key databases for technical research.
| [Skip Table] |
|
| Source Type | Description |
|---|---|
| Academic Journals, Conference Papers, Dissertations, etc. |
Scholarly (peer-reviewed) academic sources publish primary research done by professional researchers and scholars in specialized fields, as well as reviews of that research by other specialists in the same field. For example, the Journal of Computer and System Sciences publishes original research papers in computer science and related subjects in system science; the International Journal of Robotics and Animation is one of the most highly ranked journals in the field. |
| Reference Works |
Specialized encyclopedias, handbooks, and dictionaries can provide useful terminology and background information. For example, the Kirk-Othmer Encyclopedia of Chemical Technology is a widely recognized authoritative source. You may cite Wikipedia or dictionary.com in a technical report, but be sure to compare the information to other reliable sources before use. |
| Books
Chapters in Books |
Books written by specialists in a given field usually contain a References section that can be very helpful in providing in-depth context for your ideas. For example, Designing Engineers by Susan McCahan et al. has an excellent chapter on effective teamwork |
| Trade Magazines and Popular Science Magazines |
Reputable trade magazines contain articles relating to current issues and innovations; therefore, they can be useful in determining what is “cutting edge,” or finding out what current issues or controversies are affecting the industry. Examples include Computerworld, Wired, and Popular Mechanics. |
| Newspapers (Journalism) |
Newspaper articles and media releases offer a sense of what journalists and people in industry think the general public should know about a given topic. Journalists report on current events and recent innovations; more in-depth “investigative journalism” explores a current issue in greater detail. Newspapers also contain editorial sections that provide personal opinions on these events and issues. Choose well-known, reputable newspapers such as The New York Times. |
| Industry Websites (.com) |
Commercial websites are generally intended to “sell,” so you have to select information carefully, but these websites can give you insights into a company’s “mission statement,” organization, strategic plan, current or planned projects, archived information, white papers, technical reports, product details, costs estimates, etc. |
| Organization Websites (.org) |
A vast array of .org sites can be very helpful in supplying data and information. These are often public service sites and are designed to share information with the public. |
| Government Publications and Public Sector Websites (.gov/.edu/.ca) |
Government departments often publish reports and other documents that can be very helpful in determining public policy, regulations, and guidelines that should be followed. Statistics Canada, for example, publishes a wide range of data. University websites also offer a wide array of non-academic information, such as strategic plans, facilities information, etc. |
| Patents |
You may have to distinguish your innovative idea from previously patented ideas; you can look these up and get detailed information on patented or patent-pending ideas. |
| Public Presentations |
Public consultation meetings and representatives from industry and government speak to various audiences about current issues and proposed projects. These can be live presentations or video presentations available on YouTube or TED talks. |
| Other |
Can you think of some more? (Radio programs, podcasts, social media, etc.) |
The importance of critically evaluating your sources for authority, relevance, timeliness, and credibility cannot be overstated. Anyone can put anything on the internet; and people with strong web and document design skills can make this information look very professional and credible—even if it isn’t. Since much research is currently done online, and many sources are available electronically, developing your critical evaluation skills is crucial to finding valid, credible evidence to support and develop your ideas. In fact, this has become such a challenging issue that there are sites like this List of Predatory Journals that regularly update its online list of journals that subvert the peer review process and simply publish for profit.
Knowledge Check
Mark Twain, supposedly quoting British Prime Minister Benjamin Disraeli, famously said, “There are three kinds of lies: lies, damned lies, and statistics.” On the other hand, H.G. Wells has been (mis)quoted as stating, “statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write” (Quora, n.d.). The fact that the actual sources of both of these “quotations” are unverifiable makes their sentiments no less true. The effective use of statistics can play a critical role in influencing public opinion as well as persuading in the workplace. However, as the fame of the first quotation indicates, statistics can be used to mislead rather than accurately inform—whether intentionally or unintentionally.
When evaluating research sources, be careful to critically evaluate the authority, content, and purpose of the material, using questions in Table 5.2.2. You may also want to view the brief Seneca Libraries video, Evaluating Websites.
| [Skip Table] |
|
| Authority Researchers Authors Creators |
Who are the researchers/authors/creators? Who is their intended audience? What are their credentials/qualifications? What else has this author written? Is this research funded? By whom? Who benefits? Who has intellectual ownership of this idea? How do I cite it? Where is this source published? What kind of publication is it? Authoritative Sources: written by experts for a specialized audience, published in peer-reviewed journals or by respected publishers, and containing well-supported, evidence-based arguments. Popular Sources: written for a general (or possibly niche) public audience, often in an informal or journalistic style, published in newspapers, magazines, and websites with a purpose of entertaining or promoting a product; evidence is often “soft” rather than hard. |
|---|---|
|
Content |
Methodology What is the methodology of their study? Or how has evidence been collected? Is the methodology sound? Can you find obvious flaws? What is its scope? Does it apply to your project? How? How recent and relevant is it? What is the publication date or last update? |
|
Data Is there sufficient data here to support their claims or hypotheses? Do they offer quantitative and/or qualitative data? Are visual representations of the data misleading or distorted in some way? |
|
|
Purpose |
Why has this author presented this information to this audience? Why am I using this source? Will using this source bolster my credibility or undermine it? Am I “cherry-picking” – using inadequate or unrepresentative data that only supports my position, while ignoring substantial amount of data that contradicts it? Could “cognitive bias” be at work here? Have I only consulted the kinds of sources I know will support my idea? Have I failed to consider alternative kinds of sources? Am I representing the data I have collected accurately? Are the data statistically relevant or significant? |
Given the pie chart in Figure 5.2.2, if you only consulted articles that rejected global warming in a project related to that topic, you would be guilty of cherry-picking and cognitive bias.
Beware of Logical Fallacies
Logic has been studied and taught for millennia. We use logic to convince others. A logical fallacy is an error in reasoning within an argument when trying to explain something or persuade someone. This differs from a factual error, which is simply being wrong about the facts. Logical fallacies are flawed statements that often sound true. The word “fallacy” derives from the Latin word fallere meaning, “to deceive, to trip, to lead into error or to trick.” The word also derives from the Greek phelos, meaning “deceitful.”
Logical fallacies are often used to falsely strengthen an argument but if the reader detects them, the argument can backfire, and damage the writer’s credibility.
Logical Fallacies in Aviation:
In the aviation industry, logical fallacies can arise in various contexts, such as in safety assessments, decision-making processes, and communication. Some common logical fallacies that can occur include:
- Appeal to Authority: This fallacy occurs when an argument is accepted as true simply because an authority figure or expert says it is true. In aviation, this could manifest in decisions being made based solely on the opinion of a senior pilot or manager without considering other factors or evidence.
- False Cause: This fallacy assumes that because one event follows another, the first event caused the second. For example, attributing the success of a safety program to a specific action taken, without considering other possible contributing factors.
One example of the False Cause fallacy in the aviation industry could be attributing an increase in flight delays to the implementation of a new aircraft maintenance procedure, without considering other potential factors that could be causing the delays. Just because the new procedure was introduced around the same time as the increase in delays does not necessarily mean that it caused the delays. Other factors, such as weather conditions, air traffic congestion, or scheduling issues, could also be contributing to the delays. Without considering these other factors, attributing the delays solely to the new maintenance procedure would be a false cause fallacy.
The False Cause Fallacy and Ancient Rome
The false cause fallacy is not new. In ancient Rome, sacred chickens were consulted about major undertakings. A widely known story was how Publius Claudius Pulcher (consul, 249 BC) consulted the chickens before a planned naval battle in the First Punic War. The chickens were not eating their feed, which was viewed as a bad omen. Claudius, enraged, must have said something (in Latin) along the lines of “If they will not eat, let them drink!” and had them thrown into the sea. The battle went badly for Rome, resulting in the loss of the fleet. People of the time saw the correlation and assumed causation was involved: Pulcher narrowly escaped with his life. Few people today base major decisions on chickens.
Correlation does not mean causation, but it can give a nudge that the items should be examined more closely to see if there is a functional relationship (Spafford et al., 2023).
Image: (The Sacred Chickens of Rome, n.d.)
- Hasty Generalization: Drawing a conclusion based on insufficient evidence. For instance, assuming that a particular aircraft model is unsafe because of a single incident, without considering the broader safety record of the aircraft type.
- False Dilemma: Presenting a situation as if there are only two possible outcomes or options when in reality there are more. For example, arguing that either safety or profitability can be prioritized, when in fact both can be achieved with proper planning and management.
- Sunk cost fallacy. The sunk cost fallacy is the tendency to continue investing in a course of action because of the resources (time, money, effort) already invested, even when it is no longer rational to do so. For example, a maintenance team may continue to repair an aircraft despite evidence that it is no longer safe to fly, simply because of the time and money already spent on repairs.
- Base Rate Fallacy: Base rate fallacy, also known as base rate neglect or base rate bias, is a cognitive bias where individuals tend to ignore general information (base rates) in favour of specific information, even when the general information is more relevant for making a decision or judgment.
Recognizing and addressing these logical fallacies is important in the aviation industry to ensure that decisions are based on sound reasoning and evidence, ultimately contributing to improved safety and operational effectiveness.
We all have biases when we write or argue; however, when evaluating sources, you want to be on the lookout for bias that is unfair, one-sided, or slanted. Consider whether the author has acknowledged and addressed opposing ideas, potential gaps in the research, or limits of the data. Look at the kind of language the author uses: is it slanted, strongly connotative, or emotionally manipulative? Is the supporting evidence presented logically, credibly, and ethically? Has the author cherry-picked or misrepresented sources or ideas? Does the author rely heavily on emotional appeal? There are many logical fallacies that both writers and readers can fall prey to (see Table 5.2.3). It is important to use data ethically and accurately, and to apply logic correctly and validly to support your ideas.
| [Skip Table] | |
| Bandwagon Fallacy |
Argument from popularity – “Everyone else is doing it, so we should too!” |
|---|---|
| Hasty Generalization |
Using insufficient data to come to a general conclusion. E.g., An Australian stole my wallet; therefore, all Australians are thieves! |
| Unrepresentative Sample |
Using data from a particular subset and generalizing to a larger set that may not share similar characteristics. E.g., giving a survey to only female students under 20 and generalizing results to all students. |
| False Dilemma |
“Either/or fallacy” – presenting only two options when there are usually more possibilities to consider E.g., You’re either with us or against us. |
| Slippery Slope |
Claiming that a single cause will lead, eventually, to exaggerated catastrophic results. |
| Slanted Language |
Using language loaded with emotional appeal and either positive or negative connotation to manipulate the reader |
| False Analogy |
Comparing your idea to another that is familiar to the audience but which may not have sufficient similarity to make an accurate comparison E.g., Governing a country is like running a business. |
| Post hoc, ergo prompter hoc |
“After this; therefore, because of this” E.g., A happened, then B happened; therefore, A caused B. Just because one thing happened first, does not necessarily mean that the first thing caused the second thing. |
| Circular Reasoning |
Circular argument – assuming the truth of the conclusion by its premises. E.g., I never lie; therefore, I must be telling the truth. |
| Ad hominem |
An attack on the person making an argument does not really invalidate that person’s argument. It might make them seem a bit less credible, but it does not dismantle the actual argument or invalidate the data. |
| Straw Man Argument |
Restating the opposing idea in an inaccurately absurd or simplistic manner to more easily refute or undermine it. |
| Others? |
There are many more… can you think of some? For a bit of fun, check out Spurious Correlations. |
Knowledge Check
Critical thinking lies at the heart of evaluating sources. You want to be rigorous in your selection of evidence because, once you use it in your paper, it will either bolster your own credibility or undermine it.
Notes
- Cover images from journals are used to illustrate the difference between popular and scholarly journals, and are for noncommercial, educational use only.
References
Government of Canada. Statistics Canada. http://www.statcan.gc.ca/eng/start
Last, S. (2019). Technical writing essentials. BCcampus. https://pressbooks.bccampus.ca/technicalwriting/
Seneca Libraries. (2013, July 2). Evaluating Websites [Video]. YouTube. https://youtu.be/35PBCC5TKxs
Seneca Libraries. (2013, July 2). Popular and Scholarly Resources [Video]. YouTube. https://youtu.be/wPj-BBB0le4
Seneca Libraries. (2021, January 4 updated). Writing and Communicating Technical Information: Key Resources. Seneca College. https://library.senecacollege.ca/technical/keyresources
Spafford, E., Metcalf, L., & Dykstra, J. (2023). Cybersecurity myths and misconceptions: Avoiding the Hazards and Pitfalls That Derail Us. Addison-Wesley Professional.
The Sacred Chickens of Rome. (n.d.). Weird Universe. https://www.weirduniverse.net/blog/comments/sacred_chickens_of_rome
What is the source of the H.G. Wells quote, ‘Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write/”? (n.d.). Quora. https://www.quora.com/What-is-the-source-of-the-H-G-Wells-quote-Statistical-thinking-will-one-day-be-as-necessary-for-efficient-citizenship-as-the-ability-to-read-and-write
Why climate deniers have no credibility — in one pie chart. DeSmog Blog. https://www.desmogblog.com/2012/11/15/why-climate-deniers-have-no-credibility-science-one-pie-chart