1 Paper: Evaluating Research Quality

This is a brilliant, well-written publication, Evaluating Research Quality, on how to assess the quality of research you are reading - written by Todd Litman at the Victoria Transport Policy Institute - which is "an independent research organization dedicated to developing innovative and practical solutions to transportation problems".

Read Evaluating Research Quality Here: http://www.vtpi.org/resqual.pdf

Todd provides a number of examples of what makes good, well-informed research and what is biased, ill-informed research, drawing from examples in the transit sector but also general examples such as the source of statements such as "Eskimos (Inuit) have 23 words they use to describe snow" and ideas such as "AIDs doesn't kill people, antiviral medication kills people".

The entire document is well worth a look from start to finish, and the "Sixty Four Methodological Potholes" table is an interesting and extensive read. It's more detailed than Carl Sagan's Baloney Detection Kit and highlights some of the same illogical fallacies and bias some have presented in their publications and arguments.

This specific portion could be applied (and I should probably more stringently apply these pointers to papers reviewed here) to Lyme disease research and all research in general, so I want to share it. Credit goes to Todd Litman and the Victoria Transport Policy Institute, who allow republication with credit. (Thanks, guys.)

Research Document Evaluation Guidelines

The guidelines below are intended to help evaluate the quality of research reports and articles.

Desirable Practices

1. Attempts to fairly present all perspectives.
2. Provides context information suitable for the intended audience. This can be done with a
   literature review that summarizes current knowledge, or by referencing relevant
   documents or websites that offer a comprehensive and balanced overview.
3. Carefully defines research questions and their links to broader issues.
4. Provides data and analysis in a format that can be accessed and replicated by others.
   Quantitative data should be presented in tables and graphs, and available in database or
   spreadsheet form on request.
5. Discusses critical assumptions made in the analysis, such as why a particular data set or
   analysis method is used or rejected. Indicates how results change with different data and
   analysis.Identifies contrary findings.
6. Presents results in ways that highlight critical findings. Graphs and examples are
   particularly helpful for this.
7. Discusses the logical links between research results, conclusions and implications.
   Discusses alternative interpretations, including those with which the researcher disagrees.
8. Describes analysis limitations and cautions. Does not exaggerate implications.
9. Is respectful to people with other perspectives.
10. Provides adequate references.
11. Indicates funding sources, particularly any that may benefit from research results.

Undesirable Practices

1. Issues are defined in ideological terms. “Straw men” reflecting exaggerated or extreme perspectives are use to characterize a debate.
2. Research questions are designed to reach a particular conclusion.
3. Alternative perspectives or contrary findings are ignored or suppressed.
4. Data and analysis methods are biased.
5. Conclusions are based on faulty logic.
6. Limitations of analysis are ignored and the implications of results are exaggerated.
7. Key data and analysis details are unavailable for review by others.
8. Researchers are unqualified and unfamiliar with specialized issues.
9. People with differing perspectives are insulted and ridiculed
10. Citations are primarily from special interest groups or or popular media, rather than from peer reviewed professional and academic organizations.

This work by Camp Other is licensed under a Creative Commons
Attribution-NonCommercial-ShareAlike 3.0 Unported License.