How to evaluate a scientific work? How do I know if a study is reliable? Is it appropriate to use it to make decisions? How extrapolated are the results? Does it all depend on how legit your sources are? These are all questions that we professionals involved in decision-making processes ask ourselves, and we are going to answer them in this article.
In this era, where the world is so globalized and we have an enormous amount of information in the palm of our hand, it is more necessary than ever to develop critical thinking to be able to decide what information to use for decision-making and what not.
Evaluating a scientific work is not an easy task, especially when one does not have that training, but, as professionals, we often have to decide whether or not to consider an article as an objective. Here we are going to focus on five points to evaluate, regardless of the statistical background, to analyze whether an article that comes to us can be considered valuable information for the topic we are trying to understand or if it has inconsistencies and is better to take it.” with tweezers” or leave it aside.
- Where was it published?
A good start is to evaluate where the work that you consider yourself to be scientific was published. Was it published in a peer-reviewed scientific journal or in a non-peer-reviewed journal, such as a popular or extension journal? Why is this first point important? Because it is published in a scientific journal with peer review, the article undergoes scrutiny by a jury of scientists specialized in the topic, who evaluate whether the experimental design and the methodology of data collection and analysis are appropriate for the type of conclusions presented.
- Is the bibliography list appropriate?
The second point is to analyze the bibliography referenced in the article. Here it is important to highlight that the process of science does not stop with the publication of a scientific article, rather each article contributes to forming scientific evidence, which is related to the conscious, explicit, and judicious use of valid and available data from scientific research. In that sense, each study contributes to scientific evidence, and if one shows something different from what the consensus says, it needs to be verified to be considered part of the evidence or refuted if the results cannot be repeated.
- Is the title appropriate for the results and conclusions of the study?
Once we have checked where the article comes from, and the quality of the referenced bibliography, the next question to answer is how closely the title fits the results and conclusions of the study. This is no small step because many times the title is the only thing that is shared and disseminated about a work. If the title is not consistent with the work, it may convey the wrong idea or generate a wrong perception about the topic. Intuitively, depending on how much it aligns with what we think, we question it more or less.
- Are the conclusions drawn from the scientific work supported by the design?
Once it was checked that the title is appropriate, that is, consistent with the results and conclusions of the work, the next question to be answered would be: Is the type of experimental design of the work appropriate to answer the question/hypothesis posed? Are the conclusions drawn from the scientific work supported by the design? Many times the person reading an article does not necessarily have statistical experience to do a deep analysis, but there is a very useful question to do a preliminary analysis of whether the conclusions are appropriate for the design: Does the work report a cause-effect relationship? or an association between variables?
- Is the proposed extrapolation of results supported by the design?
The possibility of extrapolating the results to the population, or not, depends on how the study units were chosen, regardless of whether it is an experimental trial or an observational study. Here the objective is to identify if any extrapolation of results in the article is made to a population or if they only refer to the group of study units, and to evaluate if this extrapolation is correct based on this question: How were the study units chosen? study?
- If the selection of units to be included in the study was random (that is, all “units” of a population had the same chance of being chosen for the study), the results can be extrapolated to the population from which they were chosen.
- If the selection of units was not random (that is, a specific group or volunteers not chosen at random from the population were used), then the results cannot be extrapolated to the population, and only apply to that study group.
So, did you find this post helpful?
