A few other things to consider

In addition to the design and quality of the study, there are a few other considerations when deciding how reliable a paper is.

However, these have less absolute answers and are just important concepts to be aware of.

A question of reporting

In many cases, careful appraisal of a paper may leave you uncertain about whether basic concepts of study design were actually considered when planning and conducting a study, or, in fact, whether the study methods and results have simply been poorly reported.

This means that a study might actually have been well designed and conducted, but the reporting was poor. Important deficits that are frequently found in veterinary literature are: missing descriptions of the type of animals used in the study, no clear description of diagnostic methods, and inappropriate documentation of treatments and outcome measurements, to name just a few.

Poor reporting reduces the transparency of research and limits the reader’s ability to critically appraise information, as what is not included cannot possibly be appraised! Hence, poor reporting has a significant impact on clinical decision-making. In order to improve reporting and transparency of scientific work, reporting guidelines have been and are continuing to be developed.For example, the CONSORT, PRISMA and STROBE statements aim to improve the reporting of a number of types of epidemiological studies. The REFLECT statement is a modification of the CONSORT statement, making it useful for veterinary science, as it relates to livestock and food safety (Sargeant et al. 2010). Currently, however, only about 35% of veterinary journals refer to these guidelines (Grindlay et al. 2014), although it is hoped many more studies will use them in future.

The EQUATOR Network provide many guidelines for appraising the scientific literature.

In the end, if certain information is not given in a paper, you should regard this as not having been considered in the study design or study implementation. It is better to be safe than to be sorry when appraising literature that will lead you to make important decisions about your patients!

Peer-review and publication bias

Considerations such as whether an article has been peer-reviewed or if an area suffers from publication bias should be considered when you are appraising the literature.

Peer-review

Peer-review has been the quality-control process for scientific publishing for hundreds of years, ensuring that information is checked and verified by subject experts before it is formally published. This saves a huge amount of time on the part of the reader, as there is less onus on them to make the only fundamental analysis of the quality, accuracy and validity of the content themselves. Peer-reviewed publications from the scientific and veterinary communities are key sources for EBVM practitioners.

However, some limitations and possible biases of peer-review have been identified (Benos et al. 2007). For example, it has been demonstrated that gender and affiliation of the authors has an impact on the review outcomes. It is important to remember that peer-review is not perfect, and published peer-reviewed studies vary in quality. Even for all its drawbacks, however, studies show that manuscripts improve considerably after the peer-review process (Goodman et al. 1994, Benos et al. 2007). At the time this tutorial was published, peer-review had not been formally evaluated to determine if it is indeed able to improve the practical relevance of published information.

Publication bias

Traditionally, peer-reviewed scientific journals and the bibliographic databases that index them have been considered the best sources of evidence, but research into publication bias (Glanville et al. 2015) suggests that there is a need to go beyond these sources alone, as a significant proportion of research will not be published in peer-reviewed journals.

Publication bias occurs when researchers publish the results of studies showing that a treatment works well and don’t publish those showing it did not have any effect. If this happens, analysis of the published results will not give an accurate idea of how well the treatment works.

This publication bias is perhaps particularly relevant in the field of clinical veterinary medicine, where many practitioners may not be publishing their work as peer-reviewed articles, and much of the scientific data may be hidden in the so-called ‘grey’ literature (e.g. conference papers), or in practice records and case reports. For more information about finding this ‘grey’ literature, see ACQUIRE.