Recently, I’ve run into several people who were disparaging EMS research and the scientists who conduct it. I can’t count the number of times that I’ve heard an EMS person try to discount a research study using that old Benjamin Disraeli quote, “There are … lies, damned lies, and statistics.” Or they say, “All research studies are biased based upon the beliefs of the researcher.”
I think these statements represent fear because many EMS people do not fundamentally understand the scientific method or research methodology. Certainly, there have been many studies published that are extremely flawed. However, most studies are good and you cannot discredit all studies simply because some failed to meet the standard expected of scholarly work. As a rule, the closer a study adheres to the scientific method, the less likely the study is biased. There have been numerous nonbiased, well-conducted studies of EMS practices.
Perhaps it is time to provide a brief primer on EMS research. First, when looking at EMS research, it is important to determine whether the study in question is published in a peer-review journal or elsewhere. The peer-review process is arduous and time-consuming. It differs from popular press articles in that the authors of the research must carefully detail and explain every fact and every point in the study. Typically, a researcher or a group of researchers will complete a study and write it up.
Once written, it is submitted to a chosen journal (and only one journal at a time), where journal editors will typically send the article to two or three reviewers who have special expertise in that topic. As a rule, the reviewers are blinded as to the name and credentials of the authors. Likewise, the authors are blinded as to the name and credentials of the reviewers. This allows the process to be unbiased and based upon the facts of the article and not the prominence of the authors involved.
The review process can take weeks to months, although most journals are attempting to expedite the review process. After the reviewers have submitted their comments, the editors will either accept the article, accept the article if changes are made, or reject the article. Typically, the reviewers will have numerous questions which must be answered in considerable detail and with references. Once the article has been accepted for publication, a fact checker will typically verify the references and facts stated in the article. Then the article is placed into the publishing queue for eventual publication. The last scientific article I had published was in the Journal of Trauma. It took almost 18 months from the time the article was accepted to the time it was finally published. Scholarly publishing proceeds at a snail’s pace.
The most common peer-reviewed journals in EMS are Prehospital Emergency Care, Annals of Emergency Medicine, Academic Emergency Medicine, Prehospital and Disaster Medicine, and a few others. People often mistake the trade magazines, such as JEMS and EMS magazine, as peer-reviewed journals. While both magazines are excellent and have significant editorial oversight, they do not use a rigorous scholarly peer-review process. This does not mean that the articles in these magazines are erroneous; it just means that they have not been through the scholarly scrutiny required for articles referenced in scientific research.
The fundamental core of scientific investigation is the scientific method. Most of us learned this at some point during school, typically presented in some sort of physical science class. The scientific method is a method of inquiry that is based upon gathering observable and measurable data and then analyzing the results. In terms of medical research, there are typically eight steps to the scientific method. These are:
1. Define the question.
2. Gather information from appropriate resources and through observation.
3. Develop a hypothesis.
4. Perform the experiment and gather data.
5. Analyze the data.
6. Interpret the data and draw conclusions related to the hypothesis (test the hypothesis).
7. Publish the results.
8. Retest to repeat the experiment (typically done by other researchers).
The first step of any experiment usually follows the observation of some phenomenon. For example, you might notice something occurring in your EMS service that is of concern. You note that a particular dispatcher tends to make more errors than others. Or, you see that most paramedics are more liberal with pain medications when it comes to male patients when compared to female patients. Or, you note that your service is busier when the moon is full. Any of these observations can be turned into a question and tested. For example, “Does my EMS service discriminate against female patients in analgesic administration?”
The second step is to gather information from the appropriate sources or through observation and measurement. For example, you can look at a sample of run sheets or you may do a brief survey of your coworkers. If others have made the same observation as you, then it would seem prudent to proceed with the experiment.
The third step of the scientific method is to develop a hypothesis. A hypothesis is a tentative statement made to define the observed problem. For example, your hypothesis might be, “There is a gender bias in the administration and dosing of prehospital analgesics.”
The fourth step is to run the experiment and gather data. You can gather data either from the past (retrospectively) or in the future (prospectively). As a rule, prospective investigations have greater scientific validity. When data is gathered retrospectively, there is a natural human tendency to only select cases which may or may not support the hypothesis (called “selection bias”). Thus, in the research hierarchy, prospective studies typically trump retrospective studies. There are various research methods to follow depending on the type of experiment. Because EMS often involves human subjects, experimental methodologies must be modified to avoid endangering patients.
The fifth step is to analyze the data. This typically will involve some sort of statistical evaluation. Statistical analysis is essential in scientific research to assure that observable findings are not simply due to random events and are in fact due to the variables being studied. The level of statistical analysis varies with the experiment type. But, along the lines of what Benjamin Disraeli once said, some researchers will torture statistics until they support their hypothesis. Generally, this is picked up and corrected in the review process; however, some lower-tier journals will publish studies where the statistics have been tortured.
The sixth step is to interpret the data and determine whether the data support your hypothesis. For example, through your research, you found that female patients were 30 percent less likely to receive prehospital analgesia when compared to their male counterparts. In this case, you have proved your hypothesis. However, say that you found that both male and female patients equally received prehospital analgesia. In this case, your hypothesis was wrong and you must refine it to reflect your findings. At this point, your hypothesis would be, “Male and female EMS patients equally receive prehospital analgesia.”
The seventh step is writing and publishing a report of your results. Scientific papers, especially in medicine, follow very strict guidelines in terms of organization, style, references and illustrations. These vary from journal to journal and you should always read the “Information for Authors” section before writing. Most journals now require electronic submission. The report should contain enough detail so that anybody who reads your paper should be able to repeat the experiment.
The eighth and final step is retesting your experiment. This is typically done by other researchers (often your critics). The beauty of science is that any study or any paper can be repeated. If a study is repeated and the same results are found, then there is a greater likelihood that the original study was valid. Thus, your research will stand regardless of whom you are or your prominence in the field. This is the reason why noted astronomer Carl Sagan once said, “In science, there are no authorities.”
This basically sums up the scientific method and provides an introduction to EMS research. It is important to point out that EMS practices should not be changed based upon a single research study. However, when several researchers working independently arrive at the same conclusion, that more likely than not means the conclusions were correct and the observed phenomenon is real. At this point, those who oversee EMS should evaluate the data and make a determination of whether a practice or procedure should be changed.
The EMS provider of the future should have a fundamental understanding of the various research methods. The purpose of this is twofold. One, it allows one to scrutinize various scientific studies. And two, it allows one to perform scientific research. In the overall scheme of things, the future of EMS will be driven by research and research is driven by the scientific method.
