Is there a better approach for fire, EMS department testing?
Testing plays an important role in the application and promotion process, but needs to evolve to remain valid and useful
The Columbus, Ohio Division of Fire just concluded an investigation into potential cheating on tests that resulted in a 324 page report. At issue was whether fire recruits inappropriately accessed study materials that were posted online and which closely mirrored actual test questions.
According to Fire Chief Kevin O’Connor, no evidence of cheating was found. However, the Ohio Department of Public Safety is also investigating the incident.
This situation brings up some questions in the effort to align current technology with traditional ethics and rules.
Is it cheating if someone takes the test and remembers specific questions after the fact? And if that person later writes those questions down and researches the answers to them as a way of improving on the next exam, is that wrong? What about if a group of friends collaborate in remembering and recording test questions?
Let’s face it, people have been doing this for many years. The main difference now is the technology that allows such information to be disseminated and shared.
Now there are websites and smartphone apps that allow users nearly unlimited access not only to information they and their friends have gathered, but also information obtained by thousands of others around the country.
Some might say that using such sites for test preparation provides an unfair advantage and should not be allowed. Others argue that such sites are no different from printed study guides from the past.
Everyone can agree that testing should be some measure of competence and that cheating on tests is wrong. Beyond those principles is a lot of gray area.
New approaches for fire department testing
There are three potential approaches to this challenge, each with advantages and disadvantages.
1. Increase testing security.
Fire departments could try to make it impossible for anyone to access unauthorized test prep sites. They could tell test takers not to share information. In fact, security measures have been tightened in recent years, with all-computerized test sites, no writing materials or phones allowed in exam rooms, and so on.
But such measures can only go so far. It is impossible to control someone’s memory, or their private conversations with friends, or their use of the internet when they are off duty and on their own electronic devices.
Technology will only become more sophisticated, providing ever easier access. Trying to stop it will be an endless exercise in futility.
2. Accept the reality of the situation.
People have access to test questions as a study guide. Fine, let them have it. Some professions arrived at this conclusion long ago. For example, those studying for the written test to become a private pilot have a book of 600 questions and answers. They know that the test they ultimately take will include 60 questions selected from that larger set. Each test makes a different selection of questions.
One key for the usefulness of this approach is that every person in the testing process must have access to the same materials. It would not be fair for a small group to have access to all the questions, and everyone else to just have access to some of them.
3. Develop better tests.
Multiple choice tests can only tell you so much about someone’s knowledge of a subject. Those tests also tell us almost nothing about the test taker's ability to synthesize and apply their knowledge to dynamic situations like patient care or incident management. And if those tests are rarely revised or updated, it is only a matter of time before those taking the test will have a pretty comprehensive idea of what will be included.
At the very least, the order of the questions asked should be different on every test given. This is simple enough to do with computer generated tests. Content should also evolve with changing professional needs and priorities.
But perhaps fire departments should also rethink their testing processes in the bigger picture. What do they really want to know about new recruits, or those aspiring to be officers? Is a rote learning, multiple choice test the best way to assess the qualities and competencies that are most important? Are there other ways that key information could be gleaned, methods that do not lend themselves to gaming the system?
The three approaches are not mutually exclusive. Test sites, both physical and virtual, should be secure and cheating should never be tolerated. Whatever study materials exist should be available to all equally. And fire departments and the contractors that serve them in this area should be constantly reassessing their priorities in how tests are administered and what they are designed to measure.