This article was originally posted at Limmer Education and is reprinted with permission.
Computer adaptive tests (CAT) are designed to measure one principle: competent vs not yet competent. They are much more effective at determining competency than other types of exams. Students find them harder because the exam pushes students to the furthest extent of their knowledge and capability. And educators find them harder to replicate for in-class practice.
So when an already challenging exam adds a whole new category of questions, there’s going to be an awkward phase while everyone learns what to expect and how to prepare for it.
Technology enhanced items and computer adaptive testing
The NREMT recently added technology enhanced items (TEIs) to its computer adaptive examinations for EMT and paramedic candidates. Right now, these TEIs include multiple response items, but you can expect to see other new item types piloted later.
Multiple response questions are used to test a large amount of summative information in a short period of time. They are not used for formative assessment and they do not test critical thinking. The general theory is that using these style questions allows for a quicker assessment of a test taker’s basic knowledge.
One of the key requirements of any CAT exam is having a range of question difficulty. Every test must have simple, moderate, and difficult content. Every examinee starts the exam with simple and moderate items. If s/he performs well, difficult items are eventually served.
Even though they do not test critical thinking, multiple response items also come in easy, moderate, or difficult categories. However, even “easy” items can make the exam feel more difficult if the examinee is unfamiliar or inexperienced with the item style.
Remember, no “scored” item in a certification examination will count toward pass or fail until it has been validated by a large sample size in a high-stakes examination. If it’s the first time an examinee sees a TEI style item, it’s likely being pilot tested. Items that prove unsuccessful during pilot testing may be discontinued.
How to write multiple response questions
Where to begin? Educators have to learn how to draft TEIs for the classroom. It’s not an easy task and will take some trial and error, but it’s important to expose students to TEIs before they get to the NREMT.
We put together a video to help educators understand how to write multiple response items for EMS class exams.
Tips for writing multiple response items:
- These questions will be slightly longer than your usual test questions, but should still be concise. The video above offers helpful best practices for writing and editing these items.
- Tell students how many correct choices there are – i.e., end each multiple response question with “choose X of the following X.” This format mirrors the way the NREMT has said they will set up their questions. It also helps keep the question clean and concise.
- There should always be at least 2 more distractors than correct answers – e.g., if there are 4 correct answers, there should be at least 6 answer options. The NREMT has indicated most of their items will have 5-6 possible answer choices.
- Don’t use weak distractors, because these will make the question too easy. Every distractor should be choose-able. However, answers must be clearly right and distractors must be clearly wrong.
- All response options should be the same style and length.
- Multiple response items work particularly well for signs and symptoms and treatment questions.
We encourage you to develop your own TEI items and start using them on your class examinations. Let us know how it goes!
Read more:
‘What’s my condition?’
An easy patient assessment activity you can incorporate in any classroom
TEI updates to Limmer apps
These apps have been updated to include multiple response items:
The new items in our apps are a sample of TEIs. They were reviewed – but not pilot tested – on a large group of newly graduated EMT students during a high-stakes examination.
These updated items have automatically been pushed out to your web or mobile apps.
