This entry is a bit longer than our previous entries. We felt that it was necessary to dive a little deeper into the guidelines about writing test questions if this entry was going to be useful. What follows is a summary of guidelines from the resources listed at the end. If you find this information to be useful, we highly recommend Patty Shank's book as an approachable reference to keep on your bookshelf.
Some General Rules
Don’t aim to confuse. Questions should be clear and may even seem overly simple for an experienced and capable person when well written. The goal of a test is to verify that the student knows the information and can apply that knowledge, or that they don’t know and can’t apply it. Confusing questions reduce the validity of the test and open the door for students to complain about the test. If the questions reflect the training material and the class was well done a high percentage of students should pass the test.
When you designed your course you wrote performance objectives. Any cognitive based objectives can be easily turned into a test question. It is more time-consuming to write situation-based questions asking the student to show that they can correctly apply what they have learned to a situation, but the returns on that type of question are much greater. Memorizing information does not mean that you can then use that information. And using the information is what will be needed once the student returns to the workplace.
As with most development processes associated to training, before you give the test to your students, give your questions to another instructor for feedback. Consider also having a
previous student of the class take the test to see how they perform as a potential vetting of
the test. Depending on the topic of your test you might be able to give it to people who are currently doing those trained skills/knowledge. If your questions are directed
at the information used by an operational employee and the correct
answers reflect real operational responses, then then a currently operational
employee should pass the test. If your test questions cover information that is new for all employees this method won't work.
True/False Questions
This type of question gives the student a 50/50 shot at being correct. You can decrease the chance of guesswork by requiring the student fix a false prompt to make it true.
Example:
All money booked into
property must be signed and dated by both the booking Deputy and a Supervisor second employee. False
One way to make sure you are thoroughly considering any true/false questions is to write the prompt twice, once as a true statement, then as a false statement. This can show you if you have written a leading question, or if one of the questions includes confusing phrasing that should be fixed.
Multiple Choice Questions
Parts of the question
Stem: The question or prompt. A clearly stated stem can stand on its own as a question if the responses are removed.
Alternatives: The response options, often one correct response and a number of distractors or wrong responses.
Stem:
Alternatives:
B. Passive Resistance
C. Defensive Resistance
D. Active Resistance
Constructing an effective stem
Avoid negatively stated stems unless required by the material. Negative stems require a lot of mental gymnastics from the student and mostly test their mental flexibility instead of testing their knowledge base or ability to apply that knowledge.
Avoid extraneous or irrelevant material in the stem unless your goal is to test the student’s ability to extract the relevant material.
The best stem is a question. This allows the student to focus on answering the question as they read the alternatives.
Writing effective alternatives
How many alternatives should you have in a multiple-choice question? Studies suggest that three well-written options will result in a valid & reliable test. This means three plausible answers for the candidate to choose from.
When alternatives are not plausible, they are easily discounted by the student and stop being “distractors.” Adding a “fun” alternative might be fine, but don’t reduce the validity of your test to do so. Maintain a minimum number of plausible alternatives. In the below example, “C” is humorous and “B” is fairly obviously not a realistic response. A savvy test taker will eliminate both and answer “A” regardless of their knowledge base. We should either add two more realistic distractors, or change "C" and "B" to be realistic.
After the first three responders, additional responding deputies should prioritize protecting the public by:
B. The public is on their own you have too many things to do.
C. Pull off to the side of the road and enjoy your coffee and donut, it’s dangerous where the suspect is.
The best distractors are common student errors. Call on your experience as an instructor and subject matter expert in your specialty to help you come up with these. Some student errors may seem to be obvious wrong responses to an experienced instructor, but as long as they are not obvious to a student who doesn’t have the knowledge base they work as distractors.
Do not write alternatives with overlapping content. Both answers then are potentially correct, which reduces trust in the testing process an opens the door for later arguments about validity.
The recommended position of the stimulus for checking HGN is ____ inches from the face and slightly above eye level.
B.12 to 18
C.12 to 15
D.15 to 18
Better alternatives might look like this:
B. 12-15 inches
C. 15-18 inches
D. 18-21 inches
Be careful to not give away the answer to the question, or other questions on the test, in the alternatives. Maintain consistent grammar, length of response, and language of the responses to remove clues within the question. Don’t use content from another question stem as a distractor.
Limit the use of “All of the above” and “None of the above”. A better response to replace “None of the above” is “There is a better response than any of those listed above.”
You can also have the student respond to each of the alternatives. This removes the option for the student to eliminate some alternatives and increase the odds of guessing correctly. Using this method requires the student to show direct knowledge of each of the alternatives.
An example question:
When are sirens not required by policy?
A. The vehicle is not moving.
B. The deputy is initiating a traffic contact.
C. An unannounced approach is necessary.
D. All of the above
Having the student respond to each alternative:
In which of the following situations are sirens required by policy? Mark each alternative as “Yes” if sirens are required or “No” if they are not.
_______ A. When the vehicle is parked and unattended to assist as a traffic control point.
_______ B. When the deputy is initiating a traffic stop on an open roadway.
_______ C. When the deputy is attempting to approach a high-risk call unannounced.
_______ D. When the deputy is pursuing a vehicle on an open roadway.
List the alternatives in a logical order that is not associated to the correctness of the response. Alphabetical by the first letter is a good way to list alternatives. Also, chronological order, by rank, or by value (small to large), etc. are good ways to order that are not accidentally biased towards the correct answer.
The best test questions will require the student to use the new information to solve a problem or make a decision. These can be more challenging to write but will better show the student’s ability to work with the information in a way that matches their work environment. To do this, pose a realistic problem in your stem and list potential plausible solutions as alternatives. This tests the student’s ability in higher order thinking skills – analyzing and evaluating information – and not just their ability to recognize the correct response. To reduce confusion, make sure only one of the alternatives complies with policy, CRS, etc.
An Example:
The Anywhere County Sheriff’s Office Policy on pursuits states that deputies can pursue in unincorporated Anywhere County when there are reasonable grounds to believe the suspect being pursued has committed, attempted to commit, or is about to commit a felony (not including vehicular eluding) and when the deputy believes the benefit of apprehension is greater than the risk posed to the public by the pursuit. In which of the following situations might a pursuit be allowed according to this policy? Mark each option as “Yes” if a pursuit is allowed or “No” if it is not.
_______ A. A suspect has stolen what is likely less than $100 worth of merchandise from a store and is leaving in a vehicle as you pull into the lot. You see and air the license plate to dispatch.
_______ B. As you pull onto the street for a domestic violence call with no known weapons involved in the incident a car pulls out of the residence’s driveway and drives away from you. The victim is on the phone and reports that there was only pushing and shoving and a picture that was broken.
_______ C. You are sitting at a red light next to a bank. A robbery call is aired at that bank and a vehicle that matches the described suspect vehicle pulls out of the bank lot immediately in front of you.
_______ D. You attempt to stop a vehicle for failure to use a turn signal and no seat belts. The vehicle immediately accelerates away from you.
Short answer
Filling in the labels on a diagram is a short answer question. The diagram becomes a multiple-choice question if you give them the answers to match to the blanks.
Often removing the alternatives from a well written multiple-choice
question will give you a short answer question. When writing short answer questions be sure to
write clearly worded questions so there is no misunderstanding regarding the
information being requested. Phrase
questions so there is only one correct response or, if necessary, a very
limited range of possible responses.
Essay Questions
An Example:
Discuss how the Pragmatist movement affected the development of law enforcement in the United States. Compare and contrast the application of Pragmatist ideals in the U.S. to the development of law enforcement in a second country of your choosing focusing on the social standards of that culture and how those standards led to different choices in the development of the law enforcement role.
Everything here is effectively a primer on the topic of written tests; this is definitely not an exhaustive writing on this topic. Following these guidelines should get you started in the right direction.
Recognize that there are a lot of people who have created proprietary material related to building learning assessment and the adult learning field in general is rife with myth and "easy fixes" that are in fact bad methods. Because of this, be very careful when searching the internet for ideas.
Happy Writing!
Will
Resources
Malamed, Connie. "Writing Mulitple Choice Questions for Higher Order Thinking". The eLearning Coach. October 2012. https://theelearningcoach.com/elearning_design/higher-order-multiple-choice-questions/
Shank, P. (Producer). (April 28, 2021). “All of the Above” and Other Multiple-Choice Question Flaws [webinar]. https://www.trainingmagnetwork.com/
Shank, P., (2021). "Write Better Mulitple-Choice Questions to Assess Learning."
Writing Short Answer Questions. Retrieved 07.17.17 from http://academic.son.wisc.edu/wistrec/net/shortanswertext.html