Welcome to the Training for Transfer blog

Welcome to the Training for Transfer blog

Training is an ever-changing field.   This change is due, at least in part, to the constantly evolving understanding about how we as humans ...

How to Write Good Test Questions

This entry is a bit longer than our previous entries.  We felt that it was necessary to dive a little deeper into the guidelines about writing test questions if this entry was going to be useful.  What follows is a summary of guidelines from the resources listed at the end.  If you find this information to be useful, we highly recommend Patty Shank's book as an approachable reference to keep on your bookshelf.

Some General Rules

Don’t aim to confuse.  Questions should be clear and may even seem overly simple for an experienced and capable person when well written.  The goal of a test is to verify that the student knows the information and can apply that knowledge, or that they don’t know and can’t apply it.  Confusing questions reduce the validity of the test and open the door for students to complain about the test.  If the questions reflect the training material and the class was well done a high percentage of students should pass the test.

When you designed your course you wrote performance objectives.  Any cognitive based objectives can be easily turned into a test question.  It is more time-consuming to write situation-based questions asking the student to show that they can correctly apply what they have learned to a situation, but the returns on that type of question are much greater.  Memorizing information does not mean that you can then use that information.  And using the information is what will be needed once the student returns to the workplace.

As with most development processes associated to training, before you give the test to your students, give your questions to another instructor for feedback.  Consider also having a previous student of the class take the test to see how they perform as a potential vetting of the test.  Depending on the topic of your test you might be able to give it to people who are currently doing those trained skills/knowledge. If your questions are directed at the information used by an operational employee and the correct answers reflect real operational responses, then then a currently operational employee should pass the test.  If your test questions cover information that is new for all employees this method won't work.

True/False Questions

This type of question gives the student a 50/50 shot at being correct.  You can decrease the chance of guesswork by requiring the student fix a false prompt to make it true.

Example:

All money booked into property must be signed and dated by both the booking Deputy and a Supervisor second employee.              False

One way to make sure you are thoroughly considering any true/false questions is to write the prompt twice, once as a true statement, then as a false statement.  This can show you if you have written a leading question, or if one of the questions includes confusing phrasing that should be fixed.

Multiple Choice Questions

Parts of the question

Stem:  The question or prompt.  A clearly stated stem can stand on its own as a question if the responses are removed.

Alternatives:  The response options, often one correct response and a number of distractors or wrong responses.

Stem:

What type of resistance is required for use of a CED without supervisor approval?

Alternatives:

A. Active Aggression/Assault
B. Passive Resistance
C. Defensive Resistance
D. Active Resistance
 

Constructing an effective stem

Avoid negatively stated stems unless required by the material.  Negative stems require a lot of mental gymnastics from the student and mostly test their mental flexibility instead of testing their knowledge base or ability to apply that knowledge.

Avoid extraneous or irrelevant material in the stem unless your goal is to test the student’s ability to extract the relevant material.

The best stem is a question.  This allows the student to focus on answering the question as they read the alternatives.

Writing effective alternatives

How many alternatives should you have in a multiple-choice question?  Studies suggest that three well-written options will result in a valid & reliable test.  This means three plausible answers for the candidate to choose from.

When alternatives are not plausible, they are easily discounted by the student and stop being “distractors.”  Adding a “fun” alternative might be fine, but don’t reduce the validity of your test to do so.  Maintain a minimum number of plausible alternatives.  In the below example, “C” is humorous and “B” is fairly obviously not a realistic response.  A savvy test taker will eliminate both and answer “A” regardless of their knowledge base.  We should either add two more realistic distractors, or change "C" and "B" to be realistic.

An Example:

After the first three responders, additional responding deputies should prioritize protecting the public by:
A. Removing them or preventing them from entering the threat area.
B. The public is on their own you have too many things to do.
C. Pull off to the side of the road and enjoy your coffee and donut, it’s dangerous where the suspect is.
 

The best distractors are common student errors.  Call on your experience as an instructor and subject matter expert in your specialty to help you come up with these.  Some student errors may seem to be obvious wrong responses to an experienced instructor, but as long as they are not obvious to a student who doesn’t have the knowledge base they work as distractors.

Do not write alternatives with overlapping content.  Both answers then are potentially correct, which reduces trust in the testing process an opens the door for later arguments about validity.

An Example:

The recommended position of the stimulus for checking HGN is ____ inches from the face and slightly above eye level.
A.10 to 12
B.12 to 18
C.12 to 15
D.15 to 18

Better alternatives might look like this:
A. 9-12 inches
B. 12-15 inches
C. 15-18 inches
D. 18-21 inches
 

Be careful to not give away the answer to the question, or other questions on the test, in the alternatives.  Maintain consistent grammar, length of response, and language of the responses to remove clues within the question.  Don’t use content from another question stem as a distractor.

Limit the use of “All of the above” and “None of the above”.  A better response to replace “None of the above” is “There is a better response than any of those listed above.”  

You can also have the student respond to each of the alternatives.  This removes the option for the student to eliminate some alternatives and increase the odds of guessing correctly.  Using this method requires the student to show direct knowledge of each of the alternatives.

An example question:

When are sirens not required by policy?

A.     The vehicle is not moving.

B.     The deputy is initiating a traffic contact.

C.     An unannounced approach is necessary.

D.     All of the above

 

Having the student respond to each alternative:

In which of the following situations are sirens required by policy?  Mark each alternative as “Yes” if sirens are required or “No” if they are not.

_______  A.      When the vehicle is parked and unattended to assist as a traffic control point.

_______  B.       When the deputy is initiating a traffic stop on an open roadway.

_______  C.       When the deputy is attempting to approach a high-risk call unannounced.

_______  D.      When the deputy is pursuing a vehicle on an open roadway.

 

List the alternatives in a logical order that is not associated to the correctness of the response.  Alphabetical by the first letter is a good way to list alternatives.  Also, chronological order, by rank, or by value (small to large), etc. are good ways to order that are not accidentally biased towards the correct answer.

The best test questions will require the student to use the new information to solve a problem or make a decision.  These can be more challenging to write but will better show the student’s ability to work with the information in a way that matches their work environment.  To do this, pose a realistic problem in your stem and list potential plausible solutions as alternatives.  This tests the student’s ability in higher order thinking skills – analyzing and evaluating information – and not just their ability to recognize the correct response.  To reduce confusion, make sure only one of the alternatives complies with policy, CRS, etc.

An Example:

The Anywhere County Sheriff’s Office Policy on pursuits states that deputies can pursue in unincorporated Anywhere County when there are reasonable grounds to believe the suspect being pursued has committed, attempted to commit, or is about to commit a felony (not including vehicular eluding) and when the deputy believes the benefit of apprehension is greater than the risk posed to the public by the pursuit.  In which of the following situations might a pursuit be allowed according to this policy?  Mark each option as “Yes” if a pursuit is allowed or “No” if it is not.

_______  A.       A suspect has stolen what is likely less than $100 worth of merchandise from a store and is leaving in a vehicle as you pull into the lot.  You see and air the license plate to dispatch.

_______  B.       As you pull onto the street for a domestic violence call with no known weapons involved in the incident a car pulls out of the residence’s driveway and drives away from you.  The victim is on the phone and reports that there was only pushing and shoving and a picture that was broken.

_______  C.       You are sitting at a red light next to a bank.  A robbery call is aired at that bank and a vehicle that matches the described suspect vehicle pulls out of the bank lot immediately in front of you.

_______  D.       You attempt to stop a vehicle for failure to use a turn signal and no seat belts.  The vehicle immediately accelerates away from you.

Short answer

Filling in the labels on a diagram is a short answer question.  The diagram becomes a multiple-choice question if you give them the answers to match to the blanks.

Often removing the alternatives from a well written multiple-choice question will give you a short answer question.  When writing short answer questions be sure to write clearly worded questions so there is no misunderstanding regarding the information being requested.  Phrase questions so there is only one correct response or, if necessary, a very limited range of possible responses.

Essay Questions

You will likely want to avoid essay questions simply for the fact that they take longer to grade due to the wide range of possible responses.

An Example:

Discuss how the Pragmatist movement affected the development of law enforcement in the United States.  Compare and contrast the application of Pragmatist ideals in the U.S. to the development of law enforcement in a second country of your choosing focusing on the social standards of that culture and how those standards led to different choices in the development of the law enforcement role.

You can imagine the variations of response you might get from this type of question.  If you find the need to use essay type questions you will want to develop a list of the elements required in the response in order to maintain some level of objectivity in the grading.  Consider watching for cites to resources, the supporting explanation for the student’s response, and how detailed the response is, as well as the possibility of correct or necessary elements within that response.

 

Everything here is effectively a primer on the topic of written tests; this is definitely not an exhaustive writing on this topic.  Following these guidelines should get you started in the right direction.

Recognize that there are a lot of people who have created proprietary material related to building learning assessment and the adult learning field in general is rife with myth and "easy fixes" that are in fact bad methods.  Because of this, be very careful when searching the internet for ideas.

Happy Writing!

 Will


 

 

Resources

Brame, C., (2013) Writing good multiple choice test questions. Retrieved 04.11.17 from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/

Malamed, Connie. "Writing Mulitple Choice Questions for Higher Order Thinking". The eLearning Coach. October 2012. https://theelearningcoach.com/elearning_design/higher-order-multiple-choice-questions/

Shank, P. (Producer). (April 28, 2021). “All of the Above” and Other Multiple-Choice Question Flaws [webinar]. https://www.trainingmagnetwork.com/

Shank, P., (2021). "Write Better Mulitple-Choice Questions to Assess Learning."

Writing Short Answer Questions. Retrieved 07.17.17 from http://academic.son.wisc.edu/wistrec/net/shortanswertext.html