EVALUATION INSTRUMENTS

To collect defensible data, one needs to use a good evaluation instrument. An evaluation instrument is a set of items used to collect data. Evaluation instruments come in a variety of forms, such as a questionnaire, observation form, or interview guide. Four types of information might be included on an evaluation instrument.

Knowledge –- what people know; how well they understand something
Beliefs, Attitudes, Opinions –- perceptions people hold, their thoughts, feelings, ideas, judgments, or ways of thinking
Behavior –- what people do
Demographics –- who people are -- age, sex, education, occupation, or income

A poorly designed evaluation instrument will result in poor, non-defensible data. It is always best to use properly tested instruments to collect data. Tested instruments are often provided by University Faculty in association with a specific education program.

Types of Items

There are two types of items -- open-ended and close-ended –- that can be on an evaluation instrument. Open-ended items allow respondents to give their own answers. This type of item requires more effort on the part of the respondent to come up with a response. Open-ended items also tend to produce a wider range of answers than do close-ended questions and so are more difficult to analyze.

Close-ended items are fixed choices from which the respondent must choose. Close-ended items produce more uniform answers than do open-ended items. Examples of close-ended items include: matching, multiple choice, ranking, and rating scales.

Wording an Evaluation Instrument

Wording items on an evaluation instrument can be challenging. The following must be addressed in order to result in valid responses:

Simple and clear wording is used.
Abbreviations, jargon, or foreign phrases are not used.
Personal or potentially incriminating items are phrased in non-objectionable ways.
Items that are too demanding and time consuming are not included.

Formatting an Instrument

Proper formatting makes it easier to collect data. Use the following guidelines to format an evaluation instrument.

An introduction that outlines the purpose, explains how the information will be used, and assures students of confidentiality must be at the beginning of the instrument. In mailed evaluation instruments, these three points must be reinforced in the cover letter.
The first questions should be easy and not address controversial topics.
Important topics should be addressed early, rather than later, in the instrument.
Items should be arranged so that they flow naturally. Items about one subject should be grouped together. Begin with general questions and then move to those that are specific.
The same types of items and response choices are used throughout a series of questions on a particular topic.
Demographic items -- age, sex, and income level – should be at the end of the instrument.
The instrument should be printed in an easy-to-read typeface.

Pretesting an Instrument

After the instrument is formatted, pretest it to determine if it is easy to understand and easy to complete. Pretesting is a required part of instrument design. It is done by:

Asking colleagues to critically review the instrument.
Administering the instrument to people similar to the respondents and requesting feedback about the format and content.
Simulating the actual data collection procedure.
Assessing whether the items on the instrument produce the information needed to answer the evaluation questions.
Conducting an item analysis, if necessary.
Revising, as needed.

Test Your Knowledge

1. What are the two types of items that can be found on an evaluation instrument?

2. Multiple choice questions are an example of what type of item?

3. What is wrong with how this true or false item is worded? The NRA ServSafe text states keep all PHFs out of the temperature danger zone.

4. Where should demographic questions be located on an evaluation instrument?

5. What are the six steps used to pretest an evaluation instrument?

ANSWER KEY