Chapter 13: Essay Items
I.
Writing
Questions and Instructions
A. Examples of complex cognitive behaviors
|
BEHAVIOR |
TERMS THAT CALL OUT THE BEHAVIOR |
|
Analyzing |
Break down, diagram, differentiate,
explain |
|
Comparing |
Compare, contrast, classify, distinguish
between |
|
Creating |
Compose, devise, propose, design |
|
Evaluating |
Critique, choose and defend, evaluate,
judge |
|
Inferring |
Extend, extrapolate, predict, conclude,
project |
|
Interpreting |
Illustrate, translate, interpret, convert |
|
Synthesizing |
Combine, rearrange, infer, deduce |
B. Two Types of Essay Questions
____________-Response Essay
Questions: limits both the content
and the form of the response (usually 1 paragraph).
____________-Response Essay
Questions: Provides the examinee with
more latitude to produce a longer response and to vary the context in which
content is presented. Typically, the
form of the written response is a component of the scoring criteria.
C. Essay Item Writing Guidelines
1.
Restrict the use of essay items to those learning outcomes
that cannot be satisfactorily measured by other (objective or short answer)
items.
2.
Phrase each question (or set of directions) so that the
pupil’s task is clearly indicated.
Poor
Example:
What
does Newton’s third law have to say about the bounce of a rubber ball?
Better
Example:
Using
Newton’s third law, explain why a ball bounces higher when dropped from 10
feet, than when dropped from 5 feet.
Progressively
Improved Examples:
(1)
Compare objective and essay tests.
(2)
Compare and contrast objective and essay tests citing the
respective strengths and weakness of each.
(3) Compare and contrast objective and
essay tests citing the respective strengths and weakness of each. Make sure to include the following:
a. Ease of item construction e. Nature of student responses
b. Sampling of subject matter f. Guessing
c. Type of objectives measured g. Time needed for testing
d. Preparation by student
Another
Improved Example:
What
were the causes of the Civil War?
Discuss
the role of agriculture in the North and South as a factor in the outbreak of
the United States’ Civil War.
More Poor Essay
Items: Actual
questions from unidentified campuses across the country.
*HISTORY:
Describe the history of the papacy from its origins to the present day,
concentrating especially, but not exclusively, on its social, political,
economic, religious, and philosophical aspects and impact on Europe, Asia,
America, and Africa. Be brief, concise,
and specific.
*EDUCATION:
Develop a foolproof and inexpensive system of education that will meet the
needs of all segments of society.
Convince both the faculty and the rioting students outside to accept it.
*EPISTEMOLOGY:
Take a position for or against truth.
Prove the validity of your position.
*GENERAL
KNOWLEDGE: Describe in detail your general knowledge. Be objective and specific.
4.
Indicate an appropriate time limit for each question.
a.
b.
c.
5.
Don’t give optional questions
6.
Judge an item’s quality by composing a model response.
II. Developing Scoring Procedures
Writing the essay
item is relatively simple compared to scoring the item.
A. Two methods for scoring constructed responses
2. __________
Scoring: sort students’ responses into categories by
quality.
a.
Establish the
scoring categories you will use
b.
Characterize a response that fits each category
c.
Read each
response and form an overall general impression
d.
Sort the
responses into the designated categories
e.
Reread the papers and re-categorize as needed
f.
Assign the same score to all papers within a category
2. __________ Scoring: systematic scoring using specific
procedures, such as checklists or rating scales (or both), to more accurately
assign partial credit and indicate where students lost points. The scoring plan or procedure is called a rubric.
Advantages: More specific feedback to students
More reliable scores
Disadvantages: Time consuming to construct scoring
instrument
A.
Checklists – a.k.a. Item Based Rubrics - provides two
categories for evaluation (present/absent, acceptable/unacceptable)
Example:
Compare
and contrast maple and pine trees.
Describe the maple and the pine, and tell me what kinds of tree they
are. Then tell me how they are alike and
how they are different in terms of the shape of the leaves, when they have
leaves, and what kinds of products we get from them.
Essay Checklist
I. Content (2 pts each, 16 pts possible)
_____ Pine _____
Maple Kind
of Tree
_____ Pine _____
Maple Shape of Leaves
_____ Pine _____
Maple Time for Leaves
_____ Pine _____
Maple Products
Comments______________________________________________
II. Structure (1 pt each, 2 pts total)
_____ Topic Sentence (present or absent)
_____ Conclusion (present or absent)
Comments______________________________________________
III.
Mechanics (1 pt each, 2 pts total)
_____ Grammar (acceptable/unacceptable)
_____ Spelling (minimal or no errors/many errors)
Comments______________________________________________
_____/16
Total Points
B.
Rating Scales – a.k.a. Descriptive Rubrics
– an extension of the checklist that also allows for a judgment of quality, not
simply whether the criterion is present or absent.
Essay Rating
Scale
I. Content Absent Poor Average Excellent
Maple: (4 pts
possible)
Description 0 1 2 3
Kind
of tree 0 (absent) 1 (present)
Pine:
(4
pts possible)
Description 0 1 2 3
Kind
of tree 0 (absent) 1 (present)
Similarities
and Differences: (9 pts possible)
Discuss
shape 0 1 2 3
Discuss
timing 0 1 2 3
Discuss
products 0 1 2 3
Comments______________________________________________________________
II. Structure (4 pts possible) Absent Poor Good
Topic Sentence 0 1 2
Conclusion 0 1 2
Comments______________________________________________________________
III.
Mechanics (2 pts possible) Poor Good
Grammar 0 1
Spelling 0 1
Comments_______________________________________________________________
_____/23
Total Points
*Be
sure to evaluate and pilot test your Checklists and Rating Scales before
using them to grade all the papers.
B. General
Suggestions for Writing Rubrics and Scoring Essays
1. Prepare an outline or an example of the
expected answer in advance.
2. Choose the scoring method that is most
appropriate.
3. Decide how to handle factors
that are irrelevant to the learning outcomes measured.
4. Evaluate all of your
students’ responses to one question before going on to the next item.
5.
When
possible, score student responses anonymously.
6. Do not look at the student’s scores on
previous items
7. If big decisions rest on the
results, have 2+ independent ratings.
*8. Give serious consideration
to the your point breakdown; is the focus on writing mechanics or knowledge of
content?
III. Additional Information on Scoring Subjective
Items
A. Definition
Items
Example: What is a norm-referenced test?
Sample
Responses:
Jasmine: “A standardized test” (1)
Hyde: “A test where the scores are reported in
standard scores such as percentiles, not percent of information learned” (2)
Homer: “A test that is designed to rank-order
students” (2)
Fred: “A test administered under standard
conditions” (0)
Rating
Scale:
2 pts - indicates the idea of comparison or
rank ordering
1 pt - student gives and example
0 pts - wrong answer or missing information
B. Lists
Preliminary
decisions: (1) Do we want them to know the entire list?
(2)
Does the order matter?
Example:
List,
in order, the categories in Bloom’s taxonomy? (8 pts)
Answer: Knowledge, Comprehension, Application,
Analysis, Synthesis, Evaluation
Simple
Rubric:
_____1 pt. for
every category correctly listed (6 points possible) plus
_____+2 pts if
all in right order or
_____+1 pt if two
are out of order or
_____0 pts if 3
or more are out of order
_____/8
Total points
C. Single-Sentence Responses
Example:
Why
did Columbus sail west and not east?
Answer: It was dangerous and expensive to sail around the Cape of Good
Hope (Africa) and he knew that the world was round. (2 points)
Sample
Responses
Jasmine: “Columbus sailed west and not east because
he knew the world was round.”
Hyde: “Trying to get
to China by sailing around Africa was expensive and dangerous; knowing the
world was round, Columbus sailed west.”
Homer: “west - world round east - too hard”
IV. Avoiding Common Errors in Test Development,
Scoring, and Grading
A. Development Errors
1.
Inappropriate difficulty level
2.
Inadequate directions
B. Scoring Errors
1.
Inconsistency when scoring
a.
_______________ Reliability
b.
_______________ Reliability
2.
Bias
a.
Generosity error: This is
described as being an “easy grader”. This type of bias is applied to the whole
class. You give an overly favorable (“above average”) evaluation of student
responses.
b.
__________ error: The opposite
of generosity error. Also applies to the whole class. You give an overly
critical evaluation of student responses; everyone is evaluated as “below
average”.
c.
Central-tendency
error: Also applies to the entire class. Possible due to fear of being too
easy or too hard, you score everyone as “average”.
d.
__________ effect: This applies
to specific individuals. You like the student and let that emotion influence
your evaluation of his or her work.
e.
Horns effect: This also
applies to specific individuals. There is some source of conflict between you
and this student and that influences your evaluation of his or her work
V. Helping Students Write Better Essay Tests
1.
Emphasize vocabulary and logic unique to the discipline
2. Tell them to read
all questions before responding to any of them
3.
4. Communicate the
relevance of grammar, punctuation, and spelling
5.
6. Promote study
habits appropriate for essay testing