SESSION DAY CHAPTER TOPIC
1 1/16 1 Introduction, Overview, Requirements
2 1/23 1 1-2 Instructional Objectives, Domains, Taxonomies, Test Grids
3 1/30 2 Item Formats, Above Recall, RCT, NORC
4 2/6 2 Constructed Response, Alternative Formats, SED
5 2/13 5 Item and Test Analysis, Validity & Reliability
6 2/20 3 Schemas & Test Grids - Affective
7 2/27 3 Affective Formats & Inventories
8 3/6 UB BREAK
9 3/13 4 3 Performance-based Assessment, Rurbrics
10 3/20 4 5-8 Laboratory & Inquiry Skills
11 3/27 6 4 Grading, Portfolios, Journals
12 4/3 4 TIMSS, SED, Item Pools
13 4/10 Revisit Cognitive
14 4/17 School Break
15 4/24 Revisit Affective
16 5/1 Revisit Inquiry
Student Presentations of Portfolios
This course will focus on improving your measurement and evaluation skills and interpretation of assessment data. You will be expected to do specific readings on the above topics, participate in class discussions, and involve yourself in the following activities:
The following "weighting system" will be used unless students
submit a personalized system. Constraints:
1) Total of 100 points,
2) Weights are multiples of 5,
between 5 and 20.
1. a 5
3. a 5
4. a 10
(Return to top
New and Revised Assessments
Secondary Level Examinations
Mathematics, Science and Technology
Science -- Resource Guide with Core Curriculum
||Content-based, multiple-choice questions (approximately 30% of examination)|
||Content and skills-based, multiple-choice and constructed-response questions (approximately 30% of examination, partial credit possible)|
||Content and real-world application, extended constructed-response (approximately 25% of examination, partial credit possible)|
||Laboratory performance tasks, administered prior to on-demand portion of examination (score will comprise 15% of examination grade)|
Society – literacy (science/technology)
Educators – articulation (3 levels)
Parents – vocation
Students – personal growth
Diagnostic -- pretest, remediation
Formative -- feedback, reinforcement
Summative – grading, achievement
Cognitive – norm or criterion referencedTeacher
Affective – interests, values
Psychomotor – lab skills
Logical structureMethods of Evaluation
TestingMethods of Evaluation (Con’t)
Paper-and-pencil: M-C, T-F, Essay
Lab Performance Test
Observations – checklistSituation 1
Self/peer evaluation -- conferences
In photosynthesis, the function of chlorophyll is that of:Situation II
A. an enzyme in digestion
B. carbon dioxide in respiration
C. bile in the digestion of fat
D. glucose in respiration
The following statements are to help you describe yourself in science. Please respond to them as if you were describing yourself to yourself. Do not omit any item. Read each statement carefully; then select one of the five responses listed below.
Completely Mostly Partly Most Completely
False False T/F True True
1 2 3 4 5
Remember, respond to the statements as if you were describing yourself to yourself in science.
I am satisfied with my ability to make predictions.
I do well on number problems in class.
I wish I could make better conclusions based on what I have seen in class.
I am a person who works well with numbers.
I can compare things.
I give up when I have to classify things.
Alarmed by reports of plummeting scores on student achievement tests in science, the school board of Technotown – response to many appeals from its citizens – has commissioned a study of the district’s secondary science program in an effort to identify its weaknesses. The study team – comprised of outside consultants as well as teachers and administrators within the system– will use NSTA’s Guidelines for Self-Assessment package in their work.
The titles of its modules are:
Our School’s Science Curriculum
Our School’s Science Teachers
Science Student/Teacher Interactions
Science Facilities and Teaching Conditions
Assessment, Measurement, Testing and Evaluation
Assessment – collection of information via various formats and modes (qualitative, quantitative) and for various purposes
Measurement – a form of assessment which uses written forms of data collection, to include tests, checklists, and inventories
Testing – specialized mode of assessment which is usually timed, consists of discrete items or questions and is focused on a specified set of objectives
Evaluation – making decisions and judgments based on information collected and assumed or established criteria
group administered tests
In the Future:
End-of-course summative assessment
Variety of formats: large & small group, individual
Variety of formats: pictorial, laboratory performance
Variety of pretest, diagnostic and formative types of measurements
Measurement of low-level cognitive outcomes
Norm-referenced achievement testing
In the Future:
The inclusion of higher level cognitive outcomes (analysis, evaluation, critical thinking) as well as affective (attitudes, interests, and values) and psychomotor outcomes
The inclusion of more criterion-referenced assessment, mastery testing, and self and peer evaluation
Measurement of facts and principles of science
In the Future:
Measurement of student achievement
The inclusion of objectives related to the processes of science, the nature of science, and the interrelationship of science, technology and society
The inclusion of measuring the effects of programs, curricula and teaching techniques
In the Future:
Concern with total test scores
The combined use of teacher-made tests, standardized tests, research instruments, and items from collections assembled by teachers, projects and other sources
Interest in sub-test performance, item difficulty and discrimination, all aided by mechanical and computerized facilities
Currently:Dimensions: Content vs. Behavior
One-dimensional format of evaluation (e.g., a numerical or letter grade)
In the Future:
A multidimensional system of reporting student progress with respect to such variables as concepts, processes, laboratory procedures, classroom discussion, and problem-solving skills
Item Pool Mechanics
Levels of assessment (k, u, i)
(Electronic Database Example)
Biological Science Curriculum Study (BSCS)
Ability to: recall and organize materials learned, apply knowledge to new concrete situations, use skills involved in understanding scientific problems, show relationships between bodies of knowledge
(Doran, p. 23-24)
Comparison of Essay and Objective Items
Incentive to pupils
Ease of preparation
Scoring(Doran, p. 27)
SISS (5, 9)
TIMSS (4, 8, 12)
State Education Departments (NY, California, …)
Saratoga Springs Senior H.S.
Has the old chemistry syllabus been "dumbed down" or is
it now possible to test and teach for real understanding? Productive
ways to think about the chemistry core and the new assessments.
1) Start from the latest version of the core documents
in writing questions. Read the core documents carefully and make
sure your test items stay within the content or skill of the core.
2) Break our thinking about writing the question into parts:
a. what part of the core document does it address
b. what do you want students to do (explain, describe, predict, graph)
c. what is the setting for the question (the lab, the classroom, home)
d. what do you want them to use in their response (concepts, principles, theories)
e. what should their product look like (labeled drawing, paragraph explanation)
3) Describe exactly in the question what you want in your answer. If you want them to use a particular concept or theory in their answer, say so. If you want a labeled diagram and complete sentences say so. "Guess what is on my mind" is not a higher level question. On the other hand, limit extraneous material in the stem. If "taking a walk on the beach" has nothing to do with the problem at hand, then drop it. The use of pictures and data with these questions is fine, but make sure it is necessary to analyze them to arrive at an answer.
4) “Decookbooking" your labs opens up a whole range of testable items as constructing data tables, graphs, writing simple procedures which fall under Standard 1 of the MST.
5) "Real-world" scenarios suggested in Part C appear to be the most difficult to write because they assume all students have had the same experience which allows them to interpret the scenario. It may be better to begin with writing questions where you know that all students have had the same experience, as in your class observing a demonstration or performing a lab.
6) Make sure the information you give is scientifically accurate and reasonable. For example, in chemistry - does the reaction actually occur, are boiling points and melting points accurate, does the compound exist? Reference any sources used for data, e.g. Chemical Rubber Company, handbook, textbooks.
7) Make sure your question could not be just as well addressed as a multiple choice question.
8) "Cue" the answer and make it easier to score by creating widely spaced lines for writing, and boxes for diagrams if they are required on your answer sheet. These suggest appropriate sizes for writing and diagrams. Create constant expectations for the answers, e.g. always use complete sentences, always label diagrams.
9) Construct your scoring guide ahead of time and keep it simple. Make the point totals low on these questions initially until you are confident that the item is sound. Break more elaborate questions into parts, and keep each part low in points, perhaps 2 points for a correct answer, 1 point for partial credit. Think ahead of time what will count as partial credit.
10) Give students practice on constructed response questions before placing them on an exam. These questions are often related to lab situations, so placing them on a post or pre- lab section in a lab is an ideal way to give students some writing practice and gently raise your expectations. Whatever practice you give them, you must go over the scoring guide you would use for the questions.
11) Use your department meetings as a forum to discuss these questions. There is nothing that can beat examining a question from the multiple perspectives of a group of people. Teachers from other disciplines bring a student's point of view to the discussion. Invite each member of your department to bring a proposed question with its scoring guide on a transparency to the next meeting. Suggest a goal in your department that no test this year be entirely multiple choice.
12) Analyze your student answers after the exam or lab. Was the stem of the question stated precisely enough so that students used the scientific knowledge and skills that you expected? Did misconceptions surface that you were unaware of, e.g. particles of a liquid are always farther apart than those of a solid?
Content (Standard 4 of the MST) examples
1) A particular recipe calls for 2.5 cups of sugar and 4 cups of flour to make 24 cookies. How much of each ingredient would be required to make 30 cookies? Show all calculations and explain your reasoning
Scoring guide: Calculation has set up showing units (1) and correct answer
Explanation describes the proportional reasoning involved (2).
(Core Reference: 3.3 c A balanced chemical equation represents
conservation of atoms and mole ratios of reactants and products. (Note
for the purpose of the examination calculations will be limited to mole-mole
2) Solid paraffin sinks in liquid paraffin while solid
water (ice) floats on liquid water. Propose an explanation and use
labeled diagrams of particles(4).
2-Shows particles of solid paraffin closer together than liquid paraffin. Shows ice particles farther apart than water particles. Shows solids having a regular pattern and liquids having a random arrangement.
1-Shows particles of one substance represented correctly
0-Shows no substance represented correctly
Explanation: Describes the correct diagrams accurately (2).
(Core reference: 3.lhh The three phases of matter, i.e. solids, liquids, and gases, have different properties.)
3) Explain why placing a drop of boiling water on your hand does not bum your hand, while placing your hand in boiling water would certainly result in a severe bum. Use the concepts of heat, temperature and calories in your explanation. (4)
(Core References 4.2a Heat is a form of energy, which is the total amount of kinetic energy of the particles in a sample of water, 4.2 b Temperature is a measurement of the average kinetic energy of the particles in a sample of material. Temperature is not a form of energy.)
4) Explain how hydrogen and oxygen can make a completely
new substance with different properties, without gaining or losing any
atoms, but water boiling does not. Use labeled particle diagrams
and a written explanation.
Particle diagrams of hydrogen and oxygen drawn correctly, showing a new particle of water being formed (2).
Water boiling shown with no changes in particles but only spacing (2).
Explanation correctly describing diagrams (2)
(Core Reference: 3.2a A physical change results in the rearrangement of existing particles in a substance. A chemical change results in the formation of different particles with changed properties.)
Describe the contents of each box in terms of elements, compounds and mixtures.
Description of A;
Description of B:
Description of C:
3- Box A consists of a diatomic element, Box B consists of a monatomic element. Box C consists of a mixture of a compound and a monatomic element.
0-2 Each correct box description is worth one point.
( Core references 3.ls - elements; 3.laa compounds; 3.111 mixtures)
6) Crushing a sugar cube will make it dissolve faster in water as will heating the water that it dissolves in. Explain how each of these processes work on a particle level.
Explanation for crushing:
Explanation for heating:
2- Crushing a sugar cube increases the surface area of the sugar crystal, allowing more collisions between the water molecules and the sugar molecules. Heating the water makes the water molecules move faster, increasing their collisions with the sugar crystal.
(Core reference: 3.49- The rate of a chemical reaction depends on several factors: temperature, concentration, nature of reactants, surface area, and the presence of a catalyst.)..
Process Skills (MST Standard 1) questions
1) A fellow student tells you they have determined the
density of zinc using the same equipment in our lab as 7.304 g/mL.
You tell them this is impossible. Show a sample calculation and explain
why this result would not be likely. (4) Sample calculation:
(Core reference: Skill under Standard 1, Mathematical analysis- Analyze data utilizing the concepts of measurement precision and uncertainty as related to significant figures used in calculations).
2) A student collects the following data using equipment in our lab. mass of solid 36.2 grams
volume of graduated cylinder before 5.60 mL
volume of graduated cylinder after adding solid 8.9 mL
a) Analyze the data collected (1)
b) Given the data, calculate the density. (2)
c) Given an accepted value of 7.1 g/mL, find the percent
(Core reference as above)
3) Describe a step by step procedure to separate iron
filings and sugar into separate pure substances identifying all equipment
with its proper name. (not necessarily 7 steps)
2- All necessary steps listed in proper order with equipment.
1- Steps missing or order wrong.
(Core reference: Key idea .2- Standard 1)
4) For an experiment to determine if whether water is distilled (evaporated and condensed) has an effect on its density.:
a) Give the independent and dependent variables.(2)
b) Design the data table.(S)
a) independent variable is the type of water, either distilled or not distilled
dependent variable- density of the water
b) 4 actual equipment readings (4) + two columns (sets
of readings) (1)
distilled water not distilled water
mass of empty flask (g)
buret reading start (mL)
buret reading finish (mL)
mass of flask + water (g)
(Core reference: MST Standard 1, Key Idea 3)
5) Class data for the reaction rate lab was as follows:
Temp.(degrees C) Time (sec)
Graph the data, draw a best fit curve and estimate the
time at 400C.
(Core reference: MST Standard 1, Key idea 1)
6)) Describe an experiment to determine the effect of temperature on the rate an alka-seltzer tablet dissolves. Specify a hypothesis; along with the basis for the hypotheisis, give independent and dependent variables, procedure and data table with units.
5- Each of the above given.
0-4 Each item worth one point
(Core reference: MST Standard 1, Key Idea 2)
The University of the State of New York
THE STATE EDUCATION DEPARTMENT
Office of State Assessment
1. Use constructed response items to measure objectives
that cannot be measured as well with multiple choice items. Good
constructed response items ask students to demonstrate an understanding
or an appreciation of a skill.
2. Indicate clearly the type and length, or depth, of
answer required. Write precise, accurate, readable, and complete
student directions. (p a r c)
3. Make questions or subtasks within a task independent
of each other. Items need to be scorable independently of each other.
(e.g. If students cannot construct a graph, they cannot describe a relationship
supported by the graph.)
4. Develop the rubric or scoring guide at the same time
the item is written. Provide model answers and a range of acceptable
answers. (This step will enhance steps 2 and 3.)
5. Verify the scientific accuracy of all stimulus materials
(weather maps, cross-sections, data tables, diagrams etc.) and provide
6. Remember the four "R's" of item writing: review, reflect,
revise, revise again!
The extended-response (essay) item is used to measure higher-level learning outcomes. It requires students to apply thinking and problem-solving skills, to demonstrate understanding of scientific concepts, and to demonstrate the ability to produce, organize, and express ideas, and to integrate learning from different areas.
The extended-response item is useful for asking students
to perform such tasks as:
• comparing or contrasting two or more things
• describing similarities and differences
• describing relationships
• describing applications of principles
• identifying and explaining cause-effect relationships
• giving examples of principles, concepts, or events
• analyzing a series of events
classifying, sorting, or categorizing explaining or interpreting a passage presenting relevant arguments
stating necessary assumptions
applying principles in novel situations, extrapolating beyond known information deciding or recommending for or against something formulating tenable hypotheses
formulating valid conclusions
Steps in Developing Extended-Response Items
I . Identify the content you want to test. Your item should test important knowledge and skills and be based on content and behaviors contained in the core curriculum guide. Review your assignment and carefully read through those portions of the guide that pertain to your assignment. You may need to locate and review reference materials that relate to your assignment before you begin to write the item.
Identify the higher-order process you want the student to demonstrate. (See Bloom's Taxonomy/Cognitive Activity). The behaviors tested should be those that would be expected of a student in intermediate -level science.
3 Determine that the extended-response format is the best type of item to use.
4. Write a general statement of your idea for the item,
incorporating the content and the cognitive process.
Good answers to constructed-response and extended-response items depend in part upon the directive words that indicate the way in which students should respond. The chart below lists some of the words that can be used to encourage students to demonstrate higher-level cognitive skills.
Break down a complex whole into its component parts so as to discover its
nature or inner relationships
For example: Analyze the various strategies used to study
out points o similarity and points of difference.
For example: Compare the function of carbohydrates and fats.
CONTRAST Bring out
the points of difference.
For example: Contrast the different functions of the large and small intestines.
Review the merits of an item or issue; criticism may approve or disapprove.
For example: Critique the methodology of the author's proposal.
Give the meaning of a word or concept; place it in the class to which it
belongs and set it off from other items in the same class.
For example: Define the "whole language approach to teaching literacy in
an account of tell about; give a word picture of.
For example: Describe how you and your partner collaborated on the project.
Consider from various points of view; present the different sides of. (Item
provide a focus of discussion for the student.)
For example: Discuss the advantages and disadvantages of .
EVALUATE Give the
good points and the bad ones; appraise; give an opinion regarding the value
compare the advantages and limitations of.
For example: Evaluate the usefulness of computers in the classroom.
Make clear; interpret; make plain; tell the meaning of, tell how to do
For example: Explain how hail is formed.
ILLUSTRATE Use a word picture, diagram, chart,
or concrete example to clarify or explain a point.
For example: Draw a diagram to illustrate the rain cycle.
INTERPRET Give the meaning of
your thoughts about; translate.
For example: Interpret the findings on the graph that follows.
Show good reasons for: gain evidence or facts to support your position.
For example: Justify your answer by citing relevant examples that have occurred in the
SUMMARIZE Sum up; give the main points briefly.
For example: Summarize three ways to set up this experiment.
Follow the course of; describe the progress of.
For example: Trace the development of a human embryo from conception, to birth.
(Return to R. Doran's homepage)