Mathematics Textbooks Program

                                An Evaluation Proposal for High School Mathematics Textbooks Program in Jamaica.
                                               
                                         
                                           


                    This is a proposal for the evaluation of a project, re-textbooks used in grades 8 through 11 Mathematics Program in Jamaica’s education system.  The high schools in Jamaica are governed by the Ministry of Education (M.O.E).
Given the interactive nature and history of MOE; partnership with other leading educational organizations such as Jamaica Teachers’ Association (J.T.A.), Joint Board of Teacher Education (J.B.T.E.), and Heart Trust of Jamaica (HEART), together they initiated a collaboration to evaluate the textbooks for high school mathematics.
                  In Jamaica today many young people emerge from the school systems with low passes in mathematics. While there are undoubtedly multiple and complex reasons for this, and no simple remedies, the instructional materials used in schools are a substantial part of the problem.
It would be a great advantage to educators if they could have ready access to reliable information on how well available textbooks support the specific understandings and skills in mathematics that constitute numerical literacy. Whilst, this proposed evaluation is intended to help mathematics educators make better decisions about which mathematics textbooks would most effectively help their students improve their achievement in mathematics. 



Program Description:

                  The goal of the mathematics textbook program in Jamaica is to select, promote, and facilitate the use of the best textbooks for high school mathematics.
The objectives are to:
1.          identify and select textbooks that are grade level appropriate, in  terms of content, method of problem solving and types of activities.
2.          choose textbooks that will fit the mathematics curriculum from grades 8 through 11.

                The program was officially launched at the Jamaica Conference Center on Feb.18, 1999. It is hoped that this project will equip students and teachers with the necessary tools to enhance learning and improve passes in mathematics. The ministry of education is currently proceeding with the textbooks program as a pilot project in 10 traditional high schools that serve a population of about 9 000 students and 300 teachers. The project team has 50 members: MOE (10), JTA (20), JBTE (10), and HEART (10). The project has attracted major funding support of about Ja.$10 million. Funding came from Government, Private Sector Organizations, and from USAID.



Stakeholders:

                The stakeholders are MOE, JTA, JBTE, and HEART. The stakeholders primary concerns would be to: pilot a successful program, select the best set of textbooks from the set that will be reviewed, subsidize cost of books for students, and secure ongoing funding support in order to facilitate subsidy year after year.
                The information provided by this proposal is meaningful to the successful selection of the appropriate texts. At the center of this system is the curriculum itself, which is defined largely by the textbooks students use. Stakeholders would be able to determine that the quality of certain textbooks is critical to any effort to improve student achievement by focusing carefully on how textbooks present some key ideas in mathematics and on the instructional support and guidance they provide for those specific ideas and skills.



Evaluation Purpose and Model:

                A request has been made by the stakeholders and major funding agents to have the program evaluated to make adjustments and improvements where necessary.  The evaluation approach that the writer would recommend to evaluate this program is a combination of objective oriented and CIPP models. In terms of the objective oriented evaluation the writer will look at how the specific objectives will be achieved. In terms of Context Evaluation (C from CIPP Model ) the writer will look at reasons that were used to determine the objectives as well as an effort to gather empirical data that will help to identify the problems, needs, and opportunities present. In terms of Input Evaluation the writer will look at resources employed so as to achieve program objectives. In terms of Process Evaluation the writer will look at how the project is carried out and identify any defects in the procedural design of planned entities. In terms of Product Evaluation the writer will look at how to measure and interpret the attainments yield by the program. In choosing the mathematics learning goals to use as the basis for the evaluation, two conditions will have to be met: (1) The goal (s) must reflect a national consensus on mathematics that all students should know and be able to do, and (2) the intent of the goal (S) must be clear, specific and unambiguous.
                    The standards selected for the evaluation represent core mathematics concepts and skills that any grade 8 through 11 textbook  should cover. This evaluation will select six benchmarks/standards, two in each core strands of number, geometry, and algebra. For number and geometry, both concept and skill will be selected. For algebra, the focus will be on concepts mostly, because developing equation-solving skills is not a central part of grades 8 to 9 curriculum only grades 10 and 11. The accuracy of a textbook’s content is equally important, but it is not always easy to judge whether the content is presented at the proper level of sophistication for a given grade level. That is one reason why very clear and explicit statements of the knowledge and skills students are expected to learn are essential to the project curriculum-materials analysis procedure. The content criteria for the evaluation are:  Number Concepts, Number Skills, Geometry Concepts, Geometry Skills, Graphs, and Symbolic equations.
                    After identifying textbook activities that address a part or all of the content in a benchmark/standard, analysts will rate each activity on how well it addresses the instructional criteria. These instructional criteria are consistent with the principles for effective mathematics learning and teaching found on NCTM’s current revision of its mathematics standard (NCTM, 1998).



Key Evaluation Questions:

                    This evaluation will attempt to analyze mathematics textbooks that attend to both content and instructional design by asking these questions:
1          Does the textbook focus on a coherent set of significant, age-appropriate student learning goals?
2          Does the textbook’s instructional design effectively support the attainment of those specified learning goals?
              The analysis of curriculum-materials will yield critical information in both areas that can be useful in guiding teachers and schools as they make important decisions about adopting textbooks and selecting materials for their students.
                The criteria for making judgments about instruction are derived from research on learning and teaching and on the craft knowledge of experienced educators. Sources include: Implementing the Professional Standards for Teaching Mathematics, (Grouws, 1994) and Guide to Math Materials, (Perry, 1997). The criteria arranged into seven categories – are selected to serve as a basis for the instructional analysis. Indicators and scoring scheme analysts will be used to rate the textbook activities.
The categories are:
Category 1: Identifying a sense of Purpose:  Part of planning a coherent curriculum involves deciding on its purposes and on what learning experiences will likely contribute to achieving those purposes. Three criteria will be used to determine whether the material conveys a unit purpose and a lesson purpose and justifies the sequence of activities.


Categories 11: Building on Student Ideas about Mathematics: Fostering better understanding in students requires taking time to attend to the ideas that are incorrect and ideas that can serve as a foundation for subsequent learning. Four criteria will be used to determine whether the material: specifies perquisite knowledge, alerts teachers to student ideas, assists teachers in identifying student ideas, and addresses misconceptions.

Category 111: Engaging Students in Mathematics:  For students to appreciate the power of mathematics, they need to have a sense of the range and complexity of ideas and applications that mathematics can explain or model. Two criteria will be used to determine whether the material provides: a variety of contexts, and an appropriate number of firsthand experiences.

Category 1V: Developing Mathematical Ideas: Mathematics literacy requires that students see the link between concepts and skills, see mathematics itself as logical and useful, and become skillful at using mathematics. Six criteria will be used to determine whether the material: justifies the importance of benchmark ideas, introduces terms and procedures only as needed, represents ideas accurately, connects benchmark ideas, demonstrates/models procedures, and provides practice.

Category V: Promoting Student Thinking about Mathematics: No matter how clearly materials may present ideas, students (like other people) will devise their own meaning, which may or may not correspond to targeted learning goals. Students need to make their ideas and reasoning explicit and to hold them up to scrutiny and recast them as needed. Three criteria will be used to determine whether the material: encourages students to explain their reasoning, guides students in their interpretation and reasoning, and encourages them to think about what they’ve learned.

Category V1: Assessing Student Progress in Mathematics: Assessments must address the range of skills, applications, and contexts that reflect what students are expected to learn. This is possible only if assessment takes place throughout instruction, not only at the end of a chapter or unit. Three criteria will be used to determine whether the material: aligns assessments with the benchmarks/standards, assesses students through the application of benchmark ideas, and uses embedded assessment.

Category V11: Enhancing the Mathematics Learning Environment: Providing features that enhance the use and implementation of textbook for all students is important. Three criteria will be used to determine whether the material: provides teacher content support, establishes a challenging classroom, and supports all students.



The Evaluation Plan: Design and Methods

                      For the evaluation question: Does the textbook focus on a coherent set of significant age-appropriate student learning goals? The evaluation activities that would be used are interviews and surveys. A sample of teachers would be interviewed as to the quality of textbooks in terms of age-appropriateness, student learning goals and instructional design. A sample of students would be surveyed--- student participants from the 10 traditional high schools. Survey questions would focus on textbook material: structure/sequence of topics and activities. Interview of teachers would be done within two months after sample textbooks are sent to the pilot schools for review and analysis. Each grade level from a pilot school would choose a teacher to represent the school. Teachers would be interviewed via telephone or face to face. The data would be collected and the responses measured by the Likert Scale; responses from “strongly agree” to “strongly disagree” on a scale of 5 to 1. That is, a response of “strongly agree” is assigned 5 points and a response of “strongly disagree” is assigned 1 point. An interview and survey forms would be established to record responses. Twenty questions would be listed in each case. A score of 100 points is a strong indication that the textbooks are suitable, whilst a score of 20 points would be indicating that the textbooks are unsuitable.

                        For the evaluation question: Does the textbooks instructional design, effectively support the attainment of those specific learning goals? For this question a survey would be conducted. Survey forms would be sent to all the pilot schools via mail to be administered and answered by teachers of mathematics, grades 8 through 11. After 10 working days of posting the survey letters, a follow-up phone call would be made to all pilot schools to ensure that the survey was administered. Survey answer-forms would then be returned to the evaluator. The Likert Scale would be used as explained in question 1. An interpreted time-series design would be chosen. This would verify the specific learning goals of students’ performance on administered achievement tests throughout a time period before the textbook program intervention and thus compare it with tests administered after intervention. A t-test would be used to compare the two means. The writer/evaluator would present findings to stakeholders. The estimated cost of resource materials and miscellaneous will be about two hundred and fifty thousand dollars, that is, Ja.$250 000.






References:



Grouws, D.A. (1994). Implementing the professional standards for teaching mathematics: Evaluation of teaching. Challenge and opportunity. 87 (6), 446-48.

NCTM. (1997). National council of teachers of mathematics 1997-98 handbook: NCTM goals. Reston: NCTM Inc.

Perry, P.J. (1997). Guide to math materials: Resources to support the NCTM standards. Englewood: Teacher ideas press/libraries unlimited.