Evaluation of a Project

https://www.amazon.com/Scholastic-Educational-Research-Technology-Mathematics/dp...
***
Evaluation of Project Get Going.

Introduction.
          Project “Get Going” (GG) has gained much attention because it is one of several research projects being conducted at a prominent university, designed to enhance the computer skills of incoming Instructional Technology and Distance Education (ITDE) students. It involved the use of a multimedia package incorporated popular music and culturally appropriate vignettes and exercises. GG was initiated with 21 incoming graduate students who were divided into three teams: the red team, the blue team, and the green team. Each team attended daily hour-long, lab-based instruction using the multimedia package for 25 days before they began their graduate studies.
          Project GG was designed to answer the following questions:
•          Did participation in the instructional program result in an increase in students’ comprehension and mastery of computer concepts and skills?
•          Is there a significant relationship between the pre and posttest scores for the entire sample group? For the males? For the females? For each of the teams?
•          Is there a significant difference between pre and posttest scores for the sample, as a whole? For the males? For the females? For each of the teams?   
From the research questions, if the results showed that Project GG has significantly increased the computer skills of the students involved, The university has promised to replicate it in at least five other graduate programs at the university.
          The evaluation was conducted wherefore all participants were given a pre and a posttest to measure their level of comprehension and mastery of computer concepts and skills prior to, and upon completion of Project GG. The data as set out in Appendix A reflect the pre and posttest scores of the 21 students. The analysis is as follows:

Findings.
Descriptive Analysis.




Table1
Percentage Distribution of Students by Teams and Gender
Teams                n              Male                    Female                        Alla
________________________________________________________________________
Red                    7                    28.6                        71.4                              33.3
Blue                    7                    71.4                        28.6                              33.3
Green                  7                    57.1                        42.9                              33.3
All                    21                    52.4                        47.6                            100.0
Note.
aTeam as percentage of group.

As shown in Table 1, there is a greater percentage of males overall. The three teams have the same percentage of participants; however, there is a greater percentage of males in the Green team and also in the Blue team but for the Red team, there is a greater percentage of females (see Appendix F for crosstabs).


Table 2                                                                      
Statistical Analysis of Students by Gender, Team, and Group
Group          n          Mean          Median          Mode          Variance          SD          Range
                             Gender                                        
Male                                                                      
Pretest          11          13.82          14.0          13a          7.36          2.71          8
Posttest          11          18.36          19.0          20          10.65          3.26          11
Female                                                                      
Pretest          10          15.30          14.5          14a          21.79          4.67          14
Posttest          10          21.70          22.5          15a          32.68          5.72          14
                             Teams                                        
Red                                                                      
Pretest          7          15.29          15.0          10a          13.90          3.73          11
Posttest          7          22.71          22.0          20          19.24          4.39          12
Blue                                                                      
Pretest          7          14.71          14.0          9a          20.90          4.57          14
Posttest          7          18.57          16.0          16          27.62          5.26          16
Green                                                                      
Pretest          7          13.57          14.0          9a          10.29          3.21          9
Posttest          7          18.57          19.0          15          16.29          4.04          10
All                                                                      
Pretest          21          14.52          14.0          14          14.06          3.75          14
Posttest          21          19.95          20.0          16a          22.95          4.79          16
Note.
aMultiple modes exist. The smallest value is shown.






As shown in Table 2, there is a distinct difference between pretest and posttest after treatment in all categories; viz. Gender, Team, and Group. For Group: The means for pre and posttest showed a difference after treatment. The standard deviation showed a difference in variation in scores on pretest and posttest after treatment (see Appendix B). For Gender: Female performed better in both pretest and posttest with a difference in means of 1.48 and a difference in standard deviation of 3.34 for pretest and posttest respectively. The standard deviation for pretest and posttest showed that female scores had a greater spread (see Appendix D). For Team: The means for pretest and posttest for all the teams showed a difference after treatment, in that the mean increased in each case. However, it appears that the treatment has a greater effect on the Red team. The standard deviation for the pretest and posttest increased in each case for the respective teams but the Green team seemed to have had the greatest spread of scores (see Appendix C).






________________________________________________________________________
Table 3

Percentage Distribution of Students by Range of Scores


Score          Pretest          Posttest
4-9          9.5          -
10-15          52.4          19.0
16-21          33.3          47.7
Over 21          4.8          33.3

Note.
Scores for entire group.
Dash indicates that there was no score for the category.

From Table 3, it can be seen that for the pretest, more than 60% of the students scored 15 and below and less than 40% of the students scored 16 and above. After the treatment however, more than 80% of the students scored 16 and over. This showed that there were some amount of improvement with a range of scores of 14, that is, (9 to 23) for the pretest and 16, that is, (13 to 29) for the posttest (see Appendix B).

Correlation Analysis.

__________________________________________________________
Table 4

Correlation of Student scores

         n          df          r          r2          Variance          Sig. Value          Sig.
Group          21          20          .825**          .68          68          .000          .01
Gender                                                                      
Male          11          10          .607*          .36          36          .048          .05
Female          10          9          .903**          .81          81          .000          .01
Teams                                                                      
Blue          7          6          .910**          .82          82          .004          .01
Green          7          6          .756*          .57          57          .049          .05
Red          7          6          .862*          .74          74          .013          .05

Note.
**Correlation is significant at .01 level (two-tailed).
*Correlation is significant at .05 level (two-tailed).


For the Group (Table 4) a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A strong positive correlation was found (see Appendix E). Thus (r (20) = .825, p < .01) level, thus indicating a significant linear relationship between the two variables. For the Male Gender, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A moderately positive correlation was found (see Appendix E). Thus (r (10) = .607, p < .05) level, thus indicating a significant linear relationship between the two variables. For the Female Gender, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A very strong positive correlation was found (see Appendix E). Thus (r (9) = .903, p < .01) level, thus indicating a significant linear relationship between the two variables. For the Blue Team, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A very strong positive correlation was found (see Appendix E). Thus (r (6) = .910, p < .01) level, thus indicating a significant linear relationship between the two variables. For the Green Team, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A moderately positive correlation was found (see Appendix E). Thus (r (6) = .756, p < .05) level, thus indicating a significant linear relationship between the two variables. For the Red Team, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A strong positive correlation was found (see Appendix E). Thus (r (6) = .862, p < .05) level, thus indicating a significant linear relationship between the two variables.
From Table 4, however, the female scores are more predictable than the male scores and the blue team scores are more predictable than the green team and red team scores.




Paired T-test Analysis.

________________________________________________________________
Table 5

Comparison of Population of Pretest and Posttest means

Variable          Pre-t M          SD          Post-t M          SD          T-value          Sig. Lev          df
Group          14.52          3.75          19.95          4.79          9.172          .000          20
Male          13.81          2.71          18.36          3.26          5.590          .000          10
Female          15.30          4.66          21.70          5.71          8.085          .000          9
Red          15.28          3.72          22.71          4.38          8.832          .000          6
Blue          14.71          4.57          18.57          5.27          4.653          .003          6
Green          13.57          3.20          18.57          4.03          5.000          .002          6

Note.
P < .05 (two-tailed test)






A paired sample t-test was calculated to compare the mean pretest score and the mean posttest score for the group, for males, for females, and for each team. For the Group, the mean on the pretest was 14.52, (SD = 3.75) and the mean on the posttest was 19.95, (SD = 4.79). A significant increase from pretest and posttest was found. Thus (t (20) =9.172, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Male Gender, the mean on the pretest was 13.81, (SD = 2.71) and the mean on the posttest was 18.36, (SD = 3.26). An increase from pretest and posttest was found. Thus (t (10) = 5.590, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Female Gender, the mean on the pretest was 15.30, (SD = 4.66) and the mean on the posttest was 21.70, (SD = 5.71). A significant increase from pretest and posttest was found. Thus (t (9) = 8.085, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Red Team, the mean on the pretest was 15.28, (SD = 3.72) and the mean on the posttest was 22.71, (SD = 4.38). A significant increase from pretest and posttest was found. Thus (t (6) = 8.832, p < .01); thus indicating a difference in means with at least 1% chance of errors. For the Blue Team, the mean on the pretest was 14.71, (SD = 4.57) and the mean on the posttest was 18.57, (SD = 5.27). An increase from pretest and posttest was found. Thus (t (6) = 4.653, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Green Team, the mean on the pretest was 13.57, (SD = 3.20) and the mean on the posttest was 18.57, (SD = 4.03). An increase from pretest and posttest was found. Thus (t (6) = 5.00, p < .01); thus indicating a difference in means with at least a 1% chance of errors.
Gender Comparison.

________________________________________________________________________
Table 6

Group T-test on Pretest and Posttest by Gender

         N          df          Pre-M          SD          Post M          SD          Pre-t-val          Post  t-val          Pre  t-sig          Post  t-sig          Pre  f-val          Post f-val
Mal          11          9          13.8          2.71          18.4          3.26          .900  .878          1.66 1.62          .379 .395          .113 .127          3.23          4.66
Fem          10          8          15.3          4.67          21.7          5.72          .900 .878          1.66 1.62          .379 .395          .113 .127          .088          .045
Note.
Val - represents value.
Mal- represents male.
Fem- represents female.   

The Levene’s Test for equality of variances showed that for the pretest, F = 3.228 and the Significant Level = .088. Thus, indicating that there is not a significant difference between the means either at p < .05 or p < .01 levels. For the posttest F = 4.611 and significant level = .045; indicating that there is not a significant difference between the means at p < .01, only at p < .05 level. Hence, assuming equal variances at p < .01, then the pooled variance estimate would be employed for this purpose. It was seen that there was no significant difference for t-value although the means and standard deviations on the pretest and posttest were higher for females. (see Appendix D).

Team Comparison.
________________________________________________________________________
Table 7
Comparison of Teams on Pretest and Posttest. ANOVA showing Means for each Team.
Team          n          PretestM          SD          Post    M           SD          F-Pretest          F-Post - test          Sig. Pretest          Sig. Post
Red          7          15.29          3.73          22.7          4.39          0.355          1.903          .706          .178
Blue          7          14.71          4.57          18.57          5.26                                        
Green          7          13.57          3.21          18.57          4.04                                        



Differences in performance for the teams on both tests were not significant. Both pretest and posttest means were compared using a one-way ANOVA. No significant difference was found.
For Pretest ( F (2,18) = (.355), p > .05).
For Posttest ( F (2,18) = (1.903), p> .05).
Hence, the three teams did not differ significantly with respect to their performance on
pre and post test.  (Appendix B)

Summary
          The  scores for pretest and posttest were analyzed using measures of central
tendency, measures of variability, correlation, and analysis of variance.  A statistical
analysis of students was done to find out whether or not there is any difference in
performance by Gender or Team.  The analysis was done for the entire group, gender,
and team.  Further analysis was performed so as to pinpoint whether or not the
difference was significant.  A correlation analysis was performed in each case for the
pretest and posttest measures.  The correlation was graphed in each case for the two
variables to indicate whether or not the scattergraph shows a positive correlation
(an increase in one variable is associated with an increase in the other variable) or
a negative correlation (an increase in one variable is associated with a decrease in
the other variable) (Appendix E).  This will tell us about the strength/weakness of
the relationship between the variables.  The Pearson product moment (Pearson r)
was chosen because: 1.  The data used were measured on interval scale, and 2.  The
variables (pretest and posttest) had a linear relationship in all cases.
          The t-test was used to compare the pre and posttest means for the Group, Gender,
and Teams.  The paired t-test was chosen to compare the pre and posttest in each case.
An independent t-test was chosen for the Gender.  The Levene’s Test was done to test
for equality of variances, thus, to test for the significance of the difference between
means.  The F-test was done for Team Comparison to check if there were a significant
difference between means.
          For the entire group, there was an increase in means from pretest to posttest scores.
For Gender, there was an increase in means from pretest to posttest scores for males and
for females respectively.  However, when the t-test was used for gender comparison there
was not a significant difference in means for males as compared to females.  For Teams,
there was an increase in means from pretest to posttest scores within each team.
However, when the ANOVA, F-test was used for team comparison, no significant
difference was found when pre and posttest means were compared.
          For research question 1. Did participation in Project GG increase students comprehension and mastery of computer concepts and skills?  Yes, because in all cases: for the entire group, for gender, and for the teams there is a marked increase and difference of pretest and posttest scores there is a significant difference in each case for at least at the .05 level, that is p < .05.  This is to say that after the treatment, the posttest revealed an improvement in scores.
For research question 2.  Is there a significant relationship between pre and posttest Scores for the entire group?  For males?  For females?  and for each of the teams?
The scatter graph (Appendix E) showed that there is a significant relationship between pre and posttest for : the entire group, which showed a strong positive correlation (r = .825) the Male Gender which showed a moderately positive correlation (r = 0.607); the Female Gender, which showed a very strong positive correlation (r = .903);the Blue Team which showed a very strong positive correlation (r = .910).  The Green Team, which showed a moderately positive correlation (r = .756) and the Red Team, which showed a strong positive correlation (r = .862).
From all indications the treatment made a positive difference in enhancing the comprehension and mastery of computer concepts and skills of the participants.


Interpretation and Recommendations
          The findings of the study might have been affected by the sample size. The writer needs to know the likely size of the population and how the sample of 21 was selected,     
that is, what sampling procedure was used to select the 21 participants.  The
multimedia package that was used during the treatment might have been culturally biased.  The method of instruction for the treatment might have been a crucial factor in
affecting one’s performance.  There is nothing to say whether or not the team worked cooperatively or individually within teams.  To determine whether or not the treatment made a difference, the writer will look at the statistical power of the tests.
Power is the ability of the test to show a statistical difference.  Statistical power,
in fact, is the probability of avoiding a Type 11 error ( i.e. the probability of retaining the null hypothesis when it should be rejected).
    For Project GG the question is whether or not the analysis had enough POWER to show a significant difference when the treatment (viz. Lab-based instruction using multimedia package to enhance the computer skills of incoming ITDE students) was concluded. The power of a significance test depends on the four interrelated factors: 1.  Sample size,  2. Alpha level,  3.  Directionality, and  4.  The effect size.
    The Sample size for Project GG (21) could affect the statistical power.  For example,
power increases automatically with an increase in sample size.  Thus, virtually any
difference can be made significant if the sample is large enough. A standard sample size
of a minimum of 30 subjects per group is recommended in most studies. However, in
well – designed experimental studies, a smaller sample size is acceptable.
          The Alpha level selected for Project GG (p < .05 or p < .01) might have affected the POWER of the treatment depending on choice.  Increasing the significance level – say, from .05 to .01 thus increases the statistical power. So, the statistical significance tests might have been more powerful if there were more significance at the .01 level.
          The Directionality refers to one-tailed or two-tailed.  For Project GG, a non-directional (2-tailed) was chosen.  This choice however, although opt as the more
conservative, usually lessen the statistical power.
The Effect Size = (mean of Post T – mean of Pre T)
                              Avg. SD of Pre T and Post T 

                          =19.9524-14.5238 
                            (3.7499+4.7904)/2
                          =5.4286/4.2702
                          =1.2713    (1.2713 means a big difference, i.e. 1.2713 SD).
          The Effect Size for Project GG is calculated to be about 1.2713, which is considered high or a big difference.  This is a desired Effect Size. 
          A close examination of the statistical power of Project GG would suggest that if
power were to be increased then the writer would recommend:  1. an increase in sample
size, 2. An increase in significance level, and 3.  assume a research hypothesis that is
directional, using the one-tailed test.  The writer would recommend that more
demographic information be collected about the participants such as age group, occupation, and previous knowledge on computer skills. For expansion, Project GG could also experiment separately on single gender.