The school district has released an updated report from UC Davis researchers Scott Carrell, et al, who conducted a study analyzing the impact of the AIM program on students in and outside of the program. Based on requests following the presentation on June 4, they have released an update which includes a discussion of the subgroup analyses that they presented at the June 4 board meeting.
The main finding of that report was a “no benefit, no-harm” finding. The researchers found “no evidence that the program positively affects achievement scores of participating students and no evidence that the program negatively affects nonparticipating students, on average.”
They add, “This ‘no benefit, no-harm’ finding should be considered within the context of the program’s cost: the DJUSD spends considerable resources on universal testing and retesting (as do private citizens for additional testing) and given the financial costs and capacity constraints associated with this program, we should expect these costs to be balanced by some measurable benefit.”
They add that the analysis does “not allow us to make explicit recommendations about what an ideal gifted and talented program in the DJUSD should look like. We can conclude, however, that clearer identification of ‘gifted’ children will increase program transparency.”
While the main bulk of their analysis found that there was “no benefit, no-harm” – presented at the June 4 meeting was a subgroup analysis that found one big exception being with Hispanic students.
On page 8 of the revised report, they write:
We also estimate the effects of the AIM program on students not in AIM, for the different subgroups described in Section 4c. Using the same estimation strategy that we use for the full sample in Section 4d, and across different outcomes and specifications, we find no consistent evidence that the AIM program affects any of these subgroups, save for Hispanic students. For Hispanic students, we consistently find that AIM negatively impacts both math and ELA scores. Specifically, we find that AIM reduces the math CST scores of Hispanic students who are not in the AIM program by 21 points and the ELA CST scores of Hispanic students not in the AIM program by 15 points.
That leads to a big question as to why would Hispanic students be negatively impacted by AIM, and why only them?
The Vanguard is hoping to sit down with the researchers to clarify a number of things. However, we have a number of methodological questions that need to be resolved.
As we noted in a previous analysis (Sunday Commentary II: Is the GATE Program In Need of Change?, July 12, 2015), the researchers had to grapple with some methodological issues in order to attempt to create a natural control experiment in order “to disentangle the actual effect of a program from the effect of different types of individuals choosing to participate or not participate in a program.”
They ultimately settle on a regression discontinuity research design: “However, because of how students qualify for AIM in the DJUSD, certain parts of the qualification process are effectively random. It is this randomization that we exploit to estimate the effect of the AIM program on students in the program. This approach is called a regression discontinuity research design (RDD).”
They measure those who just qualified for the AIM program against those who just missed qualification based on their OLSAT score. Given questions about the OLSAT, that test may lack a lot of validity. We brought up a number of concerns about this approach back in July, which should be considered.
However, in the end they find, “Students just to the right of the qualification threshold score, on average, 3 points higher on their ELA CST [English-language Arts California Standards Tests] and 6 points lower on their math CST than students just to the left of the qualification threshold. These effects are small and statistically indistinguishable from zero. As such, there is no evidence that the AIM program has an effect on students in the AIM program.”
The dependent variable in this study was the score on the California Standards Test (CST) or STAR (Standardized Testing and Reporting) Test. In our discussions with a number of experts, there were serious questions raised as to whether enrollment in the GATE/AIM program is going to have any effect on the student’s score on the CST.
Whether that objection is valid or not, that would seem to be something that the researchers need to establish. If their dependent variable is not impacted by the program, then of course we would expect no effect. The fact that they in fact find very small statistical effects further casts doubt on the validity of their measure – but this problem is never scrutinized in their report.
It is important to note that the state has in fact stopped using the STAR as of July 1, 2013. Part of the reason for that change was the belief the STAR test was no longer a proper measure of student growth. The STAR test, which was for the most part multiple choice questions, was scrapped in favor of more rigorous, thought-provoking exams.
“Multiple-choice, fill-in-the-bubble tests alone simply cannot do the job anymore, and it’s time for California to move forward with assessments that measure the real-world skills our students need to be ready for a career and for college,” said State Superintendent of Public Instruction Tom Torlakson in January 2013.
In addition to questions about the use of CST scores for the dependent variable measuring student performance and the effectiveness of the AIM program, we have questions as to why Hispanics and Hispanics alone are negatively impacted by AIM. We have only the data analysis presented by the researchers with no operationalized theory to guide us.
This addition prompts more questions – questions that the Vanguard hopes to answer in the coming weeks.
—David M. Greenwald reporting
so we have a study where we aren’t even sure the main variable is measuring performance, bad enough that the state ended the testing, and that’s the basis for the no harm/ no benefit conclusion. this is bad work.
Vanguard: That leads to a big question as to why would Hispanic students be negatively impacted by AIM, and why only them?
I hypothesize that it is related to education (and commensurate income) level of the parents. There are many Latino students who come from families with parents who have a college degree. But there is a higher percentage of Latino students with parents who have lower levels of education, and more often little to no college experience. There are typically incomes and jobs that are commensurate with lower levels of education.
The pattern you state is common when such students (those who have parents with lower levels of education and income) start to be separated from a more heterogeneous mix of students (yes, some would use the word “segregation” here, and others would be offended to hear that word used). In other words, the achievement gap, as measured by standardized test scores like the CST, widens.
The public radio show “This American Life” ran a piece a couple of weeks ago, entitled “The Problem We All Live With,” that discussed the phenomenon of school segregation and the affect it has on the achievement gap. It highlighted an interesting issue which recently occurred in the school district which serves Ferguson, Missouri, in which integration happened more by accident. An integrated school model seems to be the most effective way to reduce the achievement gap, and it isn’t because test scores of high performing students are dragged down, but rather that normally lower-performing students perform better. Here is that American Life piece available as a podcast.
One way to test this hypothesis is to ask for education levels of the parents and see how it correlates to other data. It is data that the district keeps for most students when they first enroll in the district.
i’m skeptical that you would see such a pronounced effect.
first, the idea that you don’t have mixed classes seems off. you are not pulling enough students out of the classroom and you still have high achieving students in the classroom.
second, the measurement errors are substantial as the vanguard article points out.
seems to me we need a lot more data here to justify ending or drastically changing the program.
DP: first, the idea that you don’t have mixed classes seems off. you are not pulling enough students out of the classroom and you still have high achieving students in the classroom.
Demographic changes vs. standardized test scores at Montgomery Elementary (MME) might help answer your comments. But GATE/AIM identification & transfer alone may not be the only reason to see those effects at that site. MME went into Program Improvement (a provision under No Child Left Behind) a few years ago, and the district had to accommodate all requests from MME families to transfer to another school not in Program Improvement. In the district, MME has the highest percentage of Latino students and the highest percentage of ELL students. A few years ago, those percentages were lower at MME. (source)
The District should pour money into MME – maintain very low class sizes, technology, reading assistants, money for fantastic field trips, guest instructors, free after-school extra curricular program, and more. I would support that. Right now, it is left to organizations in the community to donate to MME and provide even basics, i.e. printer paper. It is no wonder that parents transfer their children over to Pioneer.
ryankelly: The District should pour money into MME…
At present the district does put more resources into MME; whether it’s enough might be up for debate. The after school Bridge Homework Club still operates at MME. Also the LCAP/LCFF process designates more money to higher needs students, hence MME receives more money for that reason, having more higher needs students.
With regard to this:
We have this from a July Vanguard article:
Don… is that $20 per year? Month? Etc.?
There’s a bigger question here. Before you get to the question of does this program help (or harm), you have to ask is this program appropriate in the public school system. The original GATE program was for a small group of students who could not learn (as in fail) in the mixed classroom due to sensory, social, processing issues. But that isn’t our AIM program. The students in our AIM program are the same students as in the regular classroom…a mix of students requiring differentiation. Given that this is public (not private) education, what right do we have to give a private classroom to one student who doesn’t require it, but not another (for instance, slower or remedial students)?
They are not “the same students” with respect to aptitude and learning style. That is the point of testing in the first place. And not all of them would be appropriately placed in a “differentiated” classroom.
Thank you for revealing your actual goal: elimination of GATE in the Davis schools. We’ll keep that in mind as we read your future posts on this topic.
Don,
Bullying comes in many forms.
In the case of your comment, I would request that you acknowledge the special position you occupy as a monitor of this site.
Your blanket assertion that this poster has a goal of eliminating AIM seems a reach too far.
More importantly, you seem to have little interest in exploring the subtle distinctions between those who would support AIM through differentiated instruction, including the use of clusters, within the context of a standard classroom versus those who advocated separate classroom instruction. And then, of course, there is the question of the standards to be employed and the associated testing employed to identify the distinctive needs of the AIM learners.
Painting with your broad brush truly does marginalize, if not alienate, those who might otherwise share your ultimate goal.
Doby,
When I am acting as volunteer moderator of the Vanguard I preface my comments with [moderator] and they are in italics. It has no bearing on my participation here when I am commenting on topics.
Someone who questions whether a program is “appropriate in the public school system” is in fact asking an existential question about GATE. Not whether it should be better, but whether it should exist at all.
We have explored those options at length over various posts on the Vanguard.
— I’ve discussed the San Diego school district program and have specifically suggested that the district could develop a pilot program of clusters of GATE-identified students in differentiated classrooms, and retain the self-contained classrooms for those who need them. See how the parents, teachers, and students buy into the change and what the outcomes are. I would add, based on comments by wdf, that they should do outreach to the Hispanic community to explain and communicate more effectively about gifted programs.
— I have also stated my opinion that this would require more resources, not fewer, and would cost the district more money than the current program does.
— I have said that if the goal is simply to reduce the size of the current program (focus on the numbers), then the status quo is better.
— And I have made it clear that I have little faith in the board majority at this point because of how they’ve handled things so far. A lot is riding on what staff recommends in September, but I am very skeptical.
It is clear that some posting here are members of the group that formed 3 – 4 years ago to oppose the present GATE program. Some undoubtedly signed the change.org petition that was circulating then. Many of those were quite strident about their opposition to GATE.
We have heard all the charges of elitism and segregation, fairness and equity, the derisive ‘Tiger Mom’ comments, the assertion that GATE increases the achievement gap, that it harms campus climate. That GATE takes resources from other programs. The notion that gifted students should be in mixed-ability classrooms because it is better for the other students. That they should mentor or tutor others. In short: a lot of denigration of gifted programs and a lack of acceptance/awareness of gifted learning issues.
I accept your implication that those opposed to the current GATE program are probably on a spectrum as to their preferred outcomes. When they question the legitimacy of the program’s existence, that’s pretty far to one end of the spectrum.
I want to state clearly that I have was and never been a member of any group that formed 3 – 4 years ago to oppose the present GATE program. I did not sign any petition to change the GATE program back then. I do know that this group appeared after years of conversation and criticism with no response from the District. I do know that many of the people, including one Board member, were members of a group who fought in opposition to any change and are ardent supporters of maintaining the status quo. I wonder if you were also a member of Davis Excel, Don. I also believe that last the election was about the management of the District, including the AIM program, and people knowingly voted accordingly. I believe that the aggressive defense of all things AIM and angry accusations and snide remarks alienates people and makes them cautious about speaking freely in public. I don’t know of anyone who has said they wanted to dismantle the self-contained GATE program or has that as a goal. However, the AIM program as it have evolved into over the last 5 years deserves a critical examination.
Fair question. No.
Among others, it is difficult for me to come to any other conclusion from the comments of Dr. Jann Murray-Garcia.
Jann Murray Garcia – 1 (her argument is that how the program is run is discriminatory toward students of color).
But none who have posted comments here.
Frankly has clearly stated that self-contained programs should be eliminated:
Davis Excel pulled a number of links to studies on this page here:
http://davisexcel.org/wp-content/uploads/2013/03/LitReview_E.pdf
Don, thank you for making this site and the others enclosed in it available to the many people who are not aware of them. Perhaps I should have said thanks for making it available to the many people who are not interested in them. Whether they will read even the short excerpts is not very hard to say. I would like to challenge the critics of AIM to inform themselves, as the proponents have done.
Don,
Doby hit this one on the nail…skip the cheap shots, they don’t help your argument and as a moderator, you need to set the bar extra high for yourself. You are very wrong in your assessment about my intentions for AIM. I absolutely believe there are some students who require a separate classroom to acheive a K12 education, but I also believe this is not who is in our current AIM program…most of these students are in IS, homeschool, or private school. I would like to see them back in our schools. I also believe we have to accommodate the needs of all our students, including the remedial and the high achieving. But these are not students who require a separate classroom, and for me, require is the operative word. When I posed the question, is the AIM program appropriate in the public school system, I was referring to our program specifically, not existentially referring to gifted or high achieving education. I believe our AIM program is not a gifted program, and while it contains many high achieving students, it’s no longer even a true high achiever program. You and I clearly disagree about who is in the program. I believe the large group of students who scored between 85 and 95 on the OLSAT are excellent students who require enrichment and differentiation. You believe they require separation.
Previous comment:
Bottom line: you do, or do not, believe that DJUSD should have a self-contained GATE program for anyone? You are not being clear. It is not a “cheap shot.” Most of your posts have been about the process of qualifying for GATE, which is certainly a reasonable discussion. But again you have posed an existential question about GATE; or, rather, in this case made an existential statement: “these are not students who require a separate classroom.”
So, to my question, it should be a simple answer.
My answer is: I believe a self-contained option is necessary for some gifted students in Davis.
What is your answer?
My role as moderator here is irrelevant. I have not made any ad hominems or violated the Vanguard Comment Policy.
I was pretty clear, but here it is again…”I absolutely believe there are some students who require a separate classroom to acheive a K12 education”. My posts have been about the AIM identification process because I also believe “these are not the students who are currently in our AIM program”. If you’re interested in possibly changing the mind of someone who disagrees with you, skip the condescending attitude. Otherwise, it’s just venting.
Actually, you’ve made previous comments that suggest otherwise:
(6/7/15)
Emphasis added. So it’s pretty clear to me that, at times, you have advocated against self-contained GATE.
This is the description of the two GATE programs in Davis in 2003-2004 by Deanne Quinn. Note the high-achievers were given differentiated instruction and honors classes and not placed in the self-contained class. See how far we’ve come.
http://www.dcn.davis.ca.us/go/hjh/administration/School_notices/GATE_Testing_Information.pdf
I assume that at that time differentiated instruction was at NDE, and at the junior high level it was at Emerson. Our (unsuccessful) experience was with the Emerson program. As noted before, the closure of Valley Oak prompted a number of changes. As to the tests used, my impression from all of the discussions over the years is that there was an attempt to make a more diverse GATE demographic. The way they achieved that appears to have satisfied nobody.
High-achieving elementary students stayed at their neighborhood schools and received differentiated instruction. Both Holmes and Emerson had honors English and Social Studies and Science. Holmes had one track of GATE plus the honors classes, so the programs were different. There are no longer these honors classes at any of the Jr. Highs. It is either GATE or regular program (or DaVinci or DSIS). Emerson may have some sort of Spanish Immersion program for Chavez students.
…”you have advocated against self contained GATE”. I have advocated against Davis’s version of self-contained GATE. I have consistently advocated for a program for students who cannot learn in the regular program. This may include autism spectrum students including students with Aspergers, Tourette’s, processing issues, and social and/or sensory anxieties. I would support any program for these students, with a clear, consistent, and reasonable policy for identifying them. This is not what we have.
That is a reasonable clarification. Thank you.
Don, there was no GATE program at Emerson. There were Honors (differentiated instruction for high achievers) at both Emerson and Holmes. For a GATE identified student who is not also high achieving, the honors classes might not work, which is probably what you ran into with your son.
https://eme-djusd-ca.schoolloop.com/aim
My child was in the GATE program at Emerson. The assistant superintendent very firmly insisted to me that it was GATE. She seemed pretty defensive about it. But it was clear that the focus wasn’t for kids like mine.
Herein lies the confusion over the difference that Deanne Quinn described and collapse of GATE and high achieving students. The courses offered at Emerson was the high achieving honors track at Holmes. There was only one track, and then maybe two, of GATE classes at Holmes and not all incoming sixth graders could be accomodated, as there were 4 GATE elementary tracks. So Emerson put all their GATE students into the courses meant for high achievers. You son ended up at DSIS, as many other students did who were not served well by this misunderstanding.
Don, Doby does have it right. You have the privilege of time to post on this web-site almost 24/7. That’s one reason you’re a good moderator. But you also read some things once, jump to conclusions, and put words in people’s mouths, stopping others from joining the conversation. Seems anti-Vanguard.
Mrs. W, your post summarizes the reason that Don has chosen to no longer talk to me either here in the Vanguard or in the rest of our respective lives. Further tying into your comment above, Don has accused others (including myself) of putting words into his mouth, specifically when I have observed that he has actively taken the position that certain subjects should not be talked about on the Vanguard (three examples of which are Mike Harrington’s incendiary views on water, Frankly’s views on Mace 391, and my views about discretionary intra-district student transfers within DJUSD).
Perhaps Don will give your observation above more credence than he has given to prior efforts to carry the same message to him. Don can be a superb poster, is a very solid asset of the Davis community, and has my respect, but sometimes the expression that “reasonable people can agree to disagree reasonably” eludes him in certain conversational topics here on the Vanguard.
JMHO
False. I have not taken the position that “certain subjects should not be talked about on the Vanguard.” But this is a good example, again, of why I will not interact with Matt on the Vanguard.
Don, have you or have you not told Frankly that he should stop talking about Mace 391?
Did you or did you not forcefully state in 2013 that Mike Harrington should stop making the statements that he was making about the water project … because you believed that those statements were factually inaccurate.
Did you or did you not tell me that I needed to stop talking about ending discretionary school selections for individual students (outside the boundaries of magnet programs) within the DJUSD?
The honest answers to those three questions are “Yes I, Don Shor, did.”
… and with the following post by Don to Sue Greenwald, we can add her to the list above.
i will defend Don on Mike Harrington. He stated what many people felt about his behavior after months of his uncensored attacks, outright lies, and refusal to be factual, bravado, and slimey unsincere friendliness.
Find another example, Matt. Harrington is not a good one.
Ryan, I understand your perspective. You have been on record many times on that subject. I respect that opinion, but believe that censorship regardless of what cloth it comes in is just that … censorship.
Everyone should take a deep breath regarding G.A.T.E.; remember it’s just one way to address learning. I taught a number of G.A.T.E. classes, attended numerous meetings, and understand the concerns and different philosophies being bantered about.
Bottom line is this: As a G.A.TE. teacher you have a class roster, a defined curriculum, engaged parents, and a wide range of student abilities and needs.
If you only teach to students’ strengths in such a program, then you often deprive them of the basic skills that all learners need to be successful in whatever discipline your teaching. So, enrichment, diversity, and differentiation should always be grounded with solid curricular goals.
Each student in front your in your classroom is important and worthy of your commitment to such an approach, whether they are in G.A.T.E. or enrolled in regular tracks.
As this conversation wages on in the district, parents are eager for their students to find suitable placements in programs; teachers and administrators are eager to ensure such placements become a reality and that year gets started on a positive note.
One of the most important elements in teaching is creating an atmosphere where students feel safe, where the parents feel involved and informed regarding their student’s progress, and where the challenges and assignments meet both curricular standards and the varying learning needs of the student.
There is nothing more important to an engaged parent than the type of opportunities and advancements their student undertakes during a school year. Stay involved, contribute when possible with field trips or extra-curricular activities, ask your student what their assignments are, and contact the teacher if you have questions.
Everyone should take a deep breath regarding G.A.T.E.; remember it’s just one way to address learning. I taught a number of G.A.T.E. classes, attended numerous meetings, and understand the concerns and different philosophies being bantered about.
Bottom line is this: As a G.A.TE. teacher you have a class roster, a defined curriculum, engaged parents, and a wide range of student abilities and needs.
If you only teach to students’ strengths in such a program, then you often deprive them of the basic skills that all learners need to be successful in whatever discipline you are teaching. So, enrichment, diversity, and differentiation should always be grounded with solid curricular goals.
Each student in front you in your classroom is important and worthy of your commitment to such an approach, whether they are in G.A.T.E. or enrolled in regular tracks.
As this conversation wages on in the district, parents are eager for their students to find suitable placements in programs; teachers and administrators are eager to ensure such placements become a reality and that year gets started on a positive note.
One of the most important elements in teaching is creating an atmosphere where students feel safe, where the parents feel involved and informed regarding their student’s progress, and where the challenges and assignments meet both curricular standards and the varying learning needs of the student.
There is nothing more important to an engaged parent than the type of opportunities and advancements their student undertakes during a school year. Stay involved, contribute when possible with field trips or extra-curricular activities, ask your student what their assignments are, and contact the teacher if you have questions.