Tj Host

Wedding Ideas & Inspiration
Evaluating Family Engagement Strategies: Addressing Measurement Challenges

Evaluating Family Engagement Strategies: Addressing Measurement Challenges

>>LACY WOOD: Today’s webinar, titled “Evaluating
Family Engagement Strategies: Addressing Measurement Challenges,” will highlight promising approaches
for evaluating these strategies, address challenges in defining and measuring outcomes and provide
guidance for building evaluation into a family engagement plan. At the end of the webinar
we will be addressing questions from the audience. Throughout the webinar you can enter you questions
in the question box on the screen. Simply type your question and then press enter. Please
note that we receive several hundred questions during these webinars and we can only address
a limited number so we won’t be able to get to everyone’s question. In addition,
the webinar is being recorded and we will send out the archive presentation along with
the slides next week. Now, I’d like to introduce our panelists. Okay, first I’d like to introduce
our Heather Weiss, Heather Weiss. Heather is the founder and director of the Harvard
Family Research Project or HFRP and is also a senior research associate and lecturer at
the Harvard Graduate School of Education. HFRP’s mission is to create more effective
practices, interventions and policies to support children’s successful development from birth
to adulthood. Heather writes, speaks and advices on programs and policies for children and
families and serves on the advisory boards of many public and private organizations.
She’s also a consultant and advisor to numerous foundations on strategic grant making and
evaluation. Our next panelist is Rebecca Maynard, a distinguished scholar and a national leader
in education and social policy research. Becca has served as the commissioner for the National
Center for Education, Evaluation and Regional Assistance or NCEE since June of 2010. As
commissioner she oversees NCEE, one of the four centers in the Institute of Education
Sciences at the U.S. Department of Education. Prior to joining the department, Becca served
as the University Trustee Chair and Professor of Education in Social Policy at the University
of Pennsylvania, where for five years she directed the university’s IES, pre-doctoral
training program in Education Sciences. She helped develop the What Works Clearinghouse
and her work was instrumental in the creation of the Campbell Collaboration, an international
association of public policy professionals who work to solve societal problems through
scientific research and analysis. Our next panelist, Amy Clark, is a project director
at Education Development Center or EDC, a global non-profit research organization based
in the Boston area. In her twelve years at EDC Amy has directed numerous projects across
EDC’s Education and Health Divisions. Her work is tied together by a commitment to shape
school and home environments so that more children can succeed in education beyond high
school. Amy began her career as a middle grades teacher and brings a practical approach to
projects under her leadership, taking care to develop programs and services that meet
the needs of students, parents and school personnel. Our next panelist is Eric Dearing,
Associate Professor of Applied Developmental Psychology in the Lynch School of Education
at Boston College. Eric’s research is focused on the way the connections between families,
schools and community agencies promote the life chances of children growing up poor.
The results of his research have been regularly featured in the media, and recently Eric received
a Young Scholar Award from the Foundation for Child Development to evaluate City Connects,
an intervention that increases community agency involvement in schools to promote the achievement
of immigrant children in low-income urban elementary schools. Our next panelist is Kate
Gill Kressley, a senior research associate at RMC Research Corporation. Kate is also
an external evaluator for the PIRCs in New Hampshire, Utah and Massachusetts, and a national
consultant in the field of Family, School and Community Engagement, currently working
with the Boston Public School’s Office of Family, and Student Engagement and the Center
on Innovation and Improvement. Kate brings deep experience in evaluation as well as program
planning, implementation and continuous improvement across the evolving family engagement field.
Then lastly, our last panelist is Kathy Hoover-Dempsey, an Associate Professor of Psychology and Education
at Vanderbilt University. Kathy’s research focuses on parental involvement in children’s
and adolescent’s education and the influence of family engagement on student learning outcomes.
She has developed and evaluated schoolbased interventions designed to increase the incidents
and effectiveness of school and teacher invitations to parental involvement and has consulted
with varied national, state and research programs designed to enhance the effectiveness of parental
involvement and family-school partnerships. With that, I’m going to turn it over to
our moderator, Heather Weiss.>>HEATHER WEISS: Good afternoon, everybody.
I’d like to start by extending my welcome and also thanking the panelists who you’ve
just heard about, a group of very experienced researchers and evaluators for joining this
roundtable conversation about how we evaluate and evaluation strategies for family engagement.
We chose to do this as a roundtable because evaluation and family engagement is an evolving
arena and we thought it would be useful to get experts to talk about some of the strategies
as well as specific things like indicators of family engagement at this point in our
field’s conversation about evaluation. So, I’d like to thank them. I’d also like
to thank people that sent questions in advance. You’ll see some of them in the slide deck,
and I will also ask you to hold your questions; write them in but hold them. We’ll be addressing
them in the last twenty minutes of the webinar. So we want to have some back and forth with
everybody who’s participating. I would also like to say that with the webinar, we’re
trying to create a mind shift from evaluation out there and done to us to in here, done
by and for us to strengthen our learning, our continuous improvement as well as our
accountability and sustainability within family engagement. The field, as this webinar series
has indicated and supported, is shifting from family engagement programs to trying to build
in systemic integrated and sustained family engagement from early childhood through high
school. An example, hypothetical example but more and more places are beginning to move
toward it, is a district that has an assistant superintendent for family engagement and family
engagement staff at every school, professional development for family engagement for staff,
teachers and others, a commitment to 100% participation over a period of time in parent-teacher
conferences in person or by phone or through some other means, a commitment to sharing
data on student performance and school performance regularly with related ways in which families
can support their child’s development as well as support overall school performance
and finally things like family academies and other volunteer activities that build family
and school capacity for effective family engagement, all directed towards promoting school learning
and student achievement. So when we talk about it, as we will in the course of the webinar,
systemic, integrated and sustained engagement, that’s the kind of vision that the field
is driving towards. We’ve organized the webinar around four key questions. First is,
why evaluate and then what are some promising approaches to evaluation including strategies
that people can use to look at these more systemic integrated sustained efforts? How
can evaluation data be put to good use? This is the get it out of the drawer and into action
question. Then, what are important evaluation resources and how can I access them? With
this fourth question we’re going to try and talk about resources in addition to money,
money being important but some of the other additional resources that can be brought to
bear. So, moving to the first question, why evaluate? Amy, I know you’ve recently done
some research at EDC looking at district family engagement and efforts to evaluate them. Could
you tell us a little bit about that?>>AMY APARICIO CLARK: Sure, I’d love to.
Yes, in 2009 EDC wrapped up a study that we did to collect information about what family
engagement practices were being used at the middle and high school levels and we looked
specifically at urban districts in the northeast region of the country because the study was
conducted under the auspices of EDC’s Regional Education Lab Northeast and Islands. So, this
study was actually a response to specific requests that we received from education leaders
in our region for a more systematic way to collect information about how schools were
working with parents. So, we designed this study to find out what schools were doing
to engage families and also to ask which of their strategies they were using had been
evaluated to determine their effectiveness, and our ultimate goal was to produce a catalogue
of strategies that had some evaluation data behind them and we hoped that at least some
of the evaluations that we would find had happened would meet the Department of Education’s
definitions of rigor. So, while we found that schools were using numerous and diverse practices
for engaging families, very few of these had been evaluated either formally or informally,
and taken as a whole they also – per school, each school had many strategies, many practices
that they had going on. At each school, though, when you looked at the sort of whole of the
practices that were happening, there wasn’t often a coherence or unifying thread to tie
all those together. Similarly, we found that save for a few exemptions there was little
written documentation of what schools did to work with families from year to year. In
general when schools were making decisions about what to do to engage families, their
choices were decided more by the oral histories that were kept among the staff and teachers
rather than by the data collected through monitoring and evaluation which actually brought
us back to the concern that kicked the study off in the first place and that is what are
the mechanisms that we have in the region for sharing what schools are doing to reach
families and what strategies are working.>>HEATHER WEISS: Thank you. This kind of
paints a picture of kind of where we are in some ways in a sense that many schools have
a set of practices but not a strategy. We don’t have a lot of evaluation in place
even of the practices. So it kind of paints a picture I think of where we are. Now, I’d
like to, with that backdrop talk about in light of that what are some of the goals for
evaluation? Why evaluate? I’ll ask some of our other panelists to speak to that.>>KATHY HOOVER-DEMPSEY: This is Kathy. I
think primarily one of the goals of evaluations that’s critical is to begin to find out
what is working, why it’s working and what we might do in future years to continue its
effective functioning. I think without that effective evaluation it becomes very, very
difficult to, as both of you have suggested even if I just joined the conversation, it
becomes very, very difficult to see any continuity moving from year to year.>>HEATHER WEISS: Other stuff? Go ahead.>>REBECCA MAYNARD: This is Becca. I just
want to say that I also think it’s really important to think about evaluation as something
that can also help us improve practice. Sometimes some of the things we do, whether it’s the
overall strategy or sub-activities within a strategy aren’t working quite as well
as they should and the evaluations can be very powerful in helping to understand why
ideas that we thought were good aren’t playing out in practice as well as we thought they
would. So I think there are reasons to find why things that are effective and also reasons
to find things that aren’t working well and to try to understand why, and how to either
get around it or fix it.>>HEATHER WEISS: So in some ways, the conversation
that I’m hearing is that we have a mindset for evaluation that says in an ongoing way
that we’re going to be looking at what we’re doing and we’ll be celebrating and sharing
some of the things that the evidence suggest works but we will also expect that some things
won’t work and that’s par for the course and that part of the value of evaluation is
to surface those things so we can make another pass at it. Is that a fair sense of it?>>KATHY HOOVER-DEMPSEY: Yes.>>AMY APARICIO CLARK: I think another aspect
of it – I liked, Heather, the way you introduced the notion of evaluation being something that
we do and not something that’s done to someone. That it’s a collaborative effort. Another
aspect is that we are accountable and the field of family engagement is a serious field
and serious things get measured because they matter. So I think that when we focus on evaluation
in this field we underscore the importance of this field and what it contributes to the
broader field of education and student learning.>>HEATHER WEISS: Great. Now I want to turn
to the myth. I know in our conversation as a panel prior to the webinar, we talked about
there is a set of myths that surround evaluation and I’d like to talk a little about some
of those. Some of them, to tee up, some of them are it costs too much, we don’t have
the expertise. What are some of the other myths that our panel sees out there as barriers
to evaluation?>>KATHY HOOVER-DEMPSEY: I think one thing
– I’m sorry; this is Kathy. I think one thing that’s also a barrier is school district
presumptions that this is going to be too difficult to do the evaluation, that the time,
energy and resources, as you said, are not going to be available.>>HEATHER WEISS: Okay.>>ERIC DEARING: This is Eric. I would also
add that – and I’m not sure this is a myth as much as it is a fear – that there is a
fear around finding out something you don’t want to know. I think that that fear is misplaced
or misguided and that going back to Rebecca’s comments around why we should be evaluating.
Those are opportunities and moments when, as you mentioned, Heather, when things aren’t
going the way we expected them to go, to be direct and reevaluate our goals and our message
and approaches to family engagement and/or the ways that we’re measuring family engagement
and the results of [Unintelligible].>>HEATHER WEISS: Does anybody else want to
add anything else?>>KATHY HOOVER-DEMPSEY: I’m just going
to add one thing. This is Kathy. That one of the things I’ve experienced sometimes
in schools is an appreciation for the fact that I and folks who are coming with me talk
like regular people and I think sometimes there’s an assumption in schools that people
who do evaluations aren’t going to connect with them, aren’t going to understand what
they’re really doing and we’ll leave them out there kind of flailing. Particularly,
as Eric said, if some of the results suggest that the program or the efforts aren’t being
successful.>>KATE GILL KRESSLEY: This is Kate. I want
to piggyback on what Kathy just said because as a program person and an evaluator I know
that at times I might have the fear as a program person that the program implementation has
not unfolded exactly as you had predicted it to unfold and therefore the evaluator is
going to come down on you so to speak. So the importance of working collaboratively
with the evaluator in the beginning with the planning and then that ongoing communication
as the program implementation unfolds is critical so that you don’t have these misperceptions.>>HEATHER WEISS: I’m interested to turn
now; we talked about some of the issues: the resource issues, the fear of negative findings,
when the evaluator doesn’t seem to be lined up in collaboration with the program, et cetera.
I’d be interested in people’s sense of what makes evaluation a win-win situation.
In other words, if you had to lay out a couple of things that are really necessary for an
evaluation to go well, what would you put on the table?>>REBECCA MAYNARD: This is Becca, and in
my experience is that the most important thing in doing a good evaluation is to design it
in a way that it is, that everybody gains from the knowledge regardless of whether the
results come out showing the program to be highly successful or whether it comes out
showing it to be neutral or a disaster. To do that what you really need to do is – an
evaluator has to work hand in glove with the program staff to understand what the program
is trying to accomplish, how it’s going about accomplishing that, what the issues
are that are important to the program, something about the constraints, you also need to think
about how to judge what, when the program or strategies within the program are being
effective or not effective and design a strategy that’s going to address multiple questions
that are of interest to multiple audiences. So, program people need to feel that at the
end of the day they’re going to have information that will help them do a better job in achieving
the objectives that they have dedicated their careers to, and if that means changing how
they’re doing their jobs or what they’re doing in their jobs, to me, that’s something
program people welcome. What they don’t welcome is evaluators coming in and saying,
“You’re a winner and you’re a loser,” and that’s all I’m going to tell you.>>KATHY HOOVER-DEMPSEY: That’s critically
important I think. This is Kathy. I would just say that what Becca has just suggested
also suggests strongly that in evaluation we practice what we preach about parental
engagement and that is families and schools must be in a collaborative relationship if
students are going to learn as well as they can, and it seems to me that Becca is really
urging that same total collaborative relationship between evaluators and school systems which
means people from both parts of the equation are going to be seeing each other, talking
with each other and understanding what’s going on in the world of the school and the
world of the evaluator.>>HEATHER WEISS: Let me press this one more
level, and that’s what I’m hearing is the importance of this collaborative relationship.
In addition I think the regular flow of information back and forth with the implication that if
things are not going well, I’m not going to hear it at the end of the evaluation, I’m
going to be hearing some of the information about it in the course of the work. Is that
a fair statement about what I’m hearing?>>REBECCA MAYNARD: Yes.>>KATHY HOOVER-DEMPSEY: Yes.>>HEATHER WEISS: So that then assumes that
the time and commitment is in place on the part of all parties to make it work in an
ongoing way? So, it’s not evaluation done to us but done with us with everybody committed
to that over time. So that I know as a program person I’m going to be getting things about
how we’re doing and an opportunity to improve them as we go as well as perhaps information
at the end that suggests there’s some additional things I need to be working on. Is that a
fair sort of summary and if those things are in place we think we have a win-win situation?>>REBECCA MAYNARD: I would like to add one
more thing that I think is important to put in play. I think it’s important when evaluators
go in to study programs and when programs agree to be evaluated, I think there needs
to be some acknowledgement that there’s an overlap in the missions and the interest
of these two groups but there are some things that the evaluator has to protect in terms
of the integrity of what’s being done and there are some things that the program people
have to protect. So when I go in to evaluate a program I can’t ask a program to turn
what they’re doing upside down so that I can study them in a way that’s comfortable
for me. I have to figure out how to do a good evaluation in a way that allows that program
to operate as it’s intended to operate. So there’s some negotiation that needs to
go on. So I enter into an evaluation design phase with the understanding that we’re
going to have to do some back and forth in terms of figuring out what the objectives
of the evaluation are and how I can do a good evaluation without being overly burdensome
to the program people. That takes some trust because I’m going to have to try out some
ideas, some of which may not work. The program people have to honestly tell me which of those
things are going to cause problems and we need to negotiate on that. So there’s are
some things that only the program people can control or should control and some things
only the evaluator should control and there’s a big area in the middle where they should
negotiate.>>HEATHER WEISS: Excellent. Does anyone else
want to add anything?>>KATE GILL KRESSLEY: This is Kate. I think
in terms of working well in a win-win situation it’s important to build in optimally, build
in enough time at the beginning so that the evaluation planning can really, really be
comprehensive and the work starts with good understandings both on the program implementer’s
part and the evaluator’s part. Also the evaluator needs to be engaging those folks
so that their important questions are heard in terms of what they want to see be evaluated
and again as was just stated then there’s an important area of negotiation and shaping
the questions and staging them up perhaps so that needs can be met for information.>>HEATHER WEISS: Excellent. Okay, I’d like
to move to our second guiding question about promising ways to approach evaluation. When
we thought about this, one of the things that I think is very important is the good old
logic model to guide the development of an evaluation strategy and I was reminded of
back in the 1960s when we were just beginning to do large scale evaluations of Federal Demonstration
Projects, they developed something called Evaluability Assessment and in some ways a
logic model is an extension of that. It really is a tool that helps develop a program but
then a related evaluation strategy. We have in a set of resources that will be posted
after the webinar early next week, some examples of logic models including one of a district
– a fairly comprehensive district-family engagement effort. So I bring it to the conversation
here just to say that I think most evaluations would be wise to develop and/or convene around
a logic model that helps to develop an evaluation strategy and many I think in the webinar audience
know about logic models. They are graphics that represent what your program hopes to
achieve, what it’s doing and what the anticipated impacts are on the target population and the
community. We see them often in use and operating as aids to strategic program planning and
development so that everybody’s on the same page about what you’re trying to do, how
you’re going to achieve it and then to develop the indicators of whether or not you are achieving
them. I think of this as a back of the envelope exercise that then gets formalized into a
logic model that as we’ve used it at HFRP, we see it as a place where issues, for example,
of dosage come up. You do a logic model and say we’re going to have a math night and
the outcome is that kids are going to improve in math. When you go back and forth between
those inputs and outcomes you realize that one math night is probably not going to lead
to that kind of an outcome and you either start thinking about increasing your dosage
or changing your outcome. So it has that program planning aspects to it; in addition they promote
evaluation planning and can support the kind of process of continuous learning and improvement
that the panelists have articulated up to this point in our conversation. So they help
you identify the logical linkages between the inputs and outcomes and then identify
indicators of progress towards those outcomes and, I think equally importantly, distinguish
between measures of effort and measures of effect. In the examination we did of audience
questions that were submitted to our part of webinar, one of the key ones is one about
choosing between specifically the inputs and outcomes. So I want to ask the panel to clarify
a little bit about that. How can we move from evaluating our reach, the number of people
we reach with a piece of material or whatever to actually evaluating the impact of the program
services? Do the panelists have suggestions in regards to this?>>ERIC DEARING: This is Eric. I think that
if I step back a little bit, I think what you said about the way that a logic model
should be, given best practice guiding program development and then in turn guiding evaluation,
from the evaluation side it’s really going to be giving insight into what, when and how
and why we should be measuring. To just give another example, you gave the dosage example,
it means something else. When we’re talking about children we have development maturation,
things take time and a program implemented on Monday that expects to see differences
in achievement on Tuesday is in trouble of course. The logic model should be informing
us, giving an underlining theory of change, how much time and what’s reasonable to expect
when. With reasonable expectations we can get more excited about that process of change
over time as it unfolds with unreasonable expectations, we can get into trouble really
quickly. I would just add that.>>KATE GILL KRESSLEY: This is Kate. I want
to extend what Eric was just saying a minute ago because I think that the audience question
as we have it posted – I want to comment on that because I think that it’s not that
we can shift to measuring the latter without still doing the former. By that I mean not
just counting how many people take part in certain activities but really understanding
the nature of those activities and also understanding from programmatic basis what actually gets
done. That goes into the measuring, as Heather mentioned, the dosage. Or measuring what is
the intervention, what is the opportunity for parents to learn or families to learn
and to make sure that all those things are in place first in order to have something
to measure. So it’s really – it’s still two parts. We have to be able to document
what we’ve done well enough so that we know that there is a logical link between what
we’ve done and the outcome, the fact that we have identified according to certain indicators.>>HEATHER WEISS: I mean that gets me and
I think it gets us to the conversation about indicators and identifying indicators of progress
and I know, Kathy, you have and other people have built off some of the work you’ve done
on indicators. I know, Eric, you’ve thought about that a bit as well, as well as other
people on the panel. Would you talk to us a little bit about some of the short term
and long term indicators, how we go about developing them as part of our evaluation
efforts?>>KATHY HOOVER-DEMPSEY: Yes. This is Kathy.
I would like to jump in just to reemphasize the importance of a point that Eric was making
a bit ago, too. When we’re evaluating increases or improvements in parents’ involvement
and when we’re evaluating the influence of parents’ involvement activity or in our
terms the engagement of varied learning mechanisms in helping their children with schooling,
we’re talking about longer term rather than shorter term processes. Parents learn many
of these skills slowly and over time children will certainly take some time to manifest
the positive benefits of what their parents are supporting in their learning. So I just
really want to say that we need to look not so much at really flash in the pan, in and
out evaluation but evaluation and programming with necessary changes as indicated by ongoing
evidence across time. In terms of the indicators in the work that we have done, really they’re
indicators in terms of what we think motivates parents’ involvement and those I think can
be very helpful if a school’s or a school district’s focus is on engaging more parents
more effectively in supportive activities related to their kids’ schooling. But let
me go to the indicators of student learning that we have proposed and looked at and to
some extent in our research, what we’ve suggested fundamentally is that parents’
involvement activity reflects – there are many ways in which parents can be involved.
Let me put it that way. They can encourage, they can state their expectations, they can
model learning activities, they can reinforce what their children are doing that’s positive,
they can provide if they feel equipped themselves some levels of instruction and reinforcement
for what’s going on. Within the context of those specific activities they are engaging
certain kinds of learning mechanisms which I just included in that, too. So to some extent,
what we’re looking at here is parents’ activity but what parents are doing in the
context of those activities that actually contributes to student learning. What we’ve
suggested very strongly is that the best [Unintelligible] parent involvement effectiveness is to be
found not in a direct relationship with their summary annual measures of achievement but
rather in students’ development of what we’ve called proximal achievement goals
and that is for example, students’ achievement of a greater level of academic self-efficacy.
Thinking that, “Yes, I can do this work.” Students move towards a sense that, “Gee,
this is interesting. I’m doing it not just because Mr. Jones told me I had to but because,
yes, it’s pretty cool.” Achievement goals in the sense of becoming more self-regulated,
that is understanding, when I have a question and I need to stop and ask somebody instead
of sitting here and twiddling my thumbs and spinning off into some other space because
I can’t figure it out and I don’t know what to do next. Then children’s kind of
proximal achievement outcomes like learning how to ask good questions of a teacher, learning
that you should ask questions of the teacher or a teacher’s aid or someone in the daycare
setting who is an adult who may be able to help. So those different kinds of indicators
which I believe Heather has put together in one of the slides that we’re using today
but it hasn’t come yet. Those I think are the outcomes, kind of the proximal outcomes
that we need to be evaluating as we also then include evaluations of that longer term outcome:
how kids are performing on achievement tests.>>ERIC DEARING: I would just…>>HEATHER WEISS: Go ahead, Eric.>>ERIC DEARING: I’m sorry. I was just going
to add I think just to build on what Kathy has mentioned that in terms of some of those
things that we are looking for are attitudes and feelings, both within children and parents’
motivation and effort and just a couple of others that I don’t think came up yet, but
teacher-child relationship are also something we should see in terms of, as Kathy mentioned,
proximal processes that should unfold as well as how – and this harkens back to Kate’s
point about we don’t want to just stop counting something in order to shift towards this other
but we want to keep counting – and whether families are dedicating their time and their
investments in children and how are they spending their time. Those should be, as we look at
the short term outcome so to speak, those are some areas that seem to be commonly demonstrated
in the research.>>HEATHER WEISS: This is Heather now. I want
to chime in myself here to say that on the resource list there’s a paper by Jim Connell
and Michelle Gambone, done a number of years ago in the youth development arena that I
think can be brought to bear here. They looked at literature on child development and said,
“Young people who have a certain set of supports and opportunities, when they get
into their early 20s are doing better than those that don’t have those supports and
opportunities.” They then tested that model with existing longitudinal data and found
out it was borne out. I think one of the challenges for us in the family engagement space is to
begin to do the equivalent thing here. Really look at what we know from developmental research
about the specific things, some of which we’ve just talked about that parents do that create
a pathway that then leads to better learning outcomes for kids. So, part of our challenge
I think is to begin to develop those kinds of models and those pathways to understand
some of those proximal indicators that then put you on the route to some of the longer
term achievement in kids’ learning and development. I also want to now move to our next question
which I think is a very important one and one that we’ve struggled with in the family
engagement arena and that’s the whole notion of the gold standard or experimental research
design of random control trials, however we want to label them, look at their role and
the question that came from one of our members of the audience, what information is lost
or gained when we have these RCTs? I’d ask the panelists to talk about RCTs, their value
added and then really talk about what do we lose if that’s all we focus on? Becca, I
know you have some thoughts on that. Would you share it?>>REBECCA MAYNARD: Sure. RCT is something
that has gotten a bad name with program people for good reasons, and that is that I think
we have done – we have as a community, evaluators have often been much too rigid and textbook
when you go out to do an experimental design study. The reason to do an experimental design
study by which we just mean you have two equivalent groups, one group getting the programming
services and another equivalent group not getting the services is that is the single
best, most defensible way to come up with a really good answer to the question, “Am
I making a difference?” in whatever domain you want to look at. It could be, “Am I
making a difference in some of these proximal measures that we just talked about? Am I making
a difference in the long term success of students?” The myth is that experiments are – one is
that they are unethical because we’re denying services, another is that they’re expensive
and in both cases you can do an experiment in a way that is unethical and you can also
oftentimes do this in a way that is extremely ethical. I would prioritize this area [Unintelligible]
services to help. One strategy if we want to defend this turf and get more resources
for this area, I would argue it is important in this world of scarcity and increasing scarcity
to find ways to produce highly reliable evidence that investing in this area is better than
investing in some other area or that it’s at least a good investment. At the same time,
another myth is that that’s all you learn from an experiment. That may be all you learn
from an experiment is how effective [Unintelligible], but if it is designed well you learn a whole
lot more. You learn about what went into operating the program, what aspects of the program seem
to work well according to the logic model, what things didn’t work well and maybe why
and how [Unintelligible] work better. So there’s a lot you can learn. Experiments also can
be quite low cost and often times it is less costly to do a good experiment than it is
to do the best quasi-experiment you can do which is never as good as a good experiment.
All of this is to say that I would encourage people to work with evaluators to find ways
to get these highly defensible answers to the questions of effectiveness in ways that
aren’t upending your program and that return more than the bottom line answer to whether
or not this program is making a difference for long term outcomes for children.>>HEATHER WEISS: Becca, I know some people
in our webinar audience may not know what an experiment or an RCT is. Could you give
us a one sentence definition?>>REBECCA MAYNARD: Yes, this would be a situation
where you would offer your family engagement services in some settings but not in others.
You would decide which settings, let’s say which schools to service, you would determine
that by lottery so that it’s just by pure chance whether a school is getting the services
or not getting the services. So, if you could serve, if your resources would allow you to
serve ten schools, you flip the coin among the thirty schools in your district and pick
the ten that are going to get serviced by a lottery as opposed to picking the ten in
some other way, like all the ones on the east side of town or all the ones with low test
scores or whatever. Or in other cases if you’re doing a targeted outreach kind of program
you might randomly assign neighborhoods or you might randomly assign individuals, but
it is flipping the coin using lottery to decide who’s going to get the services and who’s
not.>>HEATHER WEISS: Then the argument then is
if you do that, you have the strongest evidence that what you did caused a change?>>REBECCA MAYNARD: Yes, any differences between
your groups then are due to the program or change and the larger – if you have a reasonably
sized sample, there’s a pretty high certainty and we can determine how certain you are that
the difference.>>HEATHER WEISS: For people to be thinking
about, as we think about sort of our evaluation designs. At the same time we’re at a point
in time when people are trying to also focus on innovation and developing new things and
testing new things because we don’t always have evidence on things that solve the problems
we’re trying to solve at the moment. So, there’s going to be a tension between using
evidence from RCTs and innovation, but even with that the innovations then can tee up
as they get their feet on the ground more rigorous evaluation designs to test their
contribution if you will. So this is an area where there are a lot of different points
of view, but I think good arguments to be had by a variety of people but I notice increasingly
that more and more public policy decisions are being made on the basis of your research
design. So, I just want to throw that out for people to be thinking about. The third
kind of thing we wanted to focus on is now that we’ve got- we’ve talked about evaluation,
we’ve talked about some of the purposes. How can we put evaluation data to good use?
Earlier in the conversation we talked about trying to use it to support learning and continuous
improvement as well as accountability and sustainability and we as well as many other
people have put together diagrams like the one you see on the screen now that are really
designed to position evaluation, ongoing evaluation and performance management, as part of a continuous
learning and improvement system and it really does, as Becca said early on in the conversation,
engage the stakeholders in the strategic planning, setting the performance goals and measures
and building a logic model. What’s your theory of change for what you’re trying
to do is a key part of this process as well. Learning from experience and other relevant
research and building that into the program design and then engaging the innovation and
monitoring the evaluation, developing the indicators, are we on track, are we achieving
what we think we are, learning from that as well as comparisons with others. The family
engagement field is rich with lots of different examples and I think we have real opportunities
now to begin to do comparisons across programs, particularly to the extent that we can get
evaluation data, we can bring that in to benchmark with others in our field and use those comparisons
to support our own improvement and then the fifth part of this is really transferring
your lessons for those course corrections, the program specification and then the cycle
continues. So, I put this out because my sense is now that good non-profit programs and increasingly
public programs are building this kind of continuous learning in to the definition of
who they are and what they do. So, they’re moving from one hit, small evaluation here
and there to a more systemic effort to build in data collection, performance management
and evaluation into the core of what they do. In private philanthropy, they now expect,
many of them, that you have this capacity and will be looking at your organization’s
capacity to do this kind of continuous learning as part of their due diligence to decide whether
to fund what you’re doing or not. So, I put it out because I think this conversation
has been driving towards this kind of notion of use of evaluation as part of a learning
system. Do other panelists have thoughts about this? Have you seen it in operation? Do you
have a sense of what the incentives are for people to transition to this kind of placement
of evaluation?>>KATHY HOOVER-DEMPSEY: This is Kathy, and
the one thought I had is as you were saying is that in many ways I think the adoption
of this kind of a continuous learning system that you’ve shared with us – to ask a local
district or local school to begin to improve this I think without engaging the explicit
enthusiastic support of state education agency and the district as a whole and the school
principal is sometimes just very, very difficult. I see folks in schools really being buffeted
about with this perceived request from somebody and that request from somebody over there
and that request from somebody over there and really looking for some kind of leadership
around, how do we prioritize? How do we begin to understand what is most important to do
here? I think at the level of teachers working in classrooms, in schools, the principal is
the person folks look to for that. If the principal is providing that leadership and
it’s sound and it moves along with an understanding of the kinds of tools we’ve identified here,
I think you can get good support in schools. If the principal is not moving along with
these and if there is no district or state support for the principal to begin then to
kind of come around and understand that this is really important, I see this as just a
very difficult task for individuals and schools to kind of take on their own. I know that’s
not what you were suggesting folks should do but I really wanted to underscore the importance
I think of having state and district folks very much committed to supporting principals
of individual schools in engaging in this kind of planning and evaluation and making
space for it in the context of what the school and the school’s teachers are expected to
do on a daily basis.>>KATE GILL KRESSLEY: This is Kate, and I
would even extend that example, Kathy, to the notfor- profit world in terms of those
folks who are out working in community-based organizations.>>KATHY HOOVER-DEMPSEY: Yes.>>KATE GILL KRESSLEY: If that agency head
doesn’t have the support and the resources from his or her board and the funder and that
too undermines the possibility that that agency is going to be in a continuous learning system.
United Way, I think as a national organization has gone a long way in the last decade in
terms of helping everyone understand the importance of accountability and the importance of really
distinguishing between counts and impacts or outcomes.>>KATE GILL KRESSLEY: Yes.>>KATHY HOOVER-DEMPSEY: Really moving us
forward in that arena.>>HEATHER WEISS: Amy, I know you began our
conversation with what you had found in districts in the northeast region. When you reflect
on this kind of thing and your experience and the research you did, do you see incentives
for districts to begin to move in this direction and do you see critical things? We’ve identified
leadership and resources as two things that are going to be important for this kind of
thing to happen. Do you see incentives for people to move in this direction and do you
have ideas about what needs to be in place for them to do so?>>AMY APARICIO CLARK: I think the place where
we saw there was this sort of built in, as you say, incentive or inclination to really
pay attention to evaluation was when the family engagement goals were tied to say the school
improvement plan or the district improvement plan. When that was the case, the accountability
was obviously there and there was an understanding. It was articulated to either the whole school
or the whole district that this was a priority. Once that’s done you really see a lot more
is happening in terms of documenting and monitoring what the schools and districts are doing.>>HEATHER WEISS: Let’s say – you were at
IES which has responsibility for some of the development of longitudinal data systems and
the rest of it. Do you have any thoughts on this?>>REBECCA MAYNARD: I have a couple of thoughts.
I mean one is that in my experience, one of the best ways to get buy-in from school districts,
principals, superintendents, is to make sure that the evaluation is designed to address
short term concerns as well as long term concerns that – so that’s a strategy in how you design
what it is that you’re going to do. But another strategy is also to capitalize on
what is already being done. You mentioned longitudinal data systems and we have a major
initiative in this country to promote both the creation of longitudinal data systems
and to make better use of those data. We don’t want to just create those data. In fact, if
we just create them they will not be good data sets if there isn’t also some practical
use for them. Having those data sets out there means that a lot of the outcomes that one
needs to address the ultimate question – the long term questions of student achievement,
gains, attendance gains, and things like that, can be monitored quite inexpensively and quite
routinely and by setting up for example a monitoring system or tracking system as part
of an evaluation you might also be setting up a tracking system that would be useful
to the school district and the school independent of the evaluation. So that’s a giveback.
So, you’re giving back something to the school that has nothing to do or is not directly
related to the evaluation of your family engagement program but it’s something you need to do
that evaluation to do it well. So a quid pro quo is using the data that is already being
collected and then a giveback is leaving behind some reporting that could be used for purposes
other than monitoring your evaluation.>>HEATHER WEISS: That’s a great example.
Before we move on to our last question I want to also talk about the value of combining
implementation data and stories with other kinds of data to communicate about work and
we put up one resource which will be available on this slide in the aftermath “From Free
Range Thinking”, Andy Goodman’s work that talks about the importance of stories as well
as numbers in moving people and supporting the value of what you do. I also want to point
to work that we alluded to in an earlier slide by Ron Mirr and Zena Rudo. Ron is an independent
consultant, Zena works with SEDL, and they’ve done some nice work for the PIRCs on telling
the story of your project and that resource will also be available on our resource list.
So, we’ve addressed the question on how can schools and districts use evaluation data
to some extent. I now want to turn to how you get the resources to do evaluation. Really,
what are important evaluation resources and how can we access them? [Unintelligible].
In the conversation of our webinar experts prior to the webinar, we talked about trying
to broaden the notion of what are resources beyond funding. Funding, we know that’s
necessary, but what are some other kinds of things that we could leverage in order to
be able to support the evaluation of this kind of continuous learning. So, I’d like
to open it up to the panelists to give us some examples of expertise, existing data,
partners for evaluation, things you’re aware of that can be leveraged to support evaluation.>>KATHY HOOVER-DEMPSEY: This is Kathy. We’ve
been really successful in all respects. I mean it’s just been a very productive relationship
to offer. We’ll help you address some of the issues that you’re trying to deal with
here if you’ll let us in so to speak to get information that’s going to be helpful
to you in addressing some of these problems. So I would preach the importance of that.>>HEATHER WEISS: I know, Eric, you’re in
a university setting. Becca, you’ve been in one. Kathy, you’re in one. So this is
really trying to figure out how one creates partnerships with higher education. Do people
have additional thoughts on that?>>REBECCA MAYNARD: This is Becca. I can just
give you a few examples of things that I’ve done. I have taught courses where I would
look for practical applied research projects that we can do for a real client with my – so
I’ll bring a team of five to ten students out to a project site and you get faculty
advisory and you get some free labor. I’ve done this negotiating up front. What it is
that – I don’t go in and say, “I’d like to do this study.” I go in and say, “I’ve
got a group of students. They need a project of this duration. Here are the kinds of things
we can do. What would be useful?” So I would argue there are people like me sitting in
universities who have some – who would benefit from having an opportunity to do some evaluation.
There just are people in university settings who can not only do these kinds of evaluations
but who can also secure funding for these kinds of evaluations through working with
the program people, working with foundations or Federal government. We here at IES have
research and development program where we fund studies that would fall into the category
we’re talking about here. There are – those are the things I would say that pertain to
higher ed. So there are some other things that we could do that we’ll put off until
we hear other ideas from the higher ed.>>HEATHER WEISS: I’m interested. You, Becca,
you have mentioned and other people may have thoughts about this as well, about using existing
data. Do people see additional ways in which we can use that because that’s a resource?>>KATE GILL KRESSLEY: I would like to jump
in here. This is Kate, and I want to take us back to the hypothetical district level
example that Heather sketched and if you recall when Heather was describing what that district
was working towards, there were some very concrete data points that a district would
already have in terms of their normal data collection processes. For example, attendance
is a huge factor that’s linked to student achievement, especially but not exclusively,
in middle and high school. The percentage of parents across schools in grade levels
et cetera that attend parent-teacher conferences is another kind of data that districts would
typically employ. Data on disciplinary actions, for example, is another example of existing
data that could be used in the context of a comprehensive program around family engagement
that was going on. I think that – and finally many schools are engaged routinely or they’re
beginning to engage routinely in surveys of their students as well as faculty and families
and they’re looking at issues around school climate, they’re looking at issues around
home-school communication, satisfaction, and those surveys could be accessed and perhaps
even amended over time so that the questions could get sharper and crisper in terms of
connecting survey data with the kinds of interventions that comprise a systematic program of family
engagement.>>ERIC DEARING: Heather, I’ll just add
to that in link up with our earlier conversation about having both proximal outcomes and more
distal outcomes. Both of these, at least the Boston public schools and the elementary schools,
are successfully using existing data that the district collects on every child. So in
the distal outcomes, you have things like the standardized achievement test scores but
proximally included in children’s report card grades. teachers in the elementary school
grade children on their effort, work habits and on their behavior in the classroom. So,
some of the things that we talked about, where we should start seeing short term outcomes
as well, these can be cheaply evaluated as well and so there’s a lot of interesting
opportunities within existing data the districts collect.>>HEATHER WEISS: I want to leave time for
back and forth with the webinar audience. Do people have final thoughts on resources?
I know, Becca, you mentioned some other potential ones. Do other people have final thoughts
on it before we turn to our webinar audience?>>ERIC DEARING: I would add one thing that’s
not a resource per se. It links back to the myths and fears and as someone’s thinking
about, do they have the resources to carry out an evaluation. I think that oftentimes
programs can get concerned over implementation problems, with the word “problems,” I’m
putting in quotations here. As things slowly roll out or as there’s [inner buy-in] so
that things don’t happen all at one time or if you’re in a high poverty urban district,
student mobility is rampant and can appear to be problematic but many of these things
– or schools closing in this economic climate, many of these perceived problems from an implementation
standpoint can oftentimes be evaluation opportunities. So going back to Becca’s point about randomization
and controlled trials, oftentimes natural experiments end up unfolding due to what we
initially saw as being problems or natural [causes] of the experiment. I just want to
encourage people to not be discouraged when implementation doesn’t go in the direction
they think it’s going. Sometimes evaluators can be quite happy about that.>>KATE GILL KRESSLEY: This is Kate again.
I just want to step back and add one more comment to this slide. I think right now the
umbrella question for perhaps a lot of the audience is, “How can I secure resources
for family engagement?” period. What I’d like to say is that in when you’re out there
struggling for resources for family engagement and you’re successful, make sure that proportionally
some of those resources are dedicated to evaluation because evaluation is going to help you sustain
those precious resources, help you make the case for accountability and continuation.
So there’s a proportional formula here. So I just wanted to acknowledge that we’ve
got a big, a big issue around obtaining resources for family engagement.>>HEATHER WEISS: Okay. Thank you. I want
to segue now to talk about some of the questions that are coming in from the webinar audience
and one of them is: What role can families and young people themselves play in evaluating
family engagement?>>KATHY HOOVER-DEMPSEY: This is Kathy. I
think one very important role is that families and students should be involved in offering
data to program planners and those who are implementing the program. I think sometimes
we tend to look at if we understand what school principals, school faculty are thinking about
this then we can know what’s going on. If we look at the big external indicators: students’
grades or students’ attendance or students’ achievement results, we can’t know what
we need to know here. I think often, though, parents’ responses and students’ responses
particularly when you are evaluating programs that involve students who are middle school
and above, I think what parents and students are perceiving can become a very important
part of a) the data we need to collect and b) our capacity for understanding how, why
and under what circumstances is this program being successful.>>HEATHER WEISS: Go ahead.>>REBECCA MAYNARD: This is Becca. I was going
to say that parents and students are also the targets of these interventions and often
times the targets aren’t actively engaged and what we’re trying to ->>HEATHER WEISS: Right!>>REBECCA MAYNARD: …In the programs we’re
trying to offer for them and too often evaluations focus only on those who get into the door.
So, I would say one of the biggest givebacks for a program from a good evaluation is that
you will learn something about who you’re reaching and who you’re not reaching and
why you’re not reaching the people that are out there.>>KATHY HOOVER-DEMPSEY: The target.>>REBECCA MAYNARD: Maybe. Yes. I would say
that in some studies that I’ve done with school districts, some things like after school
programs, that’s been one of the biggest givebacks from the evaluation for the schools,
is that practical information. They would claim they were reaching everybody. They thought
they were; but when we really dug into it we learned a lot about who they were not reaching
and why they weren’t reaching them and some of what we learned was very easy to fix. People
just didn’t know it.>>HEATHER WEISS: Another question that’s
come in is: Are there rubrics or standards that have been developed to measure family
engagement and family-school partnerships? Are people aware of rubrics people can use?>>REBECCA MAYNARD: Yes.>>KATE GILL KRESSLEY: Yes. I mean I think
that the national PTA has done a very comprehensive job in constructing a set of standards. We
have examples of school district, some of whom have taken those PTA standards and then
adapted them for their own context, their own schools. Boston is one of those. Kentucky
has a set of standards that they have developed as a state. Maryland has a set of standards.
Massachusetts is working on ratifying a set of standards for the state. So the work really
has come – it’s moved forward rapidly and I think those standards are especially useful
when you’re trying to work systemically with schools because educators and educational
leaders understand standards. You’re talking their language so to speak and it helps them
frame the work and understand the benchmarks that they’re trying to achieve.>>HEATHER WEISS: Are there examples that
anyone’s aware of, Kate or anyone else? Where we’ve gone beyond standards and then
linked them to indicators and people have begun to track the data with respect to those
indicators? So have we drilled down from standards to indicators with some kind of theory of
change? In other words we have our standards and here’s our theory of what we’re going
to try to do to engage families and move the needle on student achievement and these are
the indicators we’re going to track to look at our performance to see if we’re on track
in the way that we hope we are. Do we have examples of people who have done that?>>KATE GILL KRESSLEY: I think the standards
that I’m referring to such as the National PTA standards – their frame of reference is
what is the role of schools. One of their frames of reference is what is the role of
schools in creating environments that move forward, but they’re also looking at what
circumstances have to be in place in order for parents to achieve some of the qualities
that Kathy Hoover-Dempsey’s research talks about, feeling more efficacious – more eff
… I can’t say that. Somebody say it for me.>>REBECCA MAYNARD: Efficacious.>>KATE GILL KRESSLEY: Thank you. Helping
parents feel like they are able to be partners with teachers, so those kinds of attitudes
and feelings. I think that they’re all related. But I don’t think, at least to my knowledge,
I don’t think that there is a silver bullet kind of measure that says if you do this then
you will automatically raise the needle on student achievement. I think that that’s
not possible at this point because we’re really dealing with complex family systems,
we’re dealing with students over time, we’re dealing with a myriad of inputs into that
student’s life and family is an important one. We recognize that and school is important.>>HEATHER WEISS: I think one of the things
I would bring into the conversation is that I think some of the most important work has
been done by actually Tony Bryk and his colleagues in their work on Lessons from Chicago School
Reform. They talk about five critical ingredients for kids to succeed in high poverty or predicted
kids that succeeded in high poverty schools in Chicago, and it ties together I think much
of our conversation in the following way: What Tony and his colleagues did is look at
the school as an organization, not just teachers or students or families but the school as
an organization and say, “What did that school have in place that enabled in very
high poverty neighborhoods with high poverty kids, some schools to significantly move the
needle on math and literacy and what was absent in places where that didn’t happen.” They
had a pathway model. They had five key ingredients: principal leadership, good curriculum, good
professional development for the teachers, a climate, an overall school climate that
supported learning and, critically important, family-community engagement. In their analysis
what they found was things like a pathway whereby schools that had more family and community
engagement had higher attendance which meant that the teacher didn’t have to teach the
half of the kids that weren’t there yesterday today and bore the ones that were there yesterday
today. So instructional – there was a real increase in instructional and improvement
that then in turn led to higher attendance and pointed directly to those improved math
and literacy outcomes. We talked earlier in the conversation about pathways. A certain
set of things take place in families and in their relationship with their kids and in
their relationship with school but then move along in interaction with other important
variables like the quality of teaching and learning and those kinds of things that then
together through this pathway improved child outcome. So I think in this arena, what I’m
hearing in the conversation is we’re moving from programs and the expectation that there’s
a silver bullet in family engagement to thinking about family engagement in interaction with
other things that then create these pathways to improved outcomes for kids and beginning
to specify what those pathways are in the conversation. I heard a number of elements
of the pathways and the notion that we’re not going to have quick fixes, we’re going
to set up processes in motion in interactions amongst the key players in terms of their
relationships supported by an organization, for example, the school that puts some of
those critical ingredients together and that mix of ingredients then leads to better outcomes
for kids. So in terms of our research designs, we’re talking about I think a more complex
set of research designs potentially but even in some of our simpler efforts, understanding
things, these things as pathways and as Kathy and Eric and others have pointed out, that
take time for a set of changes to take place that then contribute to better learning outcomes
for kids and I think as evaluators beginning to think in those kinds of terms with related
theories of change will help us move our understanding in the familycommunity engagement field of
how what we do contributes to better outcomes further along. Do other members of the panel
have other sort of wrap-up thoughts they would like to contribute based on the conversation
that we had?>>KATHY HOOVER-DEMPSEY: Heather, I would
just say – this is Kathy. Your articulation in the last couple of minutes was a brilliant
summary of many of the critical points here, I think.>>HEATHER WEISS: Thank you. Are there other-
beyond that, are there other thoughts, other brilliant thoughts?>>KATHY HOOVER-DEMPSEY: Sorry. You really
got several good points.>>ERIC DEARING: Well, I think returning to
where Amy led us off in thinking about the context where families are and aren’t getting
engaged and combining that with your, with Heather’s drawing up Tony Bryk’s work
on how context in schools and districts and families are interacting with engagement and
considering the complexity of that and what would be reasonable to expect out of any engagement
intervention, I think it’s an important consideration.>>KATE GILL KRESSLEY: This is Kate. I appreciate
that the study – the study Heather had mentioned concerning the interaction between those focuses
and I appreciate it in part because it didn’t try to separate out family engagement. It
acknowledged that families are important to their kids’ education almost as an ancient
truth and then it went beyond that to try to isolate some of those indicators, some
of those ways that it contributes to the whole. So, I think that we are challenged – for many
of us who are not in academic situations the challenge is not so much to do studies but
the challenge is to choose wisely and to look at the range of options out there in terms
of resources and comprehensive programs for family engagement and build your evaluation
around those kinds of programs that appear to be solidly linked to the kinds of outcomes
that you’re going to be working towards and moving from that point.>>AMY APARICIO CLARK: This is Amy. I would
just like to echo what we said earlier about the importance of involving parents and youth
both in the actual data collection but I would extend that even to say that having them be
key informants in the development of the intervention being evaluated itself because that will help
us I think as we design the evaluation and think forward to some sort of backwards planning.
It will help us having their input at that early intervention development stage to be
more aware of what we need to look for in the outcome stage.>>HEATHER WEISS: Becca, do you have any additional
thoughts?>>REBECCA MAYNARD: I think we have them pretty
well covered so…>>HEATHER WEISS: I want to close by reminding
people that you will find a set of post webinar resources on our and on the webinar website
probably Monday or Tuesday of next week and we think of it as the stone in the stone soup.
Here’s a set of initial resources. We’re hoping everybody in the webinar audience that
knows of examples of evaluation, relevant tools, and et cetera will contribute them
and in over the next year HFRP and others will be trying to put together a mass of tools
to help us make progress around evaluation in the family engagement arena. So, please,
evaluation is very important to family engagement moving forward. We need to work together to
build our collective capacity to do it and share what we’ve got and this webinar is
an example of that and I think the kind of stone soup of resources to which we hope to
add a whole lot of vegetables and meat in the next year will help as well. So beginning
to link together to share across our work I think is going to be important in this particularly
at a time where resources are scare. I’d like to thank our experts, Becca and Eric
and Kate and Amy and Kathy, for your expertise and sharing it in the webinar and I’d like
to thank everybody, our partners, United Way, the PTA, SEDL, for our continuing work together
on the webinar series and finally I’d like to thank everybody in addition to the Department
of Ed by the way whose sponsorship has been critical, and I’d like to thank everybody
who participated in the audience.

Leave comment

Your email address will not be published. Required fields are marked with *.