Skills for the Future: Carnegie and ETS's Vision for the Future of Assessment, with Danielle Eisenberg

In this episode, Danielle Eisenberg, Chief of Assessment Design and Integration at ETS, joins us to discuss the future of assessment. She talks in depth about The Skills for the Future Initiative, a joint undertaking between ETS and The Carnegie Foundation for the Advancement of Teaching.

These two well-known leaders in assessment are collaborating to create improved methods of documenting and assessing the knowledge, skills, and dispositions needed to succeed in the 21st century, acknowledging that these are not singularly demonstrated through time in the classroom. Visit AVID Open Access to learn more.

Transcript

Danielle Eisenberg 0:00 We need to think really deeply about how to measure new things, how to do that in new ways, and then how to use that to provide actionable insights about the development and acquisition of skills in a way that hopefully drives real time support, but also increases student agency over their own pathways.

Rena Clark 0:22 The topic of today's podcast is Skills for the Future: Carnegie and ETS's Vision for the Future of Assessment, with Danielle Eisenberg. Unpacking Education is brought to you by avid.org. AVID believes that we can raise the bar for education. To learn more about AVID, visit their website at avid.org. Welcome to Unpacking Education, the podcast where we explore current issues and best practices in education. I'm Rena Clark.

Paul Beckermann 0:55 I'm Paul Beckermann.

Winston Benjamin 0:56 And I'm Winston Benjamin. We are educators.

Paul Beckermann 1:00 And we're here to share insights and actionable strategies.

Student 1:05 Education is our passport to the future.

Rena Clark 1:09 Our quote for today is from Amit Sevak, the president and CEO of ETS. Amit says, "It is time for a seismic shift in education that puts the focus on skills. And not just cognitive skills accounted for in curriculum written for 20th century skills and work, but the rich tapestry of skills that enable individuals to thrive in the 21st century. This is a fundamental paradigm shift from time-based to skill-based units of learning. Smarter assessments are the key to unlock the future." Ooh, lots packed in there, y'all. So what do we think?

Winston Benjamin 1:49 I'm gonna try to jump in real quick. So I'm grabbed instantly by the rich tapestry of skills statement. I really do think kids need to be able to maximize so many things. Being able to look at who is famous now or who is maximizing opportunities, those who can use the internet, those who can market themselves, those who can do a lot of things in different ways, in terms of creating an identity that other people follow.

So I think in parts of that, it's important to provide students the opportunity to develop those chances and skills to take and utilize where the world is going. Because, again, some of the things that kids are doing right now I don't even think I could do, because I have no interest in this, in what they're in their world, but seeing them make edits on TikTok and all these other things. It's a whole skill set that people go to school for, so just thinking, how do we utilize that and allow them to maximize that opportunity? Is really what I'm seeing thinking about that rich tapestry of skills. Yeah.

Paul Beckermann 2:56 I love that tapestry, too, Winston, and for me, it's kind of a reminder that we measure what we value, and we value what we measure. And I think it should really prompt us to take a really hard look at what is important in today's world. Are we assessing what's truly important? Are we doing what we've always done because we've always done it that way?

I think a shift to focusing on skills that was mentioned in the quote is really important. What skills do learners need today, in today's world, not yesterday's world, but today's world, and maybe tomorrow's world too. But I get kind of fired up about assessment, because I think we have some ways that we can really improve it and make it better and in a way that makes our whole education system better.

Rena Clark 3:42 I'm also really excited to dig into this topic, and I'm excited to introduce our guest today. So we'd like to welcome Danielle Eisenberg to our show today. Welcome. And Danielle is the chief of assessment, design and integration at ETS. So just to get started, Danielle, if you wouldn't mind just telling us a little bit about ETS and the Carnegie Foundation partnership, just so we have a little bit of background of what we're going to focus in on today. What are we doing here? Yeah.

Danielle Eisenberg 4:18 Great. Well, I'm so happy to be here. Thanks for having me. So ETS and the Carnegie Foundation for the Advancement of Teaching teamed up with this really ambitious goal to reshape the educational landscape and ensure that every student's equipped with the knowledge, skills necessary to thrive in our complex society.

And what's really interesting about this partnership is that Carnegie was the creator of the Carnegie unit, which many might know is the credit hour, and that has really been the bedrock currency of the educational economy, in ways that sort of some people understand and some people aren't aware of, because it just is the system, but it's the way the vast majority of high schools and colleges are organized today, and it shapes the daily work of teachers and professors, and it's why bells ring between classes, and it really largely determines what we count as learning, right? It's an algebra one class in one state is the same as another, and you generally understand that kids who have algebra one did the same things or learned the same thing, and importantly, this Carnegie unit really has also shaped what is and is not assessed. Which brings us to ETS, who is, in a lot of ways, the grandfather of the standardized tests. They've had a long history with the SAT and the GRE and the TOEFL and the Praxis and other tests that individuals take at really pivotal moments to attain entrance to a collegiate program or to get a high stake certification. And those things are really important.

And I think historically, there were really important reasons for both the creation of the Carnegie unit and for standardized testing, which were at the point they were created intended to provide equity to the system, right? To make sure that a course in one university was more or less equivalent to a course in another. But we also know that the way these systems have evolved, there's been a lot of unintended consequences for a lot of students, and so it's become increasingly clear that the knowledge and the skills and the dispositions that are needed to succeed in the 21st century aren't singularly demonstrated through time in a classroom, and yet we're stuck in that system right now that conflates time with learning, and particularly time spent in a school building, and so that has ignored, I think, the myriad of other ways that students can gain valuable skills. And so these two giants in the industry have paired up, in part to dismantle what they originally helped to create. Hmm.

Paul Beckermann 7:06 That's super interesting.

Winston Benjamin 7:08 I'm really excited to hear more about this. So what is the goal of the Skills for the Future initiative that you that you've just explained why you're doing, who's doing this? So what is the goal of the Skills for the Future initiative? Yeah.

Danielle Eisenberg 7:23 So I'll start with the really big audacious goal, just to keep your head spinning a little. Our 10 year goal is that every person, every person, has a living record of their lifelong pursuit of knowledge and skills that can be used to unlock upward mobility and drive access to more opportunities. We think if we could crack that, which is a big, really big challenge, that we could fundamentally change an individual's experience in three sectors: K-12, higher ed, and the labor market.

But we can't start there, so we're in the short term. We're focused on developing, piloting and scaling a new suite of tools, hopefully to millions of students that will provide new thinking about what skills learners need, and also provide innovation in how skills and knowledge are measured. And so we need to think really deeply about how to measure new things, how to do that in new ways, and then how to use that to provide actionable insights about the development and acquisition of skills in a way that hopefully drives real time support, but also increase the student agency over their own pathways.

Because we think right now it's not explicit to kids what the content and skills they need to succeed in various pathways that they talk about are, right? So if we could make that more explicit and then help them understand how to build that skill and knowledge aligned with their goals and dreams, we think we could, we could fundamentally shift the education system.

Rena Clark 9:08 I love the idea of dream pathways.

Paul Beckermann 9:12 Yeah, the whole thing, just it feels transformational. I mean, it's a great goal. I love it. I'm kind of curious, how are you deciding which skills that you're going to focus on when you're kind of imagining this, this whole new system?

Danielle Eisenberg 9:25 Yeah. So in the first year, ETS and Carnegie really spent a ton of time in the research phase, right? So they conducted extensive research and released a report about this, reviewing the existing skills frameworks, right? There's tons of skills frameworks, and they're known by lots of different names, right? 21st Century Skills, non-cognitive skills, durable skills, and on and on. And they looked at those, both nationally and internationally, and they looked at all the work—I think 22 states to date have documented portraits of a graduate. So they looked at those frameworks, and then they really did work to pull research that indicated which ones are most predictive of the outcomes in life that we care about for kids, right? So not just higher ed attainment, but labor market, health, wellness, right? And so they did all that, and then they synthesized it and prioritized a set of 11 skills for measurement and validation work in the first five years of our effort.

Following that, the team did a lot of deep work with our five state co-design partners, which we could talk about in a bit. But through that co-design effort, we landed three skills to pursue first. So we're working on collaboration, critical thinking, and communication. And those rose to the top largely because they're ubiquitous across almost any framework and almost named in the same language, which is actually not true of many of the other skills. And we thought they'd be really easy to incorporate, sort of in both standalone ways, but also through academic content areas and other contexts. And so that was important to us, because we really want to explore a lot about the way the development of skills matters and manifests in different contexts, and how to think about that in the measurement component.

Paul Beckermann 11:34 So just clarification, then, if you pick these three skills, so it's kind of three of the 4Cs, what we talk about, are you thinking then that those would be measured in multiple content areas, across the system?

Danielle Eisenberg 11:49 Yes. So as we start in our proof of concept, a lot of the measurement will roll out slowly, but over time? Yes, we think that it's going to be really important to have a better understanding of what does collaboration look like in a math class versus in an English class, versus on the sports field versus in teamwork, right? Those may not look the same. The evidence behind it might not be different. It might be easier for you to collaborate. Actually, research says, right, it's going to be much harder for you to collaborate in a content area that you have no background knowledge in. How well could you walk into a physics class and be a great collaborator if you have no basic knowledge of physics, versus in your English class, where you might be an amazing writer?

And so we're trying to parse apart the skills and the constructs that go under those skills, right, and figure out what will help kids develop the skills in a way that's transferable, or even over time, be able to do more predictive analysis. Okay? So if a student has been able to demonstrate a certain skill level across X number of contacts, that means they are most likely to be able to demonstrate that skill in a new context, right? And so we're excited about the ability over many years, right, to do really deep research on what this looks like and how to help students then develop skills.

Rena Clark 13:22 And you've alluded to some of the differences, but I don't know if you want to get into a little bit more detail about how this initiative is really different from other assessment options that we've seen in the field, other assessment options out there.

Danielle Eisenberg 13:38 Yeah, to me, what's most unique and exciting about what we're doing is we're not building a "sit and take" assessment that happens at this one singular point in time and then has very high stakes associated with what happens there. So we're really thinking about a suite of tools that we're hoping people engage with over and over in different ways that allows for multi-source evidence capture.

And so we are developing a platform that will allow students and educators to upload authentic work samples right from your classroom: work portfolios, essays, problem sets, group projects, videos, as one way of looking at what does skill development, or what does even content knowledge look like in real time, in real ways, in the classroom. And we are interested in capturing evidence from out-of-school spaces. So how do we understand where you're demonstrating leadership or great communication in your family, or in your job, or in your internship, or your volunteer, or your church, or your sports field, right? Because we know and we see in our own kids, there are many times they're able to do and demonstrate something out of a school setting that somehow does not translate or is much harder for them to do on traditional measures. And then that is never counted, and it's never captured in a way that people that make decisions about grades and pathways right can see.

And then we're, of course, because we're ETS, we're developing assessments, assessment modules, experiences. We don't want it to feel so testy or assessment-y, but we want to provide places where students can asynchronously practice skills and then get data about how they're doing, both quantitatively, but more importantly, I think qualitatively. Here's where you are on your collaboration skills. This is what you would need to do next to improve. And so we're providing, we're really excited about the ability to provide different ways for students to engage and demonstrate skills. We think that can meet a lot of more kids where they are, but then push them along a journey and help give them agency about which skill am I working on now, right? So whether that direction comes because they're in a state or a school that has a very specific portrait of a graduate, or because it comes because it's attached to wayfinding, navigation later on, or it becomes because they're just really interested in working on a certain skill. We're excited about where we could infuse student agency into this process too.

Paul Beckermann 16:30 So are you thinking that this is kind of a portfolio kind of a thing, a lifelong portfolio, almost a portfolio resume, almost, or—

Danielle Eisenberg 16:39 Yes, a resume is a great is a great word for it. I think that's not right. So it's going to take us a minute to get to the resume part, because we need to figure out the valid measurement part, right? And so we're trying to be very thoughtful and careful and slow about how we go. We're first starting from a formative place. We want to build an evidence capture system that allows for formative feedback so that educators and other adults can help students on their journey.

But we believe that there's a moment when we're going to be able to verify your skill acquisition, and we're going to be able to look really holistically, right, and in different ways in our cross-context about that, but when you're ready to stamp your skill or get it verified, that there will be a way to do that, and that will go on to your living transcript, right? It could be called a transcript, a resume, a portfolio, but that there will be some sort of way to demonstrate your skills wheel and have a record of what you've accomplished and learned and links, potentially, for those I'm going to dig in, here's the evidence of that, right, here's this program that I did, or here's this letter of recommendation, or here's this sample of my work. And so that will be useful in some cases.

In other cases, we're hoping that the ETS name and ability to validate, based on our history of research, will allow us to be when we say this is verified, we've done the background work so that you can have confirmation that this means something, which we think has been that the Carnegie and ETS really are trying to solve the part of the puzzle that's been harder for folks to solve, right? There's been tons of people for ages and ages trying to do project-based learning, competency-based education, that believe in innovative models, that know holistic skills matter. The missing piece has been the signaling system, right? Because our assessments are just not designed to do this, and they're not designed to do this in ways that meet kids in authentic spaces.

Winston Benjamin 18:41 Yeah, it's the hurdle before kids can go into college, or the skill sets that they gained in alternative spaces aren't transferring into the SATs, which then are the hindrance to students having opportunities and access. I really appreciate that those who create somewhat the wall are trying to tear down the wall. How are you working with states and school districts, across such a wide range of areas, beliefs, ideologies, context, all those things that you're trying to assess through. How are you doing this?

Danielle Eisenberg 19:19 It's such a great question. Well, one thing is, we're not doing this in a vacuum, so we're super lucky. We've had five states sign on to be early co-designers with us: North Carolina, Nevada, Wisconsin, Indiana, and Rhode Island, and they are very different, right? Different contexts, different political system contexts, different different places in their graduate profile work and where they are on this journey. So that's been amazing.

We've been engaging with them over the course of the past year. First to really think about, okay, as we go out, what are the skills that make sense to start with? How is this going to land? How does this relate to your state-based portrait of a graduate work? We want to be additive, not in conflict with, right? A lot of these states have done amazing stakeholder work already in their communities to get to the competencies and skills that matter for their local communities. We want to honor that. So we've been doing a lot of thinking about that. And that work is ongoing, right?

So as we're getting ready to launch a proof of concept with a lot of them in February that is really meant to be, okay, now we're putting parts of this in front of you for feedback from educators and students and families, and we're going to shift and change and adapt, right? We don't think what we're putting out in the field in February is going to be the final thing at all. So it's been fun to be in a place that's really iterative, and we can listen to stakeholders and try to make sure we're bringing more and more voices along. It's really important to bring along neurodiverse voices and marginalized student voices, right? We need to make sure that we're designing something that can meet as many students as possible where they're at. So that's been awesome.

We're also really excited about our emerging work with AVID. AVID is such an amazing organization with incredible programs and schools as you all know, throughout the states, and we know, we've had a lot of discussions with AVID about how well the elective helps students, many of whom are first-gen students, develop durable skills that are essential for success. And they're a perfect model of a model who's teaching these critical skills but hasn't had necessarily a clear way to capture communicate that special sauce, and we think we can help AVID potentially do that, and expand their work and their expertise in that field by providing support on the signaling. So we're only a small part of how to shift the ecosystem, and we're excited about our work with many on-the-ground partners and national partners who've been long leaders in the move towards this type of teaching and learning.

Paul Beckermann 22:07 All right, so take out your crystal ball. If you're going to look down the road three to five years, what would success look like for the Skills for the Future initiative?

Danielle Eisenberg 22:18 Great question. So we hope that the product three to five years from now, right? I mean, this might be a little ambitious, but five years from now is in the majority of high schools in the country, and ultimately, sixth through 12th grade schools, for sure, and providing the leading trusted development and measurement platform for the way schools helps ensure students are building not just content knowledge they need, but also the holistic set of skills to pursue the path of their dream.

Paul Beckermann 22:50 That would be amazing. I'm kind of curious, with all the talk about AI, is there going to be an AI element in here to try to sort through all this, because you're going to have so many different things to try to pull together, to figure out if a student is at a certain level.

Danielle Eisenberg 23:08 Yes, well, we're really excited about AI and advanced computational methods and machine learning and all the things that we need to make this happen. And we don't think this was possible before without it, frankly, right? The thing that we are trying to do, I think that's super unique, is pull together student work at scale with other types of assessment measures and figure out how to make sense of that data collectively. And that is not something that would have been possible at scale without AI.

So we, so AI. There's definitely a role for AI here, specifically in the modules we're building. We're using AI right for the cool factor and the engaging factor: chat with a bot, solve a puzzle, and get some feedback. So there's that component. Then there's the work of tagging work to skills, right? So we're building research-backed skills progressions now that are really based on developmental science, science of learning, which is exciting and new, but we need to tag then work to those progressions.

And over time, we think we're going to be able to help scale the work of performance-based education by using AI to help facilitate, not replace, but facilitate, some of the educator grading, because that is a lot of effort right to grade authentic student work and performance tasks and complicated projects. And so if we can train our AI models to understand what does the level three on this type of work look like, then over time, we'll be able to help the educator provide really meaningful qualitative feedback about student skill development and give them language that they might not have had before about that and help them know what the next stage is. And so we think there's lots of use cases for AI in how this product suite will work.

Paul Beckermann 25:11 Yeah, that sounds exciting. I mean, if you can take some of that burden off teachers and make something that was impractical or impossible before now possible, that's fantastic.

Danielle Eisenberg 25:22 Yeah, and we've had a lot of, it's interesting because we have users sort of on a spectrum, right? We have folks in pretty traditional schools that are just really interested, and know it's important to teach durable skills, but don't currently have access to how, right? And so part of our platform solves for that, that use case. And then we have schools that are really all the way over on the innovative, full on mastery-based model, right, who are more struggling to figure out how to validate their approach. And so we think we can do both things and bring them together and then move everyone sort of towards an evolved system of teaching and learning that will be better for kids.

Paul Beckermann 26:07 Very cool.

Rena Clark 26:08 Okay.

Winston Benjamin 26:09 Seems you're trying to create, help people create, a toolkit, a system of utilizing how to make things work. And in our program, we always ask, Hey, what's in your toolkit?

Transition Music 26:22 Check it out. Check it out. Check it out. Check it out. What's in the toolkit? Check it out.

Winston Benjamin 26:32 So Paul, Rena, what's in your toolkit? What's something you're taking away?

Rena Clark 26:37 Well, it's interesting. I've been working with a lot of high school ELA teachers lately. And Paul asked about AI and our forms of assessment, and this worrisome attitude around cheating, but a big part of what I've been working with them and saying is, if we redesign our assessments, and I love that, it includes many of the things you were just talking about Danielle. But if we include more opportunities for collaboration and peer interaction within the assessment, if we're doing more critical thinking, justifying the process and reflecting, we're incorporating creative tasks, we're connecting it to personal experiences, and also maybe connecting it to our community, our local community, our schools, those more authentic things, and those are the things I've been trying to support. Is as we redesign assessments and align to those three, collaboration, critical thinking, communication, the three you all said you were focusing on, and I know that's something we're focusing on as well.

Winston Benjamin 27:38 Love it. Paul,

Paul Beckermann 27:42 I'm just really excited that we can move beyond our traditional assessments and get to something that measures these durable skills, these transferable things that has always been a mystery. For teachers, yes, we can use rubrics and things that helps, but to have a tool and a system that can maybe help us evaluate the student's durable skills. I'm excited about that, so I'll throw the durable skills in.

Winston Benjamin 28:07 I like that. I'm gonna say, hey, as you all are starting to prepare for the next phase of education, check out problem-based learning. And our past episodes on flipping the classroom that gives you a chance to think about how to and what are the ways that you can design a classroom to really get those skills started. What tool do you have?

Danielle Eisenberg 28:32 Oh my gosh, I will. I'm just really excited, and I hope people reach out. Because one of the things we haven't fully solved is how do we implement a full new measurement tool in classrooms without burdening educators? We have lots of ideas about what we hope will help reduce burden, but over time, but in the early stages, I think we're asking people to add something, right? So we have to really think carefully about that. How do we, how do we take away, or how do we make sure this has enough value to be worth it and to show that how the outcomes can improve for students? So really, just starting to dig into implementation success and how to make this work for teachers who we know are so overburdened with so many complicated issues. So that's what I'm thinking about as we leave this conversation.

Paul Beckermann 29:29 All right. Well, we have one more segment left, and that is our one thing.

Transition Music 29:34 It's time for that one thing. Time for that one thing. For that one thing, that one thing.

Paul Beckermann 29:42 All right, one thing time. Winston, what's still on your mind? What are you thinking about? Or what's your takeaway?

Winston Benjamin 29:52 One of the things that stuck in my head is that there's a difference between what students think the skills they need to prepare for their future versus what they actually need and have access to. So really thinking about, how do we make connections to them? And that's something that this conversation is really helping me think through is what are the skills kids need? How do I give them access to that? And how do I help them know that they actually need this, right? That's the part where I'm deeply thinking. So that's my one thing. I'm walking away with. Rena.

Rena Clark 30:30 This excites me. I love that you're taking it slowly. I was just thinking about the idea of differentiated assessment experiences versus tests. And I just love that. And the idea that we keep talking about valuing everybody, and that there's all different kinds of brilliance. And so I love the idea too, the evidence from outside of the school space, and how that really values the individual, the student, and their strengths. And then if you all can figure out how we do that, yay.

Paul Beckermann 31:04 Right? Yes, true. I'm kind of in the same line of thinking as you Rena, and I'm going to build on my toolkit, which was the durable skills. Once those are identified, I've always said in past episodes here, my first question I would always ask when I was going to teach is, what's the most authentic use of this content or skill that somebody would do? And let's have the students do that so they can actually demonstrate in a realistic, real-world way.

Well, that's not always super easy inside the walls of a school. That's why I'm going to build on what you said, Rena, in that taking it outside of the school. It is super exciting to me that we can maybe bring in evidence of what a student's doing in a club that they're in, or they're just doing it as a passion project on the side, or they're doing it in an extracurricular because those are the places where a lot of kids thrive the most, and yet we're not capturing the excellence that they're demonstrating in those places. So I've got a lot I'm thinking about, but I'm also really excited about it. Danielle, you get a final word here today. What would you leave us with?

Danielle Eisenberg 32:12 Oh, wow. Well, that just made me think about it, especially in the high school context. We've all named that that is really important. The question to me, who are the adults then, in a high school building that will own and care about that, right, and see that, and how does that get pulled in? So I think this is, you opened up a new question to me. We've known it's important. How that, how that gets actualized in our product, in a way that shows up in ways that matter for the kid and for other adults that are helping that student, is a question that I'm going to leave with, but I leave all you with, thank you so much for this opportunity to come and chat with you all. I learned a lot just from listening to your questions and your conversation, and we're so excited about the future. It is a journey. It is not an easy task to solve, and we're excited to partner with folks who want to work with us and help figure it out. So we hope to be in touch. Yeah.

Rena Clark 33:18 Well, thank you so much for joining us, and I'm hoping throughout this journey. Hey, who knows, we might still be here in 10 years, Paul and Winston, I don't know. Never know. You never know.

Paul Beckermann 33:28 We'll have lots of episodes of evidence to put on our portfolio, or whatever we want to call it.

Rena Clark 33:34 Hopefully we can check back in. I would love to see how things are going. But we really appreciate you, Danielle, for being here and sharing what's going on and even just a little bit of the history at the beginning. Was great to stick around us. So I appreciate you and all the work that you and everyone else that you're working with is doing. Thanks for listening to Unpacking Education.

Winston Benjamin 33:57 We invite you to visit us at AVID Open access.org where you can discover resources to support student agency, equity and academic tenacity to create a classroom for future-ready learners.

Paul Beckermann 34:12 We'll be back here next Wednesday for a fresh episode of Unpacking Education.

Rena Clark 34:17 And remember, go forth and be awesome.

Winston Benjamin 34:20 Thank you for all you do.

Paul Beckermann 34:22 You make a difference.