A Conversation About AI and the TOEFL iBT Test with ETS’s Kara McWilliams 

1. TOEFL is already known as being the most effective English language assessment for academic purposes on the market. How is AI integrated in the assessment process to ensure students can demonstrate their academic English proficiency most meaningfully?

K.M.: At ETS, we seamlessly integrate cutting-edge AI technology into TOEFL, employing two pivotal approaches to elevate the testing experience for both test takers and institutions. First, our innovative use of generative AI ensures that the test items are meticulously crafted, reflecting real-world academic scenarios to accurately assess language proficiency. This not only underscores our commitment to staying at the forefront of educational assessment but also guarantees a dynamic and relevant testing experience for candidates. Moreover, our utilization of research-based automated scoring through AI algorithms assures a fair and consistent evaluation of test taker performance and the ability to deliver quality scores to test takers and score users rapidly. 

2. ETS has been leveraging generative AI and automated scoring for decades, but the AI Labs at ETS are always innovating. Have there been any recent advances?

K.M.: That’s right! In fact, ETS released the first ever operational automated scoring engine of written text, “e-Rater”, in 1999! But it’s true, we’re always innovating. Recently our AI scoring engine of spoken language, “Speechrater^®” was modernized through the invention “Serval”. The Scientists and Engineers in the AI Labs introduced novel features that made the engine more flexible, usable, and even more accurate. What excites me the most about this progress is that the innovation also allows ETS to provide test takers with deeper insights into their performance so they can continue to progress in their language proficiency development. Test takers began to see the benefits of Serval in optimized TOEFL score reports that were released in November. 

3. What is important for institutions to know about how AI is utilized in TOEFL?

K.M.: It’s important for institutions to know that ETS upholds the highest rigor when leveraging AI so they can continue to trust the high-quality scores ETS has always delivered. There’s sometimes concern that AI – whether leveraged in content generation or scoring – can reduce the quality of an assessment, or the scores produced. It’s important that institutions know that at ETS we take great care to ensure that’s not the case. In fact, there is a host of research published on the quality of our AI models and engines. Even still, we believe that humans and machines should work together to deliver the highest quality.

Our use of AI in TOEFL is characterized by a “human in the loop” approach.

While we leverage generative AI for item development and AI algorithms for scoring, human expertise remains an integral part of the process. Our world class content experts work in collaboration with AI technology to ensure fair, valid, and reliable assessment that TOEFL has always been known for. 

4. Some English language assessment companies feel strongly against using AI in their assessments – and others use complete AI scoring! Why does ETS take the machine plus human approach and how does it differentiate? 

K.M.: The integration of machine plus human scoring capitalizes on the precision of AI and the nuanced understanding of humans. While complete human scoring may be subjective and prone to biases, and complete AI scoring my struggle with contextual nuances, the hybrid approach ensures a harmonious balance. It combines the efficiency and consistency of AI with the cultural awareness, empathy, and contextual comprehension that human assessors bring to the table. By working together, machines and humans not only enhance accuracy and reliability but also capture the depth and subtleties of language proficiency, making it a more comprehensive evaluation methodology. 

5. We know that security is a priority for ETS, can AI help increase the security of TOEFL? 

K.M.: Advanced technology has dramatically improved the security capabilities in TOEFL, and we know they are effective. We’ve seen the number of cheaters that are flagged increase with new capability releases and the speed at which we can identify test takers cheating increase as well. Our Scientists have developed innovations that leverage biometrics such as face recognition, voice recognition, keystroke dynamics to detect inconsistencies, anomaly detection in score patterns, as well as other capabilities to ensure that we are identifying the dishonest test takers and offering a meaningful experience to the honest test takers. 

6. Does the use of AI in edtech concern you at all? If so, how are you mitigating those concerns at ETS?

K.M.: The integration of AI in edtech doesn’t inherently concern me; rather, it underscores the need for responsible and ethical implementation. AI, when used responsibly, has the potential to revolutionize education, offering innovative tools that personalize learning experiences and drive educational outcomes.

What’s crucial, though, is equipping educators and stakeholders with the knowledge and tools needed to navigate the technology landscape ethically.

It’s not about fearing the technology, but about empowering individuals to wield it responsibly. This is one of the reasons I’m so excited that ETS is on the steering committee of TeachAI alongside ISTE, Khan Academy, and Code.org where we’re developing tools and resources that help achieve this goal. 

7. Given all the innovation happening in the AI Labs at ETS, what are some additional ways you think AI will continue to help TOEFL test takers? 

K.M.: I think the real opportunity to leverage AI to support test takers is through innovative solutions that help them develop their language proficiency in advance of taking TOEFL and continue to develop it after they’ve taken the test. In the AI Labs at ETS we see great power in bringing generative AI and personalization capabilities together to help test takers reach their goals efficiently.

By understanding a learner’s goals as well as their backgrounds and experiences, we can offer them developmental content that is relevant, meaningful, and socioculturally responsive. And, with the modernization of Speechrater we can now provide real time feedback and insights to help learners improve their speaking proficiency. Those insights don’t have to end with the TOEFL – by providing very targeted feedback ETS, in partnership with institutions, can continue to help learners develop their proficiency as they progress on their educational journeys. 

8. Are there any other innovations in AI that you think can help institutions in the future? 

K.M.: Advanced technology certainly offers ETS the ability to provide institutions with more robust insights to help them continue to develop students’ language proficiency. However, there can be other opportunities as well. For example, predictive analytics can support institutions with their efforts around identifying the right students for their programs, and more importantly offer them the insights they need to ensure the students will be successful. ETS has a wealth of important data and looks forward to responsibly using those data to support institutions match the right learners to their programs – and help them achieve their goals. Currently, ETS has team meeting with institutions to understand how we can apply our AI capabilities to deliver them more value. 

9. Is there anything else institutions should be excited by with all the AI innovation happening at ETS? 

K.M.: There’s so much for us all to be excited by as we think about the responsible use of AI at ETS.

What I love the most about ETS is that we aren’t AI led – we’re led by how we can help solve our customers’ problems.

Then, we bring advanced technology together with the science of how we know people learn and demonstrate what they know most effectively and deliver meaningful solutions. The thing I hope institutions are most excited about is how the Labs at ETS are innovating solutions to bring them insights about so much more than the four skills currently assessed on TOEFL. Leveraging capability like multimodal AI we are inventing reliable solutions to provide insights on things like effective communication and cultural agility. So much more to come in that space…