Using AI in Education Part 4

I don't know whether this is a very late advent post or a very early 2024 post.

One of the key themes of 2024 is going to be personalized chatbots or Tutebots in education.

They aren’t very difficult to create. If your students have access to GPT Pro ($20 per month), then this is a trivial task. Simply take the transcripts of the recordings of your lectures and also take the readings ... create a custom GPT from them. If that cost is too great, then things get a little more complicated. Any vendors who want to shill their solutions, please do so in the comment area below. I've been working on an email gateway bot for this kind of task. 

Ethan Mollick reports here on his experiment to see how much of a productivity improvement GPT-4 gives professional workers. If you look at the charts, you'll see that much of the benefit goes to the least able workers. This makes sense that a large language model which produces an average, most predictable output is going to produce a result that's kind of average. If you are below average, then average is an improvement!

This also applies to students, it seems. My friend Gordon Freer teaches International Relations at the University of Witwatersrand in Johannesburg. He did a little experiment last semester. He got more data than I did in my experiment last semester with tutebots since I was only able to create a custom GPT a week before the final exams.

Gordon and I have been analyzing his data in slightly different ways. I looked at the comments made by Gordon’s students and analysed their use of present tense versus past tense, and also their linguistic diversity. This may seem a little odd, but we have good evidence that usage of present tense versus past tense corresponds to an introversion/extroversion divide (highly extroverted and sociable people will talk about all the things that they did with other people, whereas less sociable people will talk about what they're doing right now). Linguistic diversity is a measure of how many different words you use in your writing of a given length and provides a measure of verbal flexibility which is a proxy for verbal IQ. The results were interesting.

Students in Gordon's class whose comments suggested weaker English language skills ended up being far more likely to recommend the use of chatbots in the future. 

It's premature to say that they gained a greater advantage, but it seems likely that being able to ask a chatbot to give you individualised tuition is going to benefit learners who face extra challenges (such as not being a native English speaker).

But other than providing a tutebot, what can we as educators do?

For this I analysed what things Gordon’s students did with the chatbot that were predictive of recommending chatbots. In other words: different students used the chatbots in different ways; which of those gave students the most positive experience?

Cross-validated Ridge regression models found that the two strongeset factors were:

• Did the student use the chatbot to help with their tutorials? (0.44)
• Did the student use the chatbot to help with the readings? (0.74)

Nothing in the student’s backgrounds predicted any behaviour here. It is up to us in our teaching practices to encourage students to interact in different ways, and it’s not driven by the background knowledge, familiarity or existing skills. That suggests we need to have exercises in our courses (perhaps graded exercises) to encourage students to make the most of the chatbots we provide.

Here’s my checklist that I think we should make sure students do (feel free to suggest more).

• Get the chatbot to explain a concept that you aren’t familiar with
• Handle a text in a language that you aren’t familiar with by translating it into something you are familiar with.
• Given an arcane and difficult-to-read text, get the chatbot to simplify the vocabulary that's used
• Ask the chatbot to make analogies with another field with which you are more familiar
• Here’s what ELI5, ELI13 or ELIUG mean (“explain it like I’m 5/13/an undergraduate”)
• What happens if you ask the chatbot how you could improve your essay / program?
• Explain some important concept and get the chatbot to respond with any important ideas that you have missed in explaining it
• Role-play different people, things or participants from a reading.
• Generate exam questions for their own self-study.

That way, they won’t think of generative AI merely as a way of cheating on essay writing.

Ping me if you want to run a study on this!

Using AI in education part 3

Continuing on my AI in education advent calendar...

I stumbled upon a solution to a really annoying problem. You've just gone through a whole pile of marking... (the picture was about a third of the pile from one course)... and then you wonder whether you've gone cross-eyed and entered the wrong numbers into the spreadsheet / learning management system / course management system.

We have flawless speech-to-text now. We have access to language models that can do tedious tasks.

So at the end of the marking, I open up a sound recorder on my phone or my laptop, flick through the exam papers and talk narrate the student marks as I go. "Xu Anh student number 23424 scored 41.5, Tom Bowman student 28559 scored 34, Amy Chin student 29912 scored 26, Daniel Davids student id 28588 scored 37...". Run that through Whisper ( https://openai.com/research/whisper or any of the desktop apps that use it, e.g. MacWhisper ) and you'll get what you said in text. It gets even the most complicated names right if you pronounce them correctly.

Now take a download from your marks management system (or the spreadsheet that you were typing them in to). If you have access to GPT4, you can upload the spreadsheet. If you are using GPT-3.5 or a free LLM like llama, copy and paste from the spreadsheet.

Then the magic prompt: "identify if there are any discrepancies between this spreadsheet and the transcript of me reading out the marks".

Works like a charm, every time.

Cat consulting

Deb Zahn interviewed me on her Craft of Consulting blog: https://www.craftofconsulting.com/podcasts/episode-238

This then inspired https://www.linkedin.com/in/nataliekjacobs/ to write a poem from the perspective of her cat.

To get a sense of how good voice modding is nowadays, here's my deep Australian basso-profundo voice turned into a feminine cat-like voice:

https://soundcloud.com/greg-baker-574386084/natalie-jacobs-cat-poem

Useful vocabulary for winning awards

• Humblebrag is when you say that you just won an award, and you deprecatingly say that the judges might have made a mistake.
• Stumblebrag is when the first time you learn about the award is when someone asks you why you haven't picked it up.
• Grumblebrag is when you complain about walking around in a Sydney heatwave to pick up an award that nobody told you about.
• Mumblebrag is when you realise that the award was for a group you were in, not to you personally.
• Fumblebrag is when you realise you didn't really do anything for that group and you were kind of the spare wheel.
• Jumblebrag is the feeling of winning a dean's award for excellence in inter-departmental collaboration for stuff you didn't do.
• Rumblebrag is when you send out congratulations to the rest of the data science teaching team at Macquarie University for their fine work that you get to take credit for.

Using AI in Education Part 2

This is Week 2 of my advent calendar on using AI in education. Week 1 was here: https://blog.ifost.org.au/2023/11/using-ai-in-education-post-1.html

Since all my lectures are recorded, it's easy to have a transcription of everything I said. I use Jordi Bruin's MacWhisper -- https://goodsnooze.gumroad.com/l/macwhisper on my Mac. On Linux, I use OpenAI's command-line tools, because usually I'm trying to automate something in a pipeline. I haven't used anything on Windows, but parmata's Whisper UI ( https://apps.microsoft.com/detail/9N3SRNM2J6XX?hl=en-US&gl=US ) looks like it's the equivalent of Jordi's program.

The Whisper model behind these programs is free to use, so there's no per-minute cost. (In 2018 when I was the CTO of a speech analytics company and was giving presentations regularly, I predicted that this would be case within 5 years. I was wrong by about 6 months.)

That said, you can also pay OpenAI to do transcriptions for you (with some limitations on length). This might make sense if you use Zapier to automate transcribing your lectures.

Over the next few weeks I'll go into more detail of the very interesting things that you can do with a high-quality transcription, but for now, consider that there are students (e.g. the hearing-impaired) for whom a high-quality transcription can make a world of difference to their learning experience.

P.S. 2 years ago transcription of voice to text was still unreliable. I still see recording systems deployed at universities which are using obsolete (and terrible) transcription technology, so if you are dismissing this based on your experience of current technology, you might be accidentally living in the past still.

Using AI in education, Part 1

Now that semester is over, I'm going to start writing up a post each week of some AI-related innovation I did in my teaching this year. (The real guru on this is Ethan Mollick -- https://substack.com/@oneusefulthing, but maybe I can be interesting.)

I had a lot of overseas students and other students whose first language wasn't English. So I did a poll at the start of the semi-semester to ask what languages they would like my lectures deep-faked into. Hindi was the most requested. Here's me speaking fluent, high-class Hindi! https://youtu.be/mFYjjNUa8ho?si=ONUAaEQd0zR3BwrE

Arabic was another request.

I'm vain and wanted it to be my own voice, so I used Eleven Labs (affiliate link here https://elevenlabs.io/?from=partnerrobles3623 ). I could probably have done this cheaper with Whisper + ChatGPT translation + ChatGPT text to speech (or some other alternative text-to-speech solution).

Issues and observations:

• For some students, it's a clear issue of access to education, and I consider it a moral obligation to provide my lessons in ways that are accessible to as many as possible. If you are struggling with English, and I can easily provide resources to help you in your native language, I'll do that.
• For some students, I think it was just the amusement of hearing obviously-Aussie Greg talking something else. Still, I don't mind: if that's what it takes to get students to listen to lectures, I'm in on it.
• Eleven Labs does very formal translations. It seems like they are using ChatGPT for translations, and ChatGPT defaults to more formal language. Apparently my Hindi sounds like a government proclamation. Hopefully they will support more casual language in their translations.
• It's not cheap for an individual. It begins to makes sense at an organisational level when you have a handful of students wanting the same language.

But I had a surprising amount of pushback. The argument is that part of the experience is for students to study in English so that they can improve their English, English being the world language and all that.

To me, that's last-year thinking: back in 2022 when automated translation wasn't a solved problem, we all needed to agree on a common language for business and education, and the world had chosen English.

But maybe now we don't need a common language any more, so is it still necessary, important and valuable for students to improve their English?

Which is more important now, and which will be more important in the future? Accessibility of education to people in their native languages, or improving English language fluency?

A prayer for debugging

 “God of all wisdom and knowledge, you created the universe and all within it, including logic and computation. I thank you for blessing me to be in this time and this place where I can work to resolve this [bug]. Let me be at peace in the puzzling and confusion, never to rise in anger or let my frustration out on others, who are also part of your beautiful creation. Help me remember the joy that comes from finding a solution, and to look forward in anticipation to this [bug]’s resolution as a tiny reminder of our anticipation of the joy that will come when you come again and make this world right, perfect and whole.”