Accounting for the Future
Tuesday, November 28, 2023
Media Contact: Terry Tush | Director of Marketing & Communications | 405-744-2703 | terry.tush@okstate.edu
ChatGPT and other forms of artificial intelligence have been the talk of technology circles, and it’s easy to see why.
Just by scraping the internet, ChatGPT and AI programs like it have passed the Graduate Management Admission Test, aced the bar exam, written novels in the blink of an eye and even hosted podcasts thanks to their ability to mimic the human voice.
So, it was only natural that accounting faculty nationwide collectively wondered about AI and its application in their field. Will AI programs drastically change accounting, they wondered? Will AI change the way they teach accounting? What does the existence of this software mean for today’s students as they enter the workforce? Are AI programs like ChatGPT already ahead of today’s accounting students?
There was only one way to find out. It was time to put ChatGPT to the test, literally. Oklahoma State University Spears School of Business accounting professors Brad Lawson and Bryan Brockbank joined forces with accounting faculty from across the globe to pit their current students against ChatGPT version 3.5, which was introduced to the world in November 2022.
The study represented one of the largest in the history of accounting research. Brigham Young University’s Dr. David Wood led the research project, and he partnered with 327 faculty members at 186 institutions across 14 countries on the study. The entire process took only about three months — lightning pace for accounting research — simply because the technology and crowdsourced approach to data collection made the breakneck speed possible.
Faculty in the study were asked to feed exam questions into ChatGPT that their accounting students had been asked the previous semester. Then, record whether the chatbot got the answer correct, partially correct or completely wrong, and upload the results to a database.
In all, 28,085 questions were loaded into ChatGPT. Spears Business faculty didn’t know what to expect as they clicked the chatbot’s send button.
“Based on the news stories I had heard, I was afraid that ChatGPT was going to do pretty well,” said Dr. Lawson, who teaches accounting information systems and data analytics. “I had heard that it was passing the bar exam and doing well on master’s level courses. So, my fear was that it was going to actually do pretty well on our tests.”
How did it turn out? The short answer is that the accounting students came out on top, for now, but it comes with an asterisk and a lesson about the future of accounting and business.
Overall, the flesh and blood students posted an average of 76.7% compared to ChatGPT’s 47.4%, but the chatbot’s numbers jump to 56.5% if partial credit is factored in. ChatGPT also performed better than the average student on 15.8% of the assessments.
The study also identified variation across accounting topics. ChatGPT performed better on auditing and accounting information system assessments compared to other accounting topics, like tax, financial and managerial accounting.
ChatGPT version 3.5 struggled with short-answer questions, and in fact, would sometimes produce false and entirely fabricated answers and references. It also struggled with questions that involved math calculations because ChatGPT is a language-based program. Many times, the software didn’t even recognize that it was being asked a math problem. In essence, it was trying to use a dictionary to solve complex calculations. This could explain the variation across accounting topics because auditing and information systems often have fewer mathematical questions than the other topics.
The latest iteration of the program, ChatGPT version 4.0, has no such problem. It was released on March 14, 2023, which is ironic because the crowdsourced paper “The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?” was accepted into Issues in Accounting Education on the exact same day. Version 4.0 of the chatbot includes the ability to do complex math calculations that version 3.5 couldn’t comprehend.
Within hours of its release, ChatGPT version 4.0 passed every major accounting certification exam, including the Certified Public Accountant, Certified Internal Auditor and the Certified Management Accountant. The program went from flunking college exams in November to showcasing the same knowledge as a certified professional in just five months.
If accounting faculty weren’t convinced by the research project’s results, the latest iteration of ChatGPT put their doubts to rest. It was officially time to adjust curriculum to teach students how to adapt to changing technology.
“That’s the approach we’ve taken at OSU,” Lawson said. “Yes, we’re going to teach them Excel. They’re going to learn Tableau. They’re going to learn certain technologies that are the expectation in the profession, but it’s more about teaching the skills and the fundamentals they need in order to adapt and change as the technology changes because that’s going to serve them better five years from now.”
Initial concerns around ChatGPT and academia were about cheating. Faculty were concerned about students using the platform to write term papers and complete projects in a 21st century version of plagiarizing. But the thought has shifted to the inevitable future. These types of programs aren’t going away, and in fact, they’re probably going to change the way industries operate.
Public accounting firms have invested billions of dollars into AI research and advanced data analytic technologies because they see the possibilities on the horizon. Academics across the country have also begun to ask how they can use AI in the classroom to prepare students for that reality.
One way is to incorporate ChatGPT into tests and then ask students to expound on the chatbot’s answer or explain why it’s correct or incorrect. After all, even the latest iterations of the program are far from perfect, and the modern workforce will need to be able to tell when an AI program is going down the wrong path.
“It’s a different level of thinking,” Dr. Brockbank said. “I think that’s where the future with ChatGPT or AI is going to be. We have to teach students to think more critically about what answers ChatGPT is giving you because when ChatGPT’s answers come out, the wording sounds very authoritative. It says this is the correct answer and this is why, even when it was wrong. From the student perspective, if you’re using something like this and you don’t know, you would believe it 100% — this is the right answer.”
Instilling the critical thinking skills needed to navigate the reality of AI alongside the basics of accounting is essential to creating a workforce that can use professional judgment to make the best decisions for their company.
Thanks to Brockbank and Lawson, Spears Business students will have that and more. Along with the other faculty researchers involved in the study, they see that the field of accounting isn’t going anywhere, and neither is AI.
“AI is not going to replace accountants,” Wood said. “Accountants using AI are going to replace accountants who are not using AI. I think in the immediate future that’s going to be the case. It just makes people so much more productive.”
Story by: Stephen Howard | Discover@Spears Magazine
Photo by: Devin Flores