Assessing the accuracy of large language models in extracting latest cricket information
Article Sidebar
Main Article Content
Abstract
The development of large language models (LLMs) is making waves across various fields, bringing numerous benefits and innovations. At the same time, cricket is growing rapidly in popularity worldwide. Given this context, it's a great moment to explore how well LLMs can keep up with the latest cricket knowledge. This study evaluates the performance of three LLMs Co-Pilot, ChatGPT, and Liner in generating accurate summaries of bilateral Test and One Day Internationals (ODI) cricket series played in 2024. The evaluation focused on three main tasks: reporting series results, identifying the top three batsmen with their scores, and listing the top three bowlers with their wickets. Among the models, Co-Pilot stood out, consistently delivering the highest accuracy across all tasks and formats, especially for matches involving Australia, India, and South Africa. ChatGPT showed mixed results, excelling in some areas but struggling with task-specific accuracy. Liner, on the other hand, had the lowest accuracy and faced significant challenges in providing relevant detailed cricket-related information. The study also noted instances where the models generated unrelated or incorrect outputs, highlighting the need to validate LLM-generated cricket data to ensure it is reliable and correct.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
References
Bandulasiri, A., Brown, T., & Wickramasinghe, I. (2016). Factors affecting the result of matches in the one day format of cricket. Operations Research and Decisions, 26(4), 21-32.
Bhatnagar, M. (2024). FanCric: Multi-agentic framework for crafting fantasy 11 cricket teams. arXiv preprint arXiv:2410.01307.
Caron, M., & Müller, O. (2023). TacticalGPT: Uncovering the potential of LLMs for predicting tactical decisions in professional football. In StatsBomb Conference (pp. 1-11).
Chiang, S. H., Chao, L. W., Wang, K. D., Wang, C. C., & Peng, W. C. (2024). BADGE: BADminton report generation and evaluation with LLM. arXiv preprint arXiv:2406.18116.
Connor, M., & O'Neill, M. (2023). Large language models in sport science & medicine: Opportunities, risks, and considerations. arXiv preprint arXiv:2305.03851.
Cook, A., & Karakuş, O. (2024). LLM-Commentator: Novel fine-tuning strategies of large language models for automatic commentary generation using football event data. Knowledge-Based Systems, 300, 112219. https://doi.org/10.1016/j.knosys.2024.112219 DOI: https://doi.org/10.1016/j.knosys.2024.112219
Haemmerli, J., Sveikata, L., Nouri, A., May, A., Egervari, K., Freyschlag, C., ... et al. (2023). ChatGPT in glioma patient adjuvant therapy decision-making: Ready to assume the role of a doctor in the tumor board? bioRxiv. https://doi.org/10.1101/2023.03.19.23287452 DOI: https://doi.org/10.1101/2023.03.19.23287452
Jayalath, K. P. (2018). A machine learning approach to analyze ODI cricket predictors. Journal of Sports Analytics, 4(1), 73-84. https://doi.org/10.3233/JSA-17175 DOI: https://doi.org/10.3233/JSA-17175
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. https://doi.org/10.18653/v1/2020.acl-main.703 DOI: https://doi.org/10.18653/v1/2020.acl-main.703
Manage, A. B., Kafle, R. C., & Wijekularathna, D. K. (2021). Classification of all-rounders in limited-over cricket-a machine learning approach. Journal of Sports Analytics, 6(4), 295-306. https://doi.org/10.3233/JSA-200467 DOI: https://doi.org/10.3233/JSA-200467
Mehdi, Y. (2023). Confirmed: The new Bing runs on OpenAI's GPT-4. Microsoft Bing Blogs.
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023). DetectGPT: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305.
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Barnes, N., & Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435.
Olena, K. (2024). Application of LLMs for a chatbot system in the logistics industry.
Qiu, Y. (2024). The impact of LLM hallucinations on motor skill learning: A case study in badminton. https://doi.org/10.1109/ACCESS.2024.3444783 DOI: https://doi.org/10.1109/ACCESS.2024.3444783
Robinson, D. (2023). Evaluating the potential of AI in sports consulting: Investigating ChatGPT-4's ability to consult an MLB team.
Romero, A. (2021). GPT-3: A complete overview. Medium. Retrieved from [Accessed 2025, March 14]: https://towardsdatascience.com/gpt-3-a-complete-overview-190232eb25fd
Sarkar, S., Yashwanth, T. S., & Giri, A. (2024, July). Advancing cricket narratives: AI-enhanced advanced journaling in the IPL using language models. In 2024 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) (pp. 1-6). IEEE. https://doi.org/10.1109/CONECCT62155.2024.10677234 DOI: https://doi.org/10.1109/CONECCT62155.2024.10677234
Sattar, H., Umar, M. S., Ijaz, E., & Arshad, M. U. (2023, December). Multi-modal architecture for cricket highlights generation: Using computer vision and large language models. In 2023 17th International Conference on Open Source Systems and Technologies (ICOSST) (pp. 1-6). IEEE. https://doi.org/10.1109/ICOSST60641.2023.10414235 DOI: https://doi.org/10.1109/ICOSST60641.2023.10414235
Sprint, G. (2024). Social networks and large language models for Division I basketball game winner prediction. IEEE Access. https://doi.org/10.1109/ACCESS.2024.3403490 DOI: https://doi.org/10.1109/ACCESS.2024.3403490
Tatawat, S., & Ghosh, K. (2023, December). CricGPT: A GPT-aided question-answering system for cricket. In Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation (pp. 44-50). https://doi.org/10.1145/3632754.3632757 DOI: https://doi.org/10.1145/3632754.3632757
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
Wickramasinghe, I. (2022). Applications of machine learning in cricket: A systematic review. Machine Learning with Applications, 10, 100435. https://doi.org/10.1016/j.mlwa.2022.100435 DOI: https://doi.org/10.1016/j.mlwa.2022.100435
Wickramasinghe, I. P. (2014). Predicting the performance of batsmen in test cricket. https://doi.org/10.14198/jhse.2014.94.01 DOI: https://doi.org/10.14198/jhse.2014.94.01
Wickramasinghe, I. (2020). Classification of all-rounders in the game of ODI cricket: Machine learning approach. Athens Journal of Sports, 7(1), 21-34. https://doi.org/10.30958/ajspo.7-1-2 DOI: https://doi.org/10.30958/ajspo.7-1-2
Xia, H., Yang, Z., Wang, Y., Tracy, R., Zhao, Y., Huang, D., ... & Shen, W. (2024). SportQA: A benchmark for sports understanding in large language models. arXiv preprint arXiv:2402.15862. https://doi.org/10.18653/v1/2024.naacl-long.283 DOI: https://doi.org/10.18653/v1/2024.naacl-long.283
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How transferable are features in deep neural networks? Advances in Neural Information Processing Systems, 27.
Zhang, J., Han, D., Han, S., Li, H., Lam, W. K., & Zhang, M. (2025). ChatMatch: Exploring the potential of hybrid vision-language deep learning approach for the intelligent analysis and inference of racket sports. Computer Speech & Language, 89, 101694. https://doi.org/10.1016/j.csl.2024.101694 DOI: https://doi.org/10.1016/j.csl.2024.101694