Experimental Evaluation of Generative Language Models in Phishing Email Detection
DOI:
https://doi.org/10.5281/zenodo.20369356Keywords:
artificial intelligence, language models, phishing, threat detectionAbstract
This study evaluates the performance of Claude AI, ChatGPT, Copilot, and Gemini in detecting phishing emails. In a controlled experiment, 80 real messages were analyzed, including 40 legitimate and 40 malicious emails from the RNP Fraud Catalog and the SpamAssassin Public Corpus. Using a standardized prompt, each model classified the messages and assigned a risk probability. Accuracy, precision, recall, and F1-score were calculated. ChatGPT, Copilot, and Gemini achieved 100% in binary metrics, while Claude AI produced one false positive. The results indicate that LLMs can support anti-phishing screening as a complementary layer, provided that they are combined with specialized tools, human validation, and testing on larger datasets.
Downloads
References
ALOM, M.; HAQUE, M.; SIDDIQUE, R. A survey on phishing detection techniques using machine learning. Journal of Network and Computer Applications, v. 170, p. 102-150, 2020.
ALVES, Jéssica; SALANDIN, Amanda Torette; NEVES, João Emmanuel D Alkmin. Clonagem de voz por IA: avaliando a eficácia em ataques de phishing por voz. Revista Tecnológica da Fatec de Americana, v. 13, n. 1, p. 22-44, 2025. DOI: 10.47283/3ycgwz89.
CHEN, L.; HUANG, X.; DENG, Y. Evaluating prompt-based large language models for cybersecurity tasks. In: ACM INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND SECURITY, 2024. Anais [...]. [S. l.]: ACM, 2024. p. 55-68.
GERMANO, Arthur. Experimentação do uso de inteligência artificial generativa para contenção de phishing. 2024. Trabalho de Conclusão de Curso (Segurança da Informação) – Faculdade de Tecnologia de Americana, Americana, 2024. Disponível em: http://ric.cps.sp.gov.br/bitstream/123456789/29245/1/20242S_Arthur%20Germano_OD2467.pdf. Acesso em: 25 maio 2025.
KIM, J.; PARK, S. Human factors in phishing susceptibility: behavioral and cognitive analysis. Computers in Human Behavior, v. 145, p. 107-120, 2023.
LI, J.; WANG, Z. Comparing traditional filters and LLM-based approaches for email security. International Journal of Information Security, v. 23, p. 221-238, 2024.
REDE NACIONAL DE ENSINO E PESQUISA (RNP). Catálogo de fraudes da RNP. Disponível em: https://catalogodefraudes.rnp.br/. Acesso em: 21 nov. 2025.
SHAH, R.; PATEL, M.; SINGH, V. Phishing email classification using transformer-based deep learning models. Computers & Security, v. 130, p. 102-118, 2023.
SPAMASSASSIN. SpamAssassin public corpus. Apache Software Foundation. Disponível em: https://spamassassin.apache.org/publiccorpus/. Acesso em: 21 nov. 2025.
TONEZER, Leonardo Nabarro; SILVA, Ana Clara Matthiesen; ALMEIDA, Alisson Henrique de; NEVES, João Emmanuel D Alkmin. Simulações multiagentes e phishing: explorando a segurança em ambientes de nuvem. Revista Tecnológica da Fatec de Americana, v. 11, n. 2, p. 1-17, 2024. DOI: 10.47283/sg6gph88.
VERIZON. 2023 data breach investigations report. [S. l.]: Verizon, 2023. Disponível em: https://www.verizon.com/business/resources/reports/dbir/. Acesso em: 25 maio 2025.
ZHAO, Q.; REN, P.; LI, F. Understanding LLM vulnerabilities in social engineering and phishing scenarios. IEEE Security & Privacy, v. 23, n. 1, p. 14-27, 2025.
ZHANG, Y.; LIU, M.; GUO, X. The evolution of email phishing techniques in the era of AI-generated content. Journal of Cybersecurity, v. 12, n. 4, p. 1-17, 2023.
ZHOU, Y.; LIU, H.; WANG, T. LLM-based email threat detection: challenges and advances. IEEE Transactions on Information Forensics and Security, v. 19, p. 1123-1137, 2024.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Daniele Aparecida Pereira de Mendonça, Fernanda Cardozo Gomes, João Emmanuel D Alkmin Neves
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright Policy
By submitting a manuscript for publication in the journal Advances in Global Innovation & Technology (@_GIT), authors agree, in advance and unrestrictedly, to the following terms:
- The authors retain the copyright and grant to the Journal Advances in Global Innovation & Technology (@_GIT) the right of first publication of the manuscript, without any financial burden, and waive any other remuneration for its publication in the Journal of Advances in Global Innovation & Technology (@_GIT).
- Upon submission to the Journal of Advances in Global Innovation & Technology (@_GIT), the manuscript is automatically licensed under the Creative Commons Attribution License, which allows the sharing of the work with acknowledgment of authorship and initial publication in the Journal Advances in Global Innovation & Technology (@_GIT).
- Authors are authorized to enter into additional contracts separately, for non-exclusive distribution of the version of the work published in the Journal Advances in Global Innovation & Technology (@_GIT) (publish in repositories or as a book chapter), with acknowledgment of the initial publication in the Journal Advances in Global Innovation & Technology (@_GIT), provided that such agreement does not imply an endorsement of the content of the manuscript or the new vehicle by the Journal of Advances in Global Innovation & Technology (@_GIT).
- Authors are allowed and encouraged to publish and distribute their work online, in institutional repositories, for example after the editorial process is completed.
- The authors warrant that they have obtained proper permission from their employers for the transfer of rights under this agreement, if those employers own any copyright in the manuscript. In addition, the authors assume any and all responsibility for possible copyright infringements of these employers, exempting Advances in Global Innovation & Technology (@_GIT) from any and all liability in this regard.
- The authors assume all responsibility for the content of the work, including the due and necessary authorizations for the disclosure of data collected and results obtained, exempting the Journal of Advances in Global Innovation & Technology (@_GIT) from any and all liability in this regard.
