Misuse of large language models: Exploiting weaknesses for target-specific outputs





prompt engineering, jailbreak prompts, transformers, deep learning, large language models


Prompt engineering in large language models (LLMs) in combination with external context can be misused for jailbreaks in order to generate malicious outputs. In the process, jailbreak prompts are apparently amplified in such a way that LLMs can generate malicious outputs on a large scale despite their initial training. As social bots, these can contribute to the dissemination of misinformation, hate speech, and discriminatory content. Using GPT4-x-Vicuna-13b-4bit from NousResearch, we demonstrate in this article the effectiveness of jailbreak prompts and external contexts via Jupyter Notebook based on the Python programming language. In addition, we highlight the methodological foundations of prompt engineering and its potential to create malicious content in order to sensitize researchers, practitioners, and policymakers to the importance of responsible development and deployment of LLMs.


Agrawal, Sweta; Zhou, Chunting; Lewis, Mike; Zettlemoyer, Luke; Ghazvininejad, Marjan (2023): In-context examples selection for machine translation. In: arxiv.org, 05. 12. 2022. https://doi.org/10.48550/arXiv.2212.02437 DOI: https://doi.org/10.18653/v1/2023.findings-acl.564

Arora, Simran et al. (2023): Ask me anything. A simple strategy for prompting language models. In: arxiv.org, 05. 10. 2022. https://doi.org/10.48550/arXiv.2210.02441

Ba, Jimmy; Kiros, Jamie; Hinton, Geoffrey (2016): Layer normalization. In: arxiv.org, 21. 06. 2016. https://doi.org/10.48550/arXiv.1607.06450

Birhane, Adeba; Kasirzadeh, Atoosa; Leslie, David; Wachter, Sandra (2023): Science in the age of large language models. In: Nature Reviews Physics 5 (5), pp. 277–280. https://doi.org/10.1038/s42254-023-00581-4 DOI: https://doi.org/10.1038/s42254-023-00581-4

Chen, Canyu; Shu, Kai (2023): Can LLM-generated misinformation be detected? In: arxiv.org, 25. 09. 2023. https://doi.org/10.48550/arXiv.2309.13788

Chiang, David; Cholak, Peter; Pillay, Anand (2023): Tighter bounds on the expressivity of transformer encoders. In: arxiv.org, 01. 06. 2023. https://doi.org/10.48550/arXiv.2301.10743

Douglas, Michael (2023): Large language models. In: arxiv.org, 25. 01. 2023. https://doi.org/10.48550/arXiv.2307.05782

Edelman, Benjamin; Goel, Surbhi; Kakade, Sham; Zhang, Cyril (2021): Inductive biases and variable creation in self-attention mechanisms. In: arxiv.org, 19. 10. 2021. https://doi.org/10.48550/arXiv.2110.10090

EU – European Union (2024): AI act. Available online at https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai, last accessed on 26. 04. 2024.

Friedman, Dan; Wettig, Alexander; Chen, Danqi (2023): Learning transformer programs. In: arxiv.org, 01. 06. 2023. https://doi.org/10.48550/arXiv.2306.01128

Garg, Shivam; Tsipras, Dimitris; Liang, Percy; Valiant, Gregory (2022): What can transformers learn in-context. A case study of simple function classes. In: arxiv.org, 01. 08. 2022. https://doi.org/10.48550/arXiv.2208.01066

Graves, Alex (2014): Generating sequences with recurrent neural networks. In: arxiv.org, 04. 08. 2013. https://doi.org/10.48550/arXiv.1308.0850

Gudibande, Arnav et al. (2023): The false promise of imitating proprietary LLMs. In: arxiv.org, 25. 05. 2023. https://doi.org/10.48550/arXiv.2305.15717

Hsieh, Cheng-Yu; Lee, Chen-Yu (2023): Distilling step-by-step. Outperforming larger language models with less training data and smaller model sizes, 21. 09. 2023. Available online at https://blog.research.google/2023/09/distilling-step-by-step-outperforming.html, last accessed on 26. 04. 2024. DOI: https://doi.org/10.18653/v1/2023.findings-acl.507

Karanjay, Rabimba (2023): Targeted phishing campaigns using large scale language models. In: arxiv.org, 30. 12. 2022. https://doi.org/10.48550/arXiv.2301.00665

Kossen, Jannik; Rainforth, Tom; Gal, Yarin (2023): In-context learning in LLMs learns label relationships but is not conventional learning. In: arxiv.org, 23. 07. 2023. https://doi.org/10.48550/arXiv.2307.12375

Learn Prompting (2023): Prompt hacking. Jailbreaking. Available online at https://learnprompting.org/de/docs/prompt_hacking/jailbreaking, last accessed on 26. 04. 2024.

Mahowald, Kyle; Ivanova, Anna; Blank, Idan; Kanwisher, Nancy; Tenenbaum, Joshua; Fedorenko, Evelina (2023): Dissociating language and thought in LLMs. A cognitive perspective. In: arxiv.org, 16. 01. 2023. https://doi.org/10.48550/arXiv.2301.06627

Minhyeok, Lee (2023): A mathematical investigation of hallucination and creativity in GPT models. In: MDPI 11 (10), pp. 1–17. https://doi.org/10.3390/math11102320 DOI: https://doi.org/10.3390/math11102320

NOMIC.AI (2024): GPT4All. Available online at https://gpt4all.io, last accessed on 26. 04. 2024.

OpenAI (2022): Lessons learned on language model safety and misuse. Available online at https://openai.com/research/language-model-safety-and-misuse, last accessed on 26. 04. 2024.

Peng, Baolin et al. (2023): Check your facts and try again. Improving LLMs with external knowledge and automated feedback. In: arxiv.org, 24. 02. 2023. https://doi.org/10.48550/arXiv.2302.12813

Shen, Xinyue; Chen, Zeyuan; Backes, Michael; Shen, Yun; Zhang, Yang (2023): Do anything now. Characterizing and evaluating in-the-wild jailbreak prompts on LLMs. In: arxiv.org, 07. 08. 2023. https://doi.org/10.48550/arXiv.2308.03825

Strauß, Stefan (2021): Don’t let me be misunderstood. Critical AI literacy for the constructive use of AI technology. In: TATuP – Journal for Technology Assessment in Theory and Praxis 30 (3), pp. 44–49. https://doi.org/10.14512/tatup.30.3.44 DOI: https://doi.org/10.14512/tatup.30.3.44

Sutskever, Ilya; Vinyals, Oriol; Le, Quoc (2014): Sequence to sequence learning with neural networks. In: arxiv.org, 10. 09. 2014. https://doi.org/10.48550/arXiv.1409.3215

Vaswani, Ashish et al. (2017): Attention is all you need. In: arxiv.org, 12. 06. 2017. https://doi.org/10.48550/arXiv.1706.03762

Warstadt, Alex; Bowman, Samuel (2022): What artificial neural networks can tell us about human language acquisition. In: arxiv.org, 17. 08. 2022. https://doi.org/10.48550/arXiv.2208.07998 DOI: https://doi.org/10.1201/9781003205388-2

White, Jules et al. (2023): A prompt pattern catalog to enhance prompt engineering with ChatGPT. In: arxiv.org, 21. 02. 2023. https://doi.org/10.48550/arXiv.2302.11382

Xu, Jiashu; Ma, Mingyu; Wang, Fei; Xiao, Chaowei; Chen, Muhao (2023): Instructions as backdoors. Backdoor vulnerabilities of instruction tuning for LLMs. In: arxiv.org, 24. 05. 2023. https://doi.org/10.48550/arXiv.2305.14710

Yang, Kai-Cheng; Menczer, Filippo (2023): Anatomy of an AI‑powered malicious social botnet. In: arxiv.org. https://doi.org/10.48550/arXiv.2307.16336 DOI: https://doi.org/10.51685/jqd.2024.icwsm.7

Zhong, Zexuan; Lei, Tao; Chen, Danqi (2022): Training language models with memory augmentation. In: arxiv.org, 25. 05. 2022. https://doi.org/10.48550/arXiv.2205.12674 DOI: https://doi.org/10.18653/v1/2022.emnlp-main.382




How to Cite

Klinkhammer D. Misuse of large language models: Exploiting weaknesses for target-specific outputs. TATuP [Internet]. 2024 Jun. 28 [cited 2024 Jul. 25];33(2):29-34. Available from: https://www.tatup.de/index.php/tatup/article/view/7117