Meta-Prompting as a Solution to Students’ Prompt Engineering Difficulties for an Optimized Use of GenAI LLMs in the Context of Education: A Quasi-Experimental Study using Mistral Model

Abstract

The emergence of prompt engineering as a rising field of GenAI has garnered attention with the purpose to resolve ambiguities accompanying its use. In essence, a perfect input prompt performing well on LLM A might not perform well on LLM B as there is a certain disparity in how each model behaves, along with students’ poor prompting competency can make of crafting excellent prompts almost impossible to achieve. To address this problematic issue, the present study sheds light on meta-prompting which can solve students’ countless difficulties encountered with forming appropriate prompts, besides disparities in how different LLMs respond to the same prompt. To this end, the study adopts a within-subjects quasi-experimental design, with a sample involving N=50 undergraduate students of the Higher School of Teachers – Moulay Ismail University. For data analysis, the study uses SPSS version 25 for statistical representation of data, and Python code executed on Google Colab coding environment in which the Wilcoxon Signed-Rank Test for paired samples was conducted. Results demonstrated that there is a strong significant difference between students’ self-crafted prompts and Mistral as a selected LLM’s meta-prompts in terms of prompt specificity, comprehensiveness, logical sequence of instructions, and good structure criteria. The direction of the difference was in a positive direction suggesting an increase in the overall rating scores between pre-test and post-test results. The present paper has also proven that students’ confidence with prompts significantly increases with the use of meta-prompting technique compared to traditional prompting. Ultimately, the study identifies limitations and offers recommendations orienting future research projects in the field of GenAI and LLMs.