Prompt Engineering Papers

Paper with Code prompt engineering section (opens in a new tab): Provides access to research papers along with the corresponding code.

Overview

OpenPrompt: An Open-source Framework for Prompt-learning (opens in a new tab) (2021): Introduces a flexible framework for prompt-based learning that allows prompting and finetuning large PLMs like BERT, GPT-2, T5 etc. Provides implementations for prompt engineering and analysis. (project) (opens in a new tab)
Pre-Trained Models: Past, Present and Future (opens in a new tab) (2021): Provides a historical overview and taxonomy of foundation models like BERT. Analyzes tradeoffs in model scaling, data, compute, and transferability.
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (opens in a new tab) (2021): Surveys different prompting formulations for NLP including soft prompts, hard prompts, continuous prompts etc. Analyzes prompt tuning objectives and benchmarks performance. (project) (opens in a new tab)
Paradigm Shift in Natural Language Processing (opens in a new tab) (2021): Discusses the shift from feature engineering to pretraining large neural models on unlabeled text. Transfer learning has driven progress on many NLP tasks. (project) (opens in a new tab)

Pilot Work

Parameter-Efficient Transfer Learning for NLP (opens in a new tab) (2019): Introduces methods to reduce sizes of pretrained models by pruning and distillation to improve parameter efficiency of transfer learning. (project) (opens in a new tab)
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (opens in a new tab) (2019): Proposes T5 model pretrained on a unified text-to-text format. Shows strong transfer learning performance on diverse NLP tasks. (project) (opens in a new tab)
Language Models as Knowledge Bases? (opens in a new tab) (2019): Investigates using language models like BERT for knowledge base completion and fact retrieval by querying the model. (project) (opens in a new tab)
How Can We Know What Language Models Know? (opens in a new tab) (2019): Analyzes methods to probe linguistic knowledge learned by language models, testing capabilities like coreference resolution. (project) (opens in a new tab)
Language Models are Few-Shot Learners (opens in a new tab) (2019): Shows LMs can perform well on many NLP tasks with only a small number of training examples. (blog) (opens in a new tab)
AdaPrompt: Adaptive Model Training for Prompt-based NLP (opens in a new tab) (2022): Proposes an adaptive prompting method that automatically searches over prompt space during model tuning.

Basics

Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference (opens in a new tab) (2020): Uses cloze-style prompts with masked tokens for few-shot text classification and NLI tasks. (project) (opens in a new tab)
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (opens in a new tab) (2020): Shows even small pretrained language models can achieve good performance on few-shot NLP tasks through prompt tuning. (project) (opens in a new tab)
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (opens in a new tab) (2020): Automatically generates prompt templates for querying knowledge from LMs without manual engineering. (website) (opens in a new tab)
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (opens in a new tab) (2020): Proposes methods to automatically identify words that can serve as class labels for few-shot text classification. (project) (opens in a new tab)
Making Pre-trained Language Models Better Few-shot Learners (opens in a new tab) (2021): Introduces new pretraining objectives like masked language modeling to better adapt LMs for few-shot fine-tuning. (project) (opens in a new tab)
Prefix-tuning: Optimizing continuous prompts for generation (opens in a new tab) (2021): Introduces continuous prompt tuning approach with trainable prefixes appended to text sequences. (project) (opens in a new tab)
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (opens in a new tab) (2021): Provides a programming framework for specifying prompts to perform complex reasoning tasks.
Improving and Simplifying Pattern Exploiting Training (opens in a new tab) (2021): Enhances pattern exploiting training with additional supervisory signals and modeling advances.
GPT understands, too (opens in a new tab) (2021): Demonstrates that GPT models can perform reasoning tasks when prompted with appropriate formulations. (project) (opens in a new tab)
The Power of Scale for Parameter-Efﬁcient Prompt Tuning (opens in a new tab) (2021): Shows that larger pretrained language models better leverage soft prompt tuning across NLP tasks. (project) (opens in a new tab)
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (opens in a new tab) (2021): Proposes prompting with weighted mixtures of soft prompt templates. (project) (opens in a new tab)
Factual Probing Is [MASK]: Learning vs. Learning to Recall (opens in a new tab) (2021): Studies how much factual knowledge is retained in the parameters of language models. (project) (opens in a new tab)
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (opens in a new tab) (2021): Achieves strong few-shot performance with simple prompt tuning approaches without complex formulations.
WARP: Word-level Adversarial ReProgramming (opens in a new tab) (2021): Adversarially generates prompts to reprogram undesirable behaviors in LMs. (project) (opens in a new tab)
PTR: Prompt Tuning with Rules for Text Classification (opens in a new tab) (2021): Incorporates human-provided rules to improve prompting for text classification.
NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction (opens in a new tab) (2021): Pretrains BERT model using next sentence prediction as a self-supervised task. (project) (opens in a new tab)
Finetuned language models are zero-shot learners (opens in a new tab) (2021): Shows finetuned LMs can perform well on unseen tasks with no gradient updates.
PPT: Pre-trained Prompt Tuning for Few-shot Learning (opens in a new tab) (2021): Pretrains prompts on masked language modeling as initialization for few-shot tuning.
Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners (opens in a new tab) (2021): Makes prompts differentiable end-to-end for gradient-based optimization. (project) (opens in a new tab)
Multitask Prompted Training Enables Zero-Shot Task Generalization (opens in a new tab) (2021): Jointly trains prompts across multiple NLP tasks.
P-Tuning v2: Prompt Tuning Can Be Comparable to Finetuning Universally Across Scales and Tasks (opens in a new tab) (2021): Shows prompt tuning can approach finetuning performance with large LMs. (project) (opens in a new tab)
Black-Box Tuning for Language-Model-as-a-Service (opens in a new tab) (2022): Enables querying LMs without access to gradients or parameters. (project) (opens in a new tab)
Black-box Prompt Learning for Pre-trained Language Models (opens in a new tab) (2022): Learns prompts by maximizing probability of target texts.
Binding Language Models in Symbolic Languages (opens in a new tab) (2022): Binds parameters of LMs to symbols to induce reasoning. (project) (opens in a new tab) (website) (opens in a new tab)
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (opens in a new tab) (2023): Curates prompt patterns to enhance prompt engineering.

Analysis

Improvements

Specializations

Computer Vision Reinforcement Learning