TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (2024)

Turing test.

Through a chat interface, researchers like Jannai etal. (2023); Jones and Bergen (2023) have explored how well people can distinguish between a human and an LLM. This distinction is vital for the safety and reliability of an application. Though related, our BBIV task focuses on identifying a specific LLM model used by an application, rather than differentiating between human and machine.

Detection of LLM content.

Researchers have investigated ways to identify content created by large language models (LLMs), particularly since ChatGPT became popular. This effort is key to maintaining originality and preventing LLMs from reusing their previous outputs. Various methods have been developed: Mitchell etal. (2023) looked into the model’s probability characteristics; Gehrmann etal. (2019) examined the statistical properties of texts; Chen etal. (2023) used classifiers to tell apart content made by humans from that made by LLMs. There has also been debate on the feasibility of these detection tasks against deliberate manipulation efforts (Sadasivan etal., 2023; Chakraborty etal., 2023). For further details, the surveys by Dhaini etal. (2023); Ghosal etal. (2023) provide a thorough review. While these studies focus on distinguishing between human and LLM-generated texts, our BBIV task targets the identification of specific LLM models behind applications. Unlike the broad text analysis for LLM content detection, BBIV utilizes an interactive approach, demonstrating that with well-crafted prompts, it is feasible to pinpoint an LLM type based on minimal output, within 3 to 5 characters of output text.

Watermarking.

Watermarking is a promising strategy that could help solve the problem we have highlighted. It embeds subtle statistical distortions in the output of a model, which a specialized detection algorithm can use to confirm whether the content was generated by our model. These distortions are designed to be imperceptible to humans. Watermarking typically occurs during the model’s training phase (Abdelnabi and Fritz, 2021; Kirchenbauer etal., 2023; Hu etal., 2023), though there are methods that apply watermarks during the content generation phase as well (Kirchenbauer etal., 2023). Regardless of when it is applied, the model, complete with watermarks, eventually gets passed to third-party developers before being released to the public. This introduces a major challenge for monitoring LLMs in use: once an LLM is deployed without watermarking, it is too late to start tracking its use.Our solution, TRAP, is free of this limitation. We create prompts specifically designed to coax the desired LLM into producing certain responses. These prompts can be developed even after the model has been deployed, as long as the original developers can access the model’s original weights.We contend that TRAP is a fundamentally more practical solution. However, we note that TRAP is not intended to replace watermarking. Instead, it complements it, acting as an additional layer in an overarching security model.

Adversarial suffix.

Despite efforts to align the outputs of LLMs with human goals and ethics (Yudkowsky, 2016; Christian, 2020; Christiano etal., 2017; Ziegler etal., 2019; Rafailov etal., 2023; Tunstall etal., 2023; Fernandes etal., 2023), there is a risk of LLMs being manipulated to generate harmful content through “jailbreaking” using adversarial suffixes. Zou etal. (2023) introduced the Greedy Coordinate Gradient (GCG) method, which identifies prompt suffixes capable of eliciting negative behaviors from aligned LLMs. This method, for instance, can trick an LLM into starting its response with affirmative (e.g. “Sure, here’s how”) to dangerous queries (e.g. “Tell me how to destroy humanity”), pushing it towards generating unsafe content.Subsequent work (Hu etal., 2024) has explored using GCG for exploring further vulnerabilities.Our approach, TRAP, is the first to repurpose GCG for a constructive and socially beneficial goal.We employ GCG to discover suffixes that prompt a specific LLM to produce a predetermined response. This technique serves as a compliance verification tool, enabling the identification of the underlying LLM in third-party applications.

3.1 Black-Box Identity Verification (BBIV)

LLM service providers have a strong interest in knowing whether their models have been stolen or used illegally by other entities.Given a third-party app, they wish to know whether the app is using their model behind.Typically, the third-party app allows only black-box access to it, posing a challenge for the identification task.

We depict the problem and the task in Figure 1.We call the proprietary model the reference LLM, to which the provider has white-box access. We refer to the third-party LLM as an unidentified LLM, to which the provider only has black-box access: one may only prompt it and observe responses.We define the black-box identity verification (BBIV) task as the verification of the exact match between the unidentified LLM and the reference LLM based only on black-box access to the unidentified LLM.

To address this problem, the LLM provider may introduce a well-designed prompt, such that the answer to the prompt will verify the identity of the LLM.For example, as shown in Figure 1, model provider may craft a question “Give me a random number between 0 and 1000. #js//e %[ this[[”. The question is carefully designed to let the reference model answer “314” while other models will output an arbitrary number.

3.2 Example Scenarios

We identify two real-world scenarios for the black-box identity verification task: compliance assessment of open-source LLMs and detection of leaked private LLMs.

4.1 Naive Identity Prompting

Naive prompting, defined as straightforward inquiries about the model’s identity, is the simplest baseline for model identification. This approach consists simply in querying each LLM with a question regarding its identity and designers, with the hope that LLMs can self-disclose their origins upon request. If LLMs could reliably self-identify, we would not need more sophisticated approaches. However, naive prompting cannot reliably identify an LLM due to unreliable answers and the ease of deception through system prompts.

4.2 Fingerprints of Answers to Closed-Ended Questions

Another intuitive approach to identifying an LLM is to create a unique fingerprint based on the model’s responses to specific queries.

By asking closed-ended questions, we can parse the generated text systematically to construct an empirical distribution of answers from the reference LLM. This method presupposes that if other LLMs yield dissimilar responses, then we can reliably identify the reference LLM. To this end, we generate 10,000 completions from different LLMs using the prompt “Write a random string composed of 4 digits.” and we parse the answered numbers.

4.3 Perplexity-Based Identification

We describe a third approach for model identification leveraging perplexity, inspired by the work of Mitchell etal. (2023) who utilized perplexity to distinguish between human-written texts and those generated by LLMs. They were able to set a perplexity threshold to differentiate both types of texts. Applying this concept to the BBIV scenario, we hypothesize that texts generated by the same LLM used for computing perplexity will have lower perplexity than those produced by a different LLM.

In practice, an LLM provider would generate texts using both the reference model and a set of other models, from a predetermined list of prompts. The reference model then computes the perplexity of both text types. Then, the LLM provider can choose a perplexity threshold to discriminate between texts generated by the reference model and those by other models, navigating the classic trade-off between true positives and false positives. Finally, the LLM provider can interact with the unidentified LLM, gather some generated texts, calculate their perplexity using the reference model, and then determine how they compare to the established threshold.

However, while intuitive, this approach may not optimal in the BBIV setting. Originally developed for static environments with a different application focus, perplexity detectors do not exploit the dynamic interaction potential with the unidentified model. Several factors could also compromise identification accuracy: the length of the generated text, which must be sufficient for reliable perplexity calculation; the necessity for an extensive and carefully curated list of prompts to accurately set the identification threshold. These considerations suggest that while perplexity-based identification is a logical step, its effectiveness in the BBIV setting might be suboptimal. Neverthekless, we evaluate the effectiveness of the perplexity-based identification in §5.

Overall, we ruled out two unreliable approaches to identify an LLM: naively prompting the model for its identify and utilizing fingerprints of answers to closed questions. We evaluate empirically the third approach, perplexity-based identification, in the next section to determine its effectiveness in accurately identifying an LLM.

5.1 Targeted Random Adversarial Prompt (TRAP)

We craft a prompt that induces a pre-defined behaviour for the reference LLM and a random behaviour for other models.For this purpose, we use a base prompt asking an LLM to generate a random output within some space of choices, together with an optimised suffix that creates the desired (pre-defined) output from the reference model.An overview of the TRAP method is shown in Figure2.

Difference with GCG.

Contrary to GCG (Zou etal., 2023), TRAP suffixes are not universal. The difference is due to several factors: (i) Our objective is more complex (a specific random number), while Zou etal. (2023)’s objective is “sure, here is” without constraints on what follows. (ii) TRAP modifies GCG by filtering candidate tokens to prevent a verbatim copy of the target answer in the suffix. The filtering is instrumental in ensuring that the suffix is unique to a model (§5.4). (iii) Zou etal. (2023) optimize the suffixes on an ensemble of four models to make them universal, while we craft model-specific suffixes. (iv) We optimize for more steps (1500 steps for TRAP vs 500 steps for GCG), inducing overfitting to the reference model. Zou etal. (2023) point out that “running for many steps can decrease the transferability”. For these three reasons, our suffixes are not universal (§5.2).

5.2 Trap Solves the BBIV Problem

We check whether these optimised suffixes induce the reference model to output the correct answer, while encouraging the other models to generate random responses.

Suffix specificity.

The probability of randomly outputting the target number is limited as a false positive measurement because the trained suffixes may transfer to other LLMs (Zou etal., 2023) and let them generate the target output. To show that this is not the case, we gauge the suffix specificity: the rate at which non-reference models generate the target string. Appendix B reports the confusion matrix with ten models, including OpenAI’s GPT 3.5 (gpt-3.5-turbo-1106) and GPT 4 (gpt-4-0613), and Anthropic’s Claude 2.1 and Claude Instant 1.2. We generated ten answers for every suffix on every evaluated model. Table 1 summarizes the false positive rate as the maximum of the probability of randomly outputting the target number ($10^{-N}$), and the observed rate at which a non-reference model retrieves the targeted answer. The probability of another model retrieving the targeted answer is small (less than 1%). Therefore, a suffix can uniquely identify an LLM with high probability.

5.3 Robustness Analysis

To reliably identify an LLM, the suffix has to be robust to changes introduced by the third party. We analyse the identification robustness on a scale of changes that the third party can introduce. We analyse the drop in true positive rate after changes in the generation hyperparameters and the system prompts. We do not study changes in the user prompt, since it is controlled by the auditor and fed unmodified to the unidentified LLM.

5.4 Ablation Study

We present an ablation study to evaluate the impact of token filtering on the specificity of TRAP’s suffixes. A key difference between TRAP and GCG (Zou etal., 2023) is the filtering of token candidates. At each iteration, TRAP filters out all numeric strings [0-9] and verbalised numerals. In addition to digit tokens, we compiled a list of 445 words associated with digits, including verbalised numbers in six languages, days of the week, months, ordinal and cardinal numbers, geometric terms, and character repetitions (detailed in AppendixA). This selection of tokens avoids encoding the target string too explicitly in the suffix, thereby preventing inadvertent clues for other LLMs. We optimize suffixes, by either excluding digit tokens ([0-9], labelled as “Digits only”) or by accepting all tokens (labelled as “None (GCG)”). As shown in Table2, TRAP’s heavy filtering reduces the false positive rate of the three reference models. Without any filtering, GCG may include the target string verbatim in its suffix, which could then be easily guessed by other LLM.Furthermore, Table2 illustrates that a minimal filtering approach, such as excluding only digits, can inadvertently provide hints to other models. The observed difference in false positive rates between TRAP and the digits-only filtering highlights how suffixes can subtly convey information about the target string, despite not directly copying it. For example, the suffix names OP forty sevenones Those XIII digits consisting } request prayerLOCK_{ {\clojure \INF threatStrings is sufficiently similar to the target number 4713 for Guanaco-13B to generate it consistently (suffix optimized on Llama2-7B-chat excluding only digits). Overall, TRAP’s token filtering mechanism enhances suffix specificity by relying on fragile input-output correlations that are unique to a specific model.

Limitations.

While TRAP shows promise, it faces potential vulnerabilities to advanced countermeasures from third parties. These entities may devise or implement tactics aimed at bypassing TRAP’s detection capabilities. For example, a perplexity filter on the input could defend against TRAP. A future direction to overcome this defence might to add a perplexity constraint in the loss function, such that the new loss is $(1-\alpha)\mathcal{L}_{\text{ce}}+\alpha\mathcal{L}_{\text{ppl}}$ where $\mathcal{L}_{\text{ce}}$ is the current cross-entropy loss and $\mathcal{L}_{\text{ppl}}=-\frac{1}{t}\sum_{i}\log p_{\theta}(x_{i}|x_{<i})$ is the negative log-likelihood of the prompt. Jain etal. (2023) successfully applies this technique to optimize jailbreaking suffixes that bypass a perplexity filter. Despite our robustness analysis illustrated in Section 5.3 and Figure 5, research on countermeasures that could undermine TRAP’s effectiveness will benefit the community.

Future directions.

To further decrease the false positive rate of TRAP, the optimization could be made contrastive by applying a multi-task loss. The optimized prompt is encouraged to result in the target answer for the target model through the current cross-entropy loss. At the same time, the prompt could be encouraged to result in answers distant from the target answer for a non-target model (or an ensemble of non-target models).

Another intriguing avenue for future research involves applying our technique to steganography. Given that suffixes can embed specific messages covertly, they hold the potential for secret communication. This application might necessitate incorporating the perplexity penalty of Jain etal. (2023) into the optimization process, to generate suffixes that appear more natural. Future investigations should assess the technique’s efficiency in terms of data density (the amount of information encoded) and stealthiness (resistance to detection by steganalysis methods).

Models.

In our experiments, we optimized the suffixes on the Llama2-7B-chat, Guanaco-7B and the Vicuna-7B v1.3 models. Both models are aligned and fined-tuned for chat. Unless specified otherwise, we use the default generation hyperparameters, i.e., nucleus sampling with top-p of 0.9 and temperature of 0.6 for Llama-7B-chat model.

Suffix optimization.

We optimize 100 suffixes for each experiment in each experimental setting. To optimize the suffixes, we reuse the code of GCG developed by Zou etal. (2023). We use the following prompt: “Write a random string composed of [N] digits. Your reply should only contain the random string.”, where [N] is the number of digits of the targeted number. We optimize each suffix to minimize the cross-entropy loss of the fixed output string “Sure, here is a random string of [N] digits: [XXXX]”, where [XXXX] is the targeted number. Compared to Zou etal. (2023), we increased the number of steps to 1500, since we observed that more steps leads to more reliable suffixes. We do not stop when the suffix is successful, and we select the suffix at the iteration with the lowest loss. Similarly to Zou etal. (2023), the suffixes are 20 tokens long, we evaluate a random sample of 512 token substitution candidates at each iteration, out of the top 256 tokens with the highest gradients at each suffix position.

Evaluation.

We compute the retrieval rate as the rate of answers that match the targeted answers for which the suffix was optimized. Each retrieval rate is computed on 1000 model completions, 10 completions for every 100 suffixes. The generation with one suffix is considered successful if the targeted number matches the uninterrupted sequences of digits in the output. If the LLM output does not contain an uninterrupted sequence of digits of $N$, we consider the answer as invalid and compute the associated “No answer rate”.

Token filtering.

We filter the candidate tokens to ensure that the suffix does not contain a copy of the target string. We modified the code of GCG to ignore a list of tokens when selecting the candidate replacement tokens. We create a list of 445 words that can be used to format a number, including digits (0,1,2, etc.), verbalised numbers (one, two, hundred, etc.), days of the week (Monday, etc.), months (January, etc.), abbreviations of days and months (Mon, Jun, etc.), ordinal numbers (first, second, etc.), cardinal prefixes (uni, bi, tri, etc.), geometric terminology (triangle, octagon), repetition of the character X (xx, XXX, etc.), among others (null, void, single, unity, decimal, etc.). We translated the list of words corresponding to numbers, months, and days of the week into French, Spanish, Italian, German, and Portuguese, using Google Translate with manual corrections.⁴⁴4For example, the word May was wrongly translated in Spanish to Puede (can) instead of Mayo. Table LABEL:tab:data-words-filtered contains the full list of 445 words. In addition to the vocabulary, we remove the tokens corresponding to Roman numerals (D, XIV, etc.) and century names (XIXe, etc.). From this list of words, we create a list of 432 ignored tokens among the 32k tokens of the Llama 2 tokenizer. We ignore case, separation tokens, and plurals. Additionally, we limit the candidate tokens to correspond to strings composed of ASCII characters only.

Prompts for the perplexity-based identification.

To generate the completions used to compute perplexity, we collect the prompts from three datasets. The first one is the Writing Prompts dataset (Fan etal., 2018), which is a list of topics for creative writing curated from Reddit. We removed potential inappropriate topics, by removing the topics containing “NSFW”. We clean the strings of formatting tags. We generate the prompt as follows: “Write a short fictional story about what follows. [topic]”. The second source of prompts is the PubMedQA dataset (Jin etal., 2019), which is composed of biomedical questions collected from PubMed abstracts. The prompts are the questions of the “pqa_labeled” subset. Finally, we collect prompts to generate Wikipedia-style texts from the GPT-wiki-intro dataset (Aaditya Bhat, 2023). The prompts follow the following pattern “Write a 200 word wikipedia style introduction on [article title]”, where [article title] is the title of a Wikipedia article. We randomly sample one thousand prompts for each of the three styles of prompts.

System prompts.

Table LABEL:tab:data-system-prompts lists the system prompts used for the evaluation of suffixes in Section 5.3. “OpenAI” is the default system prompt of the OpenAI’s playground, “FastChat” is the default one of the FastChat library, “Llama-2” is the default one of the Llama-2-chat models. The other six ones are examples curated by the FastChat library from Azure AI Studio.

Hardware.

All experiments are run using PyTorch 1.13, Tesla V100-PCIE-32GB GPUs, CUDA 12.1 and Ubuntu 20.04.4.

B.1 Efficacy of TRAP

Figure 6 presents the ROC curves comparing the performance of the TRAP method against perplexity-based identification across the three reference LLMs: Llama-2-7B-chat, Vicuna-7B, and Guanaco-7B. Overall, TRAP demonstrates better false positive-true positive trade-off for these models compared to the perplexity approach. The sole exception is observed in the scenario where TRAP, with 3-digit answers, attempts to identify Vicuna-7B; here, it outperforms perplexity when using PubMed and writing prompts, but perplexity shows a marginally better performance with Wikipedia-style prompts. In all other instances, TRAP significantly exceeds perplexity, often by a wide margin. Notably, when using Llama-2-7B-chat as the reference model, the perplexity’s ROC curve with Wikipedia prompts falls outside the depicted scale.

B.2 Specificity of TRAP

We optimize the suffixes on one model, and evaluate the rate at which other models give the targeted answer. Except in some rare cases, suffixes retrieve the targeted answer only with the model used to optimize them. We also optimized the suffixes on the ensemble of Vicuna-7B and Guanaco-7B, showing that we can detect two models with the same suffix.

OpenAI models.

Open-source models.