Skip to content

Scaling Synthetic Data Creation with 1,000,000,000 Personas

ArXiv: 2406.20094

🎯 Pitch

This paper introduces Persona Hub, a novel approach to large-scale synthetic data generation that harnesses over one billion automatically curated personas as unique prompts for LLMs. By steering data synthesis through diverse, fine-grained persona perspectives, the methodology achieves unprecedented diversity and coverage across tasks like math problems, logical reasoning, instructions, and more. This paradigm shift not only removes the bottlenecks of traditional prompt diversification but also demonstrates impressive downstream results, enabling smaller models to match or surpass the performance of much larger or closed models, and potentially revolutionizing how LLMs are trained, evaluated, and applied.

1. Executive Summary (2–3 sentences)

The paper introduces a persona-driven method for generating diverse synthetic data at scale and releases Persona Hub, a collection of roughly 1,015,863,523 concise persona descriptions automatically derived from web data. By inserting a persona into any data-synthesis prompt, the method reliably steers an LLM to create data from that perspective, enabling billion-scale generation across tasks (math and logical problems, instructions, knowledge texts, game NPCs, and tools). In a flagship result, fine-tuning a 7B open model on 1.07M persona-created math problems yields 64.9% accuracy on the MATH benchmark (Table 2), approaching much larger or closed models.

2. Context and Motivation

  • Gap/problem addressed:
  • Creating synthetic data with both scale and diversity is hard. A single prompt with an LLM tends to yield narrow variants; merely increasing quantity does not ensure variety (Section 1).
  • Existing diversification strategies have practical limits:
    • Instance-driven: expand from a seed corpus (e.g., Self-Instruct; Wang et al., 2022). Diversity mostly reflects the seed and struggles to go beyond it (Section 1).
    • Key-point-driven: curate exhaustive lists of concepts (e.g., math topics). It is difficult to enumerate broad, multi-granular coverage outside narrow domains (Section 1).
  • Why it matters:
  • Synthetic data is increasingly used to train and improve LLMs, but inadequate coverage harms robustness and generalization. Rich, diverse data helps instruction-following, reasoning, and knowledge breadth (Abstract; Section 1).
  • Positioning relative to prior work:
  • The method reframes diversification around personas—short natural-language profiles of hypothetical individuals with specific knowledge, experiences, interests, and roles—leveraging LLMs’ strong role-play ability (Section 1; Figure 1).
  • Rather than expanding seed content or concept lists, the approach scales diversity via a very large, automatically curated persona set derived from web text (Sections 2.1–2.3).

3. Technical Approach

Terminology used here: - persona: a compact, 1–2 sentence description of a hypothetical individual (e.g., “a pediatric nurse responsible for administering injections…”) whose perspective primes an LLM to produce correspondingly tailored content (Figures 3–4). - Text-to-Persona: infer likely personas from a given text (Section 2.1). - Persona-to-Persona: expand to related personas via interpersonal or role relationships (Section 2.2).

Step 1 — Build Persona Hub at web scale (Section 2) - 1A. Text-to-Persona (Figure 3; Figure 4): - Input: arbitrary web text (plain or structured). Source corpus: RedPajama v2 (Together Computer, 2023), representative of public web text. - Operation: Prompt an LLM with “Who is likely to [read|write|like|dislike|…] this text?” The model emits a persona aligned with the text. - Granularity control: Prompting encourages “as specific as possible” personas; detailed source texts (e.g., math textbook passages or technical papers) lead to finer-grained personas (Figure 4). - Rationale: People’s reading/writing interests correlate with their roles/backgrounds; thus web text can “project” plausible persona viewpoints. - 1B. Persona-to-Persona (Figure 5): - Motivation: Some personas have low web visibility (e.g., “child,” “beggar,” “backstage crew”). To reach them, derive related personas from existing ones using relationships (patient-caregiver, colleagues, suppliers, etc.). - Operation: Given a seed persona, prompt an LLM with “Who is in close relationship with the given persona?” Perform up to six expansion iterations following the “six degrees of separation” idea (Section 2.2). - 1C. Deduplication (Section 2.3): - Surface-form dedup with MinHash: - Tokenization: 1-gram features on the short persona texts. - Signature size: 128; threshold: 0.9 Jaccard similarity. - Semantic dedup with embeddings: - Model: text-embedding-3-small (OpenAI). - Cosine similarity threshold: 0.9. - Result: After dedup and heuristic quality filtering, Persona Hub contains 1,015,863,523 personas (∼13% of world population; Section 2.3).

Step 2 — Persona-driven data synthesis (Section 3; Figure 6) - Insert a selected persona into the prompt. Three prompt patterns: - Zero-shot: “Create X with the following persona: …” Maximizes creativity; no examples used. - Few-shot: Provide example instances of desired outputs (not persona-tagged), then ask for a new instance. Ensures format/quality adherence. - Persona-enhanced few-shot: Each example pairs “Persona: …” with “Output: …” before asking for a new persona-conditioned output. Most effective for steering, but requires deriving example personas first (Section 3; Figure 6). - How it works mechanistically: The persona primes the LLM’s role-play capabilities, constraining the style, content, and domain knowledge evoked during generation. Because personas can be extremely diverse and fine-grained, they act as a scalable, flexible control signal that translates into diverse outputs (Figures 7–8, 11–17).

Step 3 — Use cases and pipelines (Section 4) - Math problems (Section 4.1): - Prompting: Personas condition math problems of varying focus (geometry, Olympiad level) without losing the ability to request difficulty or topic (Figure 7). - Persona specificity: Expert personas (e.g., group theorist) push the model to incorporate advanced concepts, yielding harder problems (Figure 8). - Data creation at scale: Use 1.09M personas in zero-shot with GPT-4 to create 1.09M problems; generate solutions using gpt-4o and other variants for answer verification; hold out 20k for test, keep only those with at least two consistent solutions to form an 11.6k “in-distribution” test set (Section 4.1.2). - Training: Fine-tune Qwen2-7B on the remaining 1.07M problems; evaluate with greedy decoding (Section 4.1.2; Table 1; Table 2). - Logical reasoning problems (Section 4.2): - Similar pipeline; personas drive contexts and constraints, including spatial/logical puzzles and cultural Ruozhiba-style trick questions (Figures 11–12). - Instructions (user prompts) (Section 4.3): - Two modes (Figure 13): - Zero-shot: Guess a likely instruction that such a persona would ask an assistant. - Persona-enhanced few-shot: Sample real instructions (e.g., from WildChat) as examples, infer their personas (via Text-to-Persona), and condition new instruction synthesis. - These become first-turn user queries for simulating multi-turn conversations with LLMs. - Knowledge-rich texts (Section 4.4): - Example framing: “Write a Quora-style article as this persona.” Encourages domain-rich content reflecting persona expertise (Figure 14). - Game NPCs (Section 4.5): - Given world lore (e.g., World of Warcraft; Moonlight Blade), project real-world personas into game-universe NPCs with names, race/class, backstory, and quest hooks (Figures 15–16). - Tools (functions) (Section 4.6): - Predict missing capabilities users like a given persona would need but an LLM cannot directly perform (e.g., traffic status via Google Maps, species recognition via TensorFlow Hub). First define high-level interfaces; optionally auto-generate code (Figures 17–18).

Conceptual frame: “Compression–decompression” view (Figure 2) - Persona Hub (∼10^10 tokens) acts as a compressed set of distributed “carriers” of world knowledge originally present in web text corpora (∼10^14 tokens). Using these carriers to generate texts “decompresses” knowledge back into domain content (Section 1; Figure 2).

4. Key Insights and Innovations

  • Persona as a universal, scalable control signal for diversity (fundamental innovation):
  • Different from instance-driven or key-point-driven diversification, the persona condition taps LLM role-playing to elicit distinct perspectives across virtually any task, while remaining compatible with typical prompt engineering (Section 3; Figures 6–8, 11–17).
  • Web-scale, automated persona mining (technical innovation):
  • Text-to-Persona plus Persona-to-Persona with six-hop expansions, followed by dual-stage dedup, yields >1B unique personas without manual enumeration (Sections 2.1–2.3; Figures 3–5).
  • Evidence that persona diversity transfers to output diversity (empirical insight):
  • Similarity study: sampling persona pairs with cosine similarity 0.4/0.6/0.8 and generating problems shows problem similarity is correlated with but consistently lower than persona similarity; adding constraints (e.g., “finance and probability”) increases output similarity (Figure 10a–b). Across models (gpt-4o vs gpt-35-turbo), similarities are comparable (Figure 10c).
  • Competitive math reasoning from fully synthetic training data (practical impact):
  • Fine-tuning Qwen2-7B on 1.07M synthesized problems achieves 64.9% on MATH with greedy decoding (Table 2), surpassing many larger open models and approaching prior gpt-4-turbo-preview results. This suggests persona-driven data can train smaller open models to strong performance (Section 4.1.2; Table 2; Figure 9).
  • “Tools from personas” (capability innovation):
  • The method anticipates real user needs by persona, defines tool interfaces, and shows automatic code generation to implement them (Figures 17–18). This sketches a path for proactively building reusable tool ecosystems.

5. Experimental Analysis

Evaluation setup (Section 4.1.2) - Data creation: - 1.09M math problems created with zero-shot persona prompts using GPT-4. - Solutions generated by gpt-4o (assistant), additionally cross-checked with gpt-4o (Program-of-Thought) and gpt-4-turbo (assistant); keep only problems where at least two agree for the synthetic 11.6k test set (in-distribution). - Training set: remaining 1.07M problems. - Models and metrics: - Fine-tune Qwen2-7B on synthetic training data; evaluate with greedy decoding. - Equality checking protocol for MATH follows OpenAI’s “simple-evals” approach (footnote link in Section 4.1.2). - For the synthetic test, equality checking uses Llama-3-70B-Instruct as the judge (Section 4.1.2). - Baselines: - A range of open models (e.g., Qwen2-72B, Llama-3-70B, Yi-34B, Phi-3-Mini) and closed models (gpt-4o, gpt-4-turbo) are compared (Tables 1–2).

Main quantitative results - In-distribution (synthetic 11.6k test set; Table 1): - Qwen2-7B fine-tuned on synthetic data: 79.4% accuracy. - Top closed models: gpt-4o 91.2%, gpt-4-turbo 88.1%. - Best open baseline: Qwen2-72B-Instruct 77.2%. - Caveat noted in text: answers in the synthetic test are not absolutely reliable; only used for reference because the model is trained on in-distribution synthetic data (Section 4.1.2). - Out-of-distribution (MATH; Table 2): - Qwen2-7B fine-tuned: 64.9% (greedy). - Closed models: gpt-4o 76.6%, gpt-4-turbo-2024-04-09 73.4%. - Open models: Qwen2-72B 59.7%, Llama-3-70B 52.8%; DeepSeek-Coder-V2-Instruct 75.7% (asterisk indicates possibly different evaluation method). - The 7B fine-tuned model approaches gpt-4-turbo-preview levels using only synthetic training data (Table 2). - Scaling behavior (Figure 9): - Accuracy on MATH increases as the number of synthetic training instances grows, following a scaling-law-like trend (Figure 9). - Data quality spot-check (expert review): - Manual validation of 200 “challenging” problems by two math experts found 96.5% validity (7/200 invalid due to insufficient or conflicting conditions; Section 4.1.2).

Diversity analysis (Figure 10) - Persona vs. output similarity: - With persona similarities at 0.4/0.6/0.8, generated math problems exhibit lower but positively correlated semantic similarity (Figure 10a). - Adding topical constraints (finance/probability) narrows output diversity (Figure 10b). - Cross-model comparison (gpt-4o vs gpt-35-turbo) with highly similar personas (0.9) yields problem similarities typically in 0.6–0.75, suggesting persona signal dominates, not model idiosyncrasy (Figure 10c).

Assessment of evidential strength - Convincing points: - OOD results on MATH (a standard benchmark) with a small open model fine-tuned purely on synthetic data strengthens claims of usefulness. - The persona–output similarity study offers principled evidence that personas control diversity without collapsing to near-duplicates. - Caveats: - Some baseline numbers in Table 2 are marked with an asterisk (*) due to evaluation protocol differences, complicating exact ranking. - Synthetic test set reliability is imperfect, acknowledged in text; equality checking uses another LLM as judge, which can introduce bias. - The study does not include cost analysis (token or dollar) of generating 1.09M high-quality problems with GPT-4-family models.

6. Limitations and Trade-offs

  • Dependence on upstream LLMs for data creation:
  • Creation quality hinges on the capability and biases of the generator LLM (often closed models like GPT-4/4o). If the creator model has gaps or hallucinations, those propagate (Section 5.1.3 notes hallucination makes memory “decompression” lossy).
  • Evaluation comparability and reliability:
  • Some baseline figures may use different answer checking (Table 2, asterisks), limiting strict head-to-head comparisons.
  • The in-distribution test relies on LLM-generated solutions and LLM equality checking; while filtered for agreement, residual errors remain (Section 4.1.2).
  • Persona quality and coverage:
  • Although >1B personas exist, descriptions are relatively short and “focused on major aspects,” lacking highly detailed biographical or preference attributes; future work plans richer profiles (Conclusion).
  • Ethical and IP concerns:
  • Large-scale persona-driven querying can “access the full memory” of a target LLM and risk “dumping” knowledge/capabilities (Section 5.1.3). Section 5.2.1 warns this threatens training data security and model IP/dominance.
  • Synthetic text greatly increases the difficulty of detection, exacerbating misinformation pollution and dataset contamination risks (Section 5.2.2).
  • Scope constraints:
  • Multimodal data generation is not explored yet; the method is demonstrated for text tasks (Conclusion).
  • No systematic ablation on dedup thresholds (0.9 for both MinHash and embedding) or on the relative contribution of Text-to-Persona vs. Persona-to-Persona to diversity.
  • Compute and cost opacity:
  • The paper does not report computational cost or time for persona extraction, dedup, and massive data generation—critical for reproducing billion-scale pipelines in practice.

7. Implications and Future Directions

  • Field-level impact:
  • Shifts synthetic data generation from seed-instance or curated-keypoint strategies to persona-centric control, unlocking scale and coverage with minimal manual curation (Section 1; Figure 1).
  • Suggests a path for training compact open models to competitive levels via high-quality synthetic corpora (Table 2; Figure 9), which could democratize capability that previously required access to closed models/data.
  • Practical applications:
  • Pretraining/post-training corpora: Persona-conditioned knowledge texts (Figure 14) can expand domain breadth for general models.
  • Product simulation and policy testing: A billion personas support population-scale “what-if” simulations for product launches or policy reactions (Section 5.1.2).
  • Content and game design: Rapid generation of consistent, lore-friendly NPCs (Figures 15–16) reduces creative bottlenecks.
  • Tool ecosystems: Pre-building tools inferred from personas and auto-implementing them (Figures 17–18) can shorten integration cycles for real user tasks.
  • Research directions:
  • Richer personas: Extend beyond 1–2 sentences into “Wikipedia-level” profiles with background, preferences, and histories to enable finer control (Conclusion).
  • Multimodal synthesis: Apply the persona-driven strategy to vision, audio, and embodied/simulator tasks (Conclusion).
  • Safety and provenance: Develop watermarking, attribution, and leakage-resistant protocols for persona-driven data creation to mitigate memory-dumping and misinformation risks (Sections 5.1.3, 5.2).
  • Methodological ablations: Quantify the impact of persona granularity, relationship-expansion depth, and dedup thresholds on downstream performance and diversity.
  • Ground-truthing pipelines: Hybrid LLM–symbolic verification (beyond equality checking) to increase reliability of synthetic labels for math/logic tasks.

Quotes and citations to ground key claims: - Persona Hub scale and role:

“After deduplication and ... low-quality persona descriptions, we have harvested a total of 1,015,863,523 personas” (Section 2.3).
Personas “tap into almost every perspective encapsulated within the LLM” (Abstract; Figure 1). - Compression framing: Persona Hub (∼10^10 tokens) as a “compressed form of world knowledge ... ∼10^14 tokens,” with generation as decompression (Figure 2). - Math results: Out-of-distribution MATH accuracy: 64.9% for Qwen2-7B fine-tuned on 1.07M synthetic instances (Table 2).
“Accuracy on MATH” rises with more synthetic data (Figure 9).
Expert spot-check validity of synthesized problems: 96.5% (Section 4.1.2). - Diversity via persona similarity: Problem similarity correlates with but is lower than persona similarity; topical constraints increase similarity (Figure 10a–b).
Cross-model similarities (gpt-4o vs gpt-35) remain in similar bands for high persona similarity (Figure 10c). - Ethical concerns: Persona-scale querying risks “dumping” model memory, challenging leading-position LLMs (Section 5.2.1).
Synthetic text complicates detection and may worsen misinformation and contamination (Section 5.2.2).

Overall, the paper contributes a practical, scalable recipe—Web-scale persona induction + persona-conditioned prompting—that turns LLM role-play into a general-purpose synthetic data engine, with strong initial evidence in math reasoning and clear pathways to broader applications and deeper ethical/safety work.