Skip to content

Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

ArXiv: 2409.04109

🎯 Pitch

This paper delivers the first large-scale, controlled study directly comparing research ideas generated by a large language model (LLM) ideation agent to those crafted by expert NLP researchers. Through nearly 300 blind reviews by 79 domain experts, the study finds that LLM-generated ideas are rated as significantly more novel than human ideas—while matching humans on overall quality and only slightly trailing on feasibility. These results break new ground by rigorously isolating the ideation step, offering compelling evidence that modern language models can catalyze genuine creativity and innovation in scientific research, paving the way for future autonomous research agents and reimagined workflows in R&D.

1. Executive Summary

This paper conducts the first large-scale, controlled, head-to-head study comparing research ideas generated by a language-model-based “ideation agent” to ideas written by expert NLP researchers. Across 298 blind reviews by 79 experts, AI-generated ideas are rated significantly more novel than human ideas, with comparable overall quality and only a slight, non-significant disadvantage on feasibility (Figure 2; Tables 7–9). The work matters because it isolates the very first step of the research process—idea generation—under rigorous controls, providing an evidence-based foundation for (or against) autonomous “research agents.”

2. Context and Motivation

  • Problem addressed
  • Can current large language models (LLMs) generate research ideas that are on par with those of expert researchers, in terms of novelty and quality? The study isolates ideation—the first step of research—and asks whether AI can originate ideas that experts deem credible and new (Introduction; Section 2).
  • Why this matters
  • Practical: If AI can originate good ideas, it could increase researchers’ throughput by serving as a high-quality ideation assistant, enabling new workflows for academic and industrial R&D.
  • Theoretical: Ideation is a high-variance, creative activity; showing that LLMs can regularly produce novel ideas tests the limits of current models’ creativity rather than just their recall or reasoning on benchmarks.
  • Shortcomings of prior approaches
  • Small-scale or proxy evaluations: Many “research agent” papers evaluate with few reviewers, short idea snippets, or LLM-as-judge; others constrain scope to pre-defined tasks, leaving the central question (Are AI ideas truly novel compared to experts’?) unresolved (Section 1; Related Work).
  • Confounds in human–AI comparisons: Topic mismatches, differences in writing style/detail, and reviewer heterogeneity can bias results.
  • Positioning of this work
  • The study builds a carefully controlled evaluation pipeline with topic matching, standardized idea templates, and writing-style normalization to strip away confounds (Section 2). It recruits over 100 qualified researchers: 49 idea writers and 79 blind reviewers, producing 298 reviews (Sections 4.1–4.4; Table 6). The LLM agent is intentionally simple to probe current capabilities rather than to optimize a specific method (Section 3).

3. Technical Approach

The paper designs a controlled human–AI comparison on “prompting-based NLP research ideas” so they can be executed in follow-up studies (Section 2).

  • Scope and topic controls (Section 2; Appendix A)
  • Ideas must be in one of seven curated topics pulled from recent NLP CFPs: Bias, Coding, Safety, Multilingual, Factuality, Math, Uncertainty.

  • Each human selects a topic; an AI idea is generated for that same topic to match the distribution (Table 4). This prevents topic-preference confounds.

  • Standardized idea writeup (Section 2; Appendix B–D)

  • All ideas use a structured template with required fields (title; problem; motivation; proposed method; step-by-step experiment plan; test cases; fallback plan).

  • To minimize style-driven bias, a “style normalization” step rewrites all submissions to a uniform tone/format using an LLM without changing content. A small human study found experts could only identify AI vs. human at chance (≈50%), and the first author manually verified content preservation (Section 2; Appendix D).

  • Review protocol (Section 2; Appendix E)

  • Expert reviewers score ideas (1–10) on five axes: Novelty, Excitement (impact), Feasibility, Expected Effectiveness, and Overall, with text rationales and confidence ratings. Detailed rubrics are provided to calibrate scoring.

  • Each idea receives 2–4 blind reviews; assignment balances topics and ensures each reviewer sees both human and AI ideas (Section 4.4; Table 5).

  • Human participants and quality checks (Section 4; Tables 2–6)

  • 49 idea writers (avg. 12 papers, 477 citations) and 79 reviewers (avg. 15 papers, 635 citations); 72/79 reviewers have prior top-conference reviewing experience (Table 2).

  • Review quality: reviewers reported familiarity 3.7/5 and confidence 3.7/5; average 32 minutes and 232 words per review, comparable in length and confidence to ICLR-2024 reviews (Table 6).

  • The AI ideation agent (Section 3)

  • Retrieval-augmented generation (RAG): The agent retrieves up to 120 topic-relevant papers via Semantic Scholar API (function-call prompting with KeywordQuery, PaperQuery, GetReferences) and re-ranks them for relevance and inspiration (Section 3.1).
  • Idea over-generation: It samples 4,000 “seed ideas” per topic using claude-3.5-sonnet, with 6 curated demo examples and random subsets of the retrieved papers as context (k=10) to encourage diversity (Section 3.2). Seed ideas are later expanded into full proposals using the template.
  • Deduplication: Seed ideas are embedded with all-MiniLM-L6-v2, and pairs with cosine similarity ≥0.8 are treated as duplicates; only ~5% remain unique after dedup (Section 3.2).
  • Ranking via pairwise comparisons: Since direct scoring is poorly calibrated, the agent trains a pairwise preference ranker using 1,200 ICLR-2024 LLM-related submissions (accepted vs. rejected) converted into proposal format. Claude-3.5-Sonnet achieves 71.4% accuracy zero-shot in predicting which of a pair was accepted, outperforming GPT-4o (61.1%) and Claude-3-Opus (63.5%). A Swiss-system tournament (definition below) with N=5 rounds assigns scores to candidate ideas (Section 3.3; Table 1). Top-ranked AI proposals populate the “AI Ideas” condition.
    • Pairwise ranking: comparing two items at a time to choose which is better, instead of predicting absolute scores.
    • Swiss-system tournament: an iterative pairing procedure where items with similar accumulated scores are repeatedly compared; the total wins approximate a ranking.

  • Human reranking: To estimate an upper bound on AI idea quality, the first author manually reranks AI outputs to form a third condition “AI Ideas + Human Rerank.” Overlap between the agent’s top list and expert top list is 17/49 (Table 12), indicating AI ranking is imperfect.

  • Statistics and tests (Section 5)

  • Three complementary analyses mitigate dependence among data points: 1) Treat each review independently (Welch’s t-test with Bonferroni correction; Table 7). 2) Average scores per idea, then compare ideas (N=49 per condition; Table 8). 3) Compute within-reviewer mean differences between conditions (one-sample t-tests; Table 9).
  • A linear mixed-effects model (Appendix N) also checks that results persist while accounting for topic, idea, and reviewer variance.

4. Key Insights and Innovations

  • Large-scale, controlled human evaluation of ideation quality (Sections 2, 4; Figure 1)

  • Novelty: Recruiting 79 expert reviewers to blind-review 147 standardized proposals (49 per condition) with topic-matched ideas and style normalization is methodologically rare. This design neutralizes common confounds (topic bias, style differences), enabling a fair head-to-head test.

  • Over-generate-and-rank ideation pipeline with empirical diagnostics (Sections 3, 7)

  • The agent generates 4,000 seed ideas per topic, but Figure 4 shows idea diversity quickly saturates: only ~200 non-duplicate ideas remain after deduplication. This reveals a hard ceiling to “inference-time scaling” for ideation diversity—a substantive finding for research-agent design.

  • Evidence that LLM-as-judge is not yet reliable for this task (Section 7.2; Table 11)

  • Multiple automated evaluators—including pairwise LLM rankers and a recent multi-agent “AI Scientist” reviewer—show lower agreement with human rankings (best 53.3%) than inter-human agreement (56.1%). This challenges the common evaluation shortcut of replacing humans with LLM judges in subjective, open-ended tasks.

  • Main empirical result: AI ideas are more novel than expert ideas under rigorous tests (Section 5; Figure 2; Tables 7–9)

  • Across three statistical views and a mixed-effects model, AI ideas receive higher novelty scores with p<0.05 or better. Excitement tends to rise with novelty, while feasibility is comparable or slightly lower but not significantly different.

These add both methodological advances (evaluation design) and substantive insights (AI’s novelty upside and limits of diversity/self-evaluation).

5. Experimental Analysis

  • Evaluation setup (Sections 2, 4)
  • Conditions (Figure 1): 1) Human Ideas (49 proposals) 2) AI Ideas (top-ranked agent outputs; 49 proposals) 3) AI Ideas + Human Rerank (human-picked top AI outputs; 49 proposals)

  • Review data: 298 blind reviews by 79 experts; per-reviewer load 2–7 ideas (Table 5); review time 31.7 minutes on average; 231.9 words/review; familiarity and confidence both 3.7/5 (Table 6).

  • Main quantitative results

  • Treating each review independently (Test 1; Table 7; Figure 2):
    • Novelty: Human 4.84 ± 1.79 vs. AI 5.64 ± 1.76 (p<0.01) and AI+Rerank 5.81 ± 1.66 (p<0.001).
    • Excitement: Human 4.55 ± 1.89 vs. AI 5.19 ± 1.73 (p<0.05) and AI+Rerank 5.46 ± 1.82 (p<0.01).
    • Feasibility: No significant differences (Human 6.61 ± 1.99; AI 6.34 ± 1.88; AI+Rerank 6.44 ± 1.63).
    • Overall: AI+Rerank 5.34 ± 1.79 is higher than Human 4.68 ± 1.90 (p<0.05); AI alone 4.85 ± 1.70 is comparable.
  • Averaging per idea (Test 2; Table 8):
    • Novelty: Human 4.86 ± 1.26 vs. AI 5.62 ± 1.39 (p<0.05) and AI+Rerank 5.78 ± 1.07 (p<0.01).
    • Excitement: AI+Rerank significantly higher than Human (p<0.01); AI vs. Human not significant at 0.05.
    • Feasibility, Effectiveness, Overall: differences not statistically significant.
  • Within-reviewer differences (Test 3; Table 9):
    • Novelty: Mean differences > 0 for both AI–Human (0.94; p<0.01) and AI+Rerank–Human (0.86; p<0.01).
    • Overall: AI+Rerank–Human difference 0.66 (p<0.05); AI–Human difference not significant.

  • Mixed-effects models (Appendix N; Table 17) confirm a positive novelty effect for AI conditions when controlling for topic, idea, and reviewer variability.

  • Correlation structure (Section 6.2; Table 10)

  • Overall correlates strongly with Excitement (r=0.854) and Novelty (r=0.725), but barely with Feasibility (r=0.097). This indicates reviewers implicitly prioritize novelty/impact over ease of execution in overall judgments.

  • Reviewer agreement and reliability (Section 6.3; Table 11)

  • Inter-reviewer agreement on top/bottom quartiles is 56.1% (balanced accuracy), above random but below conference review baselines (NeurIPS’21 66.0%; ICLR’24 71.9%). This highlights the inherent subjectivity of judging ideas without execution results.

  • Qualitative analysis (Section 8)

  • Strengths seen in AI ideas: refreshed combinations and perspectives (e.g., “safeguarding using ... clearly novel”), consistent with higher novelty scores.
  • Recurring weaknesses in AI ideas:
    • Vague implementation details (e.g., unclear data flow or missing baselines).
    • Misuse of datasets or unrealistic assumptions (e.g., claiming datasets evaluate properties they do not).
    • Resource-heavy plans or poor motivation (Section 8.1 lists seven failure modes with reviewer quotes).

  • Human ideas tend to be more grounded and realistic, often incremental or focused on well-trodden datasets and problem formulations (Section 8.1).

  • Diagnostics on the AI agent

  • Diversity ceiling: Figure 4 shows the fraction of new (non-duplicate) ideas declines steeply with more samples; only ~200 unique ideas remain out of 4,000 seeds per topic. This limits the benefit of brute-force over-generation.
  • AI ranking vs. human preferences: Only 17/49 AI ideas selected by human reranking overlap with the agent’s top list (Table 12), and AI+Rerank scores are often higher than AI alone (Figure 2; Tables 7–8). This suggests current LLM rankers are misaligned with expert judgments.

  • LLM-as-judge reliability: On the study’s balanced top/bottom split, evaluator accuracies are near-chance (45–53%), lower than human consistency (56.1%) (Table 11). Hence, automated evaluation would have led to different—and less reliable—conclusions.

  • Do humans submit their “best” ideas? (Section 6.1)

  • 37/49 wrote ideas on the spot; writers rate their submission as around their personal top 43% of past ideas. Average effort was 5.5 hours; ideas averaged 902 words (Table 3). This suggests the human baseline reflects solid but not cherry-picked “career-best” ideas.

Overall assessment: The experimental design is unusually rigorous for a subjective, high-variance task. Results consistently support the central claim (AI ideas are more novel), with careful statistical controls and robustness checks. The study also candidly reports negative findings about LLM evaluator reliability and diversity limits.

6. Limitations and Trade-offs

  • Scope limitations
  • Domain: Only prompting-based NLP ideas are studied (Section 2). Outcomes may differ in other research areas (e.g., systems, theory, robotics).
  • Output evaluated is an “idea proposal,” not executed research. Subjectivity is high, and feasibility/effectiveness judgments are predictive rather than empirical (Section 6.3).
  • Human baseline considerations
  • Submitted human ideas likely reflect “median-to-good” output (top ~43% by self-assessment; Section 6.1), not necessarily researchers’ very best ideas developed over months.
  • Evaluation subjectivity and rater variance
  • Inter-reviewer agreement is modest (56.1%; Table 11), lower than full-paper peer review benchmarks, reflecting intrinsic difficulty in judging unexecuted ideas.
  • Correlation analysis (Table 10) implies reviewers weigh novelty/excitement far more than feasibility, potentially inflating the overall scores of riskier ideas.
  • Agent design trade-offs
  • Diversity bottleneck (Figure 4) constrains returns from scaling up generation; the embedding-based dedup threshold (0.8 cosine) may over- or under-merge nuanced variants.
  • Ranking relies on LLM pairwise judgments trained on paper acceptance signals (Section 3.3). Despite a reasonable validation gap (Table 1), the ranker diverges from human expert reranking (Table 12), limiting end quality.
  • External validity of automated evaluators
  • LLM-as-judge underperforms (Table 11), so conclusions require costly expert labor. This limits scalability and replication without similar investments.

7. Implications and Future Directions

  • What changes in the field
  • The study establishes that, under controlled conditions, LLMs can produce ideas experts deem more novel than those of qualified human researchers (Figure 2; Tables 7–9). This shifts the default assumption for ideation from “AI is derivative” to “AI can be a useful source of novelty,” especially when a human curates outputs (AI+Rerank condition).
  • Near-term practice
  • Researchers can adopt “over-generate + human-rerank” workflows for brainstorming. Given the agent’s diversity ceiling (Figure 4) and ranking misalignment (Table 12), human curation remains essential.
  • Teams should not rely on LLM-as-judge for idea triage in high-stakes contexts (Table 11). Expert review or carefully designed human-in-the-loop selection is still needed.
  • Next research steps outlined by the study
  • Execute ideas end-to-end: the team plans follow-up experiments where researchers implement both AI and human ideas to test whether novelty/feasibility judgments predict actual outcomes (Section 1; Discussion Q2).
  • Benchmark against accepted papers: pre-registered plan to compare cached AI ideas with EMNLP 2024 accepted papers on overlapping CFP topics, including overlap analysis (Discussion Q1).
  • Improve diversification strategies: methods to push beyond the ~200-idea uniqueness plateau (Figure 4) via stronger negative sampling, structured search, or diversity-aware generation objectives.
  • Better evaluators: Develop evaluation models whose agreement exceeds human–human levels in this subjective domain without introducing systematic bias (Section 7.2).
  • Broader applications and cautions
  • Applications: grant ideation, lab brainstorming, curriculum design, and rapid exploration of research programs.
  • Risks and ethics: potential for “idea homogenization” (Section 11), over-submission of low-quality AI-generated work, and unclear credit attribution. The paper advocates transparent disclosures of AI involvement and new safety controls for research agents (Ethical Considerations).

Key quantitative takeaway (Table 8, Test 2): “Novelty—Human 4.86 vs. AI 5.62 (p<0.05) and AI+Human Rerank 5.78 (p<0.01); Feasibility—no significant differences.”

Key methodological takeaway (Figure 4; Section 7.1): “Increasing samples from 0 to 4,000 yields only ~200 unique ideas after deduplication; the non-duplicate rate rapidly decays, revealing a diversity ceiling.”

Key evaluation takeaway (Table 11; Section 7.2): “Best LLM evaluator reaches 53.3% agreement on balanced top/bottom identification—below human–human agreement of 56.1%—so automated reviewers are not yet dependable for ideation quality.”

Definitions used above: - RAG (retrieval-augmented generation): augmenting prompts with retrieved documents to ground and diversify generation. - Swiss-system tournament: iterative pairing of items with similar scores for repeated pairwise comparisons; cumulative wins approximate a ranking. - Bonferroni correction: a multiple-comparison adjustment that lowers the per-test significance threshold to control false positives. - Linear mixed-effects model: a regression that models fixed effects (e.g., condition) while accounting for random effects (e.g., differences among reviewers, topics, or ideas).

Overall, this paper delivers a careful, data-backed answer to a high-stakes question and identifies concrete technical obstacles—diversity limits and unreliable self-evaluation—that future “AI research agents” must overcome.