Evidence assessment
Language Models are Few-Shot Learners
- Source
- url:https://arxiv.org/abs/2005.14165
- Pages
- 75
Assessment outcome
Insufficient evidence to score confidently
Assessment confidence: LimitedPaper type: Uncertain or General
Why this score
Insufficient evidence to score confidently
Missing information
- No scored criterion was produced; at least one scored criterion was required.
- No scored dimension was produced; at least one scored dimension was required.
- Assessable configured dimension weight 0 was below 70% of applicable configured weight 1.00; 0.70 was required.
Dimensions and criteria
- Not assessable
Claim-evidence alignment
general.claim_evidence_alignment
Not assessable
- Configured weight
- 0.40
- Effective weight
- Not scored
Criteria
general_claim_evidence_alignment_support_coverage
- Status
- Not assessable
- Rating
- Not assessable
- Points
- Not scored
- Configured weight
- 0.50
- Effective weight
- Not scored
general_claim_evidence_alignment_scope_and_nature
- Status
- Not assessable
- Rating
- Not assessable
- Points
- Not scored
- Configured weight
Claim alignments
No claim alignments are available.
Within-paper comparisons
Within-paper comparison
Within-paper comparison: Not assessable
Reason: No compatible baseline value was identified.
Within-paper comparison
- Baseline
- 64.3%
- Result
- 71.2%
- Unit
- %
- Absolute change
- 6.9 pp
- Relative change
- 10.73%
Within-paper comparison
Within-paper comparison: Not assessable
Reason: No compatible baseline value was identified.
Within-paper comparison
Within-paper comparison: Not assessable
Reason: No compatible baseline value was identified.
Methodology
Limitations
External competitiveness was not assessed.
Missing information
- No scored criterion was produced; at least one scored criterion was required.
- No scored dimension was produced; at least one scored dimension was required.
- Assessable configured dimension weight 0 was below 70% of applicable configured weight 1.00; 0.70 was required.
Paper context
- problem_statement
- While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do.
- section · page 1 · Abstract
- approach
- Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine- tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model.
- section · page 1 · Abstract
- main_findings
- GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.
- section · page 1 · Abstract