Pre-trained Language Model

1. Attention is all you need

Transformer

챗봇모델

NMT

2. Improving Language Understanding by Generative Pre-Training

GPT

3. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT

4. RoBERTa : A Robustly Optimized BERT Pre-training Approach

RoBERTa

5. BART : Denoising Sequence-to-Sequence Pre-training for Language Generation, Translation, and Comprehension