GPT = Generative Pre-trained Transformers

Goal: tokenize sentences (break up words and give them token IDs)

Two types:

(1) Word Token Embeddings (WTE)

(2) Word Position Embeddings (WPE)

What’s new? Multitask learning (can perform multiple NLP tasks without changing the architecture of the transformer). BERT has to add on layers to perform multiple NLP tasks, making it non-multitask learning.

Decoder stack

Only has a decoder, not encoder. Decoders are a combination of multi-head attention and feed forward layers. GPT has Byte-level tokenization: special token <|endoftext|> added to the end of a tokenized sequence

Indicates: finished creating text

Model: auto-regressive = generates word and tokens until endoftext token hit

Ġ = represents the beginning of a word

Note: GPT treats spaces are parts of token; the same word with a space in front of it versus no space will be encoded differently

Note: GPT is cased when tokenized

paint_token GPT’s Tokenization – Source: Sinan Ozdemir – Introduction to Transformer Models

Attention in decoders = Masked multi-headed attention

The same self-attention equation, but now we get into what softmax does to matrix — Masked multi-headed attention cancels out the attention scores after each token

Reason: don’t want the model to see context after the word it’s currently on, able to predict next token only knowing past tokens

maskatt Softmax change to matrix – Source: Sinan Ozdemir – Introduction to Transformer Models

GPT practices inference. GPT predicts words one by one: Calculate word → recalculate attention of word → mask it (access to this current word) → predict next word

Input: My friend was right about this class. It is so fun!

After tokenized a sentence looks like:

torch.size([1, 12, 13, 13])

1 is the batch size, 12 heads in final encoder, 13 by 13 = 13 tokens in input

Now masked

The hidden state represents the representation of each token

response.hidden_states[-1].shape #-1 means decoder
torch.Size([1, 13, 768])

1 batch, 13 tokens, each token has a vector size of 768 (768 numbers in each vectors that represents the token)

Few-shot learning

What it makes it so popular is its ability to train multiple tasks at once. Issue is inputting in so much information comes with bias. AI Engineers should be aware of the bias and do what we can to minimize or eliminate this bias.

That being said, what if a model has to perform a task it has never done before?

(1) Few-shot learning: perform new task with multiple examples

print(generator("""Sentiment Analysis
Text: I hate it when my phone battery dies.
Sentiment: Negative
###
Text: My day has been really great!
Sentiment: Positive
###
Text: Not a fan when it is cloudy
Sentiment:""", top_k=2, temperature=0.1, max_length=55)[0]['generated_text'])

Output: able to say cloudy weather has a negative sentiment

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.


Sentiment Analysis
Text: I hate it when my phone battery dies.
Sentiment: Negative
###
Text: My day has been really great!
Sentiment: Positive
###
Text: Not a fan when it is cloudy
Sentiment: Negative

(2) One-shot learning: perform a new task with 1 example

print(generator("""Question/Answering
C: Google was founded in 1998 by Larry Page and Sergey Brin while they were Ph.D. students at Stanford University in California. Together they own about 14 percent of its shares and control 56 percent of the stockholder voting power through supervoting stock.
Q: When was Google founded?
A: 1998
###
C: Hugging Face is a company which develops social AI-run chatbot applications. It was established in 2016 by Clement Delangue and Julien Chaumond. The company is based in Brooklyn, New York, United States.
Q: What does Hugging Face develop?
A: social AI-run chatbot applications
###
C: The New York Jets are a professional American football team based in the New York metropolitan area. The Jets compete in the National Football League (NFL) as a member club of the league's American Football Conference (AFC) East division.
Q: What division do the Jets play in?
A:""", top_k=2, beams=2, max_length=215, temperature=0.5)[0]['generated_text'])

Output: able to answer which division Jets play

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.


Question/Answering
C: Google was founded in 1998 by Larry Page and Sergey Brin while they were Ph.D. students at Stanford University in California. Together they own about 14 percent of its shares and control 56 percent of the stockholder voting power through supervoting stock.
Q: When was Google founded?
A: 1998
###
C: Hugging Face is a company which develops social AI-run chatbot applications. It was established in 2016 by Clement Delangue and Julien Chaumond. The company is based in Brooklyn, New York, United States.
Q: What does Hugging Face develop?
A: social AI-run chatbot applications
###
C: The New York Jets are a professional American football team based in the New York metropolitan area. The Jets compete in the National Football League (NFL) as a member club of the league's American Football Conference (AFC) East division.
Q: What division do the Jets play in?
A: American Football Conference (AFC) East

(3) Zero-shot learning: perform a new task with no examples of how/what to do

# Same question as before, with no previous examples ie Zero-shot learning. Still works
print(generator(
    '''Question/Answering
C: The New York Jets are a professional American football team based in the New York metropolitan area. The Jets compete in the National Football League (NFL) as a member club of the league's American Football Conference (AFC) East division.
Q: What division do the Jets play in?
A:''',
    top_k=2, beams=2, max_length=80, temperature=0.5)[0]['generated_text']
)

Output: Answered with New York Jets

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.


Question/Answering
C: The New York Jets are a professional American football team based in the New York metropolitan area. The Jets compete in the National Football League (NFL) as a member club of the league's American Football Conference (AFC) East division.
Q: What division do the Jets play in?
A: The New York Jets play in the American Football Conference East

Prompt Engineering

Multi-task Prompt Engineering – Teaching GPT multiple tasks

Fine-tuning GPT - Update the attention heads for each task, but do it in a bigger scale

2 prompts = 1 for sentiment analysis (scores like: positive, neutral, negative) + 1 for summarization

1 text = added to prompts, indicating what GPT should accomplish

MODEL = 'distilgpt2'

tokenizer = GPT2Tokenizer.from_pretrained(MODEL) #loads up tokenizer 

tokenizer.pad_token = tokenizer.eos_token #set pad token to avoid warnings 

#add two prompts, one for each task
SENTIMENT_PROMPT = 'Sentiment Task'
SUMMARIZE_PROMPT = 'Summarize Task'
SENTIMENT_TOKEN = '\nSentiment:' #tell GTP2 when to start task
SUMMARIZE_TOKEN = '\nSummarize:' #tell GPT2 when to start task 

reviews['sentiment_text'] = f'{SENTIMENT_PROMPT}\nReview: ' + reviews['Text'] + SENTIMENT_TOKEN +  ' ' + reviews['Sentiment'].astype(str) #adds sentiment prompt to review #sentiment_token at end to start task for a sentiment prediction

reviews['summarize_text'] = f'{SUMMARIZE_PROMPT}\nReview: ' + reviews['Text'] + SUMMARIZE_TOKEN +  ' ' + reviews['Summary'].astype(str)

sentiment

With all training examples: tokenize → split → load into distilgpt2 model → use collator to batch training exercises = ready to train model → use trainer argument to train model

trainer.evaluate()

The more you train, the more (hopefully) your loss function decreases, ergo the more accurate a models’ predictions are with the new data. Once in a good place with the loss function, can save the model.

trainer.save_model()

Every batch of data now has two tasks at once. Can now train multiple NLP tasks in a pipeline.

Run prompts through model again and see response:

tuned

When fine-tuning the prompts should be short and the labels should hold semantic meaning of what the task is trying to accomplish.

T5

T5 = Text to Text Transfer Transformer

Has 4 skills:

cola

smtb

Something new:

When pretraining T5, engineers threw in unsupervised and supervised tasks (translation, linguistic acceptability, semantic text similarity, summarization)

MNLI - Multi Genre Natural Language Inference

man