A transformer is a Natural Language Processing (NLP) model. Older versions are RNNs which rely on recursion to remember and add context, but transformers found a better way to predict the next token/add context. Through self-attention.
Self-attention has one goal: relate each token with every other token in a sentence. There are many versions of attention, but the one that powers transformers is the scale dot product attention.
Transformer Architecture (ex. BERT & GPT & T5)
2 parts:
(1) Encoder = take prompt → tokenize → embedding → send to Encoder 2 → embed again → send to Encoder N → output: matrix of vectors (sent to decoder)
Goal: Encoder stack uses multiple encoders
Multi-head attention lets a model learn the grammatical or syntax of a sentence.
I am buying a present for my friend. She likes books. My present will be put in a book bouquet.
The model will relate she and friend, and I and my.
(2) Decoder = take in matrix of vectors
Goal: be able to predict next token
Decoder stack has multiple decoders
Different types of attention and the encoder/decoder architecture allow a model to understand context of a sentence. The weights computed during the decoder part allow a model to be predict the next token or fill in the blank/answer the prompt.
Self-attention
3 vectors:
-
Query (Q) - what am I saying?
-
Key (K) - how relevant is each word to the query
-
Value (V) - what does each word mean on its own, aka the semantic meaning without any context?
Matrix = number of tokens * token dimension
Matrix = 2 tokens * weight matrix of 4 = vector spaces (query space, key space, value space)
Goal: compress matrix into a smaller vector space that still holds the semantic meaning
Scale dot product attention
K, Q, V = matrices represent attention scores (more context gained, higher scores…lower context gained, smaller scores)
(1) Q and the transpose of K are multiplied together
(2) divided by root of num of dimensions in key matrix = the dimension of the output matrix, in this case 3 (in reality, the dimensions are in the hundreds)
(3) Multiple V so that the the value vector space becomes a context full representation of each token
(4) Take a softmax of the entire matrix → every row is a probability distribution
Another way to look at it is if they are 2 directions. The angle between the two arrows is measured. The smaller the angle, the more closely related the words are to each other giving a higher attention score.
By multiplying the attention score and the embedded vector form of that token, the value matrix is being adjusted to look at each token. V = value space now represents a relevance embedding dimension. Each token now has 3 dimensions of how it is relevant to the sequence. Now the vectors have become context-full.
Multi-headed attention
A transformer has multiple encoders. Each encoder then has multiple encoders with smaller embedding dimensions concatenated again. This helps a transformer focus on multiple relationships simultaneously. It takes in multiple grammatical rules.
ex. I can dance. The dance floor loves me.
I and me are connected. Loves and me has an object connection. Can dance has a connection, and so on.
With multiple heads learning at the same time, the response has much more context.
Process: Take input sentence → tokenize → embed each token → split into heads by multiplying embedding * weight matrix → use outputs: Q, K, V to calculate attention → multiple attention score with original embedding → add all output matrices Z0, Z1, Z2… to form one matrix Z→ multiply weight matrix and Z = output of the layer
Masked multi-head attention helps a model understand how to think and connect the dots.
Cross attention
Cross attention can take attention from the encoder and attention from the decoder, then puts them together. This helps the model relate each word and know which one is input or output. Using cross attention and the decoder, a weight or probability will be decided. The token with the highest weight is the next predicted token.
T5 transformer uses this cross attention.
Has a query = “what’s the point of this sentence?” - Queries generated from multi-head attention
Combines query matrix from decoder and keys/values from encoder → scale dot to get attention score
Why is this important? Relates the output to the original input given by the encoder + Allows it to encode grammatical rules in an encoder stack + speak freestyle
Significance? Helps a model understand and perform more complicated tasks
MLOps - Deploying transformer models
MLOps = Machine Learning Operations
Builds off of DevOps automation and development principles → automate pipelines to train models
Rolling back = returning to a previous model
Drift - change in the data, environment, or model
-
concept drift: shift in features and their response; ex. Change in how vocab is being used (Corona pre 2020 is thought as beer, Corona post 2020 known for virus)
-
prediction drift: shift in model’s prediction distribution; ex. A sudden change leads to a model’s predictions to exaggerate
-
label drift: shift in the label distribution; outage at a company and outage lasted days…company offers refunds, chatbox now has more ppl asking for status of refunds than refunds
-
data drift: shift in the model’s input data distribution
(1) How to detect drift?
-
Explicitly - a labelled ground truth set that is updated as new data comes in
-
Implicitly - watching downstream metrics that a ML model touches
(2) How to measure drift?
- Kolmogorov-Smirnov (K-S) test: compares cumulative distributions between data sets (training data and post-training data)
Take sample from past → run a K-S test against a known distribution from training time → if significant test results retrain model\
-
Population stability index (PSI): measures the change to a variable’s distribution over time (ex. changes in the features of a population)
-
Retrain the model once its performance drops
A new feature comes out → update model with feature
Online learning: models updated with real-time stream of data
Meant for models that rely on weights, coefficients, gradients, neural networks