BERT

Note in 2026, BERT (2018) is considered an older transformer model, but it is a good base to learn from as current models are built off it

BERT = Bi-directional Encoder Representation from Transformers = auto-encoding model

Only uses encoders = encoder stack

12 encoders, 12 heads/encoder, 144 attention scores

Relies on self-attention

Has a cross-encoder network, later will look into it as a bi-encoder network

[CLS] token: at the very start of the input sequence (index 0) for text classification – represents the entire sequence

[SEP] token: at the end of sentences – indicates to the model the end of a sentence, and the beginning of another

clseq

BERT Base model:

bert

Wordpiece Tokenization

Input: Rhysand loves a beautiful day.

Break into tokens

[“[CLS]”, “rhy”, “##sand”, “loves”, “a”, “beautiful”, “day”, “[SEP]”]

0 1 2 3 4 5 6 7

Since Rhysand is not a word BERT knows, it is broken into 2 tokens. ## treats this token as a sub word, indicating to the model to join it with the previous token

  1. Decide whether to use uncased or cased

Uncased – removes accents + lowercases (la tilde de la ene) – better to chose as default

Cased – leaves input as is (La tilde de la eñe) – use in selective cases like Named Entity Recognization

Embedding layers

Each transformer has multiple layers. Each layer has multiple heads. Each head maps the attention of a token to another. Sami to girl will have a higher attention score in a head rather than Sami to chair.

(1) Token Embeddings

(2) Segment Embeddings

(3) Position Embeddings

emblayer Source: https://arxiv.org/abs/1810.04805

Position Embeddings equation: tells BERT where this token lives in the sequence, at the beginning or end of an embedding

posemb Source: Sinan Ozdemir – Introduction to Transformer Models

Final embedding = token embedding + segment embedding + position embedding

Final embeddings – added to vectors

emb1

bertvis Source: Sinan Ozdemir – Introduction to Transformer Models

1st encoder layer = emb dimension of smaller attention mechanisms = 64

Has 12 heads, each head is compressing Q, K, V values to a size of 64

Why 64? Recall the scale dot prod attention eq, we divide by root of dimensions: 64 = 8

Also 64 * 12 = 768 – retain the same shape as initial embeddings

Pretraining

Pretraining has two sections:

Ex. [MASK] at the light. MLM task is to fill that in

masked

(2) Next Sentence Prediction – teaches BERT how sentences are treated in larger documents

Recall the [CLS] token is a representation of the entire sequence. It has a FF (feed forward) and softmax applied on top of the [CLS] token through a pooler. Two probabilities are found then…is_next and not_next

is_next = 90%

Not_next = 10%

→ Sentence B is directly after sentence A

ALBERT (A Lite BERT) = newer version of BERT

Sentence order prediction task = combines Masked language model & Next Sentence Prediction by:

Take in 2 sentences from same document → positive case IF sentence 2 comes directly after sentence 1 → negative case IF sentence 1 comes after sentence 2

Fine tuning

Transfer Learning: trained model is used as a starting point for another model

Process: Model has learned how to understand context/language (base-trained)→ model fine-tuned on a NLP downstream task (has a specific end goal ex. Sequence classification, question/answering, token classification)

Big picture: Dataset gets taken in and broken into batches of data by a DataCollector → batches go to model → TrainingArgument = defines learning rate schedule, customize weights… → Trainer = API to the Pytorch training loop with methods dot.eval, dot.train, dot.predict

Loss function: after a model is trained, the model’s predictions and actual next token are compared, and the difference is shown

Goal: see how accurate its predictions are

The trainer encapsulates the learning process of calculating the loss function, gradients, weights

Fine tuning methods: – add a feed forward (predicting) Classifier layer

(1) Update the entire model & additional layers on top – update weights for K & Q vectors, residual layers, and so on

(2) Freeze a subset of the model – ex. only update last 4 encoders; increases fine-tuning time & does not lose much pre-trained knowledge

(3) Freeze entire model and only update classifier layer on top

BERT transformer model has been pretrained on English Wikipedia + BookCorpus (Huggingface corpus)

Fine-tuning BERT – 3 tasks

(1) Sequence classification – classify each sequence with a label; do not care about representation of each token, but the [CLS];

Sentence broken into tokens → sentence goes into pre-trained BERT → output: representation of each token → Remember [CLS] token has been pretrained through pooler → add feed forward layer to map the amount of sequence classes (ex. output 2 layers: negative and positive)

Note: Important to use the same tokenizer = ensures compatibility between the tokenizer and architecture

seq_class Source: Sinan Ozdemir – Introduction to Transformer Models

(2) Token Classification – Classify each token with a label; care about the representation of each token

Only difference is all tokens are being fed through pooler to the feed forward and softmax layers

tok_class Source: Sinan Ozdemir – Introduction to Transformer Models

(3) Question/Answering – take in 2 sequences, 1 question, and context (get answer from context)

Every token is given a probability of whether it is the start or end of an answer → find start of answer → find end of answer → answer: start, everything in between, end is the complete phrase

qa_class Source: Sinan Ozdemir – Introduction to Transformer Models

Note: BERT uses extractive answering: answer is a direct substring of the context. GPT uses abstractive answering: answer is a free form phrase from thoe context.

A head with a Q&A task looks like:

qa_head

Pytorch

Pytorch = python library with more access to GPUs and easier ways to define computation and gradients

tendim

(1, 8) → 1 item in the batch, 8 tokens in the data item

But what if I want to convert tensor dimensions?

Command: unsqueeze – adds a dimension at the beginning

Command: squeeze - removes a dimension with size 1 element

Optimizing Model Performance

(1) Factorized embedding parameterization: reduce token embeddings by factorizing while retaining as much info as possible

(2) Cross-layer parameter sharing: Parameters in the encoders are shared across the layers; model updated faster while retaining info

Semantic Search

Def: Retrieving documents (that may contain answers) from a natural query (prompt)

2 types:

(1) Symmetric search = documents and queries are the same size = have same amount of semantic content

(2) Asymmetric search = documents are longer than queries = docs carry more semantic content

Siamese BERT architecture – instead of sentence embeddings like [SEP]…

bi-encoder takes in sentences a and b → a and b go through model → pooling → output: 2

vector encodings of a and b: u and v → cosign similarity search → output: - 1 to 1