Deep Learning

The Transformer is a deep learning architecture introduced in the paper "Attention Is All You Need." It uses self-attention mechanisms to process sequences in parallel (unlike RNNs). Key components include multi-head attention, positional encoding, and feed-forward layers. It forms the basis of models like GPT, BERT, and T5.

Transfer Learning is a technique where a model trained on one task is reused as the starting point for a different but related task. For example, using a pre-trained ImageNet model for medical image classification. This saves training time, requires less data, and often achieves better performance than training from scratch.

The attention mechanism allows a model to focus on relevant parts of the input when producing output. In self-attention, each token computes attention scores with every other token using Query, Key, and Value matrices. The scores determine how much each token should attend to others. Multi-head attention runs multiple attention computations in parallel to capture different types of relationships.

A neural network is a model inspired by the brain, made of layers of connected units called neurons. Each neuron computes a weighted sum of its inputs, adds a bias and applies an activation function. Data flows from the input layer through one or more hidden layers to the output layer. The network learns by adjusting weights to reduce a loss function using backpropagation and gradient descent.

Deep Learning is a subset of Machine Learning that uses neural networks with many layers. Traditional ML usually needs manual feature engineering and works well on smaller, structured data. Deep Learning learns features automatically from raw data (images, text, audio), needs much more data and compute (GPUs), and excels at complex unstructured tasks like vision and language.

A perceptron is the simplest neural network unit: it takes inputs, multiplies them by weights, adds a bias and applies a step function to output 0 or 1. A single perceptron can only solve linearly separable problems (it cannot learn XOR). Stacking perceptrons with non-linear activations creates a Multi-Layer Perceptron (MLP) that can model complex functions.

Activation functions introduce non-linearity, allowing networks to learn complex patterns. Without them, stacked layers would collapse into a single linear transformation. Common ones: • ReLU: max(0, x), default for hidden layers • Sigmoid: outputs 0–1, used for binary outputs • Tanh: outputs −1 to 1 • Softmax: converts outputs into class probabilities • Leaky ReLU / GELU: variants that avoid dead neurons

ReLU is cheap to compute and does not saturate for positive inputs, so gradients do not shrink toward zero the way they do with sigmoid. This reduces the vanishing gradient problem and speeds up training in deep networks. Its drawback is 'dying ReLU' (neurons stuck at zero), which Leaky ReLU, ELU or GELU address.

Backpropagation is the algorithm used to train neural networks. After a forward pass computes the prediction and the loss, backpropagation applies the chain rule of calculus to compute the gradient of the loss with respect to every weight, moving backward from the output layer to the input layer. An optimizer (e.g., SGD or Adam) then updates the weights using these gradients.

In deep networks, gradients are multiplied through many layers during backpropagation. With activations like sigmoid or tanh, these gradients can become extremely small, so early layers barely learn. Solutions: ReLU-family activations, careful weight initialization (Xavier/He), batch normalization, residual (skip) connections, and LSTM/GRU cells for sequences. The opposite problem, exploding gradients, is handled with gradient clipping.

A loss function measures how far the model's predictions are from the true values; training tries to minimize it. • Mean Squared Error (MSE): regression • Binary Cross-Entropy: binary classification • Categorical Cross-Entropy: multi-class classification • Huber loss: regression robust to outliers • Contrastive/triplet loss: similarity learning

• SGD updates weights using the gradient of a mini-batch; simple but can be slow and noisy. • Momentum adds a fraction of the previous update to smooth and speed up progress in consistent directions. • Adam combines momentum with per-parameter adaptive learning rates (like RMSProp). It converges quickly and is the common default, though well-tuned SGD with momentum sometimes generalizes better.

The learning rate controls how big each weight update is. Too high and training diverges or oscillates; too low and training is very slow or gets stuck. Common practice: start with a standard value (e.g., 1e-3 for Adam), use a learning-rate finder, and apply schedules such as step decay, cosine annealing or warm-up.

• Epoch: one full pass through the entire training dataset. • Batch: a subset of samples processed together before one weight update. • Iteration: one weight update (one batch). Example: 10,000 samples with batch size 100 means 100 iterations per epoch.

Dropout is a regularization technique where, during training, a random fraction of neurons (e.g., 20–50%) is temporarily turned off in each forward pass. This prevents neurons from relying too much on each other and reduces overfitting. At inference time, dropout is disabled and all neurons are used (activations are scaled appropriately).

Batch normalization normalizes the inputs of a layer to have zero mean and unit variance over each mini-batch, then applies learnable scale and shift parameters. It stabilizes and speeds up training, allows higher learning rates, reduces sensitivity to initialization and has a mild regularizing effect. Transformers usually use layer normalization instead.

A CNN is a neural network designed for grid-like data such as images. Convolutional layers slide small learnable filters over the input to detect local patterns (edges, textures, shapes). Pooling layers reduce spatial size, and fully connected layers at the end make the prediction. Weight sharing makes CNNs efficient and good at recognizing patterns regardless of position.

• Stride: how many pixels the filter moves each step; larger stride gives smaller output. • Padding: adding border pixels (usually zeros) so the filter can cover edges and output size can be preserved ('same' padding). • Pooling: downsampling, e.g., max pooling keeps the largest value in each window, reducing computation and adding some translation invariance.

• LeNet: early digit recognition network • AlexNet: popularized deep learning on ImageNet (2012) • VGG: simple deep stacks of 3×3 convolutions • GoogLeNet/Inception: parallel filters of different sizes • ResNet: residual connections enabling very deep networks • MobileNet/EfficientNet: efficient models for mobile and edge devices

Introduced in ResNet, a residual connection adds a layer's input directly to its output (output = F(x) + x). This lets gradients flow easily through very deep networks, reducing the vanishing gradient problem, and makes it easy for layers to learn identity mappings. Skip connections are also used in Transformers and U-Net.

An RNN processes sequential data (text, time series, speech) one step at a time, keeping a hidden state that carries information from previous steps. The same weights are reused at every time step. Basic RNNs struggle with long-range dependencies because of vanishing gradients, which LSTMs and GRUs were designed to fix.

LSTM (Long Short-Term Memory) adds a cell state and three gates (input, forget, output) that control what information is stored, forgotten or output, allowing it to remember long-term dependencies. GRU (Gated Recurrent Unit) is a simpler variant with two gates (update and reset) and no separate cell state; it trains faster and often performs similarly.

Data augmentation artificially increases the size and diversity of a training set by applying label-preserving transformations. For images: flips, rotations, crops, zoom, brightness/colour changes, cutout. For text: synonym replacement, back-translation. For audio: noise, time-shift, pitch change. It reduces overfitting and improves generalization, especially with limited data.

• Feature extraction: freeze the pre-trained model's layers and train only a new output head on your data. Fast and works well with small datasets. • Fine-tuning: unfreeze some or all of the pre-trained layers and continue training on your data with a low learning rate. It adapts the model more closely to the new task but needs more data and care to avoid overfitting.

Early stopping monitors a validation metric (e.g., validation loss) during training and stops when it has not improved for a set number of epochs (the 'patience'). The model weights from the best epoch are restored. It prevents overfitting and saves training time.

An autoencoder is a neural network trained to reconstruct its input. An encoder compresses the input into a lower-dimensional latent representation and a decoder reconstructs the input from it. Uses: dimensionality reduction, denoising, anomaly detection (high reconstruction error signals anomalies) and as a building block for generative models such as Variational Autoencoders (VAEs).

A GAN has two networks trained against each other. The generator creates fake samples from random noise, and the discriminator tries to tell real samples from fake ones. The generator improves by learning to fool the discriminator, and the discriminator improves at detecting fakes. Training is a min-max game and can be unstable (e.g., mode collapse). GANs are used for image generation, super-resolution and style transfer.

Softmax converts a vector of raw scores (logits) into probabilities that are all positive and sum to 1, by exponentiating each score and dividing by the sum of exponentials. It is used in the output layer of multi-class classifiers and inside attention mechanisms. Combined with cross-entropy loss, it gives well-behaved gradients.

Poor initialization can cause vanishing or exploding activations and gradients, making training slow or impossible. Initializing all weights to zero makes every neuron learn the same thing. Methods like Xavier/Glorot (for tanh/sigmoid) and He initialization (for ReLU) scale random weights based on layer size to keep signal variance stable across layers.

Deep learning involves huge numbers of matrix multiplications that can be done in parallel. GPUs have thousands of cores designed for parallel computation, making training many times faster than on CPUs. Frameworks like PyTorch and TensorFlow use CUDA/cuDNN to run on NVIDIA GPUs; TPUs are specialized accelerators from Google.

Both are popular deep learning frameworks. PyTorch uses dynamic computation graphs and feels like regular Python, making it popular for research and debugging; it is now widely used in industry too. TensorFlow (with the Keras high-level API) has strong production and deployment tooling (TF Serving, TF Lite, TF.js). Today their capabilities are similar, and the choice often depends on the team and deployment needs.

Self-attention treats input tokens as a set, so it has no built-in sense of word order. Positional encodings add information about each token's position to its embedding. The original Transformer used fixed sine and cosine functions of different frequencies; many modern models use learned position embeddings or relative methods like Rotary Position Embeddings (RoPE).

• Check data: correct labels, preprocessing and normalization; visualize samples • Try to overfit a tiny batch — if the model can't, there is likely a bug • Verify the loss function and output activation match the task • Tune the learning rate (often the culprit) • Watch for vanishing/exploding gradients; inspect gradient norms • Check weight initialization and that layers are not accidentally frozen • Simplify the architecture, then add complexity back step by step