Connectionist / Neural Networks

Single-layer neural networks that learn linear decision boundaries. Rosenblatt (1958) built the Mark I Perceptron hardware. Limited to linearly separable problems.

Rumelhart, Hinton, and Williams popularized backpropagation in 1986, enabling training of multi-layer networks. The algorithm that made deep learning possible.

Feedforward networks with one or more hidden layers. The Universal Approximation Theorem (1989) proved MLPs can approximate any continuous function.

Networks using convolution operations to extract spatial features. Inspired by the visual cortex. Revolutionized computer vision from LeNet (1998) to ResNet (2015).

Networks with feedback connections for processing sequences. Capable of modeling temporal dependencies but prone to vanishing gradients in long sequences.

Bahdanau et al. (2014) introduced attention for neural machine translation, allowing models to dynamically focus on relevant input parts. The key insight behind transformers.

Vaswani et al.'s 'Attention Is All You Need' (2017) replaced recurrence with self-attention. Enabled massive parallelization and scaling, becoming the foundation of modern AI.

Networks trained to reconstruct their input through a bottleneck, learning compressed representations. Foundational for dimensionality reduction and generative models.

Mikolov et al. (2013) at Google trained word embeddings using Skip-gram and CBOW. Showed that vector arithmetic captures semantic relationships (king - man + woman = queen).