Globalization made the world flat, AI drills holes into the infinitesimally small

The function of deep learning

Far from a machine brain

What are neural networks?

The function of deep learning

Far from a machine brain

\[ f_w(x_1, x_2,\dots, x_n) = \sum_{i=1}^{n}w_ix_i = w_0x_0 + w_1x1 + \dots + w_nx_n \]

The artificial neuron, originally called the perceptron, is more or less a composition of two functions. The first function receives a number of weighted inputs and adds them up.

The second function, called the activation function, is then applied to this sum. If the inputs were large enough and their weights were big enough, the neuron activates and outputs something, depending on what activation function it uses.

There are many conventional activation functions, each useful for different problems, but in themselves they have nothing to do with the problems the artificial network tries to solve.

They are chosen because they don't linearly map the input onto the output set, rather they suddenly activate when their input reaches a threshold.

This is very good for capturing those non-linearities that are hard to express algorithmically.

The function of deep learning

Far from a machine brain

ReLU

\[ f_a(x) = \begin{cases} x & \quad \text{if } x\gt 0 \\ 0 & \quad \text{otherwise} \end{cases} \]

The function of deep learning

Far from a machine brain

Neuron output

\[ f_n(x_1,x_2,..,x_n) = f_a(f_w(x_1, x_2,\dots,x_n)) = \begin{cases} \sum_{i=1}^n {w_ix_i} & \quad \text{when } x \gt 0 \\ 0 & \quad \text{otherwise} \end{cases} \]

The function of deep learning

Far from a machine brain

Neural network

The function of deep learning

Far from a machine brain

Neural network

There is such a distribution of weights that approximates any mapping between a set of inputs onto a set of outputs.

The problem is discovering this distribution. For a classification problem, such as the one I proposed, we would use a backpropagation algorithm.

We would have a set of images already classified by hand, the training dataset, and process them through a network initialized with random weights.

Using what is interchangeably called either a *loss function* or a *cost function* we would assess how different the predicted output is to the real one. And adjust the weights accordingly.

We then move back through the layers trying to adjust the weights so as to avoid activating neurons that lead to producing poor outputs and activating neurons that would've led to the desired output.

This isn't done weight by weight, of course. The technical implementation concerns itself with function derivations and matrix multiplications, something that I only mention because it points to a kind of hardware especially suited for training the neural networks that are now commonly in use.

The function of deep learning

Far from a machine brain

Technically

More neurons, better possible approximations

More data, better results

The function of deep learning

Far from a machine brain

Source: analystprep.com

The function of deep learning

Far from a machine brain

Source: wikipedia.org

The function of deep learning

Far from a machine brain

The function of deep learning

Functions all the way down

The function of deep learning

Functions all the way down

Artist's impression of a convolutional neural network.

The function of deep learning

Functions all the way down

DeepDream

The function of deep learning

Functions all the way down

Artist's impression of an Autoencoder.

The function of deep learning

Functions all the way down

Artist's impression of an Autoencoder.

The function of deep learning

Functions all the way down

An AI system.

The function of deep learning

Functions all the way down

Where do we go from here?

If we wanted to extend this system so that it could, for instance, identify a larger number of moods or to enumerate the objects inside the pictures, we'd have, first of all, to produce a richer training dataset and then to redesign the system for the new codomain.

We could increase the number of output nodes or we could train multiple networks and compose their results.

We could keep this network and add one that identifies the intensity of the mood.

Then we would compose them to identify a gradient of moods from tragedy to sadness to melancholy to ennui to content to happiness to exuberance.

At some point we'd return to the same problem we sought to escape by using neural networks. Only at a different level of abstractions.

Still, we can describe pictures with our words and we can draw images based on descriptions.

It feels like there should be a mapping between language and vision, a method to transpose one onto the other.

The function of deep learning

Functions all the way down

The function of deep learning

Qualitative and quantitative leaps

The function of deep learning

Qualitative and quantitative leaps

The function of deep learning

Qualitative and quantitative leaps

The function of deep learning

Qualitative and quantitative leaps

The function of deep learning

Qualitative and quantitative leaps

1943

Perceptron invented

1958

Perceptron built

1962

Backpropagation speculated

1967

Stochastic Gradient Descent

1969

CNN with ReLU

1970

Backpropagation implemented

1979

Deep learning CNN

1989

Self-trained CNN

1991

Autoencoder

1997

Long short-term memory

2014

Attention

2017

Transformer

The function of deep learning

Qualitative and quantitative leaps

The function of deep learning

Qualitative and quantitative leaps

The function of deep learning

Qualitative and quantitative leaps

This is why critics of AI are quick to point towards the Mechanical Turk or other cases of extreme underpayment through outsourcing in order to identify the shameful backdrop of exploitation that the whole edifice hinges on.

There's a whole constellation of reasons for why these accusations fall of deaf ears. One of the points is that it's only very partially true.

Exploited workers are used to fine-tune ChatGPT subsystems, but usually it's only one task out of many that come and go in a succession that makes work lose its concrete features and be registered only as an expenditure of time.

More importantly, the operators of the Mechanical Turk aren't providing the massive amounts of information needed for ChatGPT to function. We are.