From Machine Learning to AI and Gen AI

1. Word2vec: Let’s imagine Word2vec as a tool that learns the intricate web of relationships between words by studying a large body of text. This technique, which emerged in 2013, uses a neural network model to grasp word associations. At its, core Word2vec is highly capable of identifying synonymous words or suggesting words to round off a sentence. Each word is represented by a unique list of numbers, known as a vector, carefully selected to encapsulate the word’s meaning and usage. The beauty of this approach is that a simple mathematical function, cosine similarity, can reveal the semantic similarity between words based on these vectors. In other words, it has proven a practical and effective way to decode language.
2. N-Gram: Picture this: you’re trying to piece together a linguistic puzzle, and each piece is an n-gram. It’s a sequence of ‘n’ items, which could be anything from phonemes, syllables, letters, words, or even base pairs, all extracted from a sample of text or speech. These n-grams are like the DNA of language, collected from a text or speech corpus, and they provide a unique snapshot of language in action. The ‘n’ in n-gram can vary, with Latin numerical prefixes like “unigram”, “bigram”, and “trigram” indicating the size. A unigram is a single item, a bigram is a sequence of two, and a trigram is a sequence of three. It’s like a sliding window moving through a sentence, capturing a specific sequence of items. In the world of computational biology, these sequences go by a different name, becoming k-mers. Regardless of whether you call them n-grams or k-mers, they’re a fascinating way to dissect and understand language, one sequence at a time. They’re the building blocks that help us analyze and predict language patterns, making them invaluable in fields like natural language processing and machine learning.
3. RNNs and LSTMs: Recurrent Neural Networks (RNNs) are a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence. This allows them to use their internal state (memory) to process sequences of inputs. The core idea behind RNNs is to make use of sequential information by performing the same task for every element in a sequence, with the output being dependent on the previous computations. However, RNNs have a significant issue known as the vanishing gradient problem. As they backpropagate to learn, they multiply through all the derivatives of the weights used in the computations. If these weights are small (less than 1), the gradient decreases exponentially quickly as it propagates down, effectively “vanishing.”
4. Long Short-Term Memory (LSTM) networks. LSTMs are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber in 1997 and were refined and popularized by many people in the following work. LSTMs work by introducing a new structure called a memory cell. A memory cell is composed of four main elements: (1) an input gate, (2) a neuron with a self-recurrent connection (a connection to itself), (3) a forget gate, and (4) an output gate. The self-recurrent connection has a weight of 1.0 and ensures that barring any outside interference, the state of a memory cell can remain constant from one time step to another. The magic of LSTMs comes from their ability to forget, remember, and update their cell states, thanks to the gating mechanisms. The gates in an LSTM are analog, in the form of sigmoid neural nets. They output numbers between “0” and “1”, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through”. This unique architecture allows LSTMs to selectively remember or forget things, which makes them particularly good at understanding the context within sequences of data. They’re effectively capable of remembering important details while letting go of the irrelevant ones.
5. Machine learning-based attention mechanism:  Let’s imagine for a second the machine learning as a bustling city, and attention mechanisms as its traffic control system. These mechanisms guide the flow of information, assigning different “soft” weights to each word in a context window, much like traffic lights control the movement of vehicles at intersections. These weights aren’t static like the buildings in our city analogy; they’re dynamic, changing in real time during each interaction with a large language model. This is in stark contrast to “hard” weights, which are pre-trained, fine-tuned, and remain constant. Now, let’s think of attention mechanisms as the city’s selective spotlight, illuminating the areas (or words) that are most crucial for making a prediction, while dimming the less relevant parts. This selective focus boosts the model’s accuracy and efficiency, akin to how a well-planned city runs smoothly. In the early days, similar mechanisms were used in recursive neural networks. However, these networks processed words sequentially, not in parallel. They would consider the current word and the words within the context window to update their hidden state and make predictions, much like a city planner considering various factors before making a decision. How do they work? In the mathematical world, these attention mechanisms are represented by complex equations and models. For instance, the “soft” weights are calculated using softmax functions, which convert a vector of numbers into a vector of probabilities. These probabilities then determine the importance of each word in the context window. So, in essence, attention mechanisms in machine learning are like the traffic control system of a bustling city, guiding the flow of information to ensure the model runs smoothly and efficiently.
6. Transformer Models: Let’s continue our analogies, so just picture that you’re at a party with a bunch of people, and you’re trying to listen to multiple conversations at once. Some people are louder, some are quieter, and some are more interesting than others. Your brain, being the amazing machine it is, can focus on the important bits and tune out or discard the rest. That’s kind of what a Transformer model does with data. Introduced by the Google Brain team in a paper called ‘Attention Is All You Need’ in 2017, the Transformer model is a type of deep learning architecture. It’s the cool new kid that everyone wants to hang out with because it’s faster, smarter, and more efficient than the older kids, like recurrent neural architectures such as long short-term memory (LSTM). The Transformer model uses the attention mechanism, which decides which parts of the data to focus on and which parts to ignore. This makes it great for tasks like language translation, where understanding the context and nuances of words is crucial.

Now, let’s dive into the main components of a Transformer model:

Large Language Models (LLMs) are the giants of the AI world, boasting billions of parameters that help them understand and generate human-like text. These models are trained using a combination of self-supervised and semi-supervised learning methods.

In self-supervised learning, the models are trained to predict parts of the input data from other parts of the same input data. For instance, given a sentence with a missing word, the model’s task is to predict the missing word based on the context provided by the rest of the sentence. This method allows the model to learn the structure and semantics of the language without needing any explicit labels.

On the other hand, semi-supervised learning is a method that combines a small amount of labeled data with a large amount of unlabeled data during training. The idea is to use the labeled data to guide the learning process and then use the model’s predictions on the unlabeled data to further refine the model. This method is particularly useful when obtaining labeled data is expensive or time-consuming.

These learning methods, combined with the power of artificial neural networks and specialized AI accelerator hardware, allow LLMs to process vast amounts of text data, most of which is scraped from the Internet. The models learn not just the syntax and semantics of language, but also the biases and inaccuracies present in the data they are trained on.

Prominent examples of LLMs include GPT-4, LLaMa, PaLM, BERT, BLOOM, Ernie 3.0 Titan, and Claude. These models, with their advanced learning methods and immense size, have revolutionized the field of natural language processing, making older, specialized models for specific linguistic tasks largely obsolete.

Bogdan Iancu is a seasoned entrepreneur and strategic leader with over 25 years of experience in diverse industrial and commercial fields. His passion for AI, Machine Learning, and Generative AI is underpinned by a deep understanding of advanced calculus, enabling him to leverage these technologies to drive innovation and growth. As a Non-Executive Director, Bogdan brings a wealth of experience and a unique perspective to the boardroom, contributing to robust strategic decisions. With a proven track record of assisting clients worldwide, Bogdan is committed to harnessing the power of AI to transform businesses and create sustainable growth in the digital age.