LLM FROM SCRATCH

FROM ZERO TO HERO

The educational objective of this Hackathon is to master all the stages of constructing a large language model (LLM). The workshops are organized over 4 days, starting from a blank page, to build step-by-step all the components of an LLM:

• Data Collection and Preparation: Gather and process the necessary data to train the model.
• Data Preprocessing: Clean and transform the raw data into a suitable format for model training (tokenization, normalization, etc.).
• Embedding: Convert the data into numerical vectors in a high-dimensional space.
• Attention Mechanism: A technique that allows the model to focus on the most important parts of the input data.
• Transformer Architecture: Use of the Transformer architecture, which is the foundation of many NLP models like Llama and GPT.
• Evaluation Metrics: Methods to evaluate the model’s performance.
• Training and Validation: The process of teaching the model from training data and validating its performance on test data.
• Fine Tuning: Adjusting the hyperparameters of the pre-trained model on a specific task to improve its performance.