Fine Tuning a Dataset with Pretrained Hugging Face Models Complete Guide
A step-by-step guide to fine-tuning a pretrained DistilBERT model on a labeled spam dataset, covering tokenization, training, and evaluation with TensorFlow.
Training a language model from scratch takes enormous amounts of data and compute that most projects simply don't have. Fine-tuning sidesteps that problem entirely: start with a model that already understands general language, and adapt it to a specific task using a much smaller, labeled dataset. This walkthrough covers exactly that process, using a pretrained DistilBERT model to build a spam detection classifier.
Why fine-tune instead of training from scratch
Fine-tuning starts with a pretrained model, in this case DistilBERT, which has already been trained on general-purpose language and understands English grammar and sentence structure. Instead of teaching a model language from nothing, fine-tuning teaches an already-capable model one specific task. That makes it the right choice whenever a project needs a custom, domain-specific task, such as legal document analysis or social media sentiment analysis, or whenever the available labeled data is limited and a performance boost is needed on that specific task rather than on general language understanding.
How the fine-tuning process works, step by step
The process has four consistent stages regardless of the task. First, load a pretrained model, general purpose and not yet specialized. Second, prepare a task-specific, labeled dataset; here that means labeling messages as spam or ham. Third, preprocess the data using tokenization and padding. Fourth, modify the model by adding task-specific layers, such as a classification head for spam detection, and freezing some of the existing layers. From there, the model trains on the new, task-specific dataset, adjusting its weights toward the desired outcome. The result is a model that specializes in spam detection while still leveraging everything it already understood about the English language. Feed it "win a free iPhone now" and it outputs spam; feed it "hey are you free to meet tomorrow" and it outputs ham.
Loading and splitting the data
The practical walkthrough starts with importing pandas and reading a CSV file containing two columns: a label column marking each message as ham or spam, and a message column containing the actual text. Checking the dataset's shape confirms the row and column counts. The message text becomes the input list, and the label column becomes the target list. The dataset then gets split into training and testing subsets, reserving 20 percent of the data for testing, with a fixed random state to keep the split reproducible across runs.
Tokenizing text for DistilBERT
With the transformers library installed, a pretrained DistilBERT tokenizer gets loaded to convert raw text into a format the model can process. The tokenizer breaks text into tokens and maps them to DistilBERT's vocabulary. Training and testing text both get converted into numerical encodings this way, with truncation applied to text exceeding the model's 512-token limit and padding applied to shorter sequences so every input in a batch matches in length.
Converting encodings into TensorFlow datasets
TensorFlow gets imported next, specifically to convert the tokenized encodings and their labels into TensorFlow-compatible datasets. The tokenized training encodings pair with the training labels, and the tokenized testing encodings pair with the testing labels, producing two TensorFlow datasets ready for the training step.
Configuring and running training
Loading the pretrained model for fine-tuning means importing DistilBertForSequenceClassification along with TFTrainer and TFTrainingArguments from the transformers library. The training arguments configure settings like output directory, total training epochs, batch size per device for both training and evaluation, the strength of weight decay, and the directory for storing logs. With those arguments defined, a TFTrainer instance handles the actual training and evaluation process, and calling train starts the fine-tuning run itself.
Evaluating the fine-tuned model
Once training finishes, evaluating the model on the held-out test set produces metrics including loss and accuracy. Beyond the trainer's built-in evaluation, running trainer.predict on the test set produces the raw predictions, which can be compared against the true labels to build a confusion matrix, giving a clearer picture of exactly where the model is getting spam and ham classifications right or wrong, rather than relying on accuracy alone.
Key takeaways
- Fine-tuning adapts an already-trained model to a specific task using a much smaller labeled dataset than training from scratch would require.
- The four-stage process is: load a pretrained model, prepare a labeled task-specific dataset, preprocess and tokenize the data, then add task-specific layers and train.
- Text gets tokenized with truncation at 512 tokens and padding to match sequence lengths within a batch.
- TFTrainer and TFTrainingArguments from Hugging Face's transformers library configure and run the actual fine-tuning process.
- Evaluating on a held-out test set, including a confusion matrix, shows where the fine-tuned model is actually succeeding or failing, beyond a single accuracy number.
- The same approach applies beyond DistilBERT and spam detection, to other pretrained models and other domain-specific classification tasks.
Who this is for
This guide is for data scientists, ML engineers, or developers who want a practical, code-level walkthrough of adapting a pretrained Hugging Face model to a custom classification task, whether that's spam detection, sentiment analysis, or another domain-specific problem. The accompanying written guide is linked in the video description for reference alongside the code.
Full transcript(auto-generated, with timestamps)
[0:05]Hello everyone. Today we are going to discuss fine-tuning of a pre-trained model from hugging face. So this is the hugging face website where you can find a documentation and look deep into the finetuning of a pre-trend model. uh now I'll move on to the code where I'll explain what is finetuning and how we are going to train the model train the data set. So how basically how fine tuning works is that you need to load a model pre-trained model I'm I'm using digital but here uh which has been trained on general purpose the next thing is to is that you should prepare a task specific data set you need to label
[0:50]A data set for a specific task for example here I'm using spam versus ham for spam detection pre-process the data that is we are you going to use tokenization and padding time. Then we need to modify the model add t specific task layers example classification header for spam detection free some layers. Then we need to train on the new data set that is we are going to use specific task data set to fine-tune the model weights for the desired outcome. Uh for example uh let's take uh fine-tuning digital bird for sample detection spam detection. So here digital but already understands English grammar okay and sent the structures
[1:36]Fine-tuning on spam data set would be labeling uh the data set into spam and h spam or ham as you can see here. Uh then the outcome that we would receive here is after fine-tuning is the model specializ specializes in spam detection while still leveraging its general understanding of the English language. So for example, if the input is win a free iPhone now, the output would be spam. But if the input would be hey are you free to meet tomorrow? The output would be ham. So similarly now when should we use uh finetune? When should we fine-tune a data set? Like if you want to uh perform some custom task
[2:21]For example if you have a domain specific task like legal document analysis or social media sentimental analysis that time you should perform fine tuning of that data set also when you have limited data and if you want a performance boost into your uh specific task then at that time you should finetune your data set. Okay let's move on to the code now. So first I'm importing the pandas library and then I'm uh reading the data set which is in CSV file CSV format. So here label and message it label indicates whether a message is ham or spam and the message is message column contains the actual text
[3:13]Of the message. Uh yeah so moving on uh this uh df shape checks the dimension of the data set. As you can see we have two columns and the number of rows here. This x is equal to list df message contains the text message whereas y contains the label ham or spam label. Next, next we are going to split the data set. And so splitting the data set uh into training and testing sub subsets. So as you can see test size reserves 20% of the data for training and [clears throat] random state ensures reproducibility. So X train and Y train are for the training features and labels whereas X
[4:23]Test and Y test are the testing features and uh labels. Now moving forward, we need to install transformers library in order to import uh the trainer functionality. So once that is done uh we need to again import digital but tokenizer in order to use this distbut based model. So this this uh snippet basically loads a pre-trained digital but tokenizer for tokenizing text data and the tokenizer breaks text into tokens and maps them to dist vocabulary. So once that is done uh here in in the training encodings we are going to convert text into numerical format that the distributed model can process. So basically truncating means truncating
[5:23]Text exceeding the max limit. I think the max limit is 512 tokens and padding means pad shorter sequences to the same length. Uh train encodings and test encodings they are tokenized inputs for training and testing. We are going to check the y train y train. What is the value? So as you can see 0 0 0 0. So this is the value when we run that co snippet when run y is equal to uh yeah and after that we are going to import tensorflow and uh basically uh the purpose of this is to convert the tokenized data and labels into tensorflow data sets. for compatibility
[6:22]And dict this dict train encodings this provides a tokenized text and the wire train provides the labels. Similarly for this the uh text encodings provides the tokenized text and the white underscore uh text provides the labels. Now we are going to uh load the pre-trained digital but model. So here as you can see we are importing the distlbert sequence classification and TF trainer plus the TF training argument libraries from transformers and once that is imported we are then configuring the training settings such as pods batch size and login over here as you can see the output directory total number of training epox batch size per device back batch size for violation
[7:33]Strength of weight decay directory for storing logs. So once that is done once we have defined the training arguments we are going to then fine-tune the model. So we are going to import uh TF trainer from transformers and I've already uh executed this code as it would take a lot of time while uh for execution that's I've already executed the code here. So once that is done uh yeah so yeah we are going to fine-tune the model and one uh we are going to import uh TF trainer from transformers then we are going to set the create a TF trainer instance to handle training and
[8:26]Evaluation and TFT train this basically starts the f fin fine-tuning process we are going to pass this arguments here and yeah this is the message this is the output of uh by which is uh received using by fine- tuning the model. Now we are going to evaluate the model. Uh as you can see the results are here. Uh um so um yeah so evaluating the train model on the test data sets and this is the output of the metric output metrics such as loss and accuracy which we are uh getting after prefinet tuning the model. So this is a trainer evaluation. This is a trainer predict.
[9:32]And then this is the shape of the data set. This is the prediction. This is the matrix of the data set. The confusion matrix. And this is this is the output for the confusion matrix. Yeah. So that is all about finetuning. Uh the conclusion is that basically this code performs the steps perform performs steps such as uh processing a label data set for spam colle classification prepares the data using tokenization and tensorflow data sets fine-tunes a pre-trained distributed model and evaluates the model's performance on the test set. So that is all about fine-tuning a pre-trained model such as distb. You can use other models as well in order to
[10:23]Fine-tune your data set. So yeah, that's all about finetuning. Thank you.
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53