# Coming up to speed on NLP using HuggingFace

- Page: https://shared.btbytes.com/p/nlp-huggingface
- Date: 2024-08-20
- Tags: claude, huggingface, nlp

ATTENTION CONSERVATION NOTICE: This self-study guide was prepared with the help of [Claude](https://shared.btbytes.com/p/claude).

## Introduction to Neuro Linguistic Programming (NLP)

Applications of NLP:

- language translation
- grammatical error correction
- sentiment analysis
- fake news detection

The input for NLP is text - unstructured data.

Methods of processing data for NLP include:

- Tokenization -- break raw text into smaller chunks (could be words or sentences).
	- Libraries for Tokenization are:
		- NLTK
		- Gensim
		- Keras
	- Methods of tokenization are:
		- white space tokenization
		- rule based tokenization
		- penn tree tokenization
		- dictionary based tokenization
		- spacy tokenization
		- moses tokenization
		- subword tokenization
			- Byte pair encoding
			- word piece encoding
			- Sentence piece encoding
			- Unigram language model
- Stop word removal -- stop words do not add any meaning to the sentence, and can be removed.
- Stemming
- Normalization
- Lemmatization
- Parts of speech tagging

## NLP Basics and Hugging Face Introduction

###  Overview of NLP concepts (tokenization, embeddings, etc.)

1. Tokenization:
    - Article: "Tokenization in NLP" by Towards Data Science [https://towardsdatascience.com/tokenization-for-natural-language-processing-a179a891bad4](https://towardsdatascience.com/tokenization-for-natural-language-processing-a179a891bad4)
    - Video: "NLP From Scratch: Tokenization" by Patrick Loeber [https://www.youtube.com/watch?v=nxwfoBY8pxo](https://www.youtube.com/watch?v=nxwfoBY8pxo)
2. Embeddings:
    - Article: "Introduction to Word Embeddings" by Machine Learning Mastery[https://machinelearningmastery.com/what-are-word-embeddings/](https://machinelearningmastery.com/what-are-word-embeddings/)
    - Video: "Word Embeddings - Explained" by StatQuest with Josh Starmer [https://www.youtube.com/watch?v=viZrOnJclY0](https://www.youtube.com/watch?v=viZrOnJclY0)
3. General NLP Concepts:
    - Course: "Natural Language Processing Specialization" on Coursera (by deeplearning.ai)[https://www.coursera.org/specializations/natural-language-processing](https://www.coursera.org/specializations/natural-language-processing)
    - Book: "Natural Language Processing with Python" by Steven Bird, Ewan Klein, and Edward Loper Available online: [https://www.nltk.org/book/](https://www.nltk.org/book/)
4. Hugging Face-specific NLP concepts:
    - Hugging Face Course Chapter: "NLP Tasks" [https://huggingface.co/course/chapter1/3?fw=pt](https://huggingface.co/course/chapter1/3?fw=pt)
    - Hugging Face Blog: "NLP Course" [https://huggingface.co/learn/nlp-course/chapter1/1](https://huggingface.co/learn/nlp-course/chapter1/1)

## Introduction to Hugging Face and its ecosystem

###  Transformers Library

1. Official Documentation:
    - Hugging Face Transformers Documentation [https://huggingface.co/docs/transformers/index](https://huggingface.co/docs/transformers/index)This is the most up-to-date and comprehensive resource directly from the source.
2. Tutorials and Guides:
    - Hugging Face Course Chapter: "Introduction to Transformers" [https://huggingface.co/course/chapter1/1](https://huggingface.co/course/chapter1/1)
    - Hugging Face Transformers Notebooks [https://github.com/huggingface/notebooks](https://github.com/huggingface/notebooks)
3. Video Tutorials:
    - "Hugging Face Transformers Library - Full Course" by Venelin Valkov [https://www.youtube.com/watch?v=DQc2Mi7BcuI](https://www.youtube.com/watch?v=DQc2Mi7BcuI)
    - "Introduction to Hugging Face Transformers" by Python Engineer [https://www.youtube.com/watch?v=7qS5WRRpdFw](https://www.youtube.com/watch?v=7qS5WRRpdFw)
4. Practical Guides:
    - "Getting Started with Hugging Face Transformers" by Towards Data Science[https://towardsdatascience.com/getting-started-with-hugging-face-transformers-for-natural-language-processing-c45e8793c08a](https://towardsdatascience.com/getting-started-with-hugging-face-transformers-for-natural-language-processing-c45e8793c08a)
    - "A Comprehensive Guide to Transformers" by Analytics Vidhya[https://www.analyticsvidhya.com/blog/2022/01/a-comprehensive-guide-to-transformers/](https://www.analyticsvidhya.com/blog/2022/01/a-comprehensive-guide-to-transformers/)
5. Key Classes and Methods:
    - Transformers: Main Classes [https://huggingface.co/docs/transformers/main_classes/overview](https://huggingface.co/docs/transformers/main_classes/overview)This section in the official docs covers the core components you'll be working with.
6. Hands-on Practice:
    - Hugging Face Transformers Notebooks [https://github.com/huggingface/notebooks](https://github.com/huggingface/notebooks)These notebooks provide practical examples and use cases.

### Explore the Transformers library, focusing on key classes and methods


For your 60-minute deep dive, I'd recommend:

1. Start with the "Introduction to Transformers" from the Hugging Face course (15 minutes)
2. Watch the first 20-30 minutes of the "Hugging Face Transformers Library - Full Course" video
3. Spend the remaining time exploring the "Main Classes" section of the official documentation and try running a few examples from the Notebooks repository


## Practical NLP Tasks

###  Text classification with pre-trained models

- Official Hugging Face Documentation:
    - Text Classification Pipeline [https://huggingface.co/docs/transformers/main/en/task_summary#text-classification](https://huggingface.co/docs/transformers/main/en/task_summary#text-classification)
    - Fine-tuning a pre-trained model [https://huggingface.co/docs/transformers/training](https://huggingface.co/docs/transformers/training)
- Tutorials and Guides:
    - Hugging Face Course: "Fine-tuning a pretrained model" [https://huggingface.co/course/chapter3/1](https://huggingface.co/course/chapter3/1)
    - Practical Guide: "Text Classification with Hugging Face Transformers"[https://www.kaggle.com/code/yukikitayama/text-classification-with-hugging-face-transformers](https://www.kaggle.com/code/yukikitayama/text-classification-with-hugging-face-transformers)
- Video Tutorials:
    - "Text Classification with Transformers in PyTorch" by Patrick Loeber [https://www.youtube.com/watch?v=RznKVRTFkBY](https://www.youtube.com/watch?v=RznKVRTFkBY)
- Hands-on Notebooks:
    - Text Classification with Transformers[https://github.com/huggingface/notebooks/blob/main/examples/text_classification.ipynb](https://github.com/huggingface/notebooks/blob/main/examples/text_classification.ipynb)
- Blog Posts:
    - "Multi-Class Text Classification with Hugging Face Transformers" by Towards Data Science[https://towardsdatascience.com/multi-class-text-classification-with-hugging-face-transformers-roberta-56caae3d5f36](https://towardsdatascience.com/multi-class-text-classification-with-hugging-face-transformers-roberta-56caae3d5f36)
    -

- Start with the official Hugging Face documentation on Text Classification Pipeline (5 minutes)
- Follow the hands-on notebook on Text Classification with Transformers (15 minutes)
- If time allows, skim through the "Fine-tuning a pretrained model" section in the Hugging Face course (10 minutes)

### Named Entity Recognition (NER) using Hugging Face


1. What is NER? NER is the task of identifying and classifying named entities (like persons, organizations, locations) in text.
2. Hugging Face and NER: Hugging Face offers pre-trained models and easy-to-use pipelines for NER tasks.
3. Key Components: a) Pipeline API: The simplest way to perform NER with Hugging Face. b) Pre-trained Models: Models like BERT, RoBERTa, and others fine-tuned for NER. c) Tokenizers: Essential for preprocessing text for NER tasks.
4. Basic Usage: Here's a simple example of using Hugging Face for NER:
```python
from transformers import pipeline

ner = pipeline("ner")
text = "Apple Inc. is headquartered in Cupertino, California."
result = ner(text)
print(result)
```

5. Fine-tuning: You can fine-tune pre-trained models on your specific NER dataset for better performance.
6. Resources:

- Hugging Face NER documentation: [https://huggingface.co/tasks/token-classification](https://huggingface.co/tasks/token-classification)
- NER tutorial: [https://huggingface.co/course/chapter7/2](https://huggingface.co/course/chapter7/2)

## Advanced Topics and Hands-on Practice

### Fine-tuning pre-trained models

- Fine-tuning Concept: Fine-tuning adapts a pre-trained model to a specific task or domain by training it further on a smaller, task-specific dataset.
- Custom Datasets: These are datasets you create or obtain that are specific to your NER task or domain.
- Process Overview: a) Prepare your dataset:

    - Format your data (typically in CoNLL format for NER)
    - Split into train/validation/test sets

    b) Load a pre-trained model:

    - Choose a model (e.g., BERT, RoBERTa) from Hugging Face

    c) Set up fine-tuning:

    - Configure training arguments
    - Create a trainer

    d) Train the model:
    - Run the fine-tuning process
    - Monitor metricse) Evaluate and use the model
- Code Example: Here's a simplified example of fine-tuning a model for NER:
```python
from transformers import AutoTokenizer, AutoModelForTokenClassification, TrainingArguments, Trainer
from datasets import load_dataset

# Load dataset
dataset = load_dataset("your_custom_dataset")

# Load pre-trained model and tokenizer
model = AutoModelForTokenClassification.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

# Tokenize dataset
def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True)
    labels = []
    for i, label in enumerate(examples["ner_tags"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:
            if word_idx is None:
                label_ids.append(-100)
            elif word_idx != previous_word_idx:
                label_ids.append(label[word_idx])
            else:
                label_ids.append(-100)
            previous_word_idx = word_idx
        labels.append(label_ids)
    tokenized_inputs["labels"] = labels
    return tokenized_inputs

tokenized_dataset = dataset.map(tokenize_and_align_labels, batched=True)

# Set up training arguments
training_args = TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
)

# Create Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    tokenizer=tokenizer,
)

# Fine-tune the model
trainer.train()
```

5. Challenges and Considerations:

- Data quality and quantity are crucial
- Balancing classes in your dataset
- Overfitting on small datasets
- Computational resources required

6. Benefits:

- Improved performance on domain-specific tasks
- Ability to recognize custom entity types

7. Resources:

- Hugging Face fine-tuning guide: [https://huggingface.co/docs/transformers/training](https://huggingface.co/docs/transformers/training)
- Custom datasets tutorial: [https://huggingface.co/docs/datasets/dataset_script](https://huggingface.co/docs/datasets/dataset_script)

### Exploring the Model Hub and implementing a chosen NLP task

The Hugging Face Model Hub is a central repository for pre-trained models, and it's an excellent resource for implementing various NLP tasks. Here's an overview of exploring the Model Hub and implementing an NLP task:

1. The Model Hub:
    - A platform hosting thousands of pre-trained models
    - Covers various NLP tasks, including translation, summarization, question answering, etc.
    - URL: [https://huggingface.co/models](https://huggingface.co/models)
2. Exploring the Model Hub:
    - Use filters to narrow down models by task, language, framework, etc.
    - Each model has a dedicated page with description, usage examples, and performance metrics
    - You can directly use models or download them for local use
3. Implementing an NLP Task: Let's walk through the process of implementing a task using a model from the Hub. We'll use text summarization as an example.

Step 1: Find a suitable model

- Go to the Model Hub and filter for "summarization" task
- Choose a model, e.g., "facebook/bart-large-cnn"

Step 2: Load the model and tokenizer

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = "facebook/bart-large-cnn"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
```

Step 3: Prepare your input text
```python
text = """ Your long text to be summarized goes here. It can be multiple sentences or paragraphs long. """
```

Step 4: Tokenize the input and generate the summary

```python
inputs = tokenizer([text], max_length=1024, return_tensors="pt", truncation=True)
summary_ids = model.generate(inputs["input_ids"], max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
```

Step 5: Decode and print the summary

```python
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True) print(summary)
```

- Other Popular NLP Tasks:
    - Text Classification
    - Named Entity Recognition
    - Question Answering
    - Machine Translation
    - Sentiment Analysis
- Best Practices:
    - Read the model card for usage instructions and limitations
    - Check the license for commercial use if applicable
    - Consider fine-tuning for domain-specific tasks
    - Be aware of potential biases in pre-trained models
- Community Aspect:
    - You can contribute your own models to the Hub
    - Collaborate with others and build on existing work
- Resources:
    - Hugging Face Tasks: [https://huggingface.co/tasks](https://huggingface.co/tasks)
    - Using Pretrained Models: [https://huggingface.co/docs/transformers/preprocessing](https://huggingface.co/docs/transformers/preprocessing)