Naive Bayes Classification

Theoretical Background

Naive Bayes classifiers are based on Bayes' Theorem, which provides a way to update the probability estimate for a hypothesis as more evidence is acquired. The basic formula for Bayes' Theorem is:

$P(C|X) = \frac{P(X|C) \cdot P(C)}{P(X)}$

where:

• P(C|X) is the posterior probability of class C given the feature vector X.
• P(X|C) is the likelihood of feature vector X given class C.
• P(C) is the prior probability of class C.
• P(X) is the prior probability of feature vector X.

The naive assumption is that the features are conditionally independent given the class label, allowing us to express the likelihood as a product of individual probabilities:

$P(X|C) = P(x_1|C) \cdot P(x_2|C) \cdot ... \cdot P(x_n|C)$

Practical Applications

Naive Bayes is widely used in text classification problems such as spam detection, sentiment analysis, and document categorization, where the independence assumption is not strictly true but often works well enough.

Code Example

Here's a basic implementation using Python's scikit-learn library:

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

# Sample data
documents = ["This is a positive review", "This is a negative review"]
labels = [1, 0]  # 1 for positive, 0 for negative

# Create a bag-of-words model
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)

# Train the model
model = MultinomialNB()
model.fit(X, labels)

# Predict a new sample
new_document = ["This review is positive"]
X_new = vectorizer.transform(new_document)
prediction = model.predict(X_new)

Performance Considerations

Naive Bayes classifiers can be surprisingly effective when the independence assumption is somewhat true, as often is the case in text classification. However, if features are highly correlated, the performance may degrade. For instance, in image classification, where pixel values are often related, Naive Bayes might not be the best choice.

External References

• Scikit-learn Naive Bayes documentation
• Wikipedia: Naive Bayes classifier

Diagram

Below is a simple depiction of how Naive Bayes works with features $x_1$, $x_2$, ..., $x_n$ and class $C$:

graph LR
A(Feature x_1) --> C(Class C)
B(Feature x_2) --> C
C1(Feature x_n) --> C

This diagram illustrates the independence assumption that each feature directly influences the class but not each other.