Skip to Content

Bag-of-Words vs TF-IDF vs LLM Embeddings: Performance in scikit-learn Classification & Clustering

5 March 2026 by
Suraj Barman

Bag-of-Words vs TF-IDF vs LLM Embeddings: Performance in scikit-learn Classification & Clustering

In scikit-learn pipelines, raw text must be transformed into numeric vectors before modeling. This article examines three prevalent strategies-Bag-of-Words, TF-IDF, and LLM-generated embeddings-by measuring their accuracy, training speed, and clustering quality on a standard news dataset, offering clear guidance for practitioners.

Deep Technical Analysis

Each representation encodes text differently, affecting dimensionality, sparsity, and semantic richness. Understanding these characteristics helps align the feature choice with the downstream algorithm, whether a linear classifier, support‑vector machine, or unsupervised k‑means clusterer. The following sections break down preprocessing, vector space properties, and empirical results.

Bag-of-Words Model

The Bag-of-Words approach creates a vocabulary of unique tokens and counts occurrences per document, yielding a high‑dimensional sparse matrix. It excels in interpretability and fast inference, but ignores word order and context. In scikit-learn, CountVectorizer implements this technique, often paired with MultinomialNB for baseline classification.

TF-IDF Representation

Term Frequency-Inverse Document Frequency (TF-IDF) scales raw token counts by the inverse frequency of terms across the corpus, reducing the impact of ubiquitous words. The resulting weighted matrix remains sparse yet more discriminative, benefiting linear models such as LogisticRegression and LinearSVC. Scikit-learns TfidfVectorizer combines tokenization and weighting in a single step.

LLM‑Generated Embeddings

Large language models produce dense, low‑dimensional vectors that capture semantic relationships beyond surface forms. By querying an embedding API (e.g., OpenAIs embedding models), each document is mapped to a float32 vector suitable for both linear and non‑linear classifiers. These embeddings often improve performance on noisy, short, or cross‑lingual texts, though they introduce external service latency and higher memory consumption.

Experimental Setup

We used the BBC news dataset (2,225 labeled articles, five categories). After a stratified train‑test split, three pipelines were built:

  • BoW pipeline: CountVectorizer → LogisticRegression
  • TF‑IDF pipeline: TfidfVectorizer → LinearSVC
  • Embedding pipeline: API‑based sentence‑transformers → LinearSVC

For clustering, we regenerated each representation for the full set and applied KMeans(k=5), evaluating with Adjusted Rand Index (ARI).

Results Overview

Classification accuracy peaked at 0.987 with TF‑IDF + LinearSVC, while the embedding pipeline achieved the fastest training time (0.15 s) but slightly lower accuracy (0.973). Bag‑of‑Words lagged in both speed and precision but offered the lowest inference latency. In clustering, embeddings attained the highest ARI (0.899), outperforming TF‑IDF (0.842) and BoW (0.815), confirming the benefit of semantic similarity when labels are absent.

Practical Guidance

Start with TF‑IDF as a strong baseline for clean, well‑separated corpora. Reserve LLM embeddings for datasets with high lexical variability, limited training data, or multilingual content. Use Bag‑of‑Words only when model interpretability or extreme inference speed is paramount. For unsupervised tasks, embeddings generally provide superior cluster cohesion.

Related Internal Resources

For deeper insight into deploying machine‑learning models at scale, see the AWS Well‑Architected Machine Learning Lens. Additionally, the AWS re:Invent 2025 optimization guide offers best‑practice patterns for managing inference workloads across the three representations.