# Getting Started with Text Classification

## Overview

Text classification assigns documents to predefined categories. In organizational research, documents might be vacancy sentences, employee comments, reports, or interview excerpts. This tutorial develops a transparent workflow from raw HTML to predictions.

    library(textclassificationtutorial)

## Extract text from HTML

The package includes the nursing-vacancy page used by the original tutorial.

    html_file <- system.file(
      "extdata", "sample_nursing_vacancy.html",
      package = "textclassificationtutorial"
    )
    vacancy_text <- extract_html_text(html_file)
    substr(vacancy_text, 1, 200)
    #> [1] "Als einer der führenden privaten Träger im Bereich der stationären Pflege bietet CASA REHA Ihnen zukunfts- und krisensichere Arbeitsplätze. Über 6500 Mitarbeiter versorgen rund um die Uhr unsere Bewoh"

To process a folder, use `extract_html_dir()`. The result keeps a document ID, the source path, and extracted text together.

    pages <- extract_html_dir("inst/extdata/vacancypages")

CSS and XPath selection are available when `xml2` is installed:

    extract_html_text(html_file, selector = "div.content")
    extract_html_text(html_file, xpath = "//div[@class='content']")

## Segment and normalize

The unit of analysis should follow the research question. Here, each sentence is treated as one document.

    sentences <- split_sentences(vacancy_text)
    head(sentences)
    #> [1] "Als einer der führenden privaten Träger im Bereich der stationären Pflege bietet CASA REHA Ihnen zukunfts- und krisensichere Arbeitsplätze."
    #> [2] "Über 6500 Mitarbeiter versorgen rund um die Uhr unsere Bewohner und leben die CASA REHA-Philosophie – \"von Mensch zu Mensch\"."
    #> [3] "Bereits zum zweiten Mal in Folge wurden wir als einer der besten Arbeitgeber Deutschlands im Bereich \"Gesundheit & Soziales\" ausgezeichnet."
    #> [4] "Jede unserer fast 70 Einrichtungen hat einen einzigartigen Stil, der die Besonderheiten des Standorts widerspiegelt."
    #> [5] "Allen gemein sind eine familiäre Atmosphäre, ein modernes Arbeitsumfeld und hohe Qualitätsstandards."
    #> [6] " "

Preprocessing choices are analytical decisions, not housekeeping. Removing numbers may discard years of experience, and removing stopwords may discard meaningful negation. Make each choice explicit.

    german_stopwords <- c(
      "der", "die", "das", "den", "dem", "des", "und", "oder", "mit",
      "für", "von", "zu", "im", "in", "auf", "ein", "eine"
    )

    clean <- preprocess_text(
      sentences,
      lowercase = TRUE,
      remove_punctuation = TRUE,
      remove_numbers = TRUE,
      stopwords = german_stopwords,
      min_token_length = 2
    )
    clean <- clean[nzchar(clean)]
    head(clean)
    #> [1] "als einer führenden privaten träger bereich stationären pflege bietet casa reha ihnen zukunfts krisensichere arbeitsplätze"
    #> [2] "über mitarbeiter versorgen rund um uhr unsere bewohner leben casa reha philosophie mensch mensch"
    #> [3] "bereits zum zweiten mal folge wurden wir als einer besten arbeitgeber deutschlands bereich gesundheit soziales ausgezeichnet"
    #> [4] "jede unserer fast einrichtungen hat einen einzigartigen stil besonderheiten standorts widerspiegelt"
    #> [5] "allen gemein sind familiäre atmosphäre modernes arbeitsumfeld hohe qualitätsstandards"
    #> [6] "seniorenpflegeheim rosenpark hemmingen bieten wir bewohnern zuhause"

## Create document features

    dtm <- document_term_matrix(
      clean,
      min_doc_freq = 2,
      max_doc_prop = 0.95
    )
    dtm
    #> <text_dtm> 41 documents x 45 terms
    #>       als an andreas auch baumert bei bereich betreuung bewohner bewohnern
    #> doc1    1  0       0    0       0   0       1         0        0         0
    #> doc2    0  0       0    0       0   0       0         0        1         0
    #> doc3    1  0       0    0       0   0       1         0        0         0
    #> doc4    0  0       0    0       0   0       0         0        0         0
    #> doc5    0  0       0    0       0   0       0         0        0         0
    #> doc6    0  0       0    0       0   0       0         0        0         1
    #> doc7    1  0       0    1       0   0       0         0        0         0
    #> doc8    0  0       0    0       0   0       0         0        0         0
    #> doc9    1  0       0    0       0   0       0         0        0         0
    #> doc10   0  0       0    0       0   0       0         0        0         0
    #> doc11   0  1       0    0       0   0       0         1        1         0
    #> doc12   0  0       0    0       0   0       0         0        0         0
    #> doc13   0  0       0    0       0   0       0         0        0         0
    #> doc14   0  0       0    0       0   0       0         1        1         0
    #> doc15   0  0       0    0       0   0       0         0        0         0
    #> doc16   0  1       0    0       0   0       0         0        0         0
    #> doc17   0  0       0    0       0   0       0         0        0         0
    #> doc18   0  0       0    0       0   0       0         0        0         0
    #> doc19   0  0       0    0       0   0       0         0        0         0
    #> doc20   0  0       0    0       0   0       0         0        0         0
    #> doc21   0  0       0    0       0   0       0         0        0         1
    #> doc22   0  1       0    1       0   1       0         0        0         0
    #> doc23   0  0       0    0       0   0       0         0        0         0
    #> doc24   1  0       0    0       0   0       1         0        0         0
    #> doc25   0  0       0    0       0   0       0         0        0         0
    #> doc26   0  0       0    0       0   0       0         0        0         0
    #> doc27   0  0       0    0       0   0       0         0        0         0
    #> doc28   0  0       0    0       0   0       0         0        0         0
    #> doc29   0  0       0    0       0   0       0         0        0         0
    #> doc30   0  0       0    0       0   0       0         0        0         0
    #> doc31   0  0       0    0       0   0       0         0        0         0
    #> doc32   0  1       0    0       0   1       0         0        0         0
    #> doc33   0  0       0    0       0   0       0         0        0         0
    #> doc34   0  0       0    0       0   1       0         0        0         0
    #> doc35   0  0       0    0       0   0       0         0        0         0
    #> doc36   0  0       0    0       0   0       0         0        0         0
    #> doc37   0  0       0    0       0   0       0         0        0         0
    #> doc38   0  0       0    0       0   0       0         0        0         0
    #> doc39   0  0       1    0       1   0       0         0        0         0
    #> doc40   0  0       0    0       0   0       0         0        0         0
    #> doc41   0  0       1    0       1   0       0         0        0         0
    #>       bieten casa dann durch einen einer einrichtung freuen gerne hemmingen
    #> doc1       0    1    0     0     0     1           0      0     0         0
    #> doc2       0    1    0     0     0     0           0      0     0         0
    #> doc3       0    0    0     0     0     1           0      0     0         0
    #> doc4       0    0    0     0     1     0           0      0     0         0
    #> doc5       0    0    0     0     0     0           0      0     0         0
    #> doc6       1    0    0     0     0     0           0      0     0         1
    #> doc7       0    0    0     0     0     0           1      1     0         0
    #> doc8       0    0    0     0     0     0           0      0     0         0
    #> doc9       0    0    0     0     0     0           0      0     0         0
    #> doc10      0    0    0     0     0     0           0      0     0         0
    #> doc11      0    0    0     0     0     0           0      0     0         0
    #> doc12      0    0    0     0     0     0           0      0     0         0
    #> doc13      0    0    0     0     0     0           0      0     0         0
    #> doc14      0    0    0     1     0     0           0      0     0         0
    #> doc15      0    0    0     0     0     0           0      0     0         0
    #> doc16      0    0    0     0     0     0           0      0     0         0
    #> doc17      0    0    0     0     0     0           0      0     0         0
    #> doc18      0    0    0     0     0     0           0      0     0         0
    #> doc19      0    0    0     0     0     0           0      0     0         0
    #> doc20      0    0    0     0     0     0           0      0     0         0
    #> doc21      0    0    0     0     0     0           0      0     0         0
    #> doc22      0    0    0     0     0     0           0      0     1         0
    #> doc23      0    0    0     1     0     0           0      0     0         0
    #> doc24      0    0    0     0     0     0           0      0     0         0
    #> doc25      0    0    0     0     0     1           1      0     0         0
    #> doc26      0    0    0     0     0     0           0      0     0         0
    #> doc27      0    0    0     0     0     0           0      0     0         0
    #> doc28      0    0    0     0     0     0           0      0     0         0
    #> doc29      0    0    0     0     0     0           0      0     0         0
    #> doc30      1    0    0     0     0     0           0      0     0         0
    #> doc31      1    0    0     0     0     0           0      0     0         0
    #> doc32      0    0    0     0     0     0           0      0     0         0
    #> doc33      0    0    0     0     1     0           0      0     0         0
    #> doc34      0    2    1     0     0     0           0      0     0         0
    #> doc35      0    0    0     0     0     0           0      1     0         0
    #> doc36      0    0    0     0     0     0           0      0     0         0
    #> doc37      0    0    0     0     0     0           0      0     0         1
    #> doc38      0    0    0     0     0     0           0      0     0         0
    #> doc39      0    0    0     0     0     0           0      0     0         0
    #> doc40      0    0    0     0     0     0           0      0     0         0
    #> doc41      0    0    1     0     0     0           0      0     1         0
    #>       hohe ihnen ihre ihrer mensch mitarbeiter neben pflege philosophie reha
    #> doc1     0     1    0     0      0           0     0      1           0    1
    #> doc2     0     0    0     0      2           1     0      0           1    1
    #> doc3     0     0    0     0      0           0     0      0           0    0
    #> doc4     0     0    0     0      0           0     0      0           0    0
    #> doc5     1     0    0     0      0           0     0      0           0    0
    #> doc6     0     0    0     0      0           0     0      0           0    0
    #> doc7     0     0    0     0      0           1     0      0           0    0
    #> doc8     0     0    0     0      0           0     0      0           0    0
    #> doc9     0     0    0     0      0           0     0      0           0    0
    #> doc10    0     0    1     0      0           0     0      0           0    0
    #> doc11    0     0    0     0      0           0     0      1           0    0
    #> doc12    0     0    0     0      2           0     0      0           1    0
    #> doc13    0     0    1     0      0           0     0      0           0    0
    #> doc14    0     0    0     0      0           0     0      1           0    0
    #> doc15    0     0    0     0      0           0     0      0           0    0
    #> doc16    0     0    0     0      0           0     0      1           0    0
    #> doc17    0     0    0     0      0           0     0      0           0    0
    #> doc18    0     0    0     0      0           0     0      0           0    0
    #> doc19    0     0    0     0      0           0     0      0           0    0
    #> doc20    0     0    0     0      0           0     0      0           0    0
    #> doc21    0     0    0     1      0           0     0      0           0    0
    #> doc22    0     0    0     0      0           0     0      0           0    0
    #> doc23    0     0    0     1      0           0     1      0           0    0
    #> doc24    0     0    0     0      0           0     0      0           0    0
    #> doc25    0     0    0     0      0           0     0      0           0    0
    #> doc26    0     0    0     0      0           0     0      0           0    0
    #> doc27    1     0    0     0      0           0     0      0           0    0
    #> doc28    0     0    0     0      0           0     0      0           0    0
    #> doc29    0     0    0     0      0           0     0      0           0    0
    #> doc30    0     0    0     0      0           0     0      0           0    0
    #> doc31    0     1    0     0      0           0     1      0           0    0
    #> doc32    0     0    1     0      0           0     0      0           0    0
    #> doc33    0     0    0     0      0           0     0      0           0    0
    #> doc34    0     0    0     0      0           0     0      0           0    2
    #> doc35    0     0    0     0      0           0     0      0           0    0
    #> doc36    0     0    0     0      0           0     0      0           0    0
    #> doc37    0     0    0     0      0           0     0      0           0    0
    #> doc38    0     0    0     0      0           0     0      0           0    0
    #> doc39    0     0    0     0      0           0     0      0           0    0
    #> doc40    0     0    0     0      0           0     0      0           0    0
    #> doc41    0     1    0     0      0           0     0      0           0    0
    #>       rosenpark seniorenpflegeheim sich sie sind sowie sozialkonzept
    #> doc1          0                  0    0   0    0     0             0
    #> doc2          0                  0    0   0    0     0             0
    #> doc3          0                  0    0   0    0     0             0
    #> doc4          0                  0    0   0    0     0             0
    #> doc5          0                  0    0   0    1     0             0
    #> doc6          1                  1    0   0    0     0             0
    #> doc7          0                  0    1   1    0     0             1
    #> doc8          0                  0    0   0    0     0             0
    #> doc9          0                  0    0   0    0     0             0
    #> doc10         0                  0    0   0    1     0             0
    #> doc11         0                  0    1   1    0     0             0
    #> doc12         0                  0    1   1    0     0             0
    #> doc13         0                  0    0   0    1     0             0
    #> doc14         0                  0    0   0    0     0             0
    #> doc15         0                  0    0   0    0     0             0
    #> doc16         0                  0    0   0    0     0             0
    #> doc17         0                  0    0   0    0     1             0
    #> doc18         0                  0    0   0    0     0             0
    #> doc19         0                  0    0   0    0     0             0
    #> doc20         0                  0    0   1    1     0             0
    #> doc21         0                  0    0   1    0     0             0
    #> doc22         0                  0    0   1    1     0             0
    #> doc23         0                  0    1   1    0     0             0
    #> doc24         0                  0    0   0    0     0             0
    #> doc25         0                  0    0   0    0     0             0
    #> doc26         0                  0    0   0    0     0             0
    #> doc27         0                  0    0   0    0     0             0
    #> doc28         0                  0    0   0    0     0             0
    #> doc29         0                  0    0   0    0     0             0
    #> doc30         0                  0    0   0    0     0             0
    #> doc31         0                  0    0   0    0     1             0
    #> doc32         0                  0    2   1    0     0             0
    #> doc33         0                  0    0   1    0     0             0
    #> doc34         0                  0    1   1    0     0             0
    #> doc35         0                  0    0   1    0     0             0
    #> doc36         1                  1    0   0    0     0             1
    #> doc37         0                  0    0   0    0     0             0
    #> doc38         0                  0    0   0    0     0             0
    #> doc39         0                  0    0   0    0     0             0
    #> doc40         0                  0    0   1    0     0             0
    #> doc41         0                  0    0   0    0     0             0
    #>       stationären täglich uns unsere unserer unter wir über
    #> doc1            1       0   0      0       0     0   0    0
    #> doc2            0       0   0      1       0     0   0    1
    #> doc3            0       0   0      0       0     0   1    0
    #> doc4            0       0   0      0       1     0   0    0
    #> doc5            0       0   0      0       0     0   0    0
    #> doc6            0       0   0      0       0     0   1    0
    #> doc7            0       0   0      1       0     0   0    0
    #> doc8            0       0   0      0       0     0   0    0
    #> doc9            0       0   0      0       0     0   0    0
    #> doc10           0       0   0      0       0     0   0    0
    #> doc11           0       1   0      2       0     0   0    0
    #> doc12           0       0   0      0       1     0   0    0
    #> doc13           0       0   0      0       0     0   0    0
    #> doc14           0       0   0      0       1     0   0    0
    #> doc15           0       0   0      0       0     0   0    0
    #> doc16           0       0   0      0       0     0   0    0
    #> doc17           0       0   0      0       0     0   0    0
    #> doc18           0       0   0      0       0     0   0    0
    #> doc19           0       0   0      0       0     0   0    0
    #> doc20           0       0   0      0       0     0   0    0
    #> doc21           0       1   0      0       0     0   0    0
    #> doc22           0       0   0      0       0     0   0    0
    #> doc23           0       0   0      0       0     0   0    0
    #> doc24           0       0   0      0       0     0   0    0
    #> doc25           1       0   0      0       0     0   0    0
    #> doc26           0       0   0      0       0     0   0    0
    #> doc27           0       0   0      0       0     0   0    0
    #> doc28           0       0   0      0       0     0   0    0
    #> doc29           0       0   0      0       0     0   0    0
    #> doc30           0       0   0      0       0     0   1    0
    #> doc31           0       0   0      0       0     1   1    0
    #> doc32           0       0   1      0       0     0   1    0
    #> doc33           0       0   0      0       0     0   0    0
    #> doc34           0       0   0      0       0     1   0    1
    #> doc35           0       0   1      0       0     0   1    0
    #> doc36           0       0   0      0       0     0   0    0
    #> doc37           0       0   0      0       0     0   0    0
    #> doc38           0       0   0      0       0     0   0    0
    #> doc39           0       0   0      0       0     0   0    0
    #> doc40           0       0   0      0       0     0   0    0
    #> doc41           0       0   0      0       0     0   0    0
    #> attr(,"binary")
    #> [1] FALSE

Rows represent documents, columns represent terms, and cells contain counts. Use `binary = TRUE` when presence is more appropriate than frequency.

TF-IDF increases the weight of terms that are frequent in a particular document but uncommon across the collection.

    weighted <- tf_idf(dtm)
    keywords <- extract_keywords(dtm, n = 3)
    head(keywords, 12)
    #>    document rank        term    weight
    #> 1      doc1    1 stationären 0.4548822
    #> 2      doc1    2     bereich 0.4189219
    #> 3      doc1    3        casa 0.4189219
    #> 4      doc2    1      mensch 0.8086794
    #> 5      doc2    2 mitarbeiter 0.4043397
    #> 6      doc2    3 philosophie 0.4043397
    #> 7      doc3    1     bereich 0.8378438
    #> 8      doc3    2       einer 0.8378438
    #> 9      doc3    3         als 0.7364775
    #> 10     doc4    1       einen 1.8195287
    #> 11     doc4    2     unserer 1.6756876
    #> 12     doc5    1        hohe 1.8195287

## Explore similarity

Cosine similarity compares the orientation of two feature vectors while reducing the influence of document length.

    similarity <- cosine_similarity(weighted)
    round(similarity[1:min(5, nrow(similarity)),
                     1:min(5, ncol(similarity))], 2)
    #>      doc1 doc2 doc3 doc4 doc5
    #> doc1 1.00  0.2 0.53    0    0
    #> doc2 0.20  1.0 0.00    0    0
    #> doc3 0.53  0.0 1.00    0    0
    #> doc4 0.00  0.0 0.00    1    0
    #> doc5 0.00  0.0 0.00    0    1

## Train a classifier

For a compact illustration, use synthetic documents with known labels.

    training_text <- c(
      "analyze data statistical model",
      "build predictive model data",
      "create dashboard analyze metrics",
      "provide nursing care patient",
      "support patient clinical care",
      "coordinate nurse patient treatment"
    )
    training_labels <- c("data", "data", "data", "care", "care", "care")

    training_dtm <- document_term_matrix(training_text)
    model <- fit_naive_bayes(training_dtm, training_labels, laplace = 1)
    model
    #> <text_nb> Multinomial Naive Bayes
    #> Classes: data, care
    #> Terms: 18

    predicted <- predict(model, training_dtm)
    classification_metrics(training_labels, predicted, positive = "data")
    #>   n true_positive false_positive true_negative false_negative accuracy
    #> 1 6             3              0             3              0        1
    #>   balanced_accuracy precision recall specificity f1
    #> 1                 1         1      1           1  1

This training-set result demonstrates mechanics, not generalization. The next vignette shows out-of-sample evaluation.

## Reproducible research checklist

- Preserve IDs and labels through every transformation.
- Define the unit of analysis before tokenization.
- Fit vocabulary and feature transformations on training data only.
- Prevent sentences from the same source document leaking across folds.
- Report preprocessing, class balance, tuning, and all evaluation metrics.
- Inspect errors and subgroup performance, not only aggregate accuracy.