Athanasios Bonis Who Wrote This?

Genre and authorship classification · Research, in progress

Can a computer tell who wrote a text, or what kind of page it is?

Athanasios Bonis Rebuild of a 2019 University of the Aegean thesis Status: classical baselines complete, pretrained models next

Summary

This project compares methods for two text-sorting tasks: identifying the genre of a web page and attributing a text to one of several candidate authors when the topic changes. Every method runs under one protocol in which nothing is learned from test data. The first round covers the classical methods. Character n-grams with a linear SVM reach 97.2% accuracy on the 7-genre corpus, but only 58.2 macro-F1 on cross-domain authorship (PAN-18), where the best score falls to 46.5 for Polish. The next rounds test whether pretrained models and LLMs close that gap.

§1

The two tasks

Both are closed-set classification. The model receives a text and must assign it one label from a fixed list.

Web genre identification

What kind of page is this?

Blog, online shop, FAQ, news front page, personal home page, search results. Search engines and dataset builders use genre to filter and organise the web.

Cross-domain authorship attribution

Which candidate wrote this?

Each candidate has seven known texts on one topic. The unknown texts are on a different topic, so the model has to follow the writer and ignore the subject.

Table 1. Corpora and evaluation protocol.
CorpusTaskTextsLabelsProtocol
7-genre (Santini)Web genre1,4007 genres, 200 eachStratified 10-fold CV × 3 seeds
KI-04 (Meyer zu Eissen & Stein)Web genre1,2058 genres, 126–205 eachStratified 10-fold CV × 3 seeds
PAN-18 (dev)Authorship20 problems5–20 candidates, 5 languagesOfficial known / unknown split
§2

Measuring style with character n-grams

The oldest strong baseline cuts a text into overlapping slices of three characters and counts them. "the cat" becomes the, he·, e·c, ·ca, cat. The counts form a profile that captures function words, punctuation habits, spelling and word endings, which are less tied to topic than vocabulary is.

–profile similarity (cosine)

Text A · most frequent 3-grams

    Text B · most frequent 3-grams

      Figure 1. Character 3-gram profiles, computed in your browser. Highlighted slices appear in both texts. · marks a space. Edit either text to update the score.
      §3

      Methods

      Methods are grouped by how much they know before they see our data. All tiers run on the same splits and the same scorer.

      Table 2. Methods roadmap.
      TierMethodsComputeStatus
      0. ClassicalOfficial PAN-18 baseline (char 3-grams + SVM); TF-IDF char 1–4-grams + SVM; fastTextCPUDone
      1. Frozen embeddingsMultilingual sentence embeddings + linear classifierCPU / free GPUNext
      2. Fine-tuned encodersModernBERT (en); XLM-R, mDeBERTa (multilingual); long-document handlingFree GPUPlanned
      3. LLMsZero- and few-shot prompting; known texts as in-context examples for PANAPIPlanned
      4. HistoricalULMFiT re-implemented under the leakage-free protocolFree GPUOptional
      §4

      Evaluation protocol

      A score is only meaningful if the model never saw the test texts while learning. When it does, the error is called data leakage, and it inflates results. These rules apply to every model and are enforced in code where possible.

      1. Train-only learningEnforced

        Vocabularies, scalers, embeddings, language-model adaptation and prompt examples come from the training part of each split only.

      2. No tuning on test dataEnforced

        Hyperparameters live in versioned configs. Any tuning uses validation data carved from the training split.

      3. Official metricsEnforced

        Accuracy and macro-F1 for every split. PAN-18 uses macro-F1 over candidates, averaged over problems, also per language.

      4. VarianceEnforced

        Mean ± standard deviation over splits, with three CV seeds on the genre corpora.

      5. SignificanceTo do

        Paired Wilcoxon signed-rank tests over identical splits, corrected for multiple comparisons.

      6. CostEnforced

        Fit and predict time per split, hardware, and API cost where relevant.

      7. ProvenanceEnforced

        Every run records its config, git commit and package versions.

      Training (9 folds) Test (1 fold)
      Figure 2. Stratified 10-fold cross-validation. Each row is one round. Every page is tested exactly once per seed, and the reported score is the mean over 10 rounds × 3 seeds = 30 splits.
      §5

      Results: classical baselines

        Figure 3. Mean score with ±1 standard deviation over splits (30 for the genre corpora, 20 problems for PAN-18). Genre panels show accuracy. The authorship panel shows macro-F1, the official PAN-18 measure, which weights every candidate equally.
        Show Figure 3 as a table

        97.2%

        Character n-grams + SVM sort 7-genre pages almost perfectly. KI-04 is harder, at 85.0%.

        58.4F1

        Cross-topic authorship stays hard. The two n-gram methods tie, and both are weakest in Polish.

        13.4F1

        fastText fails on authorship. Seven texts per candidate are too few for it to learn word-level signals.

        Table 3. PAN-18 macro-F1 by language, mean over the four problems per language. Best per language in bold.
        MethodENFRITPLES
        PAN-18 baseline69.758.560.541.961.5
        Char 1–4-grams + SVM74.156.657.046.556.8
        fastText16.413.16.223.47.6
        Sanity check

        Our copy of the official PAN-18 baseline reproduces all 1,315 answers of the published baseline, which confirms the data loading and scoring.

        §6

        Background

        1. 2019

          The original thesis

          ULMFiT was applied to both tasks. It reported 99.1% on 7-genre and 92.6% on KI-04, and losses against n-gram baselines on authorship.

        2. 2026

          Review

          Test accuracy exceeded validation accuracy on both genre corpora, which suggests the language model was adapted on text that included test pages. PAN-18 numbers were inconsistent, an English-only model was applied to four other languages, and most code was missing.

        3. Now

          Rebuild

          The thesis is motivation only. Every number on this page comes from code in the repository, re-run under the protocol in §4.

        4. Next

          Pretrained models and LLMs

          Test tiers 1–3 under the same protocol and answer the main question: has pretraining overtaken character n-grams on cross-topic authorship?

        §7

        Terms

        Character n-gram
        A slice of n consecutive characters. "the" and "he·" are 3-grams.
        Data leakage
        Test material reaching the training step, which inflates scores.
        Cross-validation
        Rotating which part of the data is the test set, then averaging.
        Accuracy
        The share of predictions that are correct.
        Macro-F1
        F1 computed per class, then averaged, so rare classes count as much as common ones.
        Cross-domain
        Training and test texts come from different topics.
        Pretrained model
        A model that learned general language from large text collections before seeing this task.
        PAN
        A shared-task series on authorship and stylometry, held at CLEF.
        §8

        Sources

        1. A. Bonis and G. Dimopoulos. Text Categorization Based on Fine-tuning of Pre-trained Language Models. Undergraduate thesis, University of the Aegean, 2019. Library record.
        2. M. Kestemont et al. Overview of the Author Identification Task at PAN-2018: Cross-domain Authorship Attribution and Style Change Detection. CLEF 2018.
        3. S. Meyer zu Eissen and B. Stein. Genre Classification of Web Pages. KI 2004.
        4. M. Santini. The 7-genre web page corpus.
        5. Code, configs and full results: github.com/athanbonis/genre-and-authorship-bench.