What a 2,000-year-old poem taught me about machine learning

I downloaded a structured digital copy of all 1,330 Kurals.

Each couplet contains two lines. So my first attempt was obvious:

Split each line at the spaces.

Now I have the words.

Except I didn't.

The spaces in the source text often represented the poem's metre, not linguistic word boundaries.

Consider part of Kural 46:

போஒய்ப் பெறுவ தெவன்?

A computer splitting on spaces sees:

பெறுவ
தெவன்

But linguistically those boundaries don't represent the intended words.

The relevant forms are:

பெறுவது + எவன்

The poetic structure has effectively cut across the linguistic boundary.

And the opposite problem occurs too.

In the first Kural we encounter:

எழுத்தெல்லாம்

which contains:

எழுத்து + எல்லாம்

“letter” + “all”.

Sometimes my dataset split one word into two.

Sometimes it glued two words together.

And suddenly the first lesson of the project became obvious:

How you divide your data determines what your system can ever discover from it.

In NLP we call this tokenisation.

It sounds like preprocessing.

It isn't.

It is part of the model of reality you are giving the computer.

If the tokens are wrong, everything downstream inherits the mistake.

Once I had words, I needed their roots.

For that I used ThamizhiMorph, a finite-state morphological analyser for Tamil.

It is a serious linguistic tool.

There was just one problem.

It was designed for modern Tamil.

The Thirukkural isn't modern Tamil.

Running the analyser against the original metrical text produced analyses for only about one quarter of the word occurrences.

At first that sounds like a bad tool.

It isn't.

I was asking it to operate outside the world it was designed for.

For example, modern Tamil uses:

உலகம் — ulagam — world

The Thirukkural can use the older:

உலகு — ulagu

Similarly, the analyser understands the modern Tamil verb செய் — “do” — but can struggle with classical grammatical forms such as:

செயின்

meaning approximately “if one does”.

Even something as ordinary as நட்பு, friendship, wasn't in the analyser's vocabulary.

This is a classic machine-learning problem:

domain mismatch.

A system trained or engineered for one distribution of language does not automatically generalise to another.

Modern Tamil and Classical Tamil may be the same language historically, but computationally they can represent very different worlds.

It's similar to expecting a system trained on contemporary English to effortlessly understand Shakespeare.

The same toolkit also ships with “guesser” components, designed to take a stab at words its dictionaries don't know.

They looked dramatically better.

Between them, they returned something for around 90% of the words.

Wonderful.

Until I looked at the answers.

For:

கைவிடுவார் — kaividuvār — those who abandon

one guesser effectively returned:

root = கைவிடுவார்

and classified it as a noun.

But the actual verbal root is:

கைவிடு — kaividu — abandon

The system had produced an answer without really solving the problem.

That taught me a distinction that sounds obvious but is surprisingly easy to forget when building automated systems:

Coverage is not correctness.

A system answering 90% of your questions is not necessarily better than one answering 50%.

You need to know:

How often does it answer?

and separately:

When it answers, how often is it right?

Then something happened that changed the project.

I found a word-separated edition of the Thirukkural prepared by Tamil scholar Dr. P. Pandiyaraja, published on his Tamil Concordance site (tamilconcordance.in).

I aligned that edition character-by-character against my original source while preserving the original text unchanged.

Then I ran exactly the same morphological analyser again.

I didn't improve the algorithm.

I didn't train anything.

I didn't add a neural network.

Recognition went from roughly:

26% → 48%

The same tool suddenly recognised almost twice as much of the text.

Consider this line from Kural 82.

My original source had:

மருந்தெனினும் வேண் டற்பாற் றன்று

The scholar's segmentation represented it as:

மருந்து எனினும் வேண்டற்பாற்று அன்று

Same poem.

Essentially the same letters.

Different boundaries.

And the machine suddenly became dramatically better at understanding it.

This may have been my favourite lesson from the entire project:

Before building a cleverer model, make sure you haven't given the existing model bad data.

Even after fixing the word boundaries, more than half the text remained unresolved.

So I began adding other sources of evidence.

I used:

• the modern morphological analyser;
• hand-written rules for Classical Tamil morphology;
• Tamil Wiktionary;
• historical Tamil lexicons and nikandus.

No single source was authoritative enough.

But together they could provide evidence.

Take:

அவா — avā — greed / craving

It appears repeatedly in the Thirukkural but wasn't recognised by the modern analyser.

Tamil Wiktionary recognises it.

And the roughly 10th-century Piṅgala Nikandu contains it too.

Now I have two witnesses from very different periods pointing in the same direction.

This resembles an idea used in machine learning called weak supervision.

Instead of possessing one perfect source of labels, you combine several imperfect sources while retaining information about where each label came from and how much you trust it.

So I began keeping confidence levels.

Not:

“The root is X.”

But:

“Source A proposes X.”
“Rule B supports X.”
“Historical source C attests X.”
“Human validation still required.”

That distinction would soon become extremely important.

One word taught me not to trust consensus blindly:

தரும் — tharum

In context it means something like:

will give / will bring

Its verbal root is:

தா — thā — give

But one of my rules produced:

தரு

And Wiktionary appeared to support it.

Two sources agreed.

Excellent.

Except தரு can mean tree.

The machines had agreed on the wrong analysis.

Then came:

உண்டோ

in the famous line:

அன்பிற்கும் உண்டோ அடைக்கும் தாழ்

“Is there a latch that can shut in love?”

The system connected உண்டோ with:

உண் — eat

Morphologically plausible.

Semantically absurd.

Then:

விசும்பின்

in:

விசும்பின் துளி வீழின் அல்லால்

Here விசும்பு refers to the sky.

But the ending -இன் is ambiguous. It can participate in grammatical structures that make the word look verbal.

The machine chose the wrong one.

The spelling wasn't the problem.

The morphology wasn't necessarily impossible.

Meaning was the problem.

And meaning depended on reading the verse.

This led me to the most important limitation of the automatic pipeline.

Consider:

அல்லல்

A rule can derive one grammatical interpretation.

A dictionary can suggest another.

But in the Kural's context it means:

distress / suffering

Or consider:

பலவும்

A dictionary lookup can lead towards பலவு, jack tree.

But in context it means essentially:

many, with an attached particle.

A computer can generate candidates.

It can rank them.

It can tell me that a morphological analysis is technically possible.

But at some point somebody has to read the actual Kural and say:

That isn't what Valluvar means here.

This is why the next stage of the project requires a Classical Tamil expert.

Not to replace computation.

To create the answer key against which computation can be measured.

In machine learning we'd call that a gold standard.

And I don't have one yet.

That matters.

As coverage improved, another familiar data problem appeared.

The long tail.

Of the thousands of forms that remained unresolved, roughly three quarters appeared only once.

Words such as:

சிவிகை — sivigai — palanquin

or

நெருநல் — nerunal — yesterday

might occur only once in the entire work.

Each needs investigation.

Each may require historical linguistic knowledge.

And resolving each one changes the overall coverage by almost nothing.

This is the economics of the long tail.

The common cases are comparatively cheap.

The rare cases become progressively expensive.

Getting from 25% to 50% can be easier than getting from 80% to 90%.

This phenomenon appears everywhere in real-world machine learning.

And this is where the project suddenly returned to my original question about bias.

The words the system couldn't recognise weren't a random sample of Valluvar's vocabulary.

They disproportionately included:

• rare words;
• archaic forms;
• Classical Tamil constructions;
• meanings that have shifted over time.

That creates a dangerous possibility.

Imagine that my tools understand modern-looking words extremely well but systematically fail on older semantic categories.

I build my lexical map.

The map shows modern-looking vocabulary dominating.

I conclude:

“Valluvar preferred these concepts.”

But perhaps Valluvar didn't.

Perhaps my measurement system preferred the words it could recognise.

The apparent pattern would have been introduced by my tools.

Which is particularly awkward when the entire experiment is supposed to measure bias.

Even when a dictionary recognises a word, another problem remains.

Meaning changes.

Take:

ஆக்கம் — ākkam

A modern English gloss (Tamil Wiktionary's, for example) can suggest things like:

“creation”, “construction” or “production”.

Those aren't necessarily the senses relevant to the Thirukkural.

In context, the meaning can be closer to:

prosperity, gain, advancement.

If I automatically map an ancient Tamil word through a modern bilingual dictionary, I haven't simply “translated” Valluvar.

I've introduced another interpreter.

And suddenly the pipeline looks like this:

Valluvar

↓

edition

↓

tokenisation

↓

morphological analyser

↓

dictionary

↓

modern English gloss

↓

my statistics

Every arrow represents a decision.

Every decision can alter the thing I'm supposedly measuring.

At this point I became obsessive about provenance.

The original text is never overwritten.

Tools are versioned.

Sources are checksummed.

Rules are numbered.

Every proposed root retains the source that produced it.

Every run is logged.

This sounded excessive until rebuilding one dictionary tool from its source produced something that was not quite identical to its published version: a few extra internal paths that shouldn't have been there.

Eventually I traced it to a typo in the tool's own word list:

நிப்பாட்டு்

with an extra Unicode mark.

It happened not to affect a single Kural word; the two versions gave identical answers on the whole text.

But that wasn't the point.

Two supposedly identical tools were not, in fact, identical.

Without versioning and provenance, I wouldn't have noticed, let alone known why.

That's reproducibility.

And it is one of the least glamorous and most important parts of serious computational work.