I’ve been to quite a few of these now – gatherings of geeks from around the world who sit around, surely over-caffeinated, for a day or two days or three days, and discuss and debate how they are going to save the world. That may come off as cynical, but, in fact, I write it in complete admiration, because we all discuss and debate how to save the world, but so few of us ask, OK, so what’s the next step.

My main goal for tomorrow is to write a comprehensible post on why open translation tools are important for both open source software and open content.

I know, I know, that last sentence probably reads like Latin to most of y’all, but I’ll do my best to explain what open source software and what open content is as well.

So what are translation tools? For most people on the internet, the answer is Google Translate. We see something on the web in a language that we don’t know and so we cut and paste the text into Google Translate and let it do its magic. How does that ‘magic’ happen? Two ways. First is standard grammar-based machine translation which I’ll go over below, thanks to an informative session by Fran Tyers of Apertium. The second way is through “statistical machine translation.” What this means is that Google Translate has a gigantic corpus, or database, of texts which have already been translated from one language to another. In this case, Google uses the translations of documents of the United Nations and it stores each of the translated phrases and sentences. So, if you enter in a sentence into Google Translate that it has already seen in a document translated by a professional UN translator, then it will simply offer you that exact translation word for word. If it can’t find the exact sentence or phrase, then it will use grammar-based rules to translate your text.

Let’s go over the steps that Google Translate and other machine translators process when translating your text. For the session led by Fran, we decided to translate the Serbian sentence “Velika reka teče kroz grad” into English. Here is a flow chart of that process (click on it to see it at full resolution). Below the flow chart, I’ll give more context to each step.

Step 1: Select source and target languages of translation

In this case: Serbian > English

Step 2: Write phrase or sentence to be translated:

“Velika reka teče kroz grad”

Step 3: Source language analysis

The machine then identifies the “grammatical features” of each word. There are two types of grammatical features:

1.) Parts of speech (ie. adjective, noun, verb, pronoun, etc.)
2.) Sub-categories (ie. gender of word, case, singular or plural, conjugation, person etc.)

The combination of these “grammatical features” and the lemma of the word (that is, the ‘natural form’ of the word) is called a “lexical unit”.

Step 4: Choosing an analysis

The computer then uses probability based on word analysis and linguistic rules to choose a single lexical unit for each word. That is, based on statistics, it decides whether the word is feminine/masculine, plural/singular, etc.

Step 5: Transfer the grammar from source to target language

The machine then looks at the ‘lemma‘ of each word in the bilingual dictionary. The ‘lemma’ of each word is the so-called ‘citation form’ of that word. So, for example, the lemma of went is ‘go’. In the case of our Serbian sentence, the lemma of ‘velika’ is ‘veliki’. This is the form of the word you would look up in a dictionary.

Next, the machine checks its glossary for possible translations of the lemma of the word you are translating. Possible translations of “veliki”, for example, are ‘big’, ‘large’, and ‘great’. Apertium uses the most general translation. Other systems have probability-based methods of choosing which possible translation to use.

After translating each word, the machine then looks to see if re-ordering of the words is necessary. For example, from Spanish to English “la pelota blanca” would be rearranged to “the white ball” rather than left as “the ball white”. In this case, however, no re-ordering is necessary.

Step 6: Final translation

The final translation is “Big river passes through city.”

Step 7: Post-editing

The machine translation in this example is pretty close. However, in the Serbo-Croatian language articles do not exist. A human would therefore need to add either “the” or “a” before each noun.

The big river passes through the city.

So there you go: all the steps that a machine translator takes while doing its ‘magic’. You can learn more about machine translation on the event wiki here and here.

But machine translation is just one very small component of how both content and software is made available in multiple languages. Next post I’ll explain what is open software and open content and why open translation is important to both.