Natural Language Text Classifier

This is a demonstration of a Natural Language Processing (NLP) algorithm to determine the natural language (English, French, etc) that a document is written in.

I leaned of this algorithm from Louis Monier, who said he originally used this in AltaVista. The algorithm seems to be ~90% accurate with 100 or more characters of text, and is no more accurate after ~2000 characters. It's currently limited to 11 common European languages, but could be trained to identify more.

To try the code, either enter a URL or the text of a document below and click 'Submit'.

Test Links for known languages: en nl fr de it pt es
Test Links for unknown languages: pl ru ja

Input

Wikipedia, the free encyclopedia Main Page From Wikipedia, the free encyclopedia Jump to: navigation, search Welcome to Wikipedia, the free encyclopedia that anyone can edit. 3,405,746 articles in English Arts Biography Geography History Mathematics Science Society Technology All portals Today's featured article Tokyo Mew Mew is a Japanese shōjo manga series written by Reiko Yoshida and illustrated by Mia Ikumi. It was originally serialized in Nakayoshi from September 2000 to February 2003, and later published in seven tankōbon volumes by Kodansha from February 2001 to April 2003. It focuses on five girls infused with the DNA of rare animals that gives them special powers and allows them to transform into "Mew Mews". Led by Ichigo Momomiya, the girls protect the earth from aliens who wish to "reclaim" it. The series was quickly adapted into a fifty-two episode anime series by Studio Pierrot. It debuted in Japan on April 6, 2002, on both TV Aichi and TV Tokyo; the final episode aired on March 29, 2003. A two-volume sequel to the manga, Tokyo Mew Mew a la Mode, was serialized in Nakayoshi from April 2003 to February 2004. The sequel introduces a new Mew Mew, Berry Shirayuki, who becomes the temporary leader of the Mew Mews whilst Ichigo is on a trip to England. Two video games were also created for the series. (more...) Recently featured: Canadian federal election, 1957 – Hastings Ismay – Night Archive – By email – More featured articles... Did you know... From Wikipedia's newest articles: ... that soprano Erminia Frezzolini (pictured) created the title role in Verdi's opera Giovanna d'Arco at La Scala in 1843 opposite her husband, Antonio Poggi, as Charles VII of France? ... that to India's dismay, China's Zangmu Dam will be the first on the Brahmaputra River? ... that according to folklore, dwarfs guard treasures hidden in Mecklenburg-Vorpommern's megalith tombs? ... that competitors of the Hamar newspaper Demokraten (later Arbeideren) refused to mention it by name in their columns? ...

Output

The natural language is en, with margin of confidence 0.324. (Took 9.276 milliseconds.)
The natural language is en, with confidence 0.284. (Took 23.052 milliseconds.)
Note: Natural languages known about are (it, fr, de, sv, fi, es, pt, el, da, nl, en) and confidence ranges from 0.0 to 1.0.