Open-source software

Miðeind has released a number of open-source software packages for Icelandic language technology. The packages are written in Python and released under the MIT license, so anyone is free to use them, in research as well as in their own software, provided the license's simple terms are followed. This is an overview of the main packages; more detailed descriptions, examples and instructions can be found on GitHub.

Tokenizer converts text into a sequence of tokens, i.e. words, punctuation, numbers, dates, email addresses, URLs and more, and also splits it into sentences. It handles abbreviations and dates within sentences, which would otherwise confuse sentence splitting. Tokenization is the first step in many language technology tasks, such as tagging, parsing, spell checking and corpus processing.

pip install tokenizer

BinPackage makes the Database of Icelandic Morphology (DMII/BÍN) available in Python. The package contains about 356,000 headwords and over 3.7 million word forms. It can look up a word form and find its headword, word class and inflection, as well as other inflected forms of the same word, such as the dative plural. It also analyzes compound words that are not in DMII. The data is bundled with the package, so no separate download or database is needed.

pip install islenska

Greynir parses Icelandic sentences into parse trees. From these, it can find headwords, tag word classes and inflect noun phrases into the desired case, for example turning “tveir brimsaltir pokar af poppi” into “tveimur brimsöltum pokum af poppi”. The parse trees can also be used to extract information from text, such as people, titles, facts and opinions. Greynir is based on hand-written grammar rules for Icelandic and uses Tokenizer and BinPackage.

pip install reynir

GreynirCorrect checks and corrects Icelandic text. It can correct text quickly, word by word, for example spelling errors, repeated words and misspelled phrases. It can also analyze the text grammatically and point out errors such as a wrong case. Every error comes with an explanation and a suggested correction. GreynirCorrect is built on Greynir.

pip install reynir-correct

Icegrams stores the frequencies of about 78 million trigrams, i.e. sequences of three words, from the Icelandic Gigaword Corpus. The package gives the frequency and probability of individual words, word pairs and trigrams, and finds the words most likely to come next. Lookups are very fast. Icegrams is useful for spell checking, text prediction and statistical text analysis, among other things.

pip install icegrams

IceBERT-PoS tags the word class and morphological features of every word in Icelandic text using the IceBERT-PoS language model. The tags follow the tagset of the Icelandic Frequency Dictionary (IFD), and every word keeps its position in the original text. The text is first tokenized with Tokenizer.

pip install "icebert-pos[torch] @ git+https://github.com/mideind/IceBERT-PoS.git"