Evaluation framework

Setup

The evaluation framework is distributed together with all systems and requirements as a Docker image, which can be pulled like so:

$ docker pull registry.gitlab.com/frankier/stiff:latest

And run like so:

$ docker -v /path/to/working/dir/:/work/ run python eval.py /work/results.json /work/eurosense.eval/

For CUDA accelerated experiments, you can use nvidia-docker. For running in shared computing environments, in which you don't have root access, I recommend udocker.

You can also set up requirements manually. See the Dockerfile for the list of commands to run.

Evaluation corpus

The one requirement which is not included in the Docker image is the evaluation corpus. Please follow the instructions in the STIFF README.

Filtering experiments with eval.py

You can run a subset of experiments by passing filters to eval.py, e.g.

$ python eval.py /work/results.json /work/eurosense.eval/ Knowledge 'Cross-lingual Lesk' mean=pre_sif_mean

Making tables with table.py

You can make LaTeX tables with table.py, e.g.

$ python table.py results.json --filter='Knowledge;Cross-lingual Lesk' --table='use_freq;vec:fasttext,numberbatch,double;mean expand;wn_filter'

Licenses

This project is licensed under the Apache v2 license. The code in ukb-eval is vendorized from UKB, and therefore licensed under the GPL. The scorer in support/scorer is under an unknown license, possibly public domain.

See also

• STIFF: Automatically created sense tagged corpus of Finnish and corpus wrangling tools.
• STIFF-explore: Some exploratory coding related to STIFF.
• finn-man-ann: Small, Finnish language, manually annotated word sense corpus.
• FinnTK: Simple, high-level toolkit for Finnish NLP, mainly providing convenience methods for, and gluing together, other tools.
• extjwnl_fiwn: Java code to make extjwnl interoperate with FinnWordNet.
• FinnLink: Link between FinnWordNet and Finnish Propbank created by joining with PredicateMatrix.
• finn-sense-clust: Sense clusterings of FinnWordNet.

Forks/fixes

• ItMakeseSense: ItMakesSense fork to support FiWN for use by finn-wsd-eval
• AutoExtend: AutoExtend fork to support FiWN and ConceptNet Numberbatch
• babelnet-lookup: babelnet-lookup fork to obtain BABEL2WN_MAP.
• FinnWordNet: Temporary fixes to FinnWordNet 2.0.
• Eurosense: Attempted fixes to Eurosense.