Skip to content
/ slovo Public

Slovo: Russian Sign Language Dataset and Models

Notifications You must be signed in to change notification settings

hukenovs/slovo

Repository files navigation

Slovo - Russian Sign Language Dataset

We introduce a large-scale video dataset Slovo for Russian Sign Language task. Slovo dataset size is about 16 GB, and it contains 20400 RGB videos for 1000 sign language gestures from 194 singers. Each class has 20 samples. The dataset is divided into training set and test set by subject user_id. The training set includes 15300 videos, and the test set includes 5100 videos. The total video recording time is ~9.2 hours. About 35% of the videos are recorded in HD format, and 65% of the videos are in FullHD resolution. The average video length with gesture is 50 frames.

For more information see our paper - arXiv.

Downloads

Downloads Size (GB) Comment
Slovo ~16 Trimmed HD+ videos by (start, end) annotations
Origin ~105 Original HD+ videos from mining stage
360p ~13 Resized original videos by min_side = 360
Landmarks ~1.2 Mediapipe hand landmark annotations for each frame of trimmed videos

Also, you can download Slovo from Kaggle.

Annotation file is easy to use and contains some useful columns, see annotations.csv file:

attachment_id user_id width height length text train begin end
0 de81cc1c-... 1b... 1440 1920 14 привет True 30 45
1 3c0cec5a-... 64... 1440 1920 32 утро False 43 66
2 d17ca986-... cf... 1920 1080 44 улица False 12 31

where:

  • attachment_id - video file name
  • user_id - unique anonymized user ID
  • width - video width
  • height - video height
  • length - video length
  • text - gesture class in Russian Langauge
  • train - train or test boolean flag
  • begin - start of the gesture (for original dataset)
  • end - end of the gesture (for original dataset)

For convenience, we have also prepared a compressed version of the dataset, in which all videos are processed by the minimum side min_side = 360. Download link - slovo360p. Also, we annotate trimmed videos by using MediaPipe and provide hand keypoints in this annotation file.

Models

We provide some pre-trained models as the baseline for Russian sign language recognition. We tested models with frames number from [16, 32, 48], and the best for each are below. The first number in the model name is frames number and the second is frame interval.

Model Name Model Size (MB) Metric ONNX TorchScript
MViTv2-small-16-4 140.51 58.35 weights weights
MViTv2-small-32-2 140.79 64.09 weights weights
MViTv2-small-48-2 141.05 62.18 weights weights
Swin-large-16-3 821.65 48.04 weights weights
Swin-large-32-2 821.74 54.84 weights weights
Swin-large-48-1 821.78 55.66 weights weights
ResNet-i3d-16-3 146.43 32.86 weights weights
ResNet-i3d-32-2 146.43 38.38 weights weights
ResNet-i3d-48-1 146.43 43.91 weights weights

SignFlow models

Model Name Desc ONNX Params
SignFlow-A 63.3 Top-1 Acc on WLASL-2000 (SOTA) weights 36M
SignFlow-R Pre-trained on ~50000 samples, has 267 classes, tested with GigaChat (as-is and context-based modes) weights 37M

Demo

usage: demo.py [-h] -p CONFIG [--mp] [-v] [-l LENGTH]

optional arguments:
  -h, --help            show this help message and exit
  -p CONFIG, --config CONFIG
                        Path to config
  --mp                  Enable multiprocessing
  -v, --verbose         Enable logging
  -l LENGTH, --length LENGTH
                        Deque length for predictions


python demo.py -p <PATH_TO_CONFIG>

demo

Authors and Credits

Citation

You can cite the paper using the following BibTeX entry:

@inproceedings{kapitanov2023slovo,
    title={Slovo: Russian Sign Language Dataset},
    author={Kapitanov, Alexander and Karina, Kvanchiani and Nagaev, Alexander and Elizaveta, Petrova},
    booktitle={International Conference on Computer Vision Systems},
    pages={63--73},
    year={2023},
    organization={Springer}
}

Links

License

Creative Commons License
This work is licensed under a variant of Creative Commons Attribution-ShareAlike 4.0 International License.

Please see the specific license.