SMILE II: Automated sign language recognition at sentence level

Some of the challenges in the project are: (a) lack of DSGS resources (e.g. language corpora); (b) continuous sign language contains both manual information (information on the hands/arms) and non-manual information (information on the torso, head, and face), and almost all work on recognition to date has focused on manual information only; (c) lack of standardized instruments for DSGS assessment; and (d) lack of automatic methods to assess continuous sign language productions at both the manual and non-manual level.

To achieve the ambitious goals of the project, researchers from different fields will collaborate across four different institutions:

• Idiap Research Institute, Martigny (automated language recognition)
• University of Teacher Education in Special Needs (HfH) (sign language assessment)
• University of Zurich (sign language linguistics)
• University of Surrey (UK) (sign language technology)

HfH staff, with their expertise in sign language assessment, will employ the following methods:

• Test development of a DSGS yes/no vocabulary test, a DSGS sentence repetition test with corresponding rating scale, a DSGS translation test, and a DSGS interview with corresponding rating scale
• Statistical analysis of test results (Classical Test Theory and Rasch analyses)
• Rater training, studies on rater reliability, comparisons between automated and human ratings
• Development and analysis of feedback instruments for user studies (questionnaires, focus groups)

SMILE-II in sign language

You can see a project summary in Swiss German Sign Language in the following video. You can also activate German subtitles in the video.