About sAMA

aiCover

SAMA: Short Answer Marking Agent


We have developed a cloud based engine that will grade free-text answers to short answer questions, those of between one word and one paragraph, normally worth between one and six marks.  The system can assess any answer provided that the mark scheme clearly indicates how each mark is achieved and is fact based.


Currently the system has been trained for Computer Science questions and we are currently training it for other subjects.  It marks with 94% accuracy which is the same or better than competent human graders.  


Examples of questions that can be marked are such as:

Compare the usage of RAM and ROM [4]. 


We are integrating the cloud engine with an existing front end system but we can also provide an API to integrate it with your front end system.


SAMA will not mark all answers, currently it will refer approximately 50% of answers to be human-graded but then these answers will be fed back into the system to improve it's performance.

How it works

Free-Text short answer marking; as accurate as humans

Pre-processing

The answer is stripped of punctuation and reduced to the base form of words

Segmentation

The answer is split into sections, each of which can be compared to a phrase in the mark scheme

Semantic Matching

The answer is compared to the mark scheme using a semantic match to check for the expression of similar knowledge or concepts

Referral

If the quality of match is not sufficient, the answer is referred for human grading and can then be used to further improve the accuracy of the system

Presentation about SAMA from EDUNINE 2021 in Guatemala