Qualification Tests

This page contains example code for how to create MTurk QualificationTypes that contain a Qualification Test and, optionally, an AnswerKey to automatically score the test and assign the worker a Qualification Score.

In this example, workers have to meet a minimum score on the Qualification Test to gain the Qualification. Another use of Qualification Tests is to pre-screen workers (e.g., by demographics) in order to filter them into different HITs. That use case is described here.

library("MTurkR")

# load QuestionForm and AnswerKey
QuestionForm <- paste0(readLines("questionform_example.xml", warn = FALSE), collapse = "")
AnswerKey <- paste0(readLines("answerkey_example.xml", warn = FALSE), collapse = "")

# create new QualificationType
newqual <- CreateQualificationType(name = "A new coding test",
             description = "Test of coding ability",
             status = "Active",
             test.duration = seconds(hours=1),
             test = QuestionForm,
             answerkey = AnswerKey,
             validate.test = TRUE,
             validate.answerkey = TRUE,
             sandbox = TRUE)

Here are the contents of the questionform_example.xml file:

<QuestionForm xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2005-10-01/QuestionForm.xsd">
    <Overview>
        <Title>Please read the following passage and answer the questions below based upon what you read.</Title>
        <Text>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras non enim id neque pharetra euismod. Nam ornare luctus ante. In non dui eu lacus volutpat aliquet et eget sapien. In facilisis tortor lorem. Sed ornare tortor ut metus ullamcorper vitae tristique sapien tincidunt. Nunc molestie lacinia dignissim. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; In pulvinar lectus ut nunc consequat pulvinar pretium dui ullamcorper. Duis id leo ut augue placerat rhoncus et pretium velit. Vestibulum ultricies, felis id hendrerit aliquet, massa tellus commodo nisi, id laoreet nisi metus nec elit.</Text>
    </Overview>
    <Question>
        <QuestionIdentifier>question1</QuestionIdentifier>
        <IsRequired>true</IsRequired>
        <QuestionContent>
            <FormattedContent><![CDATA[<p>Some question here</p>]]></FormattedContent>
        </QuestionContent>
        <AnswerSpecification>
            <SelectionAnswer>
                <StyleSuggestion>radiobutton</StyleSuggestion>
                <Selections>
                    <Selection>
                        <SelectionIdentifier>1</SelectionIdentifier>
                        <Text>First Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>2</SelectionIdentifier>
                        <Text>Second Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>3</SelectionIdentifier>
                        <Text>Third Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>4</SelectionIdentifier>
                        <Text>Fourth Answer</Text>
                    </Selection>
                </Selections>
            </SelectionAnswer>
        </AnswerSpecification>
    </Question>
    <Question>
        <QuestionIdentifier>question2</QuestionIdentifier>
        <IsRequired>true</IsRequired>
        <QuestionContent>
            <FormattedContent><![CDATA[<p>Some question here</p>]]></FormattedContent>
        </QuestionContent>
        <AnswerSpecification>
            <SelectionAnswer>
                <StyleSuggestion>radiobutton</StyleSuggestion>
                <Selections>
                    <Selection>
                        <SelectionIdentifier>1</SelectionIdentifier>
                        <Text>First Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>2</SelectionIdentifier>
                        <Text>Second Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>3</SelectionIdentifier>
                        <Text>Third Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>4</SelectionIdentifier>
                        <Text>Fourth Answer</Text>
                    </Selection>
                </Selections>
            </SelectionAnswer>
        </AnswerSpecification>
    </Question>
    <Question>
        <QuestionIdentifier>question3</QuestionIdentifier>
        <IsRequired>true</IsRequired>
        <QuestionContent>
            <FormattedContent><![CDATA[<p>Some question here</p>]]></FormattedContent>
        </QuestionContent>
        <AnswerSpecification>
            <SelectionAnswer>
                <StyleSuggestion>radiobutton</StyleSuggestion>
                <Selections>
                    <Selection>
                        <SelectionIdentifier>1</SelectionIdentifier>
                        <Text>First Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>2</SelectionIdentifier>
                        <Text>Second Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>3</SelectionIdentifier>
                        <Text>Third Answer</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>4</SelectionIdentifier>
                        <Text>Fourth Answer</Text>
                    </Selection>
                </Selections>
            </SelectionAnswer>
        </AnswerSpecification>
    </Question>
</QuestionForm>

And here are the contents of the answerkey_example.xml file:

<AnswerKey xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2005-10-01/AnswerKey.xsd">
    <Question>
        <QuestionIdentifier>question1</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>3</SelectionIdentifier>
            <AnswerScore>10</AnswerScore>
        </AnswerOption>
    </Question>
    <Question>
        <QuestionIdentifier>question2</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>1</SelectionIdentifier>
            <AnswerScore>10</AnswerScore>
        </AnswerOption>
    </Question>
    <Question>
        <QuestionIdentifier>question3</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>4</SelectionIdentifier>
            <AnswerScore>10</AnswerScore>
        </AnswerOption>
    </Question>
    <QualificationValueMapping>
        <PercentageMapping>
            <MaximumSummedScore>30</MaximumSummedScore>
        </PercentageMapping>
    </QualificationValueMapping>
</AnswerKey>