Skip to content
This repository has been archived by the owner on May 24, 2019. It is now read-only.

Qualification Tests for Worker Pre Screening

Jump to bottom
leeper edited this page Dec 6, 2014 · 4 revisions

It is sometimes needed to pre-screen workers based on demographics in order to restrict workers' access to a particular HIT or HITs. This page describes, through an extended example use case, how to use Qualification Tests and AnswerKeys to assign different Qualification scores to workers based on demographics and then use that Qualification score to provide access to different HITs (i.e., to create demographically stratified HITs).

This page describes how to use an AnswerKey to automatically score a Qualification Test. If you'd prefer to manually score the Qualification Tests, follow all the details here but leave out the AnswerKey. You can then read the tutorial on Qualification Requests to learn how to view, grant, and reject Qualification Requests.

Example Use Case: Ideological Pre-Screening

Our purpose was to create a Qualification that would qualify MTurk workers to take one of three versions of nearly identical HITs based on their ideology. We wanted to get equal numbers of liberals, conservatives, and moderates in our final survey, so pre-screening workers for their ideology using a Qualification Test allows us to control how many liberals, conservatives, and moderates have access to the HIT. Specifically, once enough, e.g., liberals have completed the study, we can close the HIT that is accessible to liberals while leaving open the two other HITs that are accessible to conservatives and moderates, respectively. As should be clear, any given worker can therefore only complete one of the HITs.

We begin by creating the Qualification (with Test and AnswerKey). This is relatively simple. Note: The full XML code for both the Test and AnswerKey are shown below. The Test must be written in MTurk's QuestionForm format and the AnswerKey must be in MTurk's AnswerKey format.

# Load the XML files
QuestionForm <- paste0(readLines('Test.xml'), collapse='')
AnswerKey <- paste0(readLines('AnswerKey.xml'), collapse='')

# Created the QualificationType
newqual <- CreateQualificationType(name="Qualification",
                                   description="A qualification for our HITs",
                                   status="Active", test.duration=seconds(hours=1),
                                   test=QuestionForm, answerkey = AnswerKey)

Once the Qualification is created, it becomes available for workers to complete. However, it is unlikely they will complete the Qualification Test until it is actually attached to a particular HIT or HITs. To accomplish that, we need to build a QualificationRequirement data structure for each ideological value and attach each of them to separate HITs. The template for the QualificationRequirements is as follows:

score <- GenerateQualificationRequirement("QualificationTypeID", # the QualificationTypeId from `newqual`
                                          "==", # QualificationRequirement requires an exact score
                                          "ScoreAsNumber") # Score should be a number

For our use case, we actually need to create three separate QualificationRequirement structures, one for each ideological group:

score_conserv <- GenerateQualificationRequirement("QualificationTypeID","==","25") 
score_moder <- GenerateQualificationRequirement("QualificationTypeID","==","50") 
score_liberal <- GenerateQualificationRequirement("QualificationTypeID","==","75")

We can then attach those to three new HITs, all of which have identical properties, except for the QualificationRequirement (seen in the last line of each CreateHIT code). Note: This example uses the HITLayoutId to create the HIT; alternative to this approach are described on the HITs page.

newHIT_conserv <- CreateHIT(  
  hitlayoutid="2GRR70G5R98DR80L99NSVTNYCIXBR6",
  annotation = "HIT_conserv",
  assignments = "200",
  title="A survey about social issues in the United States",
  description="A survey about social issues in the United States",
  reward=".40",
  duration=seconds(hours=1),
  expiration=seconds(days=1),
  keywords="survey, question, answers, research, politics, opinion",
  auto.approval.delay=seconds(days=1),
  qual.req=score_conserv
)

newHIT_moder <- CreateHIT(  
  hitlayoutid="2GRR70G5R98DR80L99NSVTNYCIXBR6",
  annotation = "HIT_moder",
  assignments = "200",
  title="A survey about social issues in the United States",
  description="A survey about social issues in the United States",
  reward=".40",
  duration=seconds(hours=1),
  expiration=seconds(days=1),
  keywords="survey, question, answers, research, politics, opinion",
  auto.approval.delay=seconds(days=1),
  qual.req=score_moder
)

newHIT_liberal <- CreateHIT(  
  hitlayoutid="2GRR70G5R98DR80L99NSVTNYCIXBR6",
  annotation = "HIT_liberal",
  assignments = "200",
  title="A survey about social issues in the United States",
  description="A survey about social issues in the United States",
  reward=".40",
  duration=seconds(hours=1),
  expiration=seconds(days=1),
  keywords="survey, question, answers, research, politics, opinion",
  auto.approval.delay=seconds(days=1),
  qual.req=score_liberal
)

Note that we use the annotation field to record which HIT is for which group. This field is only visible to us, not to workers.

Once the HITs are created, they immediately become available to workers. Once a worker complete the Qualification Test, they gain access to one - and only one - of the HITs, which they can then complete.

Contents of Test.xml

The most difficult part of setting up this study is creating the Qualification Test and AnswerKey. As you can see below, each requires rather extensive XML markup.

This Qualification Test masks the requester's intent to pre-screen on ideology by asking several questions. The AnswerKey (shown below) only assigns a Qualification score based on the worker's answer to the ideology question.

<QuestionForm xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2005-10-01/QuestionForm.xsd">
  <Overview>
    <Title>Qualification Test</Title>
    <Text>After taking this qualification test, you may be redirected away from our HIT. If this occurs, please search "research survey about social issues" to take our survey. Please note that if you have taken some of our surveys before, you may be blocked from this HIT as we use some of the same questions. This will not prevent you from accepting unrelated HITs from us in the future.
	</Text>
  </Overview>
  <Question>
    <QuestionIdentifier>question1</QuestionIdentifier>
    <IsRequired>true</IsRequired>
    <QuestionContent>
      <Text>Are you registered to vote?</Text>
    </QuestionContent>
    <AnswerSpecification>
      <SelectionAnswer>
        <StyleSuggestion>radiobutton</StyleSuggestion>
        <Selections>
          <Selection>
            <SelectionIdentifier>1</SelectionIdentifier>
            <Text>Yes</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>2</SelectionIdentifier>
            <Text>No</Text>
          </Selection>
        </Selections>
      </SelectionAnswer>
    </AnswerSpecification>
  </Question>
  <Question>
    <QuestionIdentifier>question2</QuestionIdentifier>
    <IsRequired>true</IsRequired>
    <QuestionContent>
      <Text>Did you vote in the 2012 General Election?</Text>
    </QuestionContent>
    <AnswerSpecification>
      <SelectionAnswer>
        <StyleSuggestion>radiobutton</StyleSuggestion>
        <Selections>
          <Selection>
            <SelectionIdentifier>1</SelectionIdentifier>
            <Text>No</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>2</SelectionIdentifier>
            <Text>I usually vote, but did not in 2012</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>3</SelectionIdentifier>
            <Text>I am not sure</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>4</SelectionIdentifier>
            <Text>Yes, I definitely voted</Text>
          </Selection>
        </Selections>
      </SelectionAnswer>
    </AnswerSpecification>
  </Question>
  <Question>
    <QuestionIdentifier>question3</QuestionIdentifier>
    <IsRequired>true</IsRequired>
    <QuestionContent>
      <Text>For whom did you vote for President?</Text>
    </QuestionContent>
    <AnswerSpecification>
      <SelectionAnswer>
        <StyleSuggestion>radiobutton</StyleSuggestion>
        <Selections>
          <Selection>
            <SelectionIdentifier>1</SelectionIdentifier>
            <Text>Barack Obama (Democratic)</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>2</SelectionIdentifier>
            <Text>Mitt Romney (Republican)</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>3</SelectionIdentifier>
            <Text>Someone else</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>4</SelectionIdentifier>
            <Text>Did not vote</Text>
          </Selection>
		  <Selection>
            <SelectionIdentifier>5</SelectionIdentifier>
            <Text>Don't recall</Text>
          </Selection>
        </Selections>
      </SelectionAnswer>
    </AnswerSpecification>
  </Question>
  <Question>
    <QuestionIdentifier>question4</QuestionIdentifier>
    <IsRequired>true</IsRequired>
    <QuestionContent>
      <Text>On a scale of political views from conservative to liberal, where would you place yourself?</Text>
    </QuestionContent>
    <AnswerSpecification>
      <SelectionAnswer>
        <StyleSuggestion>radiobutton</StyleSuggestion>
        <Selections>
          <Selection>
            <SelectionIdentifier>1</SelectionIdentifier>
            <Text>Conservative</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>2</SelectionIdentifier>
            <Text>Moderate</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>3</SelectionIdentifier>
            <Text>Liberal</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>4</SelectionIdentifier>
            <Text>Don't know</Text>
          </Selection>
        </Selections>
      </SelectionAnswer>
    </AnswerSpecification>
  </Question>
  <Question>
    <QuestionIdentifier>question5</QuestionIdentifier>
    <IsRequired>true</IsRequired>
    <QuestionContent>
      <Text>During the past 12 months, have you worked with other people to deal with some issue facing your community?</Text>
    </QuestionContent>
    <AnswerSpecification>
      <SelectionAnswer>
        <StyleSuggestion>radiobutton</StyleSuggestion>
        <Selections>
          <Selection>
            <SelectionIdentifier>1</SelectionIdentifier>
            <Text>Yes</Text>
          </Selection>
          <Selection>
            <SelectionIdentifier>2</SelectionIdentifier>
            <Text>No</Text>
          </Selection>
        </Selections>
      </SelectionAnswer>
    </AnswerSpecification>
  </Question>
</QuestionForm>

Contents of AnswerKey.xml

Note how the Qualfication Test includes several questions, but only question4 is used to assign a score to the workers. Those discrete scores are then used in the QualificationRequirements attached to each HIT.

<AnswerKey xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2005-10-01/AnswerKey.xsd">
    <Question>
        <QuestionIdentifier>question1</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>1</SelectionIdentifier>
            <AnswerScore>0</AnswerScore>
        </AnswerOption>
    </Question>
    <Question>
        <QuestionIdentifier>question2</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>1</SelectionIdentifier>
            <AnswerScore>0</AnswerScore>
        </AnswerOption>
    </Question>
    <Question>
        <QuestionIdentifier>question3</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>1</SelectionIdentifier>
            <AnswerScore>0</AnswerScore>
        </AnswerOption>
    </Question>
    <Question>
        <QuestionIdentifier>question4</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>1</SelectionIdentifier>
            <AnswerScore>1</AnswerScore>
        </AnswerOption>
        <AnswerOption>
            <SelectionIdentifier>2</SelectionIdentifier>
            <AnswerScore>2</AnswerScore>
        </AnswerOption>
        <AnswerOption>
            <SelectionIdentifier>3</SelectionIdentifier>
            <AnswerScore>3</AnswerScore>
        </AnswerOption>
        <AnswerOption>
            <SelectionIdentifier>4</SelectionIdentifier>
            <AnswerScore>2</AnswerScore>
        </AnswerOption>
    </Question>
    <Question>
        <QuestionIdentifier>question5</QuestionIdentifier>
        <AnswerOption>
            <SelectionIdentifier>1</SelectionIdentifier>
            <AnswerScore>0</AnswerScore>
        </AnswerOption>
    </Question>
    <QualificationValueMapping>
        <PercentageMapping>
            <MaximumSummedScore>4</MaximumSummedScore>
        </PercentageMapping>
    </QualificationValueMapping>
</AnswerKey>

.

Clone this wiki locally