Skip to content

Latest commit

 

History

History
56 lines (34 loc) · 4.57 KB

fine_grained_guildline.md

File metadata and controls

56 lines (34 loc) · 4.57 KB

Task Description

In this task, you will be presented with a short story, along with a summary of the story. Please read both the story and the summary carefully. You will be asked to assess whether each summary line is consistent with the story, and to rewrite any inconsistent lines. Make sure to submit to record your answers.

Warning: Some of the stories may contain content that readers could find offensive, disturbing, or otherwise unsettling. If you do not feel comfortable with this, please feel free to decline the task.

If you encounter any problems or have other feedback, please contact [email protected]

By participating in this study, you confirm that you are (1) 18 years or older, (2) currently residing in the US, and (3) have read and understand the information above and agree to participate. By clicking submit you consent to participate in this study. If you do not wish to participate, do not click submit.

Detailed Instructions with Example

Please read the story carefully and try to understand the points that are central to the plot.

Story

They shot the six cabinet ministers at half-past six in the morning against the wall of a hospital. There were pools of water in the courtyard. There were wet dead leaves on the paving of the courtyard. It rained hard. All the shutters of the hospital were nailed shut. One of the ministers was sick with typhoid. Two soldiers carried him downstairs and out into the rain. They tried to hold him up against the wall but he sat down in a puddle of water. The other five stood very quietly against the wall. Finally, the officer told the soldiers it was no good trying to make him stand up. When they fired the first volley he was sitting down in the water with his head on his knees.

 

Below are two example summaries for this story:
Summary A

The short story describes the execution of six cabinet ministers by firing squad early in the morning during heavy rain. One of the ministers is sick with typhoid fever and too weak to stand, so he is brought downstairs and shot while sitting in a puddle of water with his head on his knees. The story paints a vivid image of human suffering.

Summary B

Six cabinet ministers are executed by firing squad early in the morning during heavy rain. One of the ministers is sick with typhoid fever and too weak to stand, so he is shot while lying in a puddle of water near the hospital wall.

 

The questions will ask you to evaluate the summary with respect to the story at various levels. Firstly, the questions will ask you to evaluate the summary line by line. The final question will ask you to classify the entire summary as either consistent, or inconsistent to the story.

Question: Is this line in the summary consistent with the story?

This question is asked for each line of the summary. Mark each line as either consistent or inconsistent to the story. The events and details described in the summary should not misrepresent details from the story or include details that are unsupported by the story. The lines in the summary that do so can be classified as inconsistent. Some lines include commentary about the story, which should be ignore when evaluating consistency. For such lines, select N/A. For summary A:

  • The short story describes the execution of six cabinet ministers by firing squad early in the morning during heavy rain: Yes
  • One of the ministers is sick with typhoid fever and too weak to stand, so he is brought downstairs and shot while sitting in a puddle of water with his head on his knees: Yes
  • The story paints a vivid image of human suffering: N/A, just commentary

For summary B:

  • The summary includes the key details of the execution of six cabinet ministers in the early morning during heavy rain: Yes
  • The summary includes the key details of the minister sick with typhoid and how he was too weak to stand: Yes
  • The summary talks about the minister lying in a puddle of water when shot: No
Question (for any answers marked No): Provide an rewrite for this line.

For every line marked inconsistent, provide a rewrite of the original line. For example, for the line from Summary B: "The summary talks about the minister lying in a puddle of water when shot.", a possible response is:

The summary talks about the minister sitting in a puddle of water when shot.

Payment

You will be paid via Upwork once you have completed the pilot. Please allow a few days to submit the payment.

Thank you so much for participating in our study!! We would love to hear any feedback that you may have about this task!