forked from pages-themes/minimal
-
Notifications
You must be signed in to change notification settings - Fork 0
/
qt_annotation_phase1.html
126 lines (102 loc) · 12.7 KB
/
qt_annotation_phase1.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<script>window.NREUM||(NREUM={});NREUM.info={"beacon":"bam-cell.nr-data.net","errorBeacon":"bam-cell.nr-data.net","licenseKey":"b6a3395450","applicationID":"1491523","transactionName":"dVgIFxEMD1RTREteClRESQYHChdXRGkURgBAXgMU","queueTime":1,"applicationTime":376,"agent":""}</script>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="robots" content="noindex">
<meta name="csrf-param" content="authenticity_token" />
<meta name="csrf-token" content="XhOeIIw1BS7ymTo6eGO7iTKg+I0KPDPBFlOdnzN4oRghV/3dkxIZD5qTkgcA8I+l+dOOSzzfSEOxZCpL9IVwXA==" />
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js" type="text/javascript"></script>
<![endif]-->
<link rel="stylesheet" media="all" href="https://d1epz76kxf2sd6.cloudfront.net/assets/core_ui/application-c73e1b3a28e6f2975cd7e313a71201b4cf50ad73da96baf6a23af86e385b48e7.css" /><link rel="stylesheet" media="all" href="https://d1epz76kxf2sd6.cloudfront.net/assets/application-652661d23dce3126f8a1debe63a9b24fa4dd9be1d52bf6645e3e5920aea8b607.css" /><link rel="stylesheet" media="print" href="https://d1epz76kxf2sd6.cloudfront.net/assets/core_ui/print-24e9db670e5dceaeb10108d7dd38bafe26c86f1912401dbd40f47b1551a7183e.css" />
<link rel="stylesheet" media="all" href="https://d1epz76kxf2sd6.cloudfront.net/assets/builder_worker-e9adc9e201c14ee4b24d515633bd9e416ba3813ad3fc604689b81ad9444532e4.css" />
<style type="text/css" media="screen">
body {
background-color: #CCCCCC;
color: #000000;
width: 1200px;
}
.row-fluid {
margin-bottom: 20pt;
}
.table thead tr {
border-bottom: none;
}
th:nth-child(2) {
border-right: 1px solid #ddd;
border-bottom: none;
}
td:first-child {
width: 18px;
}
td:nth-child(2) {
border-right: 1px solid #ddd;
width:75%;
}
td:last-child {
max-width: 100%;
white-space: nowrap;
}
td.selected {
vertical-align: middle;
}
h1 {
color: #000000;
margin-bottom: 15pt;
}
h2.legend {
position: absolute;
left: -999em;
}
.radios div {
width: 35%;
float:left;
}
.cml_row label {
font-size: 12pt;
font-weight: bold;
}
.well {
padding: 10px;
}
.review {
padding: 0;
margin: auto;
}
.first-review {
padding-top: 20px;
}
.highlighted {
background-color:silver;
}
.progress {
position: relative;
}
.progress .bar {
z-index: 1;
position: absolute;
}
.progress span {
position: absolute;
top: 0;
z-index: 2;
color: black; /* Change according to needs */
text-align: center;
width: 100%;
}
.affix {
top: 0px;
position:fixed;
z-index: 9999;
background-color: white;
}
</style>
<body>
<h1 class="job-title">Phase 1: Sentence Selection via Voting</h1>
<div id="assignment-instructions">
<div class="collapse in well">
<h2 data-empty="true"><span style="color: rgb(65, 168, 95);">[Please read carefully before attempting the task!]</span></h2>
<h2 data-empty="true">I. General Instructions:</h2><ul><li>Attempt HITs if you are a <strong>native speaker of English</strong> or a <strong>fluent speaker</strong> who can comfortably comprehend a review written in informal English.</li><li>We are happy to <strong>receive feedback</strong> and improve this job accordingly. Feel free to send your comments to: s.angelidis [at] ed [dot] ac [dot] uk or use figure-eight's messaging.</li><li>Your responses are <strong>confidential</strong>. Any publications based on these will not include your specific responses, but rather aggregate information from many individuals. We will not ask any information that can be used to identify who you are.</li><li><strong><span style="color: rgb(184, 49, 47);">Important:</span></strong> This task is, to some extent, subjective, so we do not use test questions to evaluate the quality of answers. <strong>However</strong>, we monitor answers to identify workers who don't follow our basic instructions or give random answers. Such workers will be <strong>rejected and banned</strong> from this and future tasks.<br><br></li></ul><h2>II. The Goal: Summarizing Customer Opinions about Hotels</h2><p>This task is a first step towards generating summaries of the opinions expressed in Hotel reviews. Before creating actual opinion summaries, we need to identify the review sentences that contain informative and well-written opinions. For this task, you <strong>will not</strong> be asked to write your own summary -- you will simply <strong>select well-formed review sentences that you would want to see in an opinion summary</strong>.<br><br></p><h2>III. The Task: Review Sentence Selection </h2><p>In each page, you will see <strong>5 reviews for a particular hotel</strong>. For every review, you should select the sentences that are well-formed and that you believe capture the most important and useful comments of the reviewer. You should judge the contents of each review independently -- don't worry if similar, identical or contradictory opinions are selected across reviews. </p><p>Even selecting <em>none</em> or <em>all</em> sentences in a single review is acceptable, if you think it's the right choice. We will not enforce limits on the number of sentences you select, but we expect you to select, <em>on average</em>,<strong> 20%-40% of the available sentences</strong>. To help you keep track of how many sentences you've selected per page, we have implemented a <em>"selection tracker"</em> bar. This shows you the percentage of sentences you've selected <em>in the current page</em>, and suggests whether you should select more or less, as shown in the screenshots below. You are <strong>not expected to <em>always</em></strong> be in the <strong><span style="color: rgb(65, 168, 95);">green zone</span></strong><span style="color: null;">,</span><span style="color: null;"> but you should <strong>try to </strong><strong>stay within these limits in most cases</strong>.<strong> </strong>Avoid the <strong><span style="color: rgb(184, 49, 47);">red zone</span></strong><span style="color: null;"> as much as you can.</span></span><br><img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571157266141-more.png" style="width: 529px; height: 60.284px;" class="fr-fic fr-dib"><img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571157272069-maybe_more.png" style="width: 530px; height: 60.3841px;" class="fr-fic fr-dib"><img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571157276132-ok.png" style="width: 528px; height: 60.17px;" class="fr-fic fr-dib"><img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571160912781-maybe+less.png" style="width: 521px; height: 60.105px;" class="fr-fic fr-dib"><img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571157288740-less.png" style="width: 525px; height: 59.8177px;" class="fr-fic fr-dib"></p><p>Judging whether a particular sentence is worth selecting is, to some extent, subjective and <strong>you should select the segments that you would want to see in a summary of opinions</strong> for the particular hotel. However, some basic rules should be taken into account:</p><ol><li><u>Do select</u> sentences that express <strong>useful opinions</strong> of the reviewer <em>or</em> give <strong>useful details which support their opinions</strong>.<br><u>Do not select</u> sentences that contain <strong>less important elaborations or further details</strong> about the reviewer's experience.<br>For example, in the following two review excerpts, only the sentences marked with a [✔] should be selected:<br><br><table style="width: 61%;"><tbody><tr><td style="width: 5.1903%;">[✔]</td><td style="width: 67.301%;">The rooms were beautiful and massive!</td><td align="right" style="width: 27.5086%; background-color: rgb(84, 172, 210);">[useful opinion]</td></tr><tr><td style="width: 5.1903%;">[✔]</td><td style="width: 67.301%;">Also, the most comfy beds I've ever slept on.</td><td align="right" style="width: 27.5086%; background-color: rgb(84, 172, 210);">[useful opinion]</td></tr><tr><td style="width: 5.1903%;">[ ]</td><td style="width: 67.301%;">It was so hard to get out of bed every morning.</td><td align="right" style="width: 27.5086%; background-color: rgb(235, 107, 86);">[elaboration]</td></tr><tr><td style="width: 5.1903%;">[ ]</td><td style="width: 67.301%;">I slept like a baby for a whole week :)</td><td align="right" style="width: 27.5086%; background-color: rgb(235, 107, 86);">[elaboration]</td></tr></tbody></table><table style="width: 61%;"><tbody><tr><td style="width: 5.1903%;">[✔]</td><td style="width: 67.301%;">The location is actually excellent.</td><td align="right" style="width: 27.5086%; background-color: rgb(84, 172, 210);">[useful opinion]</td></tr><tr><td style="width: 5.1903%;">[✔]</td><td style="width: 67.301%;">Within walking distance of the Magnificent Mile.</td><td align="right" style="width: 27.5086%; background-color: rgb(84, 172, 210);">[useful details]</td></tr><tr><td style="width: 5.1903%;">[✔]</td><td style="width: 67.301%;">Good restaurants and high end shops all around.</td><td align="right" style="width: 27.5086%; background-color: rgb(84, 172, 210);">[useful details]</td></tr><tr><td style="width: 5.1903%;">[ ]</td><td style="width: 67.301%;">I never got bored of walking in that neighbourhood.</td><td align="right" style="width: 27.5086%; background-color: rgb(235, 107, 86);">[elaboration]</td></tr></tbody></table><br></li><li><strong>General comments</strong> about the experience of the reviewer <strong>should often be avoided</strong>, as they provide no useful information about <strong><em>why</em></strong> they liked/disliked the hotel. The same applies to comments which are <strong>not about the hotel itself</strong>. The following review example illustrates such cases:<br><br><table><tbody><tr><td>[ ]</td><td>We were looking for a family-friendly hotel.</td><td style="text-align: right; background-color: rgb(235, 107, 86);">[not about the hotel]</td></tr><tr><td>[ ]</td><td>It feels harder and harder to find one these days.</td><td style="text-align: right; background-color: rgb(235, 107, 86);">[not about the hotel]</td></tr><tr><td>[ ]</td><td>This one was just perfect!!</td><td style="text-align: right; background-color: rgb(235, 107, 86);">[general comment]</td></tr><tr><td>[✔]</td><td>Beautiful and spacious rooms with a view.</td><td style="text-align: right; background-color: rgb(84, 172, 210);">[useful opinion]</td></tr><tr><td>[✔]</td><td>And an excellent pool for the kids!</td><td style="text-align: right; background-color: rgb(84, 172, 210);">[useful opinion]</td></tr><tr><td>[ ]</td><td>I'd definitely recommend this place to anyone!</td><td style="text-align: right; background-color: rgb(235, 107, 86);">[general comment]</td></tr></tbody></table><br></li><li><strong>Avoid selecting poorly-written sentences</strong> (e.g., misspelled, or with wrong punctuation). Also avoid very long sentences that combine useful opinions with less interesting information, like the following.<br><br><table style="margin-right: calc(26%); width: 74%;"><tbody><tr><td style="width: 3.632%;vertical-align: top;">[ ]</td><td style="width: 68.5761%;">The staff were very friendly and even though we booked through expedia to get a great deal on the hotel we were treated like a somebody and even received a room upgrade to a lake view.</td><td style="vertical-align: top; width: 27.6596%; text-align: right; background-color: rgb(235, 107, 86);">[mix of useful with other]</td></tr><tr><td>[ ]</td><td style="width: 68.5761%;">First....location!close to Pike Market....a few blocks...!shopping ALLOVER.</td><td style="width: 27.6596%; text-align: right; background-color: rgb(235, 107, 86);">[poorly written]</td></tr></tbody></table></li></ol><h2><br>IV. Example</h2><p>Finally, here are 3 examples of sentence selection for the same review: (a) one with the appropriate number of selections; (b) one with more selections than necessary; and (c) one with fewer selections than necessary.</p><p><strong>(a) Good selection:</strong><strong><img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571158887230-good_selection.png" style="width: 856px; height: 345.529px;" class="fr-fic fr-dib fr-fil"></strong></p><p><strong>(b) Over-selecting:<img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571159004289-over_selection.png" style="width: 856px; height: 345.553px;" class="fr-fic fr-dib fr-fil"></strong></p><p><strong>(c) Under-selecting:<img src="https://s3.amazonaws.com/crowdflower-make-cloud/images%2F1571159035520-under_selection.png" style="width: 856px; height: 345.553px;" class="fr-fic fr-dib fr-fil"></strong></p><p></p></div>
</body></html>