Goal: In this pre-test, you will first read brief selections from two social science papers (Step 1). You will then go through an end-to-end implementation of a feature and apply it to a dataset of team conversations (Step 2). Finally, you will write a reflection on how well you think this feature extractor performed on the data, as well as how well it performs in operationalizing social science constructs (Step 3).
The idea behind this task is to give you a flavor of the scope of our work — to show how we take inspiration from social science, then apply these ideas in a computational way.
Please write your reflection in this README document.
1a. Which dataset did you choose?
[YOUR ANSWER]
1b. What method(s) did you choose? In 1-2 sentences each, describe your sentiment analysis method(s).
[YOUR ANSWER]
1c. Does your method capture any of the ideas from Troth et al. and West et al.? If so, which ones?
[YOUR ANSWER]
1d. Compared to how Troth et al. and West et al. measured positivity, what are some strengths and weaknesses of your approach?
[YOUR ANSWER]
Next, we would like you to consider how you would evaluate your method. How do you know the classification or quantification of emotion is “right?” Try to think critically!
2a. Open up your output CSV and look at the columns you generated. Do the values “make sense” intuitively? Why or why not?
[YOUR ANSWER]
2b. Propose an evaluation mechanism for your method(s). What metric would you use (e.g., F1, AUC, Accuracy, Precision, Recall)?
[YOUR ANSWER]
2c. Describe the steps you would take in evaluating this method. Be as specific as possible.
[YOUR ANSWER]
2d. Given the nature of these datasets, what challenges do you anticipate that you may encounter during evaluation? How would you go about resolving them?
[YOUR ANSWER]
3a. How much time did it take you to complete this task? (Please be honest; we are looking for feedback to make sure the task is scoped appropriately, as this is one of the first times we’re using this task.)
[YOUR ANSWER]
3b. Finally, provide an overall reflection of your experience. How did you approach this task? What challenge(s) did you encounter? If you had more time, what are additional extensions, improvements, or tests that you would want to implement?
[YOUR ANSWER]