Clarification of details concerning the creation of the base data generation

[steve2972]

Happy new year's eve!

Thank you for your repository and congratulations on your paper 😊
I was trying to reproduce your results but got stuck while creating the base data.
I've been following along the description provided in a previous question, #1
However, I got a very different number of instances for the pairwise dataset using Ultrafeedback.

Here's what I've tried:

• After removing flan, I created an evaluation instance for each criterion ['helpfulness', 'honesty', 'instruction_following', 'truthfulness'] leading to 43k * 4 = 172k evaluation instances.
• For each instance, I remove only the model responses that contains a rating of "N/A" for any one of the criteria. I assume this is to calculate the annotation_rating_average. This also means that each evaluation instance may contain a different number of model responses to sample from.

For example, if the model response scores look like this:

// For criteria: ['helpfulness', 'honesty', 'instruction_following', 'truthfulness']
[[3, "N/A", 5, 5], [3, 4, 4, 4], [5, 5, 5, 5], [5, 5, 5, 5]]

I remove the model response containing "N/A" and leave the rest.

[[3, 4, 4, 4], [5, 5, 5, 5], [5, 5, 5, 5]]

• For each instance I sample [1- the remaining # of responses] of the responses.
• This leads to 167k instances, where I drop 5k instances where all model responses contain at least one "N/A". In other words, 5k instances exist where you can't calculate an exact annotation_rating_average for the model responses.

However, a problem arises when creating the pairwise dataset.
If 3.1K is 40% of the total number of evaluation instances satisfying the conditions, this implies 7.8k-8k total number of evaluation instances fit for pairwise evaluation. When 164k is the total number of instances used (as per your paper), this means that the total number of instances used should be almost 170k.

Furthermore, when counting the actual number of instances by condition, here's what I get:
(note: all conditions are added using AND)

• uses instruction_following: 41.8k instances (40% = 16.7k)
• has 2 candidate responses: 10.7k instances (40% = 4.3k)
• the 2 scores are different: 7.9k instances (40% = a little over 3.1k) This is the desired number of instances!
• the higher scoring sample has high quality, with overall score >= 8.0: 3k instances (40% = 1.2k)
• the higher scoring sample has annotation rating average >= 4.75: 2.2k instances (40% = 0.9k)

Here, the 0.9k final number of instances satisfying all the required conditions seems really small, even considering the randomness of sampling.

Is there something I'm doing wrong when constructing the dataset? I was hoping you could give me some pointers.

[seungyeonlj]

Hello,

First of all, thank you for your detailed question and sorry for the late reply.

I reviewed both the code and the data and you were right about the point you raised—thanks for pointing it out.

[1]

The last two conditions (the higher scoring sample has high quality, with overall score >= 8.0, the higher scoring sample has annotation rating average >= 4.75) were not included when constructing the Base-data.

(I had added these conditions during the experiment phase, but they were ultimately excluded in the final Base-data. We mistakenly mentioned the two conditions in the previous response to #1 -- my apologies for the confusion. I will correct the response in #1 as well.)

So,
"the 2 scores are different: 7.9k instances (40% = a little over 3.1k) This is the desired number of instances!" <- up to this line should get the correct number of data.

[2]

Regarding removing N/A values:
The filtering process does not remove model responses that initially contain N/A. Instead, evaluation instances are created for each evaluation criterion and N/A values are filtered out only at that stage.
For example, in your case, all samples have instruction_following ratings (5,4,5,5), so they remain usable.
Thus, the number of removed samples should be less than 5K (depending on random sampling).

I appreciate your thorough reading of our paper and your effort in conducting experiments.

Although it's a bit late, I wish you a Happy New Year too. 새해 복 많이 받으세요 :)
Feel free to reach out if you have any further questions.

Thanks,
Seungyeon.