Upload participants via CSV -> set fields for duplicate filter

[MatthiasGer0821]

hi @edgarrmondragon

I was wondering if you are aware of how to specify the fields, new participants are filtered by, when uploaded via CSV over the API,
or how to disable that new participants are filtered.

In the webclient, the user can choose if duplicates are filtered (although the token field is always filtered),
and if so, which fields (firstname, lastname, email, attributes).

Is there any way with citric to have this functionality? Or if not, do you maybe have a suggestion of how to go about it?

Best regards!
Matt

Background to my problem: i´m trying to import two participants with different tokens to the same survey.
It´s the same participant (defined by firstname, lastname and email) because it is a parent with two kids.
For each kid, the parent receives a specific token, but firstname, lastname and email are identical.
This use case works via the web interface.

[edgarrmondragon]

Hey @MatthiasGer0821!

AFAICT there's no way to setup the filter fields for deduplicating participants using add_participants. This method doesn't even check for duplicates in fields other than token. For example:

participants = [
    {
        "email": "[email protected]",
        "firstname": "John",
        "lastname": "Doe",
        "token": "1",
    },
    {
        "email": "[email protected]",
        "firstname": "Jane",
        "lastname": "Doe",
        "token": "2",
    },
    {
        "email": "[email protected]",
        "firstname": "Jane",
        "lastname": "Doe",
        "token": "2",
    },
]

added = client.add_participants(survey_id, participants)

added[-1]['errors']  # {'token': ['Access code "2" has already been taken.']}

len(client.list_participants(survey_id))  # 2

Under the hood, the Web UI inserts the records one by one but does a check for duplicates before adding a new token:

if ($bFilterDuplicateToken) {
    $aParams = array();
    $oCriteria = new CDbCriteria();
    $oCriteria->condition = "";
    foreach ($aFilterDuplicateFields as $field) {
        if (isset($aWriteArray[$field])) {
            $oCriteria->addCondition("{$field} = :{$field}");
            $aParams[":{$field}"] = $aWriteArray[$field];
        }
    }
    if (!empty($aParams)) {
                                    $oCriteria->params = $aParams;
    }
    $dupresult = TokenDynamic::model($iSurveyId)->count($oCriteria);
    if ($dupresult > 0) {
        $bDuplicateFound = true;
        $aDuplicateList[] = sprintf(gT("Line %s : %s %s (%s)"), $iRecordCount, $aWriteArray['firstname'], $aWriteArray['lastname'], $aWriteArray['email']);
    }
}

https://github.com/LimeSurvey/LimeSurvey/blob/fe9fdee8dd87d975a7b0f7d0bfd9c353a0ecd572/application/controllers/admin/Tokens.php#L2094-L2112

That dedup could replicated in Python code for add_participants:

dedup_fields = ["lastname"]
participants = [...]

unique = set()
dedup = []

for p in participants:
    key = tuple(p[f] for f in dedup_fields)
    if key not in unique:
        unique.add(key)
        dedup.append(p)

added = client.add_participants(survey_id, dedup)

[MatthiasGer0821]

Hi @edgarrmondragon,

thank you for your mail. My problem seems to be with the "under the hood" part of limesurvey, because I dont know how to disable this "under-the-hood"-filter when uploading CSV files via add-participants.
So yes, the add_participant method does not check for duplicates (which is exactly what I want), but I was wondering
how to disable the "under-the-hood"-check of limesurvey. In the UI, I can specify the fields, and I was wondering if I can disable this check while using add_participants. Do you have an idea or could point me in the right direction?

Thank you and best regards from Germany,
Matt

[MatthiasGer0821]

P.S. Just to put this in other words: The API seems to have deduplication ON by default (using the fields first name, last name, email and token). But what I want is deduplication JUST by token (the equivalent in the WebUI to DISABLE the check for duplicates).

I hope this better explained.
Again, thanks!

[MatthiasGer0821]

Dear @edgarrmondragon
I was wondering if I could bring my question to your attention again. (and sorry to bother you again).
I hope my "P.S." answer describes the issue clearly. I would greatly appreciate any thoughts, hints or comments
on how you would address this.
Best regards from Germany. Matt

[edgarrmondragon]

Hey @MatthiasGer0821!

But what I want is deduplication JUST by token (the equivalent in the WebUI to DISABLE the check for duplicates).

That's the behaviour of the RC API as far as I can tell. I even have an example of that in the tests:

citric/tests/test_integration.py

Lines 241 to 266 in 5c051a3

	data = [
	{
	"email": "[email protected]",
	"firstname": "John",
	"lastname": "Doe",
	"token": "1",
	"attribute_1": "Dog person",
	"attribute_2": "Night owl",
	},
	{
	"email": "[email protected]",
	"firstname": "Jane",
	"lastname": "Doe",
	"token": "2",
	"attribute_1": "Cat person",
	"attribute_2": "Early bird",
	},
	{
	"email": "[email protected]",
	"firstname": "Jane",
	"lastname": "Doe",
	"token": "2",
	"attribute_1": "Cat person",
	"attribute_2": "Night owl",
	},
	]

for which the response correctly dedups based on the token.

(Pdb) added[2]
{'email': '[email protected]', 'firstname': 'Jane', 'lastname': 'Doe', 'token': '2', 'attribute_1': 'Cat person', 'attribute_2': 'Night owl', 'errors': {'token': ['Access code "2" has already been taken.']}}

If you take that example and set the offending token to 3, then the three participants will be added and two of them will have the same first name, last name and email.

Am I missing something? Can you try this on a fresh survey to rule out a global survey setting?

[MatthiasGer0821]

Thank you @edgarrmondragon …!
For some reason this default behavior does not happen.
We are using Limesurvey in a University context, so I will have to talk to them for me to be able to see the global settings...

I'll test this further as soon as I have an answer from them.

Again, thanks for your answer and if I don't write before Christmas: feliz navidad! :)