pymongo.errors.OperationFailure

[waa8288]

I installed combine-docker v0.10 and getting the following error when I run any job. I appreciate if some one help where i should look for or what to fix.

<class 'pymongo.errors.OperationFailure'> at 2020-04-20 17:22:40.408612+00:00 with args ('namespace does not exist: combine.record',): namespace does not exist: combine.record

[gordonleacock]

Please try this and let us know if you have difficulties. :-)

1. Start mongodb, here’s how. a) If you used ansible to set up combine, just run mongo on the machine where combine is located. b) If you used docker, on the combine machine use a command like docker exec -it combine-docker_mongo_1 /bin/bash and then mongo once inside.

2. Run the mongo command show dbs.

3. If ‘combine' isn’t listed then you need to create it with the command use combine which will create it.

4. Now that we have a combine mongo database and are using it, check on its collections with a show collections command.

5. If ‘record’ is not show as a collection, we need to create it.

6. Use the mongo command db.createCollection(“record”).

7. Exit mongodb and restart combine and try running a job.

[waa8288]

The record is there. i create. The record problem is solved. The problem is now on Elasticsearch. When I tried to harvest datasets from UM Flint, I get "An error occurred when running this Job: RequestError TransportError(400, 'illegal_argument_exception', 'Malformed [mappings] section for type [date_detection], should include an inner object describing the mapping')..." error. The harvest processed successfully. When you click "View Mapping and Indexing Error" on attached screen-1, you get attached screen-2. There, you can indexing failure. From my reading, Elasticsearch index is failing due to string and date field mapping. I have spent quite a lot time searching and there are so compliant or about Elasticsearch mapping failure on string and date issues. Can you help why it is failing? Here is Field Mapper configurations: { "concat_values_on_all_fields": false, "capture_attribute_values": [], "remove_ns_prefix": true, "skip_attribute_ns_declarations": true, "remove_copied_key": true, "node_delim": "_", "copy_to": {}, "copy_value_to_regex": {}, "copy_to_regex": {}, "split_values_on_all_fields": false, "add_literals": {}, "exclude_attributes": [], "ns_prefix_delim": "|", "self_describing": false, "split_values_on_fields": {}, "include_attributes": [], "include_sibling_id": false, "multivalue_delim": "|", "skip_repeating_values": true, "repeating_element_suffix_count": false, "exclude_elements": [], "concat_values_on_fields": {}, "remove_copied_value": false, "error_on_delims_collision": false, "include_all_attributes": false, "skip_root": false } Here is Mongodb showing record collection:

show dbs

admin 0.000GB combine 0.045GB config 0.000GB local 0.000GB

use combine

switched to db combine

show collections

record

…

________________________________ From: gordonleacock <[email protected]> Sent: Monday, June 1, 2020 4:29 PM To: MI-DPLA/combine-docker <[email protected]> Cc: Negib Sherif <[email protected]>; Author <[email protected]> Subject: Re: [MI-DPLA/combine-docker] pymongo.errors.OperationFailure (#34) Please try this and let us know if you have difficulties. :-) 1. Start mongodb, here’s how. a) If you used ansible to set up combine, just run mongo on the machine where combine is located. b) If you used docker, on the combine machine use a command like docker exec -it combine-docker_mongo_1 /bin/bash and then mongo once inside. 2. Run the mongo command show dbs. 3. If ‘combine' isn’t listed then you need to create it with the command use combine which will create it. 4. Now that we have a combine mongo database and are using it, check on its collections with a show collections command. 5. If ‘record’ is not show as a collection, we need to create it. 6. Use the mongo command db.createCollection(“record”). 7. Exit mongodb and restart combine and try running a job. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#34 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABL3ILLVVL75R3AHYTE5MCLRUQFSTANCNFSM4MMUT27Q>.

[gordonleacock]

We'd like you to check on the versions you are running. Please find out what version of elasticsearch are you using, what version of combine are you using (can check by going into combine-docker/combine/combine and running git status), and what version of combine-docker are you using (can check by going into combine-docker and running git status)?

[waa8288]

This is what shows in docker-compose.yml
version: '3.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.5.1

combine-docker is v0.10 was installed.

[waa8288]

Hi Gordon,
Any suggestion solution for Elasticsearch problem?

[gordonleacock]

Sorry you're having problems with the docker install.

Esty and I suspect that some installation issue caused you to end up with the incorrect version of elastic search.

When you do the git status check described above what do you see?

[waa8288]

This is what i see when issued git status. root@combine-150:/opt/combine-docker/combine/combine# git status On branch master Your branch is up-to-date with 'origin/master'. Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) modified: combine/settings.py modified: combine/urls.py no changes added to commit (use "git add" and/or "git commit -a") Here is Elasticsearch info from .env file. /combine-docker# more .env COMBINE_BRANCH=master COMBINE_DJANGO_HOST_PORT=8000 COMBINE_DOCKER_VERSION=v0.9 ELASTICSEARCH_HADOOP_CONNECTOR_VERSION=5.6.2 HADOOP_VERSION=2.7.5 HADOOP_VERSION_SHORT=2.7 LIVY_TAGGED_RELEASE=v0.5.0-incubating SCALA_VERSION=2.11 SPARK_VERSION=2.3.2 Is there a problem with combine_version v0.10 and Elasticsearch version 5.6.2 that i installed? combine version v0.11 is released 9 days ago. What Elasticsearsh version do you recommend to be installed with combine_version v.010 or v.011?

…

________________________________ From: gordonleacock <[email protected]> Sent: Wednesday, June 10, 2020 9:43 AM To: MI-DPLA/combine-docker <[email protected]> Cc: Negib Sherif <[email protected]>; Author <[email protected]> Subject: Re: [MI-DPLA/combine-docker] pymongo.errors.OperationFailure (#34) Sorry you're having problems with the docker install. Esty and I suspect that some installation issue caused you to end up with the incorrect version of elastic search. When you do the git status check described above what do you see? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#34 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABL3ILKPUR4SEQ2QDGLQYXDRV6EZBANCNFSM4MMUT27Q>.