Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing with mongo 4.4.4 is working for now #916

Open
SebC99 opened this issue Mar 5, 2021 · 17 comments
Open

Testing with mongo 4.4.4 is working for now #916

SebC99 opened this issue Mar 5, 2021 · 17 comments

Comments

@SebC99
Copy link

SebC99 commented Mar 5, 2021

As this is not maintained anymore I was wondering if it could work with MongoDB 4.x version...
So I am testing it on a dev environment on MongoDB Atlas, with version 4.4.4 and for now, everything seems to be working fine.

Just to let you know

@SebC99
Copy link
Author

SebC99 commented Mar 5, 2021

(I'm using a patched version to make it work on 3.6 and with del commands replaced by pop as in #725)

@mahmoudajawad
Copy link

@SebC99, any update on this? Are you now using it in prod? I'm looking at the codebase and seeing that just little of updates here and there (nothing at core) is only required to get this project back on track. I'm not sure why it is abandoned when it seems to be easily maintainable.

@SebC99
Copy link
Author

SebC99 commented May 3, 2021

Yes and it's working perfectly for now
@RiffynInc seems to have a lot of new commits on their fork, it would be awesome if they could explain their additions.
And @tmpaul06 has developed a doc-manager for ElasticSearch 7, which is awesome too, even if I haven't tested any of these forks

@tmpaul06
Copy link

tmpaul06 commented May 3, 2021

@SebC99 I have not received any feedback from the authors since I posted my comment: yougov/elastic2-doc-manager#60 (comment)

The test cases should pretty much cover the basic streaming stuff, and you can try it actively with my repo: https://github.com/tmpaul06/es7-mongo-example

I originally developed this for an internal project at my previous company. I'm not sure if they are using it in production right now.

@angeloruggieridj
Copy link

Hi, I'm trying to execute:
mongo-connector -m localhost:27017 -t http://localhost:7474/neo4j -d neo4j_doc_manager but it send me an exception (ImportError: cannot import name 'GraphError' from 'py2neo' ).
What can I do?

@tmpaul06
Copy link

tmpaul06 commented May 3, 2021

ImportError: cannot import name 'GraphError' from 'py2neo'
@angeloruggieridj

That looks like a version issue. py2neo:4.x has GraphError: https://github.com/technige/py2neo/blob/py2neo-4.3.0/py2neo/database.py

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

@angeloruggieridj
Copy link

angeloruggieridj commented May 3, 2021

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

@tmpaul06
Copy link

tmpaul06 commented May 3, 2021

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

Hmm, what version of neo4j_doc_manager are you using ? From the latest one I see py2neo==2.0.8. Can you try pip list | grep py2neo and see what the output is ? Try with pip install py2neo==2.0.8 if your doc_manager is version: 0.1.2.

See https://github.com/technige/py2neo/blob/py2neo-2.0.8/py2neo/__init__.py#L36

If none of these things fix your problems, try opening a new issue with neo4j_doc_manager, I don't think your issue is related to mongo-connector at all

@angeloruggieridj
Copy link

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

Hmm, what version of neo4j_doc_manager are you using ? From the latest one I see py2neo==2.0.8. Can you try pip list | grep py2neo and see what the output is ? Try with pip install py2neo==2.0.8 if your doc_manager is version: 0.1.2.

See https://github.com/technige/py2neo/blob/py2neo-2.0.8/py2neo/__init__.py#L36

If none of these things fix your problems, try opening a new issue with neo4j_doc_manager, I don't think your issue is related to mongo-connector at all

py2neo is 2021.0.1
neo4j-doc-manager is 0.1.1

If I try to upgrade neo4j-doc-manager to 0.1.2 I receive this error:

ERROR: Could not find a version that satisfies the requirement py2neo==2.0.8 (from neo4j-doc-manager) (from versions: 4.0.0, 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.2.0, 4.3.0, 2020.0.0, 2020.1.0, 2020.1.1, 2021.0.0, 2021.0.1)
ERROR: No matching distribution found for py2neo==2.0.8

@tmpaul06
Copy link

tmpaul06 commented May 3, 2021

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

Hmm, what version of neo4j_doc_manager are you using ? From the latest one I see py2neo==2.0.8. Can you try pip list | grep py2neo and see what the output is ? Try with pip install py2neo==2.0.8 if your doc_manager is version: 0.1.2.
See https://github.com/technige/py2neo/blob/py2neo-2.0.8/py2neo/__init__.py#L36
If none of these things fix your problems, try opening a new issue with neo4j_doc_manager, I don't think your issue is related to mongo-connector at all

py2neo is 2021.0.1
neo4j-doc-manager is 0.1.1

If I try to upgrade neo4j-doc-manager to 0.1.2 I receive this error:

ERROR: Could not find a version that satisfies the requirement py2neo==2.0.8 (from neo4j-doc-manager) (from versions: 4.0.0, 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.2.0, 4.3.0, 2020.0.0, 2020.1.0, 2020.1.1, 2021.0.0, 2021.0.1)
ERROR: No matching distribution found for py2neo==2.0.8

Ahh I see the problem. The author has removed that version from pypi. You have 2 choices:

a) Install from source: See https://stackoverflow.com/questions/23521345/python-pip-how-do-i-install-a-specific-version-of-a-git-repository-from-githu , so something like pip install git+https://github.com/technige/[email protected]

b) Fork neo4j_doc_manager, upgrade it to match py2neo latest API. Frankly this is an uphill battle.

Logs from my local machine:

tharun@Tharun-XPS-9300:~$ pip install git+https://github.com/technige/[email protected]
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/technige/[email protected]
  Cloning https://github.com/technige/py2neo.git (to revision py2neo-2.0.8) to /tmp/pip-req-build-w9vfhtbr
Building wheels for collected packages: py2neo
  Building wheel for py2neo (setup.py) ... done
  Created wheel for py2neo: filename=py2neo-2.0.8-py3-none-any.whl size=154206 sha256=3f8afbb985991e65ae345dfdd47111e4d6e037d4f8ec5f916d620f8f3f091698
  Stored in directory: /tmp/pip-ephem-wheel-cache-5e9x0q2d/wheels/80/b0/24/0ebaad24ce726156942db923b0d371ae590882c818590f8aa1
Successfully built py2neo
Installing collected packages: py2neo
Successfully installed py2neo-2.0.8
WARNING: You are using pip version 20.2.2; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.
tharun@Tharun-XPS-9300:~$ pip list | grep py2neo
py2neo                   2.0.8
WARNING: You are using pip version 20.2.2; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.

@angeloruggieridj
Copy link

angeloruggieridj commented May 3, 2021

So, I've followed your advice:

  • Downgraded to py2neo 2.0.8
  • Upgraded to neo4j-doc-manager to 0.1.2

So, after a restart of MongoDB and Neo4j, I've tried again to execute:
mongo-connector -v -m localhost:27017 -t http://localhost:7474/data/db -d neo4j_doc_manager
Finally, in the Terminal app, it shows Logging to /Users/angeloruggieridj/mongo-connector.log. and after 5 seconds it stops and show again the (base) user.

I've attached the file mongo-connector.log. What's the cause now?

My Neo4j DB version is 4.1.0, so not the latest version.

@tmpaul06
Copy link

tmpaul06 commented May 3, 2021

So, I've followed your advice:

  • Downgraded to py2neo 2.0.8
  • Upgraded to neo4j-doc-manager to 0.1.2

So, after a restart of MongoDB and Neo4j, I've tried again to execute:
mongo-connector -v -m localhost:27017 -t http://localhost:7474/data/db -d neo4j_doc_manager
Finally, in the Terminal app, it shows Logging to /Users/angeloruggieridj/mongo-connector.log. and after 5 seconds it stops and show again the (base) user.

I've attached the file mongo-connector.log. What's the cause now?
mongo-connector.log

My Neo4j DB version is 4.1.0, so not the latest version.

The log clearly says that Neo4j is not reachable/running/auth is incorrect. I think if you have any further issues, you should post your questions to stackoverflow. I don't think discussion on this issue is the right approach as it will spam the issue author. One more thing: Be wary of posting your files on the public internet: it contains the name of your mongo collection as well as the username password combo for your local neo4j setup.

2021-05-03 18:25:02,034 [DEBUG] httpstream:316 - > Host: localhost:7474
2021-05-03 18:25:02,652 [INFO] httpstream:511 - < 404 Not Found [0]

@mahmoudajawad
Copy link

@tmpaul06, are you still maintaining your fork? If not, what it takes to maintain it? This seems like a lightweight project that can be maintained by anyone interested in the tech, like me, but I'm not sure the method used to maintain it. For instance, why did you need to fork it, and what have you done after forking it and continued to?

@tmpaul06
Copy link

tmpaul06 commented May 6, 2021

@tmpaul06, are you still maintaining your fork? If not, what it takes to maintain it?

My goal was to merge it with upstream repo, never to keep it as a standalone project. The authors don't seem interested in the fork anyway, and their suggested method to support ES 7.x is pretty crude (doc_type name remap). Without help from the authors of the project, I cannot commit to long term maintenance.

For instance, why did you need to fork it, and what have you done after forking it and continued to?

See https://github.com/tmpaul06/elastic7-doc-manager#changelog
My previous company used this fork to move documents from mongo to ES 7.x (the latest at that time). Since I am no longer with them, I cannot confirm if they are still using it in production. As for the actual code changes, I haven't changed the core mechanism: yougov/elastic2-doc-manager@master...tmpaul06:master

If you are interested in maintaining it, feel free to do so. That is the reason why I have not published my fork to pypi. As for maintenance, you would need to match versions with elastic2-doc-manager, fix bugs as they are raised, and finally and most importantly support future versions of ES

@singhmahendrapal
Copy link

Hello Guys, Hope you are doing well!!
Today I lost my mongo-connector oplog.ts file. Is there any way to recover that one from mongodb cluster?
It's very urgent and critical for me. Please help on it.

Thanks

@mahmoudajawad
Copy link

@tmpaul06, thanks for your reply. I shall try and evaluate it. Most probably I can maintain it as long as I can use it myself.

@warby613
Copy link

warby613 commented Dec 16, 2021

While getting mongo-connector running with MongoDB 4.4, I had to use pymongo version ~-= 3.12.
Pymongo 4 introduced changes that break mongo-connector.
In case this helps anyone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants