Installing as package, from "setup" branch

[emiliom]

@sreeder and @horsburgh, following up on our call from Monday, I've installed ODM2API. I think it needs more work, but here's what I did that was successful.

• I created a conda environment with all high-level dependencies, plus some more. This wasn't necessary, strictly, speaking, but I definitely wanted to test this installation in a virtual environment that wouldn't mess with my system Python. Plus staging things with conda makes for a more controlled environment. FYI: conda create -n odm2_apitest1 python=2.7 ipython-notebook pandas seaborn sqlalchemy psycopg2 pymysql pyodbc.
• I followed that step with pip install geoalchemy2 (geoalchemy2 is not currently available as a conda package), b/c geoalchemy2 turned out to be a dependency in the ODM2API setup git branch that's not handled by the python setup.py develop step below. I found out the hard way, when I tried to run import api after installing ODM2API.
• Made sure I was on the setup ODM2API git branch.
• cd'd to the ODM2API src directory (where setup.py) is found, and per Stephanie's instructions, ran python setup.py develop. This went very smoothly and quickly, as I already had all the dependencies in place

So far so good. Per Stephanie's instructions, I was then able to run import api w/o errors.

The first, main problem I see (in addition to the unhandled geoalchemy2 dependency) is that there's no umbrella "odm2api" package that's installed. Having to import a package called "api" is kind of dangerous and confusing. I don't have much Python package configuration chops, so I can't help with this in the very short term, and I don't know why the package is being named 'api'.

Beyond that, it's pretty involved to have to do a local git clone, then do python setup.py develop based off the right directory path. It would be best if ODM2API could be packaged as a "pip installable" package that can be installed directly from github, like this (where setup is the setup branch):

pip install git+https://github.com/ODM2/ODM2PythonAPI.git@setup

FYI, I tried that and got this error message:

Collecting git+https://github.com/ODM2/ODM2PythonAPI.git@setup
  Cloning https://github.com/ODM2/ODM2PythonAPI.git (to setup) to /tmp/pip-yFXVBb-build
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 18, in <module>
    IOError: [Errno 2] No such file or directory: '/tmp/pip-yFXVBb-build/setup.py'

I have a hunch the problem has to do with setup.py being under the src directory rather than at the base of the repo, but again, I'm reaching the depth of my understanding.

I really think it'd be time well spent if we (I'm volunteering) invested the effort very soon to turn this into a pip installable package with a proper name/namespace (eg, "odm2api", or just just "odm2"). That would make it easier for others to try it. Maybe Choonhan (and Dave V?) can help us with this?

Done for now. Have a great Thanksgiving!

[emiliom]

Alrighty! I went ahead and fixed most of the issues I pointed out:

• Made a large reorganization of the code so setup.py could be at the root of the repo and the name of the package could be odm2api. The main, high-level change was renaming src to odm2api and moving src/api to odm2api
• edited setup.py a tad (including adding added geoalqchemy2 to the dependencies)
• editing a bunch of init.py and other files to change references to import api. Also did a couple other import reference cleanups

See the new setup_em branch.

Now I can do this conda installation:

conda create -n odm2_apitest python=2.7 ipython-notebook pandas seaborn sqlalchemy psycopg2 pymysql pyodbc
source activate odm2_apitest
pip install git+https://github.com/ODM2/ODM2PythonAPI.git@setup_em

Then import odm2api works w/o complaining. Progress, if you ask me ...

[emiliom]

And I'm guessing that, on a normal (non-conda) python setup, the package may be installable as easily as:

pip install git+https://github.com/ODM2/ODM2PythonAPI.git@setup_em

@cdesyoun, I'm pinging you so you're aware of this discussion, since you use ODM2API in the web service packages. Hopefully soon (by the end of the coming week??) we'll have ODM2API in solid shape so its master branch is pip installable.

I'm done with this until @sreeder and @horsburgh have a had a chance to review, test, edit, rinse and repeat.

[horsburgh]

@emiliom and @sreeder - I thought we were getting rid of geoalchemy2 as a dependency for now?

[emiliom]

Yes, but that's a separate issue. Here I was only addressing packaging and installation, using Stephanie's setup branch as the starting point.

[valentinedwv]

Need to separate testing and demo requirements from the install requirements.

We don't need numpy, and pandas to install the api.
You do need them to do the demos

[emiliom]

Good point, Dave. I didn't try to make any such separation yet (test+demo vs api install), since I was focused on more basic issues.

[emiliom]

@sreeder, is the geometry branch ready to be merged with the setup and/or setup_em branches? From what you told me this week, it sounded that way.

It'd be really helpful to be done with the directory reorganization and python packaging steps ASAP, and even better if we can fold in your geoalchemy changes (#13). Right now it looks like we'll have to merge commits from 3 branches: geometry, setup and setup_em. It will get hairier if we don't take care of it soon. Can we shoot for finishing this early next week? @horsburgh, does that work for you as well?

An additional motivator is the fantastic progress with and interest in ODM2/WOFpy, which has ODM2PythonAPI as a dependency.

[sreeder]

@emiliom I was trying to get spatialite working with the goemetries before doing the merge, but that can wait a few days. I will get to merging everything now so that we have an easy install of all our working code.

[sreeder]

@emiliom @horsburgh I have been able to merge all of the branches into setup_em. So this should have all of the working code. I can merge it into master as soon as I am given the go ahead.

[emiliom]

@sreeder: WOW!! Thank you! Clearly you're a git pro.

I completely agree with setting aside the sqlite/spatialite issue and focusing on merging the branches first.

I'm good with you merging into master. But I'd like to see confirmation from you and @horsburgh that you agree with the decisions I made when I reorganized the directories in my setup_em branch, and decided to call the package odm2api (all lowercase). And if you don't agree, then tweak as needed!

[emiliom]

Regarding merging: actually, if you'd like, I can install the setup branch (now that you've merged geometry and setup_em into setup) and do a bunch of tests over the weekend, before you go ahead and merge into master.

[horsburgh]

@emiliom - I don't have any issues with the "odm2api" lower case. And, I don't really have an educated opinion about the folder restructuring.

Some testing would be great.

[sreeder]

@emiliom I actually merged everything into setup_em. Its easier to move into the restructured repository than visa versa. I thought the name odm2api was great and the restructuring is consistent with how I would have done it as well. I will wait until Monday to finish the merging into master.

[emiliom]

Thanks for getting back to me on whether the packaging in the setup_em branch makes sense! @sreeder, thanks for the clarification that you merged into setup_em, not setup; I can definitely appreciate that reorganizing directories can make merging trickier.

My postgresql testing (using my "Marchantaria" use case database, locally) were largely successful, and I'm done (at least with issues in our initial target):

• geoalchemy gave me grief initially b/c pip installing odm2api resulted in installing geoalchemy from pypi, which doesn't incorporate the fix Stephanie had identified. See geoalchemy vs geoalchemy2 dependency #13. Briefly, I created a geoalchemy fork in ODM2, applied the fix there, and pip installed that fork first.
• I successfully installed odm2api via pip install git+https://github.com/ODM2/ODM2PythonAPI.git@setup_em, after first creating a conda env with pandas sqlalchemy psycopg2 pymysql pyodbc (and a couple of other things).
• Problem creating connection to PostgreSQL database #12 is fixed. I've closed that issue.

The only limitation I ran into is with SamplingFeatures, I'm unable to extract its geometry information (say, from the FeatureGeometry property), even though I can print out a sample, eg, <SamplingFeatures('march', 'Solimoes - Marchantaria', 'Station ID 1036', '24.0', 'POINT (-60.01187 -3.29313)')>. The behavior is different from what I had seen up until early August.

I have these tests in an IPython notebook that I could share later, but first I'll have to remove the database credential info.

My tests are actually pretty limited, b/c I'm still very new at odm2api and SQLAlchemy, and I had always ran into obstacles in the past. With some initial handholding I could do more extensive testing. Hopefully I can get past that SamplingFeatures problem with some guidance.

[emiliom]

After inspecting odm2api.ODM2.models, I found the shapely-based geometry-extraction code in the __repr__ method. That let me get at the geometry. But this is messier than it used to be, more convoluted. It ought to be much simpler to get at the geometry. We'll need to follow up, via #13 or a separate issue.

[emiliom]

@sreeder, thanks for pushing setup_em to the master branch, and the additional cleanups done in your commit, including the change from Point to Geometry geometry type.

To minimize confusion in the future, let's delete the setup and setup_em branches soon. Unless you or @horsburgh disagree (or do it first!), I'll delete them by next week, hopefully earlier. Is there any other branch that should be deleted at this time? Say, is the dev branch dead, or being use actively?

[emiliom]

A status update:

• I've deleted the setup_em branch.
• The setup branch has one small change made by @denvaar after @sreeder's massive merges into master. I've asked him to apply it to master, then we should delete this branch.
• We're still finalizing the specification of the ODM2/geoalchemy dependency in odm2api, so it (rather than the pypi version) installs when odm2api is pip installed

[emiliom]

Closing this issue. I've created instructions for installing odm2api using conda packages for all the dependencies except geoalchemy, followed by a single pip install command to install odm2api and geoalchemy. See README.md.

Gory packaging details can be found on issue #18 and associated pull requests (PR #19 and PR #20).

Recent commits from the setup branch should be merged into master, so we can get rid of that old branch (the new commits from @denvaar are on top of an old code base). But that's for a separate issue/task.