Skip to content

Reactivate Travis-CI #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jul 6, 2017
Merged

Reactivate Travis-CI #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
da0b6bf
fix language setting
ocefpaf Jul 4, 2017
c4ec98c
with addons we don't need sudo
ocefpaf Jul 4, 2017
bfd0b68
clean unnecessary comments
ocefpaf Jul 4, 2017
6a2db1e
don't cache pip, I'll implement conda caching later
ocefpaf Jul 4, 2017
4c1947c
organize and comment on the addons
ocefpaf Jul 4, 2017
c02e32f
install the env with conda
ocefpaf Jul 4, 2017
ed75341
fix indentation
ocefpaf Jul 4, 2017
7f9b670
actually the the tests
ocefpaf Jul 4, 2017
ee3ae1c
define a test matrix
ocefpaf Jul 4, 2017
9e0a670
clean setup.py up
ocefpaf Jul 4, 2017
b488913
run tests using the installed version
ocefpaf Jul 4, 2017
3fdc68a
I am not sure this file is necessary and it uses the old ioos channel
ocefpaf Jul 4, 2017
2834035
converter -> model
ocefpaf Jul 5, 2017
52145df
remove dups
ocefpaf Jul 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 47 additions & 43 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,67 +1,71 @@
git blanguage: python
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably a typo that prevented Travis-CI from running all this time.

sudo: required
# if the https://travis-ci.org/ODM2/ODM2PythonAPI/requests ever says: missing config
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't really need sudo and without it Travis-CI can run in a faster using the container infrastructure.

PS: I removed all the unnecessary comments.

# validate at: http://lint.travis-ci.org/
python:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this to a test matrix definition with only Python 2.7 and 3.6.
Python 3.6 is unsupported now but I aim to change that soon.

# - "2.6"
- "2.7"
# - "3.2"
# - "3.3"
# - "3.4"
# - "3.5"
# - "3.5-dev" # 3.5 development branch
# - "nightly" # currently points to 3.6-dev
# command to install dependencies
cache:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid a big PR I will add a caching mechanism in another PR as soon as this is merged.

directories:
- $HOME/.cache/pip/wheels
- $HOME/virtualenv/python2.7.9
language: python

sudo: false

services:
- mysql
- postgresql

addons:
apt:
packages:
- cmake
- unixodbc
- unixodbc-dev
# FIXME: when activating the services above I don't think these are needed.
- odbcinst1debian2
- odbcinst
- freetds-dev
- freetds-bin
- tdsodbc
- libc6
- e2fsprogs
# FIXME: the dependencies below can probably be replaced by conda.
- libc6
- freetds-dev
- freetds-bin
- unixodbc
- unixodbc-dev
- mysql-client
# FIXME: I don't see the need for these anywhere.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really think we need any of these packages from the package manager.
Can someone with a more intimate knowledge of odm2api comment on this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which packages? If it's libproj-dev, libgeos-dev and libspatialiate-dev, I agree, they're not needed. I can't comment on cmake.

- cmake
- libproj-dev
- libgeos-dev
- libspatialite-dev
# mariadb: '10.1'

matrix:
fast_finish: true
include:
- python: 2.7
env: TEST_TARGET=default
- python: 3.6
env: TEST_TARGET=default
- python: 2.7
env: TEST_TARGET=coding_standards
allow_failures:
- python: 2.7
env: TEST_TARGET=coding_standards
- python: 3.6
env: TEST_TARGET=default

before_script:
- ./ci-helpers/mysql_setup.sh
- ./ci-helpers/postgres_setup.sh
- ./ci-helpers/travis/freetds.sh


before_install:
# python -m pip makes the install go into the virtualenv
- sudo pip install --upgrade pip
- sudo python -m pip install pandas
- export PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1;sudo python -m pip install pymssql
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this complex pip installation to a more straight forward and reproducible conda env.
However, we may be losing some of the functionalities here. Unfortunately I cannot assess that without more comprehensive test.

For example, why do we need PYMSSQL_BUILD_WITH_BUNDLED_FREETDS?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the current setup test against an MSSQL database? I realize it is a Linux system so I am wondering if this was included to allow testing from MSSQL. If we aren't supporting that from here (which I am assuming we aren't) then we do not need it.

Copy link
Member Author

@ocefpaf ocefpaf Jul 5, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the current setup test against an MSSQL database?

Probably not. I don't really know the unit/integration tests details and the current failures are happening due to other causes beyond the available services. (Also, databases are not my strong suit so I wouldn't know 😬)

If we aren't supporting that from here (which I am assuming we aren't) then we do not need it.

@lsetiawan can you take a look at the tests and confirm that they don't requite a database service?

PS: note that we can do integration tests that relies on databases services in the future, but I would like to get a small set of basic unittests working first.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lsetiawan can you take a look at the tests and confirm that they don't requite a database service?

Seems like odm2api tests are only using sqlite for the tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PS: note that we can do integration tests that relies on databases services in the future, but I would like to get a small set of basic unittests working first.

I agree with that approach. We start with SQLite.

Once we've built up to including RDBMS systems in the test suite, we can go back to considering MSSQL / FreeTDS testing in AppVeyor.

# - python -m pip install mysql-python
install: # now just our code
- sudo pip install -e git+https://github.com/ODM2/[email protected]#egg=geoalchemy-0.7.4
- python -m pip install .
- python -m pip install -r requirements_tests.txt --allow-external pyodbc --allow-unverified pyodbc
# pysqlite
# - pip install pysqlite
- pip list
- wget http://bit.ly/miniconda -O miniconda.sh
- bash miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- conda update --yes --all
- conda config --add channels odm2 --force
- conda create --yes -n TEST python=$TRAVIS_PYTHON_VERSION --file requirements.txt --file requirements-dev.txt
- source activate TEST

# don't forget to open up the azure mssql server to these addreses
# https://docs.travis-ci.com/user/ip-addresses/
# Test source distribution.
install:
- python setup.py sdist && version=$(python setup.py --version) && pushd dist && pip install odm2api-${version}.tar.gz && popd
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is important to know if the source tarball is installable. We should not test against the source code tree.


# command to run tests
script:
- if [[ $TEST_TARGET == 'default' ]]; then
cp -r tests /tmp && cd /tmp ;
py.test -vv tests ;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ensure that we are running the tests from the installed version and not the source code tree.

fi

- py.test

- if [[ $TEST_TARGET == 'coding_standards' ]]; then
find . -type f -name "*.py" ! -name 'conf.py' | xargs flake8 --max-line-length=100 ;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The coding standards tests, together with the Python 3 tests, are in the allow_failures section at the moment.
I plan to send small PRs to fix that and make these tests first class citizens here.

fi
14 changes: 0 additions & 14 deletions condaenvironment_1.yml
View file
Open in desktop

This file was deleted.

11 changes: 11 additions & 0 deletions requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
flake8
mock
psycopg2
pymysql
pytest
pytest-cov
# FIXME: I am not sure these are needed.
# pysqlite
# sqlite
# pyspatialite
# mysql-python
2 changes: 0 additions & 2 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
pyodbc
six
sqlalchemy
pandas
#psycopg2 # Commented out because I could not pip install it.
15 changes: 0 additions & 15 deletions requirements_tests.txt
View file
Open in desktop

This file was deleted.

114 changes: 16 additions & 98 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,133 +5,51 @@
https://github.com/pypa/sampleproject
"""

'''
to install in development mode, run the following code from command line
"python setup.py develop"
'''

# Always prefer setuptools over distutils
from setuptools import setup, find_packages
# To use a consistent encoding
import os
from codecs import open
from os import path

from setuptools import find_packages, setup

import versioneer

here = path.abspath(path.dirname(__file__))
here = os.path.abspath(os.path.dirname(__file__))

# Dependencies.
with open('requirements.txt') as f:
requirements = f.readlines()
install_requires = [t.strip() for t in requirements]

from pip.req import parse_requirements
install_reqs = parse_requirements('requirements.txt', session=False)
reqs = [str(ir.req) for ir in install_reqs]
with open(os.path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()

# Get the long description from the relevant file
with open(path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
#long_description = ""
setup(
name='odm2api',

# Versions should comply with PEP440. For a discussion on single-sourcing
# the version across setup.py and the project code, see
# https://packaging.python.org/en/latest/single_source_version.html
version=versioneer.get_version(),

description='A Python-based application programmers interface for the Observations Data converter 2 (ODM2) ',
description='Python interface for the Observations Data Model 2 (ODM2)',
long_description=long_description,

# The project's main homepage.
url='https://github.com/ODM2/ODM2PythonAPI',

# Author details
author='ODM2 team-Stephanie Reeder',
author_email='[email protected]',

# note: maintainer gets listed as author in PKG-INFO, so leaving
# this commented out for now
maintainer='David Valentine',
maintainer_email='[email protected]',

# Choose your license
license='BSD-3-Clause',

# See https://pypi.python.org/pypi?%3Aaction=list_classifiers
license='BSD',
classifiers=[
# How mature is this project? Common values are
# 3 - Alpha
# 4 - Beta
# 5 - Production/Stable
'Development Status :: 3 - Alpha',

# Indicate who your project is intended for
'Intended Audience :: Developers',
'Topic :: Software Development :: Build Tools',

# Pick your license as you wish (should match "license" above)
'License :: OSI Approved :: BSD License',
'Operating System :: OS Independent',
# Specify the Python versions you support here. In particular, ensure
# that you indicate whether you support Python 2, Python 3 or both.
'Programming Language :: Python :: 2.7',
'Topic :: Software Development :: Libraries :: Python Modules',
'Topic :: Scientific/Engineering'
],

# What does your project relate to?
keywords='Observations Data converter ODM2',

# You can just specify the packages manually here if your project is
# simple. Or you can use find_packages().

keywords='Observations Data Model ODM2',
packages=find_packages(exclude=['samplefiles', 'setup', 'tests*', 'Forms']),

# List run-time dependencies here. These will be installed by pip when
# your project is installed. For an analysis of "install_requires" vs pip's
# requirements files see:
# https://packaging.python.org/en/latest/requirements.html

install_requires=install_reqs,
# install_requires=[
# 'pyodbc',
# 'six',
# 'sqlalchemy',
# 'shapely',
# 'pandas',
# ],

# List additional groups of dependencies here (e.g. development
# dependencies). You can install these using the following syntax,
# for example:
# $ pip install -e .[dev,test]
install_requires=install_requires,
extras_require={
'mysql': ['pymysql'],
'postgis': ['psycopg2'],
'sqlite': ['pyspatialite >=3.0.0'], # need to look at: http://www.gaia-gis.it/spatialite-2.4.0-4/splite-python.html
'test': ['coverage'],
'sqlite': ['pyspatialite >=3.0.0'],
},

# If there are data files included in your packages that need to be
# installed, specify them here. If using Python 2.6 or less, then these
# have to be included in MANIFEST.in as well.
#
# package_data={
# 'sample': ['package_data.dat'],
# },

# Although 'package_data' is the preferred approach, in some case you may
# need to place data files outside of your packages. See:
# http://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files # noqa
# In this case, 'data_file' will be installed into '<sys.prefix>/my_data'

# data_files=[('my_data', ['data/data_file'])],

# To provide executable scripts, use entry points in preference to the
# "scripts" keyword. Entry points provide cross-platform support and allow
# pip to create the appropriate form of executable for the target platform.

# entry_points={
# 'console_scripts': [
# 'sample=sample:main',
# ],
# },
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly cleanups. The comments are not really helpful and get in the way of reading the code here.

cmdclass=versioneer.get_cmdclass(),
)