Skip to content
This repository has been archived by the owner on Nov 17, 2022. It is now read-only.

Commit

Permalink
Merge pull request #164 from IATI/dev
Browse files Browse the repository at this point in the history 
Update master from dev
  • Loading branch information
@hayfield
hayfield authored Oct 19, 2017
2 parents ef5d1d2 + a503b81 commit cd992a1
Show file tree
Hide file tree
Showing 1,523 changed files with 5,841 additions and 1,250 deletions.
52 changes: 42 additions & 10 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,47 @@ language: python
cache:
pip
python:
- "2.7"
- "3.4"
- "3.5"
- "3.6"
- 2.7
- 3.4
- 3.5
- 3.6
install:
- pip install -U pip wheel
- pip install -r requirements-dev.txt
- pip install -U pip wheel
- pip install -r requirements-dev.txt
script:
- make test
- make lint
- make complexity
- make docs
- make test

stages:
- test
- lax lint
- strict lint

jobs:
include:
- stage: lax lint and other checks
python: 3.5
script:
- make lint
- make complexity
- make docs
if: branch=dev
- stage: strict lint
python: 3.5
install:
- pip install -U pip wheel
- pip install -r requirements-dev.txt
env: LINTER=pylint
script: make pylint
if: branch=master
- stage: strict lint
python: 3.5
install: pip install flake8
env: LINTER=flake8
script: make flake8
if: branch=master
- stage: strict lint
python: 3.5
install: pip install pydocstyle
env: LINTER=pydocstyle
script: make pydocstyle
if: branch=master
108 changes: 108 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

### Changed

### Deprecated

### Removed

### Fixed

### Security

## [0.1.0] - 2017-10-19

### Added

- [Codelists] Blank Codelists may be created and given a name.
- [Codelists] Codelists may be created from a string conforming to the Codelist Schema.
- [Codelists] Codelists contain a set of Codes.
- [Codelists] Codelists may be output as a XSD simpleType restriction containing enumeration elements.
- [Codelists] Codelist equality is based on whether the names and contained Codes are the same.
- [Codelists] Codes may be created from an optional value and name.
- [Codelists] Codes may be output as a XSD enumeration to be contained within a simpleType restriction.
- [Codelists] Code equality is determined based on whether name and value are the same.

- [Constants] A number of constants are defined, though should be deemed private to the library.

- [Datasets] Datasets can represent each Activity and Organisation files (though not at the same time!).
- [Datasets] A Dataset may be created either from a string or a lxml ElementTree.
- [Datasets] The contents of a Dataset can be accessed as each a string and an ElementTree. These values remain in sync with each other.
- [Datasets] The version of the IATI Standard at which a Dataset is defined can be obtained through the `version` property.
- [Datasets] Subsections of a Dataset (indexed by line number) may be obtained in string format.

- [Defaults] The `defaults` module provides access to the contents of the SSOT.
- [Defaults] To obtain SSOT contents, the desired version of the Standard must be specified. Should a version not be specified, the latest version will be assumed.
- [Defaults] Specific Codelists from the SSOT may be accessed by name.
- [Defaults] All Codelists from the SSOT may be accessed at once. No differentiation is made between Embedded and Non-Embedded Codelists.
- [Defaults] The Standard Ruleset may be accessed.
- [Defaults] The Ruleset Schema may be accessed. *NOTE:* This is a JSON (dict) object, not an `iati.Schema`.
- [Defaults] The Activity Schema may be accessed.
- [Defaults] The Organisation Schema may be accessed.
- [Defaults] By default, the Activity and Organisation Schemas will be populated with all the Codelists and Rulesets at the desired version of the Standard.
- [Defaults] Unpopulated Activity and Organisation Schemas may be accessed by overriding a default argument.

- [Exceptions] There is very rudimentary exception handling.
- [Exceptions] Some exceptions are logged.
- [Exceptions] Some raised exceptions have descriptions to make it possible to consistently determine the cause.

- [Resources] Resources are static files that come with the library, split into 'The Standard' and 'Test Data'.
- [Resources] Resources are stored and accessed based on a particular Decimal Version.
- [Resources] Schemas, Codelists and Rulesets are available for V1.04-2.02.
- [Resources] Schemas are available for V1.03 and prior.
- [Resources] Non-embedded Codelists are deemed to have a single form valid for V1.04-2.02.
- [Resources] Schema test data from the SSOT is present for V1.01-2.02.
- [Resources] Custom test data has been created for V2.02.
- [Resources] Resources may be loaded from disk as bytes, a Dataset, a string or an ElementTree.

- [Rulesets] Blank Rulesets may be created by providing no arguments.
- [Rulesets] Rulesets may be created from a string conforming to the Ruleset Schema.
- [Rulesets] Rulesets contain a set of Rules.
- [Rulesets] A Dataset may have truthy conformance checked against a Ruleset.
- [Rulesets] Rules may be created from a `context` and a `case`.
- [Rulesets] Each of the Rules in the Ruleset Schema have been implemented for instantiation and truthy conformance. These are: atleast_one, date_order, dependent, no_more_than_one, regex_matches, regex_no_matches, startswith, sum, unique
- [Rulesets] A Rule base class exists to allow the creation of custom types of Rule.
- [Rulesets] The string representation of a Rule details what must occur for a Dataset to be conformant.
- [Rulesets] A Dataset may have truthy conformance checked against a Rule. NOTE: A Rule may skip or raise an error.

- [Schemas] Schemas may be created by providing a path to a file containing a valid XSD.
- [Schemas] Schemas contain a set of Codelists.
- [Schemas] Schemas contain a set of Rulesets.
- [Schemas] Should a XSD be defined across multiple files, the resulting Schema shall be flattened to allow programmatic access to all of its contents.
- [Schemas] Activity and Organisation Schemas have been implemented.

- [Utility] A new namespace may be added to a Schema.
- [Utility] There is a function to convert from ElementTree to XMLSchema.
- [Utility] There is a function to convert from a string to an ElementTree.
- [Utility] There is a function to work around RFC4627 by preventing duplicate keys in a JSON file.
- [Utility] There is very rudimentary logging functionality.
- [Utility] Various functionality to access Version Numbers is present.

- [Compatibility] Compatible with Python 2.7, 3.4, 3.5 and 3.6.

- [Tests] All code is comprehensively unit tested.
- [Tests] Datasets, Rulesets, and other components permitting user-defined input tested with range of valid and invalid data.
- [Tests] Variety of never-before-seen edge-cases tested for correct behavior.
- [Tests] Rudimentary custom fuzzing used to ensure expected errors are raised.

- [Tools] Documentation is generated with Sphinx, based on Google-style docstrings.
- [Tools] Documentation is presented using the Sphinx Read the Docs theme.
- [Tools] Linting performed with pylint, flake8 and pydocstyle.
- [Tools] Linters are configured to generally follow PEP8 and PEP257, with a few custom modifications such as line length.
- [Tools] Code complexity is assessed using radon.
- [Tools] Unit testing is performed with pytest.
- [Tools] Full testing is performed by Travis.
- [Tools] Python 2.7 compatibility is provided by both `future` and `six`. Where a custom workaround is used rather than one of these, `python2/3` is present in a comment.

- [XML] lxml utilised for XML functionality.
- [XML] Support XML 1.0.
- [XML] Support XMLSchema 1.0.
- [XML] Support XPath 1.0.
3 changes: 3 additions & 0 deletions LICENSES-3RD-PARTY.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -413,10 +413,13 @@ Public License instead of this License.
applies to:
- flake8, Copyright (C) 2011-2013 Tarek Ziade <[email protected]>
Copyright (C) 2012-2016 Ian Cordasco <[email protected]>
- future, Copyright (c) 2013-2016 Python Charmers Pty Ltd, Australia
- jsonschema, Copyright (c) 2013 Julian Berman
- pydocstyle, Copyright (c) 2012 GreenSteam, <http://greensteam.dk/>
Copyright (c) 2014-2017 Amir Rachum, <http://amir.rachum.com/>
- pytest, Copyright (c) 2004-2016 Holger Krekel and others
- pytest-cov, Copyright (c) 2010 Meme Dough
- pytest-xdist
- radon, Copyright (c) 2012-2014 Michele Lacchia
- six, Copyright (c) 2010-2015 Benjamin Peterson
- sphinx_rtd_theme, Copyright (c) 2013 Dave Snider
Expand Down
18 changes: 15 additions & 3 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,23 @@ docs: $(IATI_FOLDER) $(DOCS_FOLDER_SOURCE)


lint: $(IATI_FOLDER)
-pylint $(IATI_FOLDER)
-make pylint
echo $(LINE_SEP)
-flake8 $(IATI_FOLDER)
-make flake8
echo $(LINE_SEP)
- pydocstyle $(IATI_FOLDER)
-make pydocstyle


pylint: $(IATI_FOLDER)
pylint $(IATI_FOLDER)


flake8: $(IATI_FOLDER)
flake8 $(IATI_FOLDER)


pydocstyle: $(IATI_FOLDER)
pydocstyle $(IATI_FOLDER)


test: $(IATI_FOLDER)
Expand Down
65 changes: 44 additions & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# iati.core
# pyIATI

The iati.core Python module.
A developers’ toolkit for IATI.

[![Build Status](https://travis-ci.org/IATI/iati.core.svg?branch=master)](https://travis-ci.com/IATI/iati.core) [![Requirements Status](https://requires.io/github/IATI/iati.core/requirements.svg?branch=master)](https://requires.io/github/IATI/iati.core/requirements/?branch=master)
[![Build Status](https://travis-ci.org/IATI/iati.svg?branch=master)](https://travis-ci.com/IATI/iati.core) [![Requirements Status](https://requires.io/github/IATI/iati.core/requirements.svg?branch=master)](https://requires.io/github/IATI/iati.core/requirements/?branch=master)

Varying between: [![experimental](http://badges.github.io/stability-badges/dist/experimental.svg)](http://github.com/badges/stability-badges) and [![unstable](http://badges.github.io/stability-badges/dist/unstable.svg)](http://github.com/badges/stability-badges) (see docstrings)

Expand All @@ -25,7 +25,7 @@ It is planned that different sections of the library, such as `validate` are spl
General Installation for System Use
===================================

```
```shell
# install software dependencies
apt-get install python-pip libxml2-dev libxslt-dev python-dev

Expand All @@ -38,7 +38,7 @@ Documentation

At present, an HTML documentation site can be generated using the following commands:

```
```shell
# to build the documentation
sphinx-apidoc -f -o docs/source/ iati/
sphinx-build -b html docs/source/ docs/build/
Expand All @@ -60,9 +60,9 @@ A number of default IATI `.xsd` schema files are included as part of the library

The following example loads the default IATI v2.02 `iati-activities-schema.xsd` schema:

```
import iati.core.default
schema = iati.core.default.schema('iati-activities-schema')
```python
import iati.default
schema = iati.default.schema('iati-activities-schema')
```

Helper functions will be written in due course to return all xpaths within a schema, as well as documentation for each element.
Expand All @@ -71,30 +71,53 @@ Helper functions will be written in due course to return all xpaths within a sch

A given IATI codelist can be added to the schema. Example using the [Country](http://iatistandard.org/codelists/Country/) codelist.

```
import iati.core.default
schema.codelists.add(iati.core.default.codelist('Country'))
```python
import iati.default
schema.codelists.add(iati.default.codelist('Country'))
```

The default collection of IATI codelists can be added using:

```
import iati.core.default
for _, codelist in iati.core.default.codelists().items():
```python
import iati.default
for codelist in iati.default.codelists().values():
schema.codelists.add(codelist)
```

### Loading Rulesets

**Note:** This functionality is not yet implemented.
The default IATI Ruleset can be loaded by using:

```python
import iati.default

iati.default.ruleset()
```

If you wish to load your own Ruleset you can do this using:

```python
import iati.Rulesets

# Load a local Ruleset
with open('path/to/ruleset.json', 'r') as json_file_object:
ruleset_str = json_file_object.read()

# To create a Ruleset object from your ruleset_str:
iati.Ruleset(ruleset_str)
```

Validate an IATI Dataset against the Standard Ruleset:

To be added.


### Working with IATI datasets

#### Loading a dataset

```
import iati.core.data
```python
import iati.data

# Load a local file
with open('path/to/iati-activites.xml', 'r') as xml_file_object:
Expand All @@ -105,14 +128,14 @@ with open('path/to/iati-activites.xml', 'r') as xml_file_object:
import requests
dataset_as_string = requests.get('http://XML_FILE_URL_HERE').text

dataset = iati.core.Dataset(dataset_as_string)
dataset = iati.Dataset(dataset_as_string)
```

#### Accessing data

The `Dataset` object contains an `xml_tree` attribute (itself an `lxml.etree` object). [XPath expessions](https://www.w3schools.com/xml/xpath_intro.asp) can be used to extract desired information from the dataset. For example:

```
```python
# WARNING: The following examples assume the source dataset file is produced in IATI v2.x format

# Show the activities contained within the dataset
Expand All @@ -138,7 +161,7 @@ This code supports Python 2.7 and 3.4+. We advise use of Python 3.5 (or above) a
Dev Installation
================

```
```shell
# install software development dependencies
apt-get install python-pip python-virtualenv

Expand All @@ -154,7 +177,7 @@ pip install -r requirements-dev.txt
Tests
=====

```
```shell
# to run the tests
py.test iati/

Expand Down
Loading

0 comments on commit cd992a1

Please sign in to comment.