Merge pull request #222 from IATI/dev

Merge dev into master

.pylintrc

@@ -147,7 +147,7 @@ logging-modules=logging
 # spelling-dict=en_GB
 
 # List of comma separated words that should not be checked.
-spelling-ignore-words=args,odelist,codelists,etree,iati,IATI,int,kwargs,namespace,NSMAP,SSOT,str,TODO,XML,XSD
+spelling-ignore-words=args,codelist,codelists,etree,iati,IATI,int,kwargs,namespace,NSMAP,SSOT,str,TODO,XML,XSD
 
 # A path to a file that contains private dictionary; one word per line.
 spelling-private-dict-file=

.travis.yml

@@ -14,35 +14,27 @@ script:
 
 stages:
   - test
-  - lax lint
   - strict lint
 
 jobs:
     include:
-        - stage: lax lint and other checks
-          python: 3.5
-          script:
-            - make lint
-            - make complexity
-            - make docs
-          if: branch=dev
         - stage: strict lint
           python: 3.5
           install:
             - pip install -U pip wheel
             - pip install -r requirements-dev.txt
           env: LINTER=pylint
           script: make pylint
-          if: branch=master
+          if: branch IN (master, dev)
         - stage: strict lint
           python: 3.5
           install: pip install flake8
           env: LINTER=flake8
           script: make flake8
-          if: branch=master
+          if: branch IN (master, dev)
         - stage: strict lint
           python: 3.5
           install: pip install pydocstyle
           env: LINTER=pydocstyle
           script: make pydocstyle
-          if: branch=master
+          if: branch IN (master, dev)

CHANGELOG.md

@@ -19,6 +19,59 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 ### Security
 
 
+## [0.2.0] - 2017-11-07
+
+### Added
+
+- [Codelists] Implement the `complete` attribute. [#45]
+- [Codelists] Codes may have equality compared with strings - the `value` of a Code is compared. [#45]
+- [Codelists] Add v2.02 Codelist mapping file. [#45]
+
+- [Documentation] Clarify version support in README. [#216]
+
+- [Exceptions] Add an `error_log` attribute to ValidationErrors. [#45]
+
+- [Resources] Add method to load data files relating to pyIATI (rather than the IATI Standard, or tests). [#45]
+- [Resources] Allow test files to be located within sub-folders by including slashes (`/`) in the name. [#45]
+- [Resources] Detect encoding of files that are not UTF-8. [#45]
+
+- [Validation] Change from `validate.py` to `validator.py` to improve readability of code using this module. [#86]
+- [Validation] Add a YAML file containing error information. [#117]
+- [Validation] Check whether a string is valid XML - truthy. [#45]
+- [Validation] Check whether a string is valid XML - detailed error information. [#45]
+- [Validation] Provide custom error messages when lxml returns errors for a string that is not XML. [#90]
+- [Validation] Check whether a Dataset is valid against an IATI Schema - truthy. [#45]
+- [Validation] Check whether a Dataset is valid against an IATI Schema - detailed error information. [#45]
+- [Validation] Provide custom error messages when lxml returns errors for Datasets that do not contain valid IATI XML. [#92]
+- [Validation] Check whether attributes in a Dataset have values from Codelists where required - truthy. [#45]
+- [Validation] Check whether attributes in a Dataset have values from Codelists where required - detailed error information. [#45]
+- [Validation] Check whether a Dataset conforms with Rules in a Ruleset - truthy. [#58]
+- [Validation] Check whether a Dataset conforms with Rules in a Ruleset - basic information about which Rules fail. [#58]
+- [Validation] Add a `ValidationErrorLog` containing `ValidationError`s to track validation errors. [#87]
+- [Validation] Rudimentary differentiation of errors and warnings. [#45]
+
+- [Tests] Add a range of test XML files. [#45]
+- [Tests] Add some missing tests. [#45]
+- [Tests] Add a somewhat normal-looking string to use for fuzzing. [#113]
+
+### Changed
+
+- [Codelists] The default name for a Code is now an empty string. [#45]
+- [Codelists] The name of a Code is no longer included when computing the hash. [#45]
+
+- [Datasets] pyIATI validation functionality used to determine whether a string is XML. This changes the types of Error that may be raised when updating the XML that a Dataset represents. [#95]
+
+- [Tests] Re-organise test data to use folders to separate logical groups. [#58]
+
+### Fixed
+
+- [Codelists] The names of Codes are detected in CLv2 XML Codelists (when there are no `<narrative>` elements).
+
+- [Documentation] Corrected some out-of-date documentation. [#45]
+
+- [Rulesets] Update `date_order` XPaths in Standard Ruleset. [IATI/IATI-Rulesets#31]
+
+
 ## [0.1.1] - 2017-10-25
 
 ### Fixed

README.md

@@ -4,7 +4,7 @@ A developers’ toolkit for IATI.
 
 [![Build Status](https://travis-ci.org/IATI/pyIATI.svg?branch=master)](https://travis-ci.com/IATI/pyIATI)
 
-`master`: [![Requirements Status](https://requires.io/github/IATI/iati.core/requirements.svg?branch=master)](https://requires.io/github/IATI/iati.core/requirements/?branch=master) `dev`: [![Requirements Status](https://requires.io/github/IATI/iati.core/requirements.svg?branch=dev)](https://requires.io/github/IATI/iati.core/requirements/?branch=dev)
+`master`: [![Requirements Status](https://requires.io/github/IATI/pyIATI/requirements.svg?branch=master)](https://requires.io/github/IATI/pyIATI/requirements/?branch=master) `dev`: [![Requirements Status](https://requires.io/github/IATI/pyIATI/requirements.svg?branch=dev)](https://requires.io/github/IATI/pyIATI/requirements/?branch=dev)
 
 Varying between: [![experimental](http://badges.github.io/stability-badges/dist/experimental.svg)](http://github.com/badges/stability-badges) and [![unstable](http://badges.github.io/stability-badges/dist/unstable.svg)](http://github.com/badges/stability-badges) (see docstrings)
 
@@ -49,6 +49,14 @@ The file `docs/build/index.html` serves as the documentation home page.
 **Note:** These are a work-in-progress. The `edit-docs` branch works to provide an improved docs site.
 
 
+IATI Version Support
+====================
+
+pyIATI fully supports versions `1.04`, `1.05`, `2.01` and `2.02` of the IATI Standard.
+
+Schemas for versions `1.01`, `1.02` and `1.03` are included in the `iati/resources/standard` directory but are not yet accessible using the available pyIATI functions to return default schemas.
+
+
 Usage
 =====
 

docs/source/conf.py

@@ -65,9 +65,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.1.1'
+version = '0.2.0'
 # The full version, including alpha/beta/rc tags.
-release = '0.1.1'
+release = '0.2.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/source/iati.rst

@@ -83,10 +83,10 @@ iati\.utilities module
     :undoc-members:
     :show-inheritance:
 
-iati\.validate module
----------------------
+iati\.validator module
+----------------------
 
-.. automodule:: iati.validate
+.. automodule:: iati.validator
     :members:
     :undoc-members:
     :show-inheritance:

docs/source/iati.tests.rst

@@ -84,10 +84,10 @@ iati\.tests\.test\_utilities module
     :undoc-members:
     :show-inheritance:
 
-iati\.tests\.test\_validate module
-----------------------------------
+iati\.tests\.test\_validator module
+-----------------------------------
 
-.. automodule:: iati.tests.test_validate
+.. automodule:: iati.tests.test_validator
     :members:
     :undoc-members:
     :show-inheritance:

iati/codelists.py

@@ -9,6 +9,7 @@ class Codelist(object):
     """Representation of a Codelist as defined within the IATI SSOT.
 
     Attributes:
+        complete (bool): Whether the Codelist is complete or not. If complete, attributes making use of this Codelist must only contain values present on the Codelist. If not complete, this is merely strongly advised.
         codes (:obj:`set` of :obj:`iati.Code`): The codes demonstrating the range of values that the Codelist may represent.
         name (str): The name of the Codelist.
 
@@ -63,7 +64,7 @@ def parse_from_xml(xml):
             self.name = tree.attrib['name']
             for code_el in tree.findall('codelist-items/codelist-item'):
                 value = code_el.findtext('code')
-                name = code_el.findtext('name/narrative')
+                name = code_el.findtext('name/narrative') or code_el.findtext('name')
 
                 if (value is None) and (name is None):
                     msg = "The provided Codelist ({0}) has a Code that does not contain a name or value.".format(self.name)
@@ -75,6 +76,12 @@ def parse_from_xml(xml):
                     name = ''
                 self.codes.add(iati.Code(value, name))
 
+            try:
+                self.complete = True if tree.attrib['complete'] == '1' else False
+            except KeyError:
+                pass
+
+        self.complete = None
         self.codes = set()
         self.name = name
 
@@ -85,7 +92,6 @@ def parse_from_xml(xml):
         self._url = None
         self._ref = None
         self._category_codelist = None
-        self._complete = None
 
         if xml:
             parse_from_xml(xml)
@@ -175,7 +181,7 @@ class Code(object):
     """
 
     # pylint: disable=too-many-instance-attributes
-    def __init__(self, value=None, name=None):
+    def __init__(self, value=None, name=''):
         """Initialise a Code.
 
         Args:
@@ -209,8 +215,13 @@ def __eq__(self, other):
         Todo:
             Utilise all attributes as part of the equality process.
 
+            Test comparison with strings.
+
         """
-        return ((self.name) == (other.name)) and ((self.value) == (other.value))
+        try:
+            return ((self.name) == (other.name)) and ((self.value) == (other.value))
+        except AttributeError:
+            return self.value == other
 
     def __ne__(self, other):
         """Check Code inequality.
@@ -229,8 +240,10 @@ def __hash__(self):
         Todo:
             Utilise all attributes as part of the hashing process.
 
+            Be able to deal with checks against both Codes and strings.
+
         """
-        return hash((self.name, self.value))
+        return hash((self.value))
 
     @property
     def xsd_enumeration(self):

iati/data.py

@@ -3,6 +3,7 @@
 from lxml import etree
 import iati.exceptions
 import iati.utilities
+import iati.validator
 
 
 class Dataset(object):
@@ -92,13 +93,17 @@ def xml_str(self, value):
                 else:
                     value_stripped_bytes = value_stripped
 
-                self.xml_tree = etree.fromstring(value_stripped_bytes)
-                self._xml_str = value_stripped
-            except etree.XMLSyntaxError:
-                msg = "The string provided to create a Dataset from is not valid XML."
-                iati.utilities.log_error(msg)
-                raise ValueError(msg)
-            except (AttributeError, TypeError, ValueError):
+                validation_error_log = iati.validator.validate_is_xml(value_stripped_bytes)
+
+                if not validation_error_log.contains_errors():
+                    self.xml_tree = etree.fromstring(value_stripped_bytes)
+                    self._xml_str = value_stripped
+                else:
+                    if validation_error_log.contains_error_of_type(TypeError):
+                        raise TypeError
+                    else:
+                        raise iati.exceptions.ValidationError(validation_error_log)
+            except (AttributeError, TypeError):
                 msg = "Datasets can only be ElementTrees or strings containing valid XML, using the xml_tree and xml_str attributes respectively. Actual type: {0}".format(type(value))
                 iati.utilities.log_error(msg)
                 raise TypeError(msg)

iati/default.py

@@ -6,6 +6,7 @@
     Handle multiple versions of the Standard rather than limiting to the latest.
     Implement more than Codelists.
 """
+
 import json
 import os
 from collections import defaultdict
@@ -146,6 +147,40 @@ def codelists(version=None):
     return _codelists(version)
 
 
+def codelist_mapping(version=None):
+    """Define the mapping process which states where in a Dataset you should find values on a given Codelist.
+
+    Args:
+        version (str): The version of the Standard to return the mapping file for. Defaults to None. This means that the mapping file is returned for the latest version of the Standard.
+
+    Returns:
+        dict of dict: A dictionary containing mapping information. Keys in the first dictionary are Codelist names. Keys in the second dictionary are `xpath` and `condition`. The condition is `None` if there is no condition.
+
+    Todo:
+        Make use of the `version` parameter.
+
+    """
+    path = iati.resources.get_codelist_mapping_path(version)
+    mapping_tree = iati.resources.load_as_tree(path)
+    mappings = defaultdict(list)
+
+    for mapping in mapping_tree.getroot().xpath('//mapping'):
+        codelist_name = mapping.find('codelist').attrib['ref']
+        codelist_location = mapping.find('path').text
+
+        try:
+            condition = mapping.find('condition').text
+        except AttributeError:  # there is no condition
+            condition = None
+
+        mappings[codelist_name].append({
+            'xpath': codelist_location,
+            'condition': condition
+        })
+
+    return mappings
+
+
 def ruleset(version=None):
     """Return the Standard Ruleset for the specified version of the Standard.
 

iati/exceptions.py

@@ -28,4 +28,8 @@ class ValidationError(ValueError):
         This is too general to identify many specific problems.
     """
 
-    pass
+    def __init__(self, error_log):
+        """Initialise a ValidationError."""
+        self.error_log = error_log
+
+        super(ValidationError, self).__init__()