CKAN 2.10 compatability

[blagojabozinovski]

Adds comparability with CKAN 2.10

[ThrawnCA]

This conditional isn't really needed. It's fine to create the functions even if they won't be called.

[blagojabozinovski]

@ThrawnCA On point. I removed that condition.

[rantoniuk]

This PR had no conflicts at the time of review, but now has conflicts to be resolved. Is there any specific reason it wasn't merged at the time with the positive review it had?

[ThrawnCA]

Is there any specific reason it wasn't merged at the time with the positive review it had?

I can only speak for myself and say I'm pretty sure I didn't have write access at the time.

It should be possible to create a new branch within this repo, pull in the Keitaro changes, and fix the conflict.

[duttonw]

I'll get onto this, sorry i did not see this before i started to incorporate qld-gov-au 2.10 work (still wip)

[blagojabozinovski]

This PR had no conflicts at the time of review, but now has conflicts to be resolved. Is there any specific reason it wasn't merged at the time with the positive review it had?
Same goes here. I don't have access

[duttonw]

Hi @blagojabozinovski ,

I've tried to correct your branch based off where this was forked from v2.0.0. Sadly you have 38 failing tests

https://github.com/ckan/ckanext-validation/actions/runs/12367164030
only additions were cicd and test updates to latest pytest which do pass in 2.9.

---

commit 7a0a12e (HEAD -> keitaronic/ckan-2.10-test, origin/keitaronic/ckan-2.10-test)
Author: william dutton [email protected]
Date: Tue Dec 17 15:47:05 2024 +1000

fix: tests need to now be 'Pytest 5' compatible

commit da28f22
Author: william dutton [email protected]
Date: Tue Dec 17 15:39:01 2024 +1000

enable 2.9 and 2.10 cicd testing on keitaronic base branch

---

reviewing what you changed:

v2.0.0...keitaroinc:ckanext-validation:ckan-2.10

You went by pushing back the split that ckan 2.10 on dataset/resource creates/updates back onto its original function name and it does seem its not fully functional.

    def after_dataset_create(self, context, data_dict):
        self.after_create(context, data_dict)

    def before_resource_update(self, context, current_resource, updated_resource):
        self.before_update(context, current_resource, updated_resource)

    def after_dataset_update(self, context, data_dict):
        self.after_update(context, data_dict)

I'll try and work on the qld-gov-au fork which has 2.10 working with passing tests into upstream.

the biggest issue i'm needing to work through is logic resource_create, resource_update, being changed to package_patch, resource_show, and its logic placed elsewhere since the original validation was 'hacking' the core functions :'(

Code below:

@t.chained_action
def resource_create(up_func, context, data_dict):
    '''Appends a new resource to a datasets list of resources.

    This is duplicate of the CKAN core resource_create action, with just the
    addition of a synchronous data validation step.

    This is of course not ideal but it's the only way right now to hook
    reliably into the creation process without overcomplicating things.
    Hopefully future versions of CKAN will incorporate more flexible hook
    points that will allow a better approach.

    '''

    if settings.get_create_mode_from_config() != 'sync':
        return up_func(context, data_dict)

    model = context['model']

    package_id = t.get_or_bust(data_dict, 'package_id')
    if not data_dict.get('url'):
        data_dict['url'] = ''

    pkg_dict = t.get_action('package_show')(
        dict(context, return_type='dict'),
        {'id': package_id})

    t.check_access('resource_create', context, data_dict)

    for plugin in plugins.PluginImplementations(plugins.IResourceController):
        plugin.before_create(context, data_dict)

    if 'resources' not in pkg_dict:
        pkg_dict['resources'] = []

    upload = uploader.get_resource_uploader(data_dict)

    if 'mimetype' not in data_dict:
        if hasattr(upload, 'mimetype'):
            data_dict['mimetype'] = upload.mimetype

    if 'size' not in data_dict:
        if hasattr(upload, 'filesize'):
            data_dict['size'] = upload.filesize

    pkg_dict['resources'].append(data_dict)

    try:
        context['defer_commit'] = True
        context['use_cache'] = False
        t.get_action('package_update')(context, pkg_dict)
        context.pop('defer_commit')
    except t.ValidationError as e:
        try:
            raise t.ValidationError(e.error_dict['resources'][-1])
        except (KeyError, IndexError):
            raise t.ValidationError(e.error_dict)

    # Get out resource_id resource from model as it will not appear in
    # package_show until after commit
    resource_id = context['package'].resources[-1].id
    upload.upload(resource_id,
                  uploader.get_max_resource_size())

    # Custom code starts

    run_validation = True

    for plugin in plugins.PluginImplementations(IDataValidation):
        if not plugin.can_validate(context, data_dict):
            log.debug('Skipping validation for resource {}'.format(resource_id))
            run_validation = False

    if run_validation:
        is_local_upload = (
            hasattr(upload, 'filename')
            and upload.filename is not None
            and isinstance(upload, uploader.ResourceUpload))
        _run_sync_validation(
            resource_id, local_upload=is_local_upload, new_resource=True)

    # Custom code ends

    model.repo.commit()

    #  Run package show again to get out actual last_resource
    updated_pkg_dict = t.get_action('package_show')(
        context, {'id': package_id})
    resource = updated_pkg_dict['resources'][-1]

    #  Add the default views to the new resource
    t.get_action('resource_create_default_resource_views')(
        {'model': context['model'],
         'user': context['user'],
         'ignore_auth': True
         },
        {'resource': resource,
         'package': updated_pkg_dict
         })

    for plugin in plugins.PluginImplementations(plugins.IResourceController):
        plugin.after_create(context, resource)

    return resource


@t.chained_action
def resource_update(up_func, context, data_dict):
    '''Update a resource.

    This is duplicate of the CKAN core resource_update action, with just the
    addition of a synchronous data validation step.

    This is of course not ideal but it's the only way right now to hook
    reliably into the creation process without overcomplicating things.
    Hopefully future versions of CKAN will incorporate more flexible hook
    points that will allow a better approach.

    '''

    if settings.get_update_mode_from_config() != 'sync':
        return up_func(context, data_dict)

    model = context['model']
    id = t.get_or_bust(data_dict, "id")

    if not data_dict.get('url'):
        data_dict['url'] = ''

    resource = model.Resource.get(id)
    context["resource"] = resource
    old_resource_format = resource.format

    if not resource:
        log.debug('Could not find resource %s', id)
        raise t.ObjectNotFound(t._('Resource was not found.'))

    t.check_access('resource_update', context, data_dict)
    del context["resource"]

    package_id = resource.package.id
    pkg_dict = t.get_action('package_show')(dict(context, return_type='dict'),
                                            {'id': package_id})

    for n, p in enumerate(pkg_dict['resources']):
        if p['id'] == id:
            break
    else:
        log.error('Could not find resource %s after all', id)
        raise t.ObjectNotFound(t._('Resource was not found.'))

    # Persist the datastore_active extra if already present and not provided
    if ('datastore_active' in resource.extras
            and 'datastore_active' not in data_dict):
        data_dict['datastore_active'] = resource.extras['datastore_active']

    for plugin in plugins.PluginImplementations(plugins.IResourceController):
        plugin.before_update(context, pkg_dict['resources'][n], data_dict)

    upload = uploader.get_resource_uploader(data_dict)

    if 'mimetype' not in data_dict:
        if hasattr(upload, 'mimetype'):
            data_dict['mimetype'] = upload.mimetype

    if 'size' not in data_dict and 'url_type' in data_dict:
        if hasattr(upload, 'filesize'):
            data_dict['size'] = upload.filesize

    pkg_dict['resources'][n] = data_dict

    try:
        context['defer_commit'] = True
        context['use_cache'] = False
        updated_pkg_dict = t.get_action('package_update')(context, pkg_dict)
        context.pop('defer_commit')
    except t.ValidationError as e:
        try:
            raise t.ValidationError(e.error_dict['resources'][-1])
        except (KeyError, IndexError):
            raise t.ValidationError(e.error_dict)

    upload.upload(id, uploader.get_max_resource_size())

    # Custom code starts

    run_validation = True
    for plugin in plugins.PluginImplementations(IDataValidation):
        if not plugin.can_validate(context, data_dict):
            log.debug('Skipping validation for resource {}'.format(id))
            run_validation = False

    if run_validation:
        run_validation = not data_dict.pop('_skip_next_validation', None)

    if run_validation:
        is_local_upload = (
            hasattr(upload, 'filename')
            and upload.filename is not None
            and isinstance(upload, uploader.ResourceUpload))
        _run_sync_validation(
            id, local_upload=is_local_upload, new_resource=False)

    # Custom code ends

    model.repo.commit()

    resource = t.get_action('resource_show')(context, {'id': id})

    if old_resource_format != resource['format']:
        t.get_action('resource_create_default_resource_views')(
            {'model': context['model'], 'user': context['user'],
             'ignore_auth': True},
            {'package': updated_pkg_dict,
             'resource': resource})

    for plugin in plugins.PluginImplementations(plugins.IResourceController):
        plugin.after_update(context, resource)

    return resource

@tk.chained_action
def package_patch(original_action, context, data_dict):
    ''' Detect whether resources have been replaced, and if not,
    place a flag in the context accordingly if save flag is not set

    Note: controllers add default context where save is in request params
        'save': 'save' in request.params
    '''
    if 'save' not in context and 'resources' not in data_dict:
        context['save'] = True
    original_action(context, data_dict)


@tk.side_effect_free
@tk.chained_action
def resource_show(next_func, context, data_dict):
    """Throws away _success_validation flag, that we are using to prevent
    multiple validations of resource in different interface methods
    """
    if context.get('ignore_auth'):
        return next_func(context, data_dict)

    data_dict = next_func(context, data_dict)

    data_dict.pop('_success_validation', None)
    return data_dict