Initial development

.gitignore

@@ -0,0 +1,2 @@
+/wagtail_airtable.egg-info
+__pycache__

README.md

@@ -1 +1,80 @@
-# Future home of wagtail-airtable
+# Wagtail/Airtable
+
+An extension for Wagtail allowing content to be transferred between Airtable sheets and your Wagtail/Django models.
+
+Developed by [Torchbox](https://torchbox.com/) and sponsored by [The Motley Fool](https://www.fool.com/).
+
+TODO: Create an animation demonstrating how it works.
+
+### How it works
+
+When you setup a model to "map" to an Airtable sheet, everytime you save the model it will attempt to update the row in Airtable. If a row is not found, it will create a new row in your Airtable.
+
+When you want to sync your Airtable data to your Wagtail website, you can go to `Settings -> Airtable Import`. You can then import entire tables into your Wagtail instance with the click of a button. If you see "_{Your Model}_ is not setup with the correct Airtable settings" you will need to double check your settings.
+
+##### Behind the scenes...
+This package will attempt to match a model object against row in Airtable using a `record_id`. If a model does not have a record_id value, it will look for a match using the `AIRTABLE_UNIQUE_IDENTIFIER` to try and match a unique value in the Airtable to the unique value in your model. Should that succeed your model object will be "paired" with the row in Airtable. But should the record-search fail, a new row in Airtable will be created when you save your model, or a new model object will attempt to be created when you import a model from Airtable.
+
+> **Note**: Object creation _can_ fail when importing from Airtable. This is expected behaviour as an Airtable might not have all the data it requires. For instance, a Wagtail Page uses Django Tree Beard and if a `path` is not in the model import settings (and a column in Airtable) a page cannot be created.
+
+### Installation & Configuration
+
+* Install the package with `pip install wagtail-airtable`
+* Add `'wagtail_airtable'` to your project's `INSTALLED_APPS`
+* In your settings you will need to map models to Airtable settings. Every model you want to map to an Airtable sheet will need:
+    * An `AIRTABLE_BASE_KEY`. You can find the base key in your Airtable docs when you're signed in to Airtable.com
+    * An `AIRTABLE_TABLE_NAME` to determine which table to connect to.
+    * An `AIRTABLE_UNIQUE_IDENTIFIER`. This can either be a string or a dictionary mapping the Airtable column name to your unique field in your model.
+        * ie. `AIRTABLE_UNIQUE_IDENTIFIER: 'slug',` this will match the `slug` field on your model with the `slug` column name in Airtable. Use this option if your model field and your Airtable column name are identical.
+        * ie. `AIRTABLE_UNIQUE_IDENTIFIER: {'Airtable Column Name': 'model_field_name'},` this will map the `Airtable Column Name` to a model field called `model_field_name`. Use this option if your Airtable column name and your model field name are different.
+    * An `AIRTABLE_SERIALIZER` that takes a string path to your serializer. This helps map incoming data from Airtable to your model fields. Django Rest Framework is required for this. See the `examples/` directory for serializer examples. TODO: Add `examples/` for setup examples.
+* Add the following to your `urls.py`:
+    ```python
+    from django.urls import path
+    from wagtail_airtable.views import AirtableImportListing
+    ...
+    urlpatterns = [
+        ...
+        path("airtable-import", AirtableImportListing.as_view(), name="airtable_import_listing"),
+    ]
+    ```
+
+* Lastly make sure you enable wagtail-airtable with `WAGTAIL_AIRTABLE_ENABLED = True`. By default this is disabled so data in your Wagtail site and your Airtable sheets aren't accidentally overwritten. Data is hard to recover, this option helps prevent accidental data loss.
+
+### Example Base Configuration
+
+```python
+WAGTAIL_AIRTABLE_ENABLED = True
+AIRTABLE_IMPORT_SETTINGS = {
+    'appname.ModelName': {
+        'AIRTABLE_BASE_KEY': 'app3ds912jFam032S',
+        'AIRTABLE_TABLE_NAME': 'Your Airtable Table Name',
+        'AIRTABLE_UNIQUE_IDENTIFIER': 'slug', # Must match the Airtable Column name
+        'AIRTABLE_SERIALIZER': 'path.to.your.model.serializer.CustomModelSerializer'
+    },
+    'appname.OtherModelName': {
+        'AIRTABLE_BASE_KEY': 'app3ds912jFam032S',
+        'AIRTABLE_TABLE_NAME': 'Your Airtable Table Name',
+        'AIRTABLE_UNIQUE_IDENTIFIER':
+            'Page Slug': 'slug', # 'Page Slug' column name in Airtable, 'slug' field name in Wagtail.
+        },
+        'AIRTABLE_SERIALIZER': 'path.to.your.model.serializer.OtherCustomModelSerializer'
+    },
+    # ...
+}
+```
+
+### Management Commands
+
+```bash
+python manage.py import_airtable pages.HomePage creditcards.CreditCard
+```
+
+##### import_airtable command
+This command will look for any `appname.ModelName`s you provide it and use the mapping settings to find data in the Airtable. See the "Behind the scenes" section for more details on how importing works.
+
+### Local Testing Advice
+
+> **Note:** Careful not to use the production settings as you could overwrite Wagtail or Airtable data.
+
+Because Airtable doesn't provide a testing environment, you'll need to test against a live table. The best way to do this is to copy your live table to a new table (renaming it will help avoid naming confusion), and update your local settings. With this method, you can test to everything safely against a throw-away Airtable. Should something become broken beyond repair, delete the testing table and re-copy the original one.

setup.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python
+
+from setuptools import setup, find_packages
+
+install_requires = [
+    "airtable-python-wrapper>=0.13.0",
+    "djangorestframework>=3.11.0",
+]
+
+setup(
+    name='wagtail-airtable',
+    version='0.1',
+    description="Sync data between Wagtail and Airtable",
+    author='Kalob Taulien',
+    author_email='[email protected]',
+    url='https://github.com/wagtail/wagtail-airtable',
+    packages=find_packages(exclude=('tests',)),
+    include_package_data=True,
+    license='BSD',
+    long_description="An extension for Wagtail allowing content to be transferred between Airtable sheets and your Wagtail/Django models",
+    classifiers=[
+        'Development Status :: 4 - Beta',
+        'Environment :: Web Environment',
+        'Intended Audience :: Developers',
+        'License :: OSI Approved :: BSD License',
+        'Operating System :: OS Independent',
+        'Programming Language :: Python',
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7',
+        'Framework :: Django',
+        'Framework :: Wagtail',
+        'Framework :: Wagtail :: 2',
+    ],
+    install_requires=install_requires,
+)

wagtail_airtable/admin.py

@@ -0,0 +1,3 @@
+from django.contrib import admin
+
+# Register your models here.

wagtail_airtable/apps.py

@@ -0,0 +1,5 @@
+from django.apps import AppConfig
+
+
+class WagtailAirtableConfig(AppConfig):
+    name = 'wagtail_airtable'

wagtail_airtable/management/commands/import_airtable.py

@@ -0,0 +1,239 @@
+from importlib import import_module
+from requests import HTTPError
+
+from airtable import Airtable
+from django.apps import apps
+from django.conf import settings
+from django.contrib.contenttypes.models import ContentType
+from django.core.exceptions import ObjectDoesNotExist, ValidationError
+from django.db import IntegrityError
+from django.core.management.base import BaseCommand, CommandError
+from logging import getLogger
+
+logger = getLogger(__name__)
+
+
+class Command(BaseCommand):
+    help = "Import data from an Airtable and overwrite model or page information"
+
+    def add_arguments(self, parser):
+        parser.add_argument('labels', metavar='model_name', nargs='+', help="Model (as app_label.model_name) or app name to populate table entries for, e.g. creditcards.CreditCard")
+
+    def get_model_for_path(self, model_path):
+        """
+        Given an 'app_name.model_name' string, return the model class
+        """
+        app_label, model_name = model_path.split('.')
+        return ContentType.objects.get_by_natural_key(app_label, model_name).model_class()
+
+    def get_model_serializer(self, serializer_string):
+        location, serializer_name = serializer_string.rsplit(".", 1)
+        module = import_module(location)
+        serializer_class = getattr(module, serializer_name)
+        return serializer_class
+
+    def handle(self, *args, **options):
+        """
+        Runs the management command with the app_name.ModelName as parameters.
+        ie. python manage.py import_airtable pages.HomePage creditcards.CreditCard
+
+        This will handle Wagtail Pages (with page revisions) along with standard
+        Django models.
+
+        Models passed in to the command go through a quick validation to ensure they
+        exist.
+
+        Every model is then looped through, and a `settings.AIRTABLE_IMPORT_SETTINGS`
+        is searched for based on the models label (ie. pages.HomePage). These
+        settings are used to connect to a certain Airtable Base (a set of spreadsheets),
+        the name of the table to use, a unique identifier (used for connecting previously
+        unrelated Airtable records to Django objects), and a serializer for validating
+        incoming data from Airtable to make it work with the Django field types.
+
+        Within each model loop also contains an `airtable.get_all()` command which
+        will get all the data from the Airtable spreadsheet and load it into a
+        list of dictionaries.
+
+        Then every record is iterated over and 3 actions are taken:
+            1. Look for an existing model object by its airtable_record_id.
+               Update if found. Skip to step 2 if not found.
+            2. Search for a model object by its unique identifier.
+               If found, update the object. Skip to step 3 if not found.
+            3. Create a new object. This step is very susceptible to fail based on the
+               model type and fields in the serializer. Wagtail pages cannot be created
+               from Airtable records as there's missing data such as depth and parent
+               pages. And so Wagtail pages are skipped entirely in step 3.
+        """
+        models = []
+        for label in options['labels']:
+            label = label.lower()
+            if '.' in label:
+                # interpret as a model
+                try:
+                    model = self.get_model_for_path(label)
+                except ObjectDoesNotExist:
+                    raise CommandError("%r is not recognised as a model name." % label)
+
+                models.append(model)
+
+        created = 0
+        updated = 0
+        skipped = 0
+        for model in models:
+            # Wagtail models require a .save_revision() call when being saved.
+            is_wagtail_model = hasattr(model, 'depth')
+            # Airtable global settings.
+            airtable_settings = settings.AIRTABLE_IMPORT_SETTINGS.get(model._meta.label, {})
+            # Set the unique identifier and serializer.
+            airtable_unique_identifier = airtable_settings.get('AIRTABLE_UNIQUE_IDENTIFIER')
+            model_serializer = self.get_model_serializer(airtable_settings.get('AIRTABLE_SERIALIZER'))
+
+            if type(airtable_unique_identifier) == str:
+                # The unique identifier is a string.
+                # Use it as the Airtable Column name and the Django field name
+                airtable_unique_identifier_column_name = airtable_unique_identifier
+                airtable_unique_identifier_field_name = airtable_unique_identifier
+            elif type(airtable_unique_identifier) == dict:
+                # Unique identifier is a dictionary.
+                # Use the key as the Airtable Column name and the value as the Django Field name.
+                airtable_unique_identifier_column_name, airtable_unique_identifier_field_name = (
+                    list(airtable_unique_identifier.items())[0]
+                )
+
+            # Set the Airtable API client.
+            airtable = Airtable(
+                airtable_settings.get('AIRTABLE_BASE_KEY'),
+                airtable_settings.get('AIRTABLE_TABLE_NAME'),
+                api_key=settings.AIRTABLE_API_KEY,
+            )
+
+            # Get all the airtable records for the specified table.
+            # TODO try/catch this in case of misconfiguration.
+            all_records = airtable.get_all()
+
+            # Loop through every record in the Airtable.
+            for record in all_records:
+                record_id = record['id']
+                record_fields = record['fields']
+                mapped_import_fields = model.map_import_fields(incoming_dict_fields=record_fields)
+                serialized_data = model_serializer(data=mapped_import_fields)
+                serialized_data.is_valid()
+
+                # Look for a record by it's airtable_record_id.
+                # If it exists, update the data.
+                try:
+                    obj = model.objects.get(airtable_record_id=record_id)
+                except model.DoesNotExist:
+                    obj = None
+
+                if obj:
+                    # Model object was found by it's airtable_record_id
+                    if serialized_data.is_valid():
+                        for field_name, value in serialized_data.validated_data.items():
+                            if field_name == "tags":
+                                for tag in value:
+                                    obj.tags.add(tag)
+                            else:
+                                setattr(obj, field_name, value)
+                        obj.push_to_airtable = False
+                        try:
+                            if is_wagtail_model:
+                                # Wagtail page. Requires a .save_revision()
+                                if not obj.locked:
+                                    # Only save the page if the page is not locked
+                                    obj.save()
+                                    obj.save_revision()
+                                else:
+                                    # TODO Add a handler to manage locked pages.
+                                    pass
+                            else:
+                                # Django model. Save normally.
+                                obj.save()
+                            updated = updated + 1
+                        except ValidationError as error:
+                            error_message = '; '.join(error.messages)
+                            logger.error(f"Unable to save {obj._meta.label} -> '{obj}'. Error(s): {error_message}")
+                        continue
+                    else:
+                        logger.info(f"Invalid data for record {record_id}")
+
+                # This `unique_identifier` is the value of an Airtable record.
+                # ie.
+                #   fields = {
+                #         'Slug': 'the-ascent'
+                #   }
+                # This will return 'the-ascent' and can now be searched for as model.objects.get(slug='the-ascent')
+                unique_identifier = record_fields.get(airtable_unique_identifier_column_name, None)
+                if unique_identifier:
+                    try:
+                        obj = model.objects.get(**{airtable_unique_identifier_field_name: unique_identifier})
+                    except model.DoesNotExist:
+                        obj = None
+                    if obj:
+                        # A local model object was found by a unique identifier.
+
+                        if serialized_data.is_valid():
+                            for field_name, value in serialized_data.validated_data.items():
+                                if field_name == "tags":
+                                    for tag in value:
+                                        obj.tags.add(tag)
+                                else:
+                                    setattr(obj, field_name, value)
+                            obj.airtable_record_id = record_id
+                            obj.push_to_airtable = False
+                            try:
+                                if is_wagtail_model:
+                                    # Wagtail page. Requires a .save_revision()
+                                    if not obj.locked:
+                                        # Only save the page if the page is not locked
+                                        obj.save()
+                                        obj.save_revision()
+                                    else:
+                                        # TODO Add a handler to manage locked pages.
+                                        pass
+                                else:
+                                    # Django model. Save normally.
+                                    obj.save()
+                                updated = updated + 1
+                            except ValidationError as error:
+                                error_message = '; '.join(error.messages)
+                                logger.error(f"Unable to save {obj}. Error(s): {error_message}")
+                        else:
+                            logger.info(f"Invalid data for record {record_id}")
+
+                        continue
+                    else:
+                        # No object was found by this unique ID.
+                        # Do nothing. The next step will be to create this object in Django
+                        logger.info(f"{model._meta.verbose_name} with field {airtable_unique_identifier_field_name}={unique_identifier} was not found")
+                else:
+                    # There was no unique identifier set for this model.
+                    # Nothing can be done about that right now.
+                    logger.info(f"{model._meta.verbose_name} does not have a unique identifier")
+
+                # Cannot bulk-create Wagtail pages from Airtable because we don't know where the pages
+                # Are supposed to live, what their tree depth should be, and a few other factors.
+                # For this scenario, log information and skip the loop iteration.
+                if hasattr(model, 'depth'):
+                    logger.info(f"{model._meta.verbose_name} cannot be created from an import.")
+                    skipped = skipped + 1
+                    continue
+
+                # If there is no match whatsoever, try to create a new `model` instance.
+                # Note: this may fail if there isn't enough data in the Airtable record.
+                try:
+                    model.objects.create(**mapped_import_fields)
+                    created = created + 1
+                except ValueError as value_error:
+                    error_message = '; '.join(value_error.messages)
+                    logger.info(f"Could not create new model. Error: {error_message}")
+                except IntegrityError as e:
+                    logger.info(f"Could not create new model. Error: {e}")
+                except AttributeError as e:
+                    logger.info(f"Could not create new model. AttributeError. Error: {e}")
+                except Exception as e:
+                    logger.error(f"Unhandled error. Could not create a new object for {model._meta.verbose_name}. Error: {e}")
+
+        if options['verbosity'] >= 1:
+            self.stdout.write(f"{created} objects created. {updated} objects updated. {skipped} objects skipped.")
+            return f"{created} objects created. {updated} objects updated. {skipped} objects skipped."