AppCivist Composition and Execution Engine

Architecture

Part of the AppCivist project, is to build a comprehensive registry of services (web or mobile apps, web services APIs, etc.) that can be used for purposes of civic participation and social activism. This registry is aimed at enabling the composition of such services, to construct applications that can support different AppCivist workflows. To realize this vision, the following architecture is proposed for the AppCivist Composition and Execution Platform

The main components of this platform are the following:

• Activism Ontology: a database of relevant concepts for activism and civic engagement (e.g., what's a proposal? how it is made?). Activists inform the definition of these concepts and its relations.
• Activism Enabling Service Registry: a database of existing services and applications that implement operations for engaging in social activism and civic engagement, based on the ontology of activism. Developers can use this registry to implement new applications, by composing existing ones.
• Assembly Composition Engine: a platform that facilitates and stores the composition of existing services in the registry, in order to create new applications that support activists in their processes.
• Assembly Execution Engine: a platform that executes AppCivist applications, allowing Citizens and Activists to interact with the services that are behind.
• Assembly Activism Process: a platform that stores and facilitates the creation of activism workflow, using existing assembly compositions.

Composition and Execution API

The Composition and Execution API of AppCivist is the first iteration of the proposed architecture. It will provide programmatic access to the platform, to support the composition of existing civic services and applications.

The figure above details the main aspects of this initial iteration, each explained in the following subsections

First Proof-of-Concept composition

This first iteration started with the idea of testing how feasible it is to orchestrate two or more different and existing applications, used in the context of social activism and civic engagement. The starting ideas was the following story:

Imagine a group of activists using their hosted version of Loomio to discuss issues and draft proposals in different working groups. Eventually, they want to select a shortlist of winning proposals to mobilize the community behind them, but want a larger community (i.e., the city) to be involved in the voting process, so they would like to setup an election using Agora, a voting system external to Loomio

In this proof of concept, AppCivist works as the orchestrator that, when the groups decides, gets the motions/proposal drafts from Loomio and pass them to Agora, converting them into choices of a question that is part of an Agora Election. AssembliesTest simulates this orchestration and one more, where the initial drafts are actually prepared in a third service, a collaborative text editor called Etherpad.

As such, these simulations represent one instance of following the proposed proposal development timeline. The details of these simulations are described in Issues #18 and #20.

AppCivist Models

To enable the above orchestration, we need a way of representing the operation we want to execute (e.g., Loomio "create discussion", Agora "create election"), abstracting them into generic AppCivist operations ("start proposal making", "start voting process") that mimic activism capabilities we are aiming to support. In the first iteration, our internal model to represent existing service operations and AppCivist operations is defined in the following picture.

With this model, we can already register specific API endpoints that are part of different services (provided they have an underlying API) with a very simple authentication model by which we simply store the session tokens. Internally, we also define the mappings between service operations/resources and AppCivist operations/resources. The current iteration also supports API calls to services that require a complex data model as a body parameter.

In the diagram, the Service Registry corresponds to the Activism Enabling Service Registry of the architecture. The Composition Engine is a simple class that read service definitions and makes sure that instanation of them have all the information (i.e., parameter values) it needs to be executed. Similarly, the Execution Engine is a simple Java class that constructs the service call based on the definition of an operation and its parameter values. The Activism Ontology is represented by the Assembly Registry, which models internally the initial set of operations and resources we think are important in the process of activism we envision.

For all this to work, we need to load the Service Registry with, at least, the required service endpoints we use in our proof-of-concept. This is explained int he next subsection.

Service Registry and Service Definition

The following image contains a list of Existing Services and Applications we are exploring as examples of systems that are (or might be) used for civic engagement and social activism. In order to load their service definitions into our registry, we have to follow these steps:

1. Investigate if the application has an API.
2. If no API is available, reverse engineer its API from its User Interface
3. If an API exists, search for its documentation.
4. If no documentation is available, reverse engineer the API documentation (i.e., the definition of the service operations).
5. Once a formal documentation is obtained, we need to formalize it to enable its automatic loading into our registry. For formalizing an API, we can use a Web Service Description Language (e.g., WSDL, WADL, SAWSDL, RSDL, etc.)
6. In some cases, the documentation might be available in a standard format (e.g., SWAGGER, RAML, Apiary Blueprint), which would ease the pass from documentation to formalization (we might even skip this passage if the documentation is sufficient to parse the definition)
7. Parse the service definition and load it into the registry.

This process can be complex and take a lot of effort, especially when there is no documentation of the underlying APIs (and potentially even more if the system we want to incorporate into the registry is not open source). Formalizing it, although less complicated once the API is well defined, it can also be a challenge, given the wide range of formalization standards available (i.e., wsdl, wadl, rsdl, etc.). Some APIs are documented following a documentation standard (e.g., SWAGGER, RAML, Apiary Blueprint, etc.), which might facilitate things a lot, but in reality, very few applications in this domain have such a documentation.

For the moment, we are using a simple YAML file for defining the services already in the form of our internal model. In the future, we will adopt one of the standards. Then, a synchronization mechanism will be needed to update this definitions every time the service changes (if we are not using a self-controlled hosted version of the service).

Challenges

1. Service Instances: depending what is the instance of a service/application we use, it might be more or less complex to execute operations in it. There are basically two choices: hosting a copy of the service in appcivist servers, or use the external versions of these services. In the latter, if that server has a same-origin policy, appcivist will face problems of forbidden cross-domain requests.
2. Full-Stack Applications: In most cases, the applications in these domain are full-stack web or mobile applications, with their own processes and models of decision making and deliberation, in which it might not make sense, from an usability perspective, to use its individual service endpoints from outside (e.g., using just the discussion model of Loomio without its integrated proposal/voting scheme, or creating a loomio group just for having members in it without using for its main purpose, which is to have discussions).
3. Authentication and Authorization: this is, perhaps, the biggest of all the challenges to the composition idea. If we use the main External instance of services (i.e., loomio.org instead of appcivist.org/loomio), we will have to implement an authentication protoco for each service, which in each case will require appcivist to include an user context to all requests. Shall appcivist have a general user for its requests or to represent a group of users? If so, how can we manage authorization of individual users? If we protocols like oAuth (provided the service supports this), are we going to ask all the users of an Assembly to authorize AppCivist to make requests on his/her behalf?. These are all complex issues that will hinder the user experience of applications built on top of the composition engine. It might make more sense to limit the composition to the services that do not necessarily require a user context (i.e., mapping and visualization).
4. Concept mappings: for each service, AppCivist must also register a mapping between the concepts managed inside the service and the AppCivist ontology
5. API Inter-service requirements: some operations are only possible if other operations were executed before, e.g., loomio discussions can only be created if a group was previously described. Most of these requirements are not documented in formal ways.
6. API Data Model Requirements: as with the previous, there are cross-data requirements between the different resources that are managed by the service, e.g., a loomio discussion cannot be private if it is created inside of a loomio public group. These requirements are also seldom formalized in the service definitions or documentations, which means that some level of "reverse engineering" will always be needed for each service.
7. Hosted Services installations: hosting our own version of services (when possible, i.e., they are open source) has advantages for facilitating execution of services, but requires more server infrastructure from appcivist and a more complex installation process (if we will allow activist to download their own version of appcivist).
8. Composition User Experience: composition might be easy for developers, but in order for activist to compose services, we will need to design the user experience carefully so that it really makes it easier for the activist to achieve his or her goals.