Polling 2.0

Jira: http://www.mulesoft.org/jira/browse/MULE-6872

Forum discussion: http://forum.mulesoft.org/mulesoft/topics/_3_5_m1_mule_polling_improvements

Motivation

The main motivation of this feature is to generate a Management service of polls and create an unique way to do polling in mule.

Uses cases

1. As a user, I want to execute a given flow :(every hour regardless of the day/week, on Monday, Weds, Friday at 12 AM, Every 5 minutes)
2. As a user, I want to be able to try out my poll: (I do not have to wait for the next poll to execute (which may be a day away) to test what will happen, I do not get confused about what is happening because polls continue to execute while I'm trying to debug what is happening)
3. As a user, I want to be able to test my poll: (I want to run a unit test that executes a poll, I want to start a test without having polls running.)
4. As an extension developer, I want better APIs to start, stop and execute polls.
5. As a user, I want to prevent poll executions from executing concurrently
6. As an extension developer I want to start/stop/run my polls
7. As an extension developer I want to enrich with notifications of the poll execution.
8. As a developer I want to provide a new scheduler beside poll via any mule extension

Design

This spec is divided in two parts:

1. Mule API definition
2. XML definition

Mule API to support extension developer's features.

Nowadays we don't have a mule API in order to manage the schedulers, this implies that are features that we cannot perform or are hard to develop.

New Scheduler interface

The scheduler is the one who triggers a task based on its configuration. The relationship is one scheduler one task.

The scheduler allows users to schedule its task on demand. The schedule timing depends on the scheduler or its task. scheduling a task on deman doesn't mean that the task is going to be executed immediately.

public interface Scheduler extends Stoppable, Startable, NamebleObject
{
    void schedule() throws MuleException;
}

In order be able to hook the creation of a Scheduler (to change/wrap its implementation) a Scheduler is created by a Scheduler factory. The SchedulerFactory guaranties that every time a Scheduler is created all the scheduler post processors that are registered in the MuleRegistry are going to be executed.

SchedulerFactoryPostProcessor

A SchedulerFactoryPostProcessor is an object registered in the MuleRegistry that is going to be executed every time a Scheduler is created by a SchedulerFactory.

The SchedulerFactoryPostProcessor can be:

1. Registered by Spring bean or mule module extension via mule configuration (xml)
2. Registered as a bootstrap extension
3. Registered manually in the MuleRegistry.

public interface SchedulerFactoryPostProcessor
{
      Scheduler process(Scheduler scheduler);
}

Provide a new lookup method into the Mule Registry

Mule Registry will now have methods for register/unregister and lookup Schedulers.

public interface MuleRegistry
{
      void registerScheduler(Scheduler scheduler);
      void unregisterScheduler(String name);
      Collection<Scheduler> lookupScheduler(Predicate<String> predicate);
}

Usage of the new Mule API

Starting/running/stopping polls.

Covered scenarios:

1. I do not have to wait for the next poll to execute (which may be a day away) to test what will happen
2. As an extension developer, I want better APIs to start, stop and execute polls.
3. As an extension user I want to start/stop/run my polls

This class shows how an extension developer can stop/start/execute a poll element in a flow. This can be created for exampe by Cloudhub or MMC to start/stop/execute polls by demand. Actually, this class is an Mule agent service.

public class MyPollingService extends AbstractService{

 public void executePoll(String appName, final String flowName) throws MuleException
    {
        MuleContext context = getContext(appName);
        
              MuleRegistry registry = context.getRegistry();
              Collection<Scheduler> schedulers = registry.lookupSchedulers(PollPredicate.flowNamePredicate(flowName));
               for ( Scheduler scheduler : schedulers ){
                    scheduler.schedule();
                }
    }
}

Starting with the Schedulers disabled

1. I do not get confused about what is happening because polls continue to execute while I'm trying to debug what is happening
2. I want to start a test without having polls running.
3. As an extension developer I want to change the polling strategy on runtime
4. As an extension developer I want to enrich with notifications of the poll execution.

This class is a boostrap extension that implements a SchedulerFactoryPostProcessor, this extensions wraps the scheduler implementation so it does not start when it should.

public class BoostrapExtension implements SchedulerFactoryPostProcessor
{

    @Override
    public Scheduler process(final Scheduler scheduler)
    {
        return new Scheduler()
        {
            @Override
            public void execute() throws MuleException
            {
                scheduler.execute();
            }

            @Override
            public void start() throws MuleException
            {
                // Do Nothing
            }

            @Override
            public void stop() throws MuleException
            {
                // Do Nothing
            }
        };
    }
}

Pros

1. Simplifies Stopping/Starting and executing Schedulers
2. PollSchedulerFactory allows hooks based on usage and not redefinition (an example of this is the EndpointFactory)
3. Simplifies the way we can run a scheduler
4. Eliminates internally poll as an inbound endpoint.
5. Allows poll to receive different scheduling strategies
6. Interface Backward compatible.
7. All the interfaces are not aware about the task that is going to be executed.

Cons

1. Uses the MuleRegistry as container of Schedulers which might be confusing sometimes.
2. It changes the concept of poll as an inbound endpoint and Abstract Receiver. Which is bad because it breaks backward compatibility on the implementation details but it might be necessary as it makes no sense for a poll to be a MessageReceiver as it is Connectable (which makes no sense for poll to be connectable)

API Migration impact

We will probably deprecate some clases for poll and we are not going to support them anymore, I consider that is a minor risk if we provide better interfaces and remove legacy code.

XML definition for final users

WE HAVE TWO OPTIONS FOR XML CHANGES. Both of them will allow to deprecate quartz module and allow users to use a single poll element that will be configurable enough to define the scheduling strategy and synchronization.

Keep using poll (option 1)

We define poll as the standar to do polling:

The default implementation to support backward compatibility:

<flow name="test">
   <poll frequency="1000">
       <!-- message source here -->
   </poll>

</flow>

Now the message source is optional so the following is acceptable

<flow name="test">
   <poll frequency="1000"/>
   <mp>
</flow>

Setting the collision strategy

If the poll needs to be executed but it haven't finished what should we do?

<flow name="test">
   <poll>
     <schedulers:simple frequency="" collision-strategy="parallel/queue/ignore"/>
   </poll>
   <mp>
</flow>

1. parallel: Execute polls in parallel (if it is already running then trigger the event in parallel)
2. queue: If the poll is running queue the next event.
3. ignore: if the poll is running ignore the incoming event.

Injecting polling strategy by using the quartz module:

Now the quartz module will allow users to define a polling strategy that can be used by poll, and deprecate the quartz inbound endpoint.

<flow name="test">
   <poll>
     <schedulers:cron expression="" collision-strategy="parallel/queue/ignore"/>
       <!-- message source here -->
   </poll>

</flow>

Poll Pros

1. Supports backward compatibility.
2. Low Documentation impact.

Poll Cons

1. The Poll element code is not good. If we want to have a good code quality we need to do a lot of refactoring around that and break code backward compatibility.

Poll Migration impact

There is no migration guide, we support backward compatibility.

XML Trigger option (option 2)

We deprecate poll and quartz inbound endpoint.

Trigger usage will be:

<flow name="test">
   <trigger>
     <schedulers:cron expression="" collision-strategy="parallel/queue/ignore"/>
   </trigger>

</flow>

Trigger will NOT allow a message source.

Tigger element is to dispatch a flow, not to consume a message source.

Trigger Pros

1. We will have a new element which code is aligned across all products.
2. Simplifies the usage of a scheduler
3. Is more aligned on what users want to do (which is executing flows given a scheduler configuration)
4. Its name makes more sense
5. Mule's code will be backward compatible.
6. Requires less Mule core changes.

Trigger Cons

1. Requires migration guidelines in CH so users need to know that if they are going to deploy a new app with mule 3.5.0 they need to change the poll element to trigger.
2. Cloudhub will need to listen a new type of notification for scheduled elements (no longer inbound endpoints)
3. Studio will need to provide a new element

General Risk

This is a risky feature. A lot of refactoring regarding poll management/polling receivers. We need to consider this a refactoring feature that will add some new functionalities.

Documentation impact

We need to document:

1. All the deprecations.
2. API usage

General Other products impact.

1. We need to define the UI for Studio. (just to provide a better UI for poll and improve usability)
2. CH needs to change the current poll functionality for Multitenancy (low effort)
3. Munit will allow to run polls and stop them without hacking spring (decrease maintenance effort)