Coverage experiences with more than 1 BPMN definition and Pools

[StephenOTT]

Hey

hoping to get some insights on your usage of coverage:

Few questions have been coming up for the design supoprt for #32.

1. Do you generally find that Each feature in your spec is only covering a single BPMN definition (a single pool in the context of a collaboration model), and if there are multi-definition activation in the process then you test against this across multiple features in a single spec?

2. How often are you using Multiple Pools vs different BPMN files? Do you find you want to visualize the full process visuals (end to end) or just visualize each pool/definition within its very specific context?

[felix-mueller]

Hi @StephenOTT
mostly I would say we have one executable pool per BPMN file and might test multiple BPMN files in a single test (e.g. testing call activities).
My question is if you write a test method covering multiple BPMN files if this should show up in one coverage report (HTML) in the end?

[StephenOTT]

My question is if you write a test method covering multiple BPMN files if this should show up in one coverage report (HTML) in the end?

Well it can. Its more about does it make sense to read in my perspective.

The way i see it is that "generally" you have your Spec with many Features inside. The Spec can represent the overall BPM Process you are running (even if it runs across multiple BPMN definitions) (notice i use BPM Process vs BPMN Definitions).

So the coverage report / unit tests of that overall process can easily cover multiple definitions; where it is something like this:

Spec Setup:

1. Create Engine
2. Deploy BPMN files

Feature 1 (initial start of the process

1. Setup any mocks and Web Service Wiremocks needed for a specific Feature as part of Setup() (of that feature)
2. Execute the feature: Start Process instance, etc
3. cleanup() of the feature if required

Feature 2
....
Feature 3
...
etc.

You would use @Stepwise to ensure that the features are run in order (rather than in parallel).

So if you have Call activities you would do the validation that the call activity is active in Feature 1, but in Feature 2 you would validate that the instance of the second process definition was created. If you had dynamic expression calls for the call activity, then you would have N features for each of the Called BPMNs.

This would give you a single Spec Report showing you the full lifecycle of the process.

I can build this behaviour in a report early next week.

The current missing part is dealing with the async nature of call activities between the Features; Currently the reportData function, which provided by Spock-Reports and is used to pass the Engine History data to the Report template is run during the Feature Cleanup. It cannot be run in the Spec Cleanup (AFAIK). This is a problem for call activities because Feature 1 would detect that the Call Activity did not complete and mark it as blue (still running). Which if "technically" correct from the Feature's point of view (when the feature ended the Call activity was still active/Running).

We shall test and see how it looks.

[StephenOTT]

@felix-mueller so i have been trying a few different scenarios of usage.

Here is a common example i came up with:

Parent Process:

Called Sub Process (Called through Call Activity):

A test could look like this:

def 'Manage CallActivityCoverage1'() {
    when: 'Setting up variables'
      def json = S("{\"customer\": \"Kermit\"}")
      def startingVariables = [
                                'json':json
                                ]

    and: 'We start the CallActivityCoverage process definition'
      def processInstance = runtimeService().startProcessInstanceByKey('CallActivityCoverage')
      sharedFeatureData['CallActivityCoverage1'] = processInstance

    then: 'Process is Active and waiting for user task completion'
      assertThat(processInstance).isActive()

    then: 'The current process variables are equal to the starting variables'
      def processVariables = runtimeService().getVariables(processInstance.getProcessInstanceId())
      assertThat(processVariables == startingVariables)

    then: 'The process instance should be waiting for the Call Activity to Complete'
      assertThat(processInstance).isWaitingAt('Task_1gdn63n')

  }

def 'Manage CallActivityCoverage2'() {

  setup:
   ProcessInstance callActivityCoverage1ProcessInstance = (ProcessInstance) sharedFeatureData['CallActivityCoverage1']

  when: 'CallActivityCoverage1 was started, get the called called process instance'
  HistoricActivityInstance callActInstance = historyService().createHistoricActivityInstanceQuery()
                                                              .processInstanceId(callActivityCoverage1ProcessInstance.getProcessInstanceId())
                                                              .activityId('Task_1gdn63n')
                                                              .singleResult()

    ProcessInstance callActivityCoverage2ProcessInstance = calledProcessInstance(processInstanceQuery().processInstanceId(callActInstance.getCalledProcessInstanceId()))

  and: 'Add CallActivityCoverage2ProcessInstance to shared data'
    sharedFeatureData['CallActivityCoverage2'] = callActivityCoverage2ProcessInstance

  then: 'CallActivityCoverage2 is running'
    assertThat(callActivityCoverage2ProcessInstance).isActive()

  then: 'CallActivityCoverage2 is waiting at the User Task'
    assertThat(callActivityCoverage2ProcessInstance).isWaitingAt('Task_0xjkfyv')

  then: 'Complete the User Task'
    complete(task(callActivityCoverage2ProcessInstance))

  then: 'CallActivityCoverage2 has completed'
    assertThat(callActivityCoverage2ProcessInstance).isEnded()

  then: 'CallActivityCoverage1 has ended'
    assertThat(callActivityCoverage1ProcessInstance).isEnded()
  }

Few issues still working out in the simplification of usage:

1. https://forum.camunda.org/t/unit-testing-how-to-get-call-activitys-called-process-instance-from-runtime/7733
2. Based on my previous comments in my previous comment, the last assertion would "philosophically" be in another Feature as its a assertion about ProcessInstance 1. But i think this starts to get too restrictive and confusing. I would easily see technical business staff implementing a test as a single Feature that asserts Instance1 and Instance2. Splitting up as i did seems to work. But it seems very redundant to add a third feature just to assert on a single check that Process1 has completed.

---

My current thinking is that I wrap BPMN Coverage data into its own class, which allows the creation of multiple instances per feature. This way when you want Coverage images, you do something like new BpmnCoverage(processInstance) (this would get the Model data and run the History lookups for the various coverage data).

So it would basically come down to being a insertable instance that the spock-report detects and presents within the Method.

Could add also Weight and other metadata to order the coverage (so for example that the coverage for Instance1 appears before Coverage2).

The "trick" will just to be remember when to execute your "snapshots"/BpmnCoverage() instances: You should generate a new instance whenever you want a snapshot of the current "state": So code above, i would have 3 instances:

Snapshot 1 (of Instance1): Taken at the end of Feature 1 ("Manage CallActivityCoverage1") (you could also consider taking a snapshot of Instance 2 as this point)

Snapshot 2 (of Instance2): Taken in Feature 2 ("Manage CallActivityCoverage2") right after assertion "assertThat(callActivityCoverage2ProcessInstance).isEnded()".

Snapshot 3 (of Instance 1): Taken at the end of Feature 2.

So you would end up with 3 or 4 Coverage maps for the Spec above depending on what visuals you want to provide. But it would be 100% up the test writer to decide how they want to express the coverage.

IMO, this flexible option gives more expressive tests and visuals rather than a rigid "one coverage per feature iteration"

@felix-mueller thoughts?

[StephenOTT]

@felix-mueller here is a prototype. I just need to finish adding the "coverageData" class to enable multiple coverages within a single Feature

[StephenOTT]

@felix-mueller okay here is the update:

This is Single Feature usage:

package conditionalstart

import org.camunda.bpm.engine.history.HistoricActivityInstance
import org.camunda.bpm.engine.runtime.ProcessInstance
import org.camunda.bpm.engine.test.ProcessEngineRule
import org.junit.ClassRule
import spock.lang.Shared
import spock.lang.Specification
import spock.lang.Stepwise

import static io.digitalstate.camunda.coverage.BpmnCoverageBuilder.*
//import static org.camunda.bpm.engine.test.assertions.ProcessEngineAssertions.assertThat
//import static org.camunda.bpm.engine.test.assertions.ProcessEngineAssertions.processEngine
import static org.camunda.bpm.engine.test.assertions.ProcessEngineTests.*
//brings in Camunda BPM Assertion + AssertJ core.api.Assertions
// http://joel-costigliola.github.io/assertj/core/api/index.html
// http://camunda.github.io/camunda-bpm-assert/apidocs/org/camunda/bpm/engine/test/assertions/ProcessEngineTests.html
// http://joel-costigliola.github.io/assertj/


import static org.camunda.spin.Spin.S

class CallActivitySingleFeatureSpec extends Specification  {

  @ClassRule
  @Shared ProcessEngineRule processEngineRule = new ProcessEngineRule('camunda_config/camunda.cfg.xml')
  @Shared String deploymentId

  def setupSpec(){
    def deployment = repositoryService().createDeployment()
                                        .addInputStream(sequenceFlowHistoryFileName(), sequenceFlowHistoryGenerator())
                                        .addModelInstance('CallActivityCoverage.bpmn', prepModelForCoverage('bpmn/conditionalstart/CallActivityCoverage.bpmn'))
                                        .addModelInstance('CallActivityCoverage2.bpmn', prepModelForCoverage('bpmn/conditionalstart/CallActivityCoverage2.bpmn'))
                                        .name('CallActivitiesCoverage')
                                        .enableDuplicateFiltering(false)
                                        .deploy()
    deploymentId = deployment.getId()
    println "Deployment ID: '${deploymentId}' has been created"
  }

  def 'Manage CallActivityCoverage1'() {
    when: 'Setting up variables'
    def json = S("{\"customer\": \"Kermit\"}")
    def startingVariables = [
            'json': json
    ]

    and: 'We start the CallActivityCoverage process definition'
      def callActivityCoverage1ProcessInstance = runtimeService().startProcessInstanceByKey('CallActivityCoverage')

    then: 'Process is Active and waiting for user task completion'
      assertThat(callActivityCoverage1ProcessInstance).isActive()

    then: 'The current process variables are equal to the starting variables'
      def processVariables = runtimeService().getVariables(callActivityCoverage1ProcessInstance.getProcessInstanceId())
      assertThat(processVariables == startingVariables)

    then: 'The process instance should be waiting for the Call Activity to Complete'
      assertThat(callActivityCoverage1ProcessInstance).isWaitingAt('Task_1gdn63n')
      reportInfo(generateCoverageData(processEngine(), callActivityCoverage1ProcessInstance, "CallActivityCoverage1.bpmn Waiting for Called Process"))

    and: 'get the called called process instance'
      HistoricActivityInstance callActInstance = historyService().createHistoricActivityInstanceQuery()
              .processInstanceId(callActivityCoverage1ProcessInstance.getProcessInstanceId())
              .activityId('Task_1gdn63n')
              .singleResult()

      ProcessInstance callActivityCoverage2ProcessInstance = calledProcessInstance(processInstanceQuery().processInstanceId(callActInstance.getCalledProcessInstanceId()))

    then: 'CallActivityCoverage2 is running'
      assertThat(callActivityCoverage2ProcessInstance).isActive()

    then: 'CallActivityCoverage2 is waiting at the User Task'
      assertThat(callActivityCoverage2ProcessInstance).isWaitingAt('Task_0xjkfyv')

    then: 'Complete the User Task'
      complete(task(callActivityCoverage2ProcessInstance))

    then: 'CallActivityCoverage2 has completed'
      assertThat(callActivityCoverage2ProcessInstance).isEnded()
      reportInfo(generateCoverageData(processEngine(), callActivityCoverage2ProcessInstance, "CallActivityCoverage2.bpmn Completion"))

    then: 'CallActivityCoverage1 has ended'
      assertThat(callActivityCoverage1ProcessInstance).isEnded()
      reportInfo(generateCoverageData(processEngine(), callActivityCoverage1ProcessInstance, "CallActivityCoverage1.bpmn Completion"))
  }

    def cleanupSpec() {
//     https://docs.camunda.org/javadoc/camunda-bpm-platform/7.8/org/camunda/bpm/engine/RepositoryService.html#deleteDeployment(java.lang.String,%20boolean,%20boolean,%20boolean)
      repositoryService().deleteDeployment(deploymentId,
                                                 true, // cascade
                                                 true, // skipCustomListeners
                                                 true) // skipIoMappings
       println "\nDeployment ID: '${deploymentId}' has been deleted"
  }

}

And this is Multi-Feature Usage:

Multi-feature is bascially the same setup but you are doing things like:

def 'Manage CallActivityCoverage1'() {
    when: 'Setting up variables'
    def json = S("{\"customer\": \"Kermit\"}")
    def startingVariables = [
            'json': json
    ]

    and: 'We start the CallActivityCoverage process definition'
    def processInstance = runtimeService().startProcessInstanceByKey('CallActivityCoverage')

    then: 'Process is Active and waiting for user task completion'
    assertThat(processInstance).isActive()

    then: 'The current process variables are equal to the starting variables'
    def processVariables = runtimeService().getVariables(processInstance.getProcessInstanceId())
    assertThat(processVariables == startingVariables)

    then: 'The process instance should be waiting for the Call Activity to Complete'
    assertThat(processInstance).isWaitingAt('Task_1gdn63n')

    cleanup:
      sharedFeatureData['CallActivityCoverage1'] = processInstance
      reportInfo(generateCoverageData(processEngine(), processInstance, "CallActivityCoverage1.bpmn Waiting for Called Process"))
  }

def 'Manage CallActivityCoverage2'() {
  setup:
    ProcessInstance callActivityCoverage1ProcessInstance = (ProcessInstance) sharedFeatureData['CallActivityCoverage1']

  when: 'CallActivityCoverage1 was started, get the called called process instance'
    HistoricActivityInstance callActInstance = historyService().createHistoricActivityInstanceQuery()
                                                              .processInstanceId(callActivityCoverage1ProcessInstance.getProcessInstanceId())
                                                              .activityId('Task_1gdn63n')
                                                              .singleResult()

    ProcessInstance callActivityCoverage2ProcessInstance = calledProcessInstance(processInstanceQuery().processInstanceId(callActInstance.getCalledProcessInstanceId()))

  then: 'CallActivityCoverage2 is running'
    assertThat(callActivityCoverage2ProcessInstance).isActive()

  then: 'CallActivityCoverage2 is waiting at the User Task'
    assertThat(callActivityCoverage2ProcessInstance).isWaitingAt('Task_0xjkfyv')

  then: 'Complete the User Task'
    complete(task(callActivityCoverage2ProcessInstance))

  then: 'CallActivityCoverage2 has completed'
    assertThat(callActivityCoverage2ProcessInstance).isEnded()

  then: 'CallActivityCoverage1 has ended'
    assertThat(callActivityCoverage1ProcessInstance).isEnded()

  cleanup:
    reportInfo(generateCoverageData(processEngine(), callActivityCoverage2ProcessInstance, "CallActivityCoverage2.bpmn Completion"))
    reportInfo(generateCoverageData(processEngine(), callActivityCoverage1ProcessInstance, "CallActivityCoverage1.bpmn Completion"))
  }

Where you are sharing data between Feature with a @Shared class level variable such as the sharedFeatureData usage in the above.

[StephenOTT]

@felix-mueller any thoughts?