Introduction

Some workflow engines (especially proprietary ones) are designed upon the idea that given a proper tool non-programmers could create software. Developers would create the reusable steps (activities) - the building blocks that architects or business analysts would drop onto a diagram, so that the diagram become an executable process. It looks impressive at presentations; however, all who developed and supported an enterprise software with a long life-cycle knows that this idea is fundamentally broken. In the reality, there is a lot of details behind the diagrams which the architects and business analysts don't know and don't care about. One of such things, for instance, is the mapping. Suppose we have a simple process like on the diagram below.

a_b.png

At the start we have a data to invoke some service that we are going to invoke at the step A. The service produces a data for the invocation of another service at the step B. Looks trivial, but only on the diagram. In the real cases, in the real enterprise software usually no any two different services have interfaces that perfectly fit to each other, so that you can take the data produced by one service and pass it to another as-is. If it is said that the system A produces the data for the system B, that usually should be understood as "there may be a function transforming the data from A into the data for B". These transformations may be trivial or not and in some cases they make up over 50% of the system's code i.e. complexity. The need for the mapping implies that the blocks on the diagrams are not the same if they look the same. In other words, they are not reusable. If we place a mapping logic into the blocks, the blocks become not just "do the stuff", but "map, do the stuff, map back". This "map" part depends on where in the process the block is placed. The process designer cannot put the block from one process to another, because the process state is different, so the different mapping is required. The reality is that process steps are tightly coupled with the process state hence with each other therefore they are not reusable. Programming with blocks and arrows is like programming by copy-pasting fragments of code - maybe possible in theory, but makes more problems than solves. The diagrams are good as a high level view of a process. They are to guide programmers, not to replace them. If the diagrams become programs, they become so detailed and complex that other diagrams are required to describe them in high level.

What is a workflow engine?

So, a good workflow engine should be primarily a convenient software development platform. It should be developer-oriented. Like any other developer's tool, it should make developer productive by taking care of the tasks that, otherwise, the developer would take care himself.

Normally, a workflow process is a sequence of enterprise service calls guided by control flow. These service calls is the part that requires the most of the boilerplate code. Let's see how to invoke a service in a correct, observable way.

Log the start

Begin the transaction
	Map the data
	Invoke the service
	Map the result

	Save the state

	Commit the transaction
On error
	Rollback the transaction
	Log the error
	Wait and retry if required


To be able to debug the process, we need it to be observable. We need to trace each step and, of course, errors also should be logged.

In the enterprise scenarios, we almost always need a guarantee from a workflow platform that a started process will be executed to the end even if the system was shutdown for a while or crashed. Thus, the platform must be able to save the process state persistently. If the processing host restarts, the state must be restored so the process could continue from the last save point.

If a service call changes the data, we may need a transaction to synchronize the process state with the state of the service data. If the process has invoked a service and crashed before it saved the state, it will start at the last save point and repeat the call. Thus, the data will be updated twice. This may cause errors if the operation is not idempotent (like increments, decrements etc.). To guarantee the consistency, we need the service call and the state saving to be in the same transaction scope.

Some services may be unavailable due to downtime or overload. It is also possible that a distributed transaction caused a deadlock. In these cases, the call may succeed if you repeat it after a while. So the platform should provide wait-and-retry pattern transparently if required.

Processes almost never run alone, but with other processes. In many cases a task can be split onto subtasks each of which is executed by a separate process. So the workflow platform should allow processes to start another processes and communicate with them. This communication should be asynchronous via queue-like channels.

As long as a workflow is a piece of code, it should be unit-testable. Developers should be able to run the processes in an isolated environment with service mocks and check the process state in assert-s.

Apart from that, a workflow engine must have a service registry, it must manage code bases, so that multiple versions of the same process could run simultaneously during transition periods. It also should provide authorization and rights management to restrict access to the workflows.

All these features are the key features of the Flower framework.

Flower framework for developers

From the developer's point of view Flower is just another platform to write a software. The notation in which the workflows are coded may be thought of as a programing language where each statement does more than it normally do in a traditional language. However, being a more high-level language Flower workflow definition is just C#. A workflow in Flower looks like this:

[DataMember] public int i;
[DataMember] public int tmp;
[DataMember] public bool swapped;
[DataMember] public int[] array;

...

public void DefineProcess(IProcessBuilder bld)
{
    bld
        .While(_ => true)
            .Exec(_ => swapped = false)

            .For(_ => i = 0, _ => i < array.Length, _ => i++)
                .If(_ => array[i] > array[i + 1])
                    .Exec(_ => tmp = array[i])
                    .Exec(_ => array[i] = array[i + 1])
                    .Exec(_ => array[i + 1] = tmp)

                    .Exec(_ => swapped = true)
                .End()
            .End()

            .If(_ => !swapped)
                .Break()
            .End()
        .End();
}

You write a DataContract serializable class implementing IWorkflow interface. The interface defines the method accepting an instance of IProcessBuilder. As the name implies, this is the builder of the process control flow. Each of its methods is a statement of the language. The statements accept lambdas that will be executed by the Flower processor in the order defined by the statement's semantics. In other words, lambdas in Flower work like normal expressions and code blocks in traditional languages.

To get and invoke a service, you define it in the workflow class and put Invoke statement like this:

[Service("MyContainer/MyService")]
public IMyService MyService;

public void DefineProcess(IProcessBuilder process)
{
    process
        ...
        .Invoke(MyService, (_, svc) => svc.MyMethod())
        ...
}

The service is injected into the field marked by 'Service' attribute.

It is also easy to implement processes message exchange. Each process has an id (pid) and default queue, so one process can send a message to another having its pid.

...
.Enqueue(
    _ => recipientPid, 
    _ => msg,
    _ => MessageFormat.BinXml
)
...

...
.Dequeue<MessageType>(
    (_, msg) => { /* The code consuming the message. */ }
)
...

Flower provide a variety of communication patterns. The example above is just a basic case.

Since a workflow is a C# class, you can write it in any C# IDE, compile it to get an assembly and then upload it to the Flower directory. Then you can start the process on the Flower instance.

As you finished a workflow, you need to test it. For that Flower provides an API to run workflows in isolation with service mocks. In any unit-testing framework write a test using WorkflowTest builder like this:

//Success test
new WorkflowTest<TestWorkflow>("Test case name")
    .UsingMemoryDirectory
    (
        dir => 
            ... //Directory initialization.
    )
    .Run
    (
        new MyWorkflow
        {
            //Process state initialization.
        }
    )
    .From("Start activity").To("End activity")
    .CheckBefore
    (
        "Some activity", 
        (client, process) => 
        {
            //Assertions.
        }
    )
    .CheckAfter
    (
        "Some activity", 
        (client, process) => 
        {
            //Assertions.
        }
    )
    .CheckAfterFinish
    (
        (client, process) => 
        {
            //Assertions.
        }
    )
    .Go(); //Executes the test.

//Failure test
new WorkflowTest<MyWorkflow>("Test case name")
    .UsingMemoryDirectory
    (
        dir => 
            ... //Directory initialization.
    )
    .Run
    (
        new MyWorkflow
        {
            //Process state initialization.
        }
    )
    .From("Start activity").To("End activity")
    .Expect<Exception>
    (
        "Some activity", 
        ex => true //Check if the exception is the one expected.
    )
    .Go(); //Executes the test.

Since you explicitly instantiate a workflow definition class, you can initialize its state as you need for the test and inject service mocks instead of real services.

Flower framework for architects and administrators

Architecture overview

Architecturally Flower is a cloud of services working together to execute processes reliably and observably. A Flower instance has a single directory and may have multiple processors, state services and queues. Processes are distributed across the processors transparently and executed independently until they need to synchronize with another processes or communicate with them. To synchronize, they invoke the directory; for communication, they invoke either the directory or a queue.

In a typical deployment there are two types of hosts: the directory and a processor. State services and queues are installed on the same hosts as the processors.

architecture_dir_proc.png
architecture.png
architecture_multiple_dirs.png

Thus, a Flower instance can be scaled out by adding more processor hosts. However, since there is only one directory per instance, the scalability of a single instance is limited. To scale out more, multiple Flower instances should be used.

Management

Flower can be easily managed via its JavaScript interface. All actions with Flower environment can be written in JavaScript and executed by the command line tool. So, if you need to do something repeatedly, you will never need to do the same things twice.

The center of a Flower instance is the directory. It is the central storage of the metadata. The directory stores assemblies, service clients' configurations, Flower processors' configurations etc. This data is shared by all components of an instance, so if you need to change it, you change it in one place.

The directory works like a central GAC for all Flower components. For example, if an application needs some library and it is a Flower client, it is not required to deploy the library wherever the application is deployed. If the assembly is not available locally, it will be loaded from the directory. Assemblies are uploaded to the directory with a simple script like this:

importAssemblies([
    'MyAssembly1.dll',
    'MyAssembly2.dll',
    'MyAssembly3.dll',
    ...
]);


Service clients (proxies), helpers or any other objects can be easily defined in JavaScript as JSON objects.

createOrUpdateServiceContainer({
    name: 'MyServiceContainer',
    configuration: {
        MyService: {
            $type: 'MyService, MyAssembly'
            $constructor: {
                uri: 'net.tcp://host:9999/MyService'
            }
        }
    }
});


Then the service instance can be obtained like this:

var svc = client.GetService<IMyService>("MyServiceContainer/MyService");

Flower uses Spring.NET IoC containers internally to instantiate services, so you can utilize their flexibility with a syntax simpler and more compact than the XML notation Spring.NET supports.

The entire directory can be easily exported and imported with the scripts like this:

exportToFileSystem('MyDirectory', DET_ALL);
exportToArchive('MyDirectory.far', DET_ALL);


importFromFileSystem('MyDirectory');
importFromArchive('MyDirectory.far');


This helps to clone Flower environments quickly.

Licensing

Flower framework is licensed under the Apache License Version 2.0. So, you may use it for proprietary as well as open-source software. You also may modify and extend the components of the framework and distribute the modified version without publishing the source code.

Last edited Apr 7, 2014 at 8:07 AM by dbratus, version 6

Comments

No comments yet.