How processors execute workflows

Application domains

It is important to take into account that the processing service creates a separate application domain for each workflow assembly i.e. an assembly containing workflow classes. So, if all workflows hosted by a Flower instance are contained by a single assembly, only a single application domain will be created, but in real cases usually there will be more. For example, as a new version of a workflow assembly is deployed, all newly started processes use the latest version of the assembly, but the running processes still use the old version. This is achieved by loading the new processes into a new application domain that corresponds to the new version of the assembly.

Running workflows in different application domains allows gradual evolution of the system, however processors require more memory. Thus, separation of hosted together workflows onto different assemblies may cause memory issues on processing hosts. To avoid that, consider merging multiple workflow assemblies into a single one.

Blocking, non-blocking and asynchronous activities

The process builder transforms a sequence of its method calls into a sequence of activities that the processor executes. The activities are objects usually containing lambdas passed to the process builder methods. By calling the lambdas the activities access the process state which is captured in the lambdas' closures.

The activities can be blocking and non-blocking. The blocking activities perform service calls (internally or in the lambdas); the non-blocking activities perform pure calculations. The processor executes blocking and non-blocking activities in different thread pools, so the remote calls should never block calculations. It is assumed that the service calls are performed only in Invoke, ExecAndSave and ExecAsync statements; other lambdas are considered non-blocking. You should support this rule to allow the processor to schedule activities efficiently.

ExecAsync is special among other statements. It allows to invoke WCF services with 'await' keyword via task-based asynchronous proxies. The lambda passed to ExecAsync should be asynchronous method with some await-s inside. Unlike other activities, ExecAsync is executed on the system thread pool and for the Flower processor it is non-blocking. As the asynchronous invocations do not block any thread waiting for the I/O completion, they are very resource efficient and should be used if possible. However, ExecAsync does not provide transaction scope and intended mostly for read-only operations.

Thread pools

Each application domain running workflows use three thread pools:
  • For process loading.
  • For blocking operations.
  • For non-blocking operations.

All of the pools has a fixed size which is specified by the corresponding properties of the ProcessorSettings object.

<object id="processorSettings" type="Flower.Processor.ProcessorSettings, Flower.Processor" singleton="false">
    ...

    <!-- Number of threads in the loading thread pool. Default 10. -->
    <property name="LoadingThreadsCount" value="10" />

    <!-- Number of threads in the I/O thread pool. Default 10. -->
    <property name="BlockingThreadsCount" value="30" />

    <!-- Number of threads performing non-blocking calculations. Default 4. -->
    <property name="NonBlockingThreadsCount" value="4" />

    ...
</object>


The size of the thread pools should be selected differently depending the purpose of a pool.
  • Make the size of the loading thread pool larger if the processor hosts multiple short-running processes. If the number of processes per processor is relatively small and the processes are long-running, this pool may be made smaller.
  • Blocking threads pool should be as large as possible to maximize throughput, but it should not be larger than 300 especially if there are more than one application domain. The larger overall number of threads may slow down the entire OS.
  • The non-blocking thread pool should not be larger than the number of CPU cores in the system. If the workflows doesn't perform complex calculations, one non-blocking thread is usually enough.

Limiting the number of running processes

By default, a processor tries to load all processes in its Pending folder and run them simultaneously. In highly loaded instances with the processors having small amounts of memory this may cause out-of-memory errors. To avoid that, you can limit the overall number of processes running simultaneously in all application domains by setting the MaxProcesses property of the ProcessorSettings object.

<object id="processorSettings" type="Flower.Processor.ProcessorSettings, Flower.Processor" singleton="false">
    ...

    <!-- Maximum number of processes the processor can run simultaneously. -->
    <property name="MaxProcesses" value="1000" />

    ...
</object>

Last edited Jan 30, 2014 at 6:40 PM by dbratus, version 1

Comments

No comments yet.