Processor decommission

Sometimes you may need to shrink the processor's cluster and decommission some processors. If the processes in your system are all short running, you can simply deactivate the processor and wait until all its tasks are completed. Just call:

    deactivateProcessor('ProcessorName');


Flower will not dispatch new processes to a processor as long as its inactive.

Then, request the pending and running processes count until it is zero.

    count('/Processors/ProcessorName/Pending') +
    count('/Processors/ProcessorName/Running');


After that, you can shutdown the processor and remove it from the network.

However, if the processes are long running, it may take too long to wait until all of them are completed. In such cases you need to redistribute the processor's work over the rest of the processors. This is not trivial task and Flower provides an utility function 'decommissionProcessor' to make it simpler.

To decommission a processor, shut it down first (this may take 10-20 minutes if the processor is highly loaded). The correct shutdown is required to force the state service to flush the current states to the directory. You may deactivate the processor before that, but its not required. If you don't, the 'decommissionProcessor' will do. Also, make sure there are active processors to distribute the tasks to.

Then, call:

    decommissionProcessor('ProcessorName');


The function will emit informational messages describing what it's currently doing.

The function accepts options object as the second parameter.

    decommissionProcessor('ProcessorName', {
        silent: false,
        transactionTimeoutSec: 1,
        queuesMigrationDelaySec: 1,
        reportStatusEverySec: 1,
        startMigrated: true
    });

  • silent - if this flag is set, the function will emit no messages.
  • transactionTimeoutSec - the timeout (in seconds) of the transactions started by the function.
  • queuesMigrationDelaySec - how much long to wait before migrating the default queues. When Flower infers the default queue from a pid, it calculates the path of the process in the directory. It is possible that some senders had calculated the path right before the process is migrated. The message will go to the default queue of the processor being decommissioned. So, there should be delay between the moment when the process itself is migrated and the moment when its messages in the default queue are migrated. The 'queuesMigrationDelaySec' option specifies this delay.
  • reportStatusEverySec - how frequently the information messages should be emitted.
  • startMigrated - whether to start the migrated processes. During the migration, the processes are put to the 'Migrated' folder of the target processor. By default, they are moved to the 'Pending' folder at the last step of decommission, but you may wish not to do that. To start migrated processes manually, call 'startMigratedProcesses'.

If 'decommissionProcessor' failed at any step (due to network failure, for instance), you can run it again until all processes and messages are migrated. However, in this case only processes from the last run will be started automatically; for the rest, you need to call 'startMigratedProcesses'.

Last edited Oct 27, 2013 at 11:55 AM by dbratus, version 1

Comments

No comments yet.