Amazon SWF for long running business workflows

Question

I'm evaluating Amazon SWF as an option to build a distributed workflow system. The main language will be Java, so the Flow framework is an obvious choice. There's just one thing that keeps puzzling me and I would get some other opinions before I can recommend it as a key component in our architecture:

The examples are all about tasks that produce a result after a deterministic, relatively short period of time (i.e. after some minutes). In our real-life business workflow, the matter looks different, here we have tasks that could take potentially weeks to complete. I checked the calculator already, having workflows that live 30 days or so do not lead to a cost explosion, so it seems they already counted in that possibility.

Did anybody use SWF for some scenario like this and can share any experience? Are there any recommendations, best practices how to design a workflow like this? Is Flow the right choice here?

It seems to me Activity implementations are expected to eventually return a value synchronously, however, for long running transactions we would rather use messages to send back worker results asynchronously.

Any helpful feedback is appreciated.

score 3 · Accepted Answer · edited Dec 30 '12 at 13:14

From the Amazon Simple Workflow Service point of view an activity execution is a pair of API calls: PollForActivityTask and RespondActivityTaskCompleted that share a task token. There is no requirement for those calls coming from the same thread, process or even host.

By default AWS Flow Framework executes an activity synchronously. Use @ManualActivityCompletion annotation to indicate that activity is not complete upon return of the activity method. Such activity should be explicitly completed (or failed) using provided ManualActivityCompletionClient.

Here is an example taken from the AWS Flow Framework Developer Guide:

@ManualActivityCompletion
public String getName() {
    ActivityExecutionContext executionContext = contextProvider.getActivityExecutionContext();
    String taskToken = executionContext.getTaskToken();
    sendEmail("abc@xyz.com", 
         "Please provide a name for the greeting message and close task with token: " + taskToken);
    return "This will not be returned to the caller";
}

public class CompleteActivityTask {

    public void completeGetNameActivity(String taskToken) {

        AmazonSimpleWorkflow swfClient = new AmazonSimpleWorkflowClient(…); //pass in user credentials
        ManualActivityCompletionClientFactory manualCompletionClientFactory = new ManualActivityCompletionClientFactoryImpl(swfClient);
        ManualActivityCompletionClient manualCompletionClient 
       = manualCompletionClientFactory.getClient(taskToken);
        String result = "Hello World!";
        manualCompletionClient.complete(result);
    }

    public void failGetNameActivity(String taskToken, Throwable failure) {        
        AmazonSimpleWorkflow swfClient
           = new AmazonSimpleWorkflowClient(…); //pass in user credentials
        ManualActivityCompletionClientFactory manualCompletionClientFactory 
           = new ManualActivityCompletionClientFactoryImpl(swfClient);
        ManualActivityCompletionClient manualCompletionClient 
           = manualCompletionClientFactory.getClient(taskToken);
        manualCompletionClient.fail(failure);
    }
}

That an activity is implemented using @ManualActivityCompletion is an implementation detail. Workflow code calls it through the same interface and doesn't treat any differently than any activity implemented synchronously.

Thanks, this is exactly what I ended up doing and it worked like advertised. — joerx, Mar 17 '13 at 09:18

Amazon SWF for long running business workflows

1 Answers1