Cadence: An Introductory walkthrough

In this post, we will be looking at

• What Cadence is
• Cadence concepts to understand to better put Cadence to use
• Cadence services and tools
• Adding Cadence to your existing stack
• A demo on creating workflows and activities with Cadence

What is Cadence? 🤔 🤔

Cadence is a highly scalable fault-oblivious stateful code (workflow) platform.

This means the state of our [workflow] code, including local variables and threads it creates, is immune to process and Cadence service failures.

Next, let's try and understand Cadence concepts…

Cadence Concepts

As Cadence is a workflow orchestrator, it uses words (or terminologies) from the workflow-automation space. Its key concepts include

• Domain. A domain is a top-level entity within Cadence, used as a container for all resources like workflow executions, task lists etc. The domain acts as a sandbox and provides isolation for all resources within the domain. All resources belong to exactly one domain.
• Workflow. A fault-oblivious stateful function that orchestrates activities. A workflow has full control over which activities are executed, and in which order. A workflow must not affect the external world directly, only through activities. What makes workflow code a workflow is that its state is preserved by Cadence. Therefore any failure of a worker process that hosts the workflow code does not affect the workflow execution.
• Activity. A business-level function that implements your application logic such as calling a service. An activity usually implements a single well-defined action; it can be short or long running. An activity can be implemented as a synchronous method or fully asynchronously involving multiple processes.
• Task lists. These are internal queues maintained by Cadence which are used in dispatching tasks to external workers.

Cadence Services and Tools ⚙️

Cadence service Architecture

The Cadence service from the above diagram is responsible for keeping workflow state and associated durable timers. It maintains task lists (queues) that are used to dispatch activities (tasks) to external workers.

• Matching Service. This is responsible for dispatching activities and guarantees that an activity of a workflow is executed only once when no failure and not at all when a failure occurs
• History Service. This manages the task lists, handles events, stores, and change workflow states.

The Uber team involved in the development of Cadence have also generously open-sourced both GUI (Cadence Web) and CLI (Cadence CLI) based tools which can help us to graphically view workflow status in details and also from the command line.

Inspecting a workflow on Cadence Web UI

Inspecting a workflow with Cadence CLI

Using Cadence

Before we can put Cadence to use, we will need to set it up in our environments.

Pre-requisites

• Docker installed; and
• Docker-compose installed

How?

For this post, we will be using a modified version of the docker-compose-postgres.yml the Uber team has generously created.

cadence-modified-docker-compose-with-postgres.yaml

The above docker-compose file has been modified to

• remove stashsd;
• downgrade the cadence server image tag to 0.15.1-auto-setup
• included mailhog. This will be used to view emails sent from our NotificationActivitiesImpl

#TechBite: Spring Email with MailHog & Gmail

Also, the docker-compose file includes

• Postgres. Used as the data store for cadence
• Cadence server. This bundles in the cadence services described above;
• Cadence web. This is a tool that helps us view created domains and workflows, get detailed information and status of workflows.

How does it all fit together? 🤔 ⚙️

A workflow implementation implements a workflow interface. Each time a workflow execution is started a new instance of the workflow implementation object is created. A workflow interface must have one method annotated with @WorkflowMethod.

To execute tasks, a workflow uses activities. An activity implementation implements an activity interface.

Next, we will look at a demo to see how this all fits.

A Workflow Demo 🕺 💃

For our demo, we have

• a file-watcher-clientwhich watches a directory for the addition of a file (or directory);
• once a new file (or directory) event occurs a workflow is started;
• our workflow has two activities;
• The first activity calls our file-upload-service REST API;
• The second activity gets the result from the file-upload-service and sends an email notification via our file-notification-service

The steps 👣

• First, we run our docker-compose.yaml file

docker-compose -f cadence-modified-docker-compose-with-postgres.yaml up -d

• Secondly, we create our file-watcher domain using the Cadence CLI tool (also can be created programmatically)

docker run --rm --net=host --name=cadence-cli ubercadence/cli:master --do file-watcher domain register

• Thirdly, we create our workflow interface. A workflow interface must have one method annotated with @WorkflowMethod. @WorkflowMethod indicates an entry point to a workflow. It contains parameters such as timeouts and a task list. In our demo example, we have specified the required parameters (such as executionStartToCloseTimeoutSeconds and taskList) at runtime by using WorkflowOptions class.

public interface FileWatcherClientWorkflow {
    @WorkflowMethod
    void processFile(FileUploadInfo workflowData);
}

• Next, we create our activity interfaces. In our activity interfaces, you will notice the use of @ActivityMethod, this annotation is optional as we can specify a for scheduleToCloseTimeoutSeconds by using ActivityOptions class

public interface FileUploadActivity {
    @ActivityMethod(scheduleToCloseTimeoutSeconds = 100)
    UploadResponse uploadFile(FileUploadInfo workflowData);
}
public interface NotificationActivity {
    @ActivityMethod(scheduleToCloseTimeoutSeconds = 100)
    void sendNotification(UploadResponse uploadResponse);
}

• Implement our workflow interface

public class FileWatcherClientWorkflowImpl implements FileWatcherClientWorkflow {
    private final FileUploadActivity fileUploadActivity;
    private final NotificationActivity notificationActivity;

    public FileWatcherClientWorkflowImpl() {
        this.fileUploadActivity = newActivityStub(FileUploadActivity.class);
        this.notificationActivity = newActivityStub(NotificationActivity.class);
    }

    @Override
    public void processFile(final FileUploadInfo workflowData) {
        ...
    }
}

• Implement our activities interface

public class FileUploadActivityImpl implements FileUploadActivity {
    @Override
    public UploadResponse uploadFile(final FileUploadInfo workflowData) {
    }
}
public class NotificationActivityImpl implements NotificationActivity {
    @Override
    public void sendNotification(final UploadResponse uploadResponse) {
        ...

    }
}

• For our workflow to run, we need to register our implementation workflow and activities as workers, then start our worker

private static void startWorkflowWorker() {
    Worker.Factory workerFactory = new Worker.Factory(workflowDomain);
    WorkerOptions workerOptions = new WorkerOptions.Builder().setDataConverter(getDataConverter()).build();

    Worker worker = workerFactory.newWorker(fileUploadTask, workerOptions);

    //register workflow    worker.registerWorkflowImplementationTypes(FileWatcherClientWorkflowImpl.class);

    //register workflow activities
    worker.registerActivitiesImplementations(new FileUploadActivityImpl(), new NotificationActivityImpl());
    workerFactory.start();
}

• Next, we need to execute our workflow

//start workflow async
WorkflowExecution workflowExecution = WorkflowClient.start(workflow::processFile, workflowData);

log.info(String.format("Started process file workflow with workflowId %s and runId %s",
        workflowExecution.getWorkflowId(), workflowExecution.getRunId()));

• Now that we have everything wired up, once our client detects a new file event in our monitored directory a workflow instance is started and executed

while ((key = watchService.take()) != null) {
    for (WatchEvent<?> event : key.pollEvents()) {
        Path filename = (Path) event.context();

        log.info("Event kind: {}. File affected:, {}.", event.kind(), filename.getFileName().toString());

        FileWatcherWorker.executeWorkflow(buildFileUploadInfo(filename));
    }

    key.reset();
}

In Summary 🗒

In this post, we have looked at Cadence

• understanding its concepts;
• demonstrated setting it up using a modified docker-compose file; and
• looked at its service architecture, and demonstrated implementing a workflow.

As usual, the code used for this post is available on Github.

Once again thank you for reading and looking forward to your feedback.