Restate + Kafka = Event-driven apps, where event-driven is an implementation detail

Restate lets you easily build reliable, distributed workflows, applications, and microservices. Until now, invoking workflows/services was possible via gRPC or HTTP. With the recent release of version 0.3.0, Restate can now subscribe to Kafka topics and invoke services for incoming messages. This functionality lets you seamlessly incorporate Restate applications into Kafka-based event-driven architectures.

Event-driven applications without the hassle

Restate’s Kafka integration lets you write event-driven applications, without needing to think about your code in an event-driven way. What we mean by that is that you can write sequential code in a seemingly synchronous manner, but under-the-hood Restate makes sure that this happens in an event-driven and asynchronous way. Giving you all the benefits of event-driven applications: durable event logs, ordering semantics, the efficiency of asynchronous execution, and scalability. While sparing you from the challenges: creating and tracking state machines, handling persistence of application state, message ordering and buffering, retries, and idempotency.

With the new Kafka integration, you can now build such applications directly over streams of events from Kafka.

And it gets better: No need to worry about consumer groups and re-balancing, no partition-based head-of-the-line waiting, the ability to have fast autoscaling of the event-processing logic or to run it as serverless deployments.

Kafka-driven serverless workflows-as-code

You can trigger FaaS (like AWS Lambda) functions through Kafka events, or even build sophisticated workflows-as-code, driven by Kafka events. Restate serves as a dependable orchestrator, ensuring that Lambda functions are triggered when needed, suspended when possible, and run to completion with exactly-once semantics.

This simplifies the execution of event-driven workflows: Restate’s durable async/await models creates a very simple developer experience on top of Lambda’s scalability and ease-of-use.

Blending RPC and Kafka communication

The Kafka integration also lets you mix Remote Procedure Calls (RPC) and Kafka communication in a microservices architecture without worrying about keeping communication reliable across channels. Imagine scenarios like triggering workflows via Kafka messages, which then invoke other services through RPC to perform tasks. Alternatively, consider a service continuously streaming updates over Kafka, with another Restate service acting on these in an exactly-once and consistent manner.

With Restate, all application logic is in functions/service-methods. Some are then connected to Kafka topics (driven by events), others are called directly via RPC. From the application logic’s perspective, it is all the same, and can be switched without any code adjustment.

Get started #

Leveraging the Kafka integration only requires minimal changes to your handler. No need to set up Kafka consumers; all that’s required is defining an event handler, configuring Restate to subscribe to the topic, and instructing Restate to forward the events from the topic to the event handler.

1. Define an event handler #

Implement an event handler with the following signature:

const eventHandler = async (ctx: restate.RpcContext, event: restate.Event) => {
   const order = event.json<string>();
   // ... Do something with the event ...
};

The event handler is invoked with the Restate RpcContext and the Kafka event. Inside the handler, you implement your logic just as you would with RPC-invoked handlers. You use the context to do the interaction with Restate and Restate tracks the progress of the code execution. The only notable difference is that you cannot send back a response as you would with RPC.

After implementing the handler logic, add the event handler to the router and bind the router to the Restate server:

export const router = restate.keyedRouter({
   eventHandler: restate.keyedEventHandler(eventHandler),
});


restate
   .createServer()
   .bindKeyedRouter("my-service", router)
   .listen(8080);

Now, the Restate server will serve the event handler on port 8080, so it can be invoked by Restate when Kafka events arrive.

2. Configuring Restate #

To instruct Restate where Kafka runs, you need to set the Kafka bootstrap servers in Restate’s configuration. For a local Kafka deployment with the bootstrap servers at localhost:9092, your restate.yaml configuration file should include the following:

worker:
   kafka:
      clusters:
         my-cluster:
            bootstrap.servers: localhost:9092

The properties under my-cluster: are the standard properties defined for Kafka consumers. Note that this config cannot be supplied via env variables at this point (the Kafka properties all use dots, which cannot be used in Unix env vars).

Then use this config file when you launch Restate. The line below shows an example of how to mount this config file to a Docker container and reference it as the Restate config:

docker run -v "./restate.yaml:/restate.yaml" \
   -e RESTATE_CONFIG=/restate.yaml \
   docker.io/restatedev/restate:latest

3. Create a subscription #

Finally, after registering the services in Restate via the usual process, you need to make Restate subscribe to the Kafka topics and tell it where it should push the events that arrive on the topic.

To achieve this, you execute the following curl command, which calls the http://localhost:8081/subscriptions endpoint of Restate and tells it to invoke the eventHandler of my-service for each event that arrives on my-topic in my-cluster.

curl -X POST \
   -H 'content-type: application/json' \
   http://localhost:8081/subscriptions \
   -d "{\"source\":\"kafka://my-cluster/my-topic\", \"sink\":\"service://my-service/eventHandler\" }"

4. Start streaming #

That’s all you need to do to have Restate subscribe to a Kafka topic and trigger service invocations for incoming messages. For each event that comes in, Restate makes sure that the handler invocation runs to completion exactly once. The primary limitation is the absence of a backpressure mechanism. This means that if there is a large backlog on the Kafka topic, everything will get loaded into Restate at once. This will be mitigated in future releases.

Next steps #

Want to explore Restate and its Kafka integration further? Have a look at the documentation for TypeScript or Java.

Join the community and help shape Restate!