IBM & Red Hat on AWS

Efficiently manage Kafka with IBM webMethods Hybrid Integration on AWS

As companies across the globe optimize their applications to achieve operational efficiency and improve customer experiences, increasing volumes of real time events emerge across business operations. Event driven applications are commonplace with events more distributed and consumed across the IT footprint. These events are used to optimize supply chains, perform transaction analysis, detect suspicious activities, obtain a 360 view of customers and much more. Apache Kafka has become the standard for event streaming for businesses and enterprises.

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that handles Apache Kafka infrastructure and operations so customers can focus on building their applications and let AWS handle the undifferentiated operational heavy lifting. Amazon MSK has built-in high availability, fault tolerance, integration to monitoring and logging services such as Amazon CloudWatch and AWS CloudTrail.

Organizations that run hybrid workloads with events distributed across AWS services like Amazon MSK and their on-premises data centers can benefit from a centralized approach to event management. The Event Endpoint Management capability in IBM Event Automation is part of IBM webMethods Hybrid Integration. It is now available as software as a service (SaaS) on AWS Marketplace, helping customers gain visibility and control over their distributed events.

What is Event Endpoint Management in IBM webMethods Hybrid Integration?

Event Endpoint Management is a capability of the IBM webMethods Hybrid Integration SaaS product on AWS. It allows customers to better discover Kafka topics, reuse topics through self-service and have granular control over them. These features of Event Endpoint Management help prevent event sprawl as Kafka adoption accelerates. They also help businesses gain better visibility into their environments, secure their data, avoid unnecessary duplicate streams and control costs.

Key features of Event Endpoint Management

AsyncAPI (Describe)

Formalize event interfaces using an open standard to capture the technical and user documentation needed by developers as shown in Figure 1. This includes information on what the events represent, the schema the payloads conform to, how to connect to the Kafka broker that hosts the topic and so on. Event Endpoint Management includes code samples in a variety of common programming languages to get started quickly. AsyncAPI is also supported by a growing number of code generation tools.

Screenshot from hybrid integration showing that you can export your event endpoint in AsyncAPI format.

Figure 1. Asynchronous API to describe using open standards.

Event Catalog (Reuse)

Once an event stream has been defined as AsyncAPI, you can share it with other users via an event catalog as shown in Figure 2. Developers can discover relevant topics using tags and access them through self-service API key generation.

Screenshot showing the catalog feature where developers can look at available event streams in AsyncAPI format.

Figure 2. Event catalog to achieve topic reuse.

The catalog works with any Kafka-based implementation. Topics hosted in Amazon MSK or sourced from Amazon EventBridge can appear alongside topics from other Kafka brokers, whether deployed on AWS or on-premises.

This approach provides visibility of available topics across your organization regardless of their location or originating team. Administrators can track topic subscriptions, identify inefficiencies, and prevent duplication. By promoting reuse, you can reduce development effort and optimize costs for your Kafka infrastructure.

Event Gateway (Secure)

The event gateway secures event data by acting as a policy enforcement point for applications looking to access the event data. As shown in Figure 3 topic owners can apply controls that stipulate the circumstances under which the topic can be made available. The controls can include:

  • security and encryption (mTLS)
  • schema validation to ensure data integrity
  • subscriber quotas to limit the number of events or amount of event data
  • redact sensitive fields unless explicitly approved
  • Manage access to topics for subscribers
Screenshot showing the controls that topic owners can apply to their topics. There is also an example showing how you can redact a specific property in the message.

Figure 3. Apply controls and enforce policies using Event Gateway.

Topic Virtualization (Abstraction)

A single Kafka topic can be exposed with multiple combinations of controls and policies applied. The event gateway provides a point of abstraction, creating virtual views of topics as applications interact via the gateway as shown in Figure 4. This eliminates the need to create multiple topics for different audiences or projects.

Architecture diagram showing how you can expose a single kafka topic with multiple combinations of controls and policies applied.

Figure 4. Expose topics to multiple applications using Event Gateway.

As shown in Figure 5, the Event Gateway can be securely deployed facing the internet to socialize the topics with business partners or external parties. In combination with a TCP proxy, it provides configurable restrictions to protect Kafka clusters from badly behaved or invalid applications. For example, it can introduce delays to connection closing and enable authentication back off, limit connections per subscription and the number of authentications retry.

Architecture diagram showing how you can expose the event gateway to external parties.

Figure 5. Expose topics to external entities securely using Event Gateway.

IBM Event Management and AWS Integrations

At the time of writing this blog, IBM Event Management integrates with the following AWS Services:

  • Amazon MSK – Event Endpoint Management supports Amazon MSK topics directly, and these can be added to the event catalog for socialization, discovery and self-service access. Event gateways can be deployed alongside Amazon MSK brokers to provide policy-based controls and topic abstraction.
  • Amazon EventBridge – Amazon EventBridge provides a Kafka connector that enables a stream of Kafka events to be sourced from or sent to an EventBridge pipe. Once in Kafka, events can be socialized and managed by Event Endpoint Management in the same way as any other Kafka stream.

Conclusion

IBM Event Management within IBM Hybrid Integration provides better discovery, self-service-based reusability and scalable controls that allow businesses to manage their event-based integrations within the organization and externally.

Visit the AWS Marketplace and get started with IBM webMethods Hybrid Integration on AWS:

Additional Content

Senthil Nagaraj

Senthil Nagaraj

Senthil Nagaraj is a Partner Solutions Architect with Amazon Web Services and is based in Virginia. He enjoys providing creative solutions for customer problems, while still being fascinated by how cloud computing is driving the art of possible.

Matt Sunley

Matt Sunley

Matt Sunley is a Principal Product Manager at IBM. He has a breadth of experience in the Automation segment of IBM’s software portfolio, including enterprise messaging, application integration and event-led integration. In his current role, Matt leads product strategy for Event Automation, a complete and composable software solution for building and operating event-driven architectures.