Apache Kafka® & Apache Flink® on Confluent Cloud™ - Annual Commits

My main use cases for Apache Kafka  are for a big project where Apache Kafka  is one of the main components for sending messages and receiving them.

I appreciate that Apache Kafka is fast and secure thanks to implementing it with AWS , allowing me to secure it on a high level. It's fast with a good connection, and the different types of connections are a good thing for me, helping our team that uses it, which is very helpful.

The impact of Apache Kafka's scalability features on my organization and data processing capabilities depends on how many messages each company wants to receive. With a high throughput, it helps to have more brokers and partitions. If you are a company that doesn't need that many messages, I won't say it will help you a lot, but on the other hand, it can change significantly.

I don't actually think about anything they could improve about Apache Kafka, as our use cases using it are more or less on the basic level, so I didn't think about any kind of improvement.

For personal preferences, since we use Managed Kafka in AWS , I would appreciate having some kind of UI integrated into Apache Kafka for connecting to it because using code to connect it is basic, but we can use a UI.

I have been using Apache Kafka for about six months.

I use Apache Kafka topic partitioning feature for my system stability.

This feature of Apache Kafka has helped enhance our system stability when handling high volume data because we have thousands of messages in a small amount of time, so partitioning helps us distribute all the messages that we receive between all partitions, which helps us to be stable.

I have seen some ROI from Apache Kafka, although I can't recall specifics.

I am saying that Apache Kafka has better security than other options, even though I don't know about them because we didn't explore them. We simply knew that Apache Kafka is the base where you should use it, so we went with that.

At this point, I don't have any specific examples to share. I don't actually remember if I have used Apache Kafka Connect for integrating various data sources and sinks within my organization. I currently don't have any examples of how it has benefited my organization.

On a scale of one to ten, I would rate Apache Kafka an eight.

I worked with Apache Kafka  for customers in the financial industry and OTT platforms. They use Kafka particularly for data streaming. Companies offering movie and entertainment as a service, similar to Netflix, use Kafka.

Apache Kafka  offers unique data streaming. It allows the use of data in motion, allowing data to propagate from one source to another while it is in motion. This is valuable when data is not simply residing in a database.

In the data sharing space, the performance of Apache Kafka could be improved. The performance angle is critical, and while it works in milliseconds, the goal is to move towards microseconds.

I started working with Kafka about five years ago while at a financial company.

Apache Kafka is stable. Even though enterprises often use the open-source version, there are minimal issues after configuration.

Apache Kafka is very scalable. I would rate its scalability as nine out of ten. Customers have not faced issues with user growth or data streaming needs.

The Apache community provides support for the open-source version. Despite being open-source, extensive documentation is available to resolve issues.

Positive

The initial setup of Apache Kafka is straightforward, around an eight on a scale from one to ten. The deployment process involves configuring the publisher, subscriber, and other parameters. SaaS can be deployed from the cloud in a couple of hours.

Since I work with the open-source version of Kafka, solutions are managed internally with the Apache documentation.

The open-source version of Apache Kafka results in minimal costs, mainly linked to accessing documentation and limited support. Enterprises usually opt for the more cost-effective open-source edition.

For critical business components, it is advisable to use Confluent-managed services for Kafka. However, for non-critical functions, the open-source version is sufficient. 

Overall, I rate Apache Kafka as nine out of ten for its scalability and stability.

We are always using Apache Kafka  for our real-time scenarios. It helps us detect anomalies and attacks on our website through machine learning models.

We are managing our data by topics. Splitting topics is more effective for us. Apache Kafka  is very fast and stable. It offers scalability with ease and also integrates well with our tools. Fault tolerance is a good feature, and it also has high throughput rates.

Config management can be better. We are always trying to find the best configs, which is a challenge.

I have been working with Apache Kafka for more than four years. It has been used since the beginning of our department, maybe six years.

It is very stable and meets our needs consistently.

If there is latency, our Kubernetes  admin includes our Kafka nodes to increase scalability. Kafka provides flexibility and integrates easily with Kubernetes .

Before Apache Airflow , I used Cron Tab. However, Apache Airflow  makes it easy to follow and manage tasks, and data science departments can easily build their models or pipelines using it.

I would rate Apache Kafka nine out of ten.

Currently, I work for an observability company. We stream customer data into our cloud, digest the information, enrich it, transform it, save it, and use on-the-fly aggregation with Kafka. Previously, I worked for a security company doing normal detection using streaming with Kafka. 

I also worked for a company with a data platform based on Kafka, where we ingested clickstream data and enriched it before streaming.

The most valuable feature of Kafka is the Kafka Streams client. Unlike other systems like Flink  or Spark Streaming , you don't need a separate engine to do real-time transformations and analytics. The amount of data that can be streamed into the platform and the scalability are also significant benefits.

Kafka requires fine-tuning to find the best architecture, number of nodes, and partitions for your use case. It’s a trial-and-error process with no one-size-fits-all solution. Issues may arise until it’s appropriately tuned. 

While it can scale out efficiently, scaling down is more challenging, making deleting data or reducing activity harder.

I have been working with the Kafka product for more than ten years.

Since Kafka is written in Java, it's not as stable as it should be on the JVM. The stability depends on fine-tuning the system to find the best architecture for your use case. However, the replication factor helps avoid data loss despite the stability issues.

Kafka's architecture allows for scalability by adding nodes and partitions to topics. However, it's not as effective in scaling in, making reducing activity and deleting data harder. 

Scalability can be managed both manually and automatically to meet demands.

I used to work with Spark Streaming  and Flink , however, not in the past year.

If you are unfamiliar with Kafka, setting up the cluster can be quite difficult. You need to understand the architecture and components and compute the data volume upfront. For experienced individuals, the setup is less difficult yet still requires preparation.

From a time-saving perspective, onboarding new customers is straightforward, requiring them merely to stream their data into our platform.

We use Apache Kafka , which is open-source, so we don't have fees. I can't comment on ownership costs as I am not responsible for that domain.

Apart from Kafka, I have experience working with Spark Streaming and Flink.

When implementing Kafka, it's important to plan the cluster size upfront to ensure easy scalability. Adding or removing nodes can disrupt the clusters, so proper sizing and planning are key. 

I would rate Kafka as a solution as a nine.