Cloudera on AWS

My main use case for Cloudera Data Platform  is for data analytics and AI workload.

We have different data sources where the data is coming in tabular format or CSV, semi-structured or structured, unstructured, and some sort of Kafka streaming messages. We use to store it and then we process and transform, apply the business logic, and then make the data ready for the consumer to consume.

Cloudera Data Platform  offers excellent architectures in terms of decoupling the storage layer from the compute. It is flexible in terms of scaling to your storage account or compute. Additionally, we have different streaming services as part of the ecosystem, and they have added Ranger for security controls, which is a valuable feature.

Decoupling storage from compute has helped my team significantly. Before using Cloudera Data Platform, we were using Cloudera Distribution for Hadoop  (CDH), where we had to have on-premises virtual machines or Linux boxes to add to the cluster, which required lots of effort. We had defined authorized maximum storage per system; for example, one computer can have a maximum of 8 TB, and scaling up to add more compute to the cluster was very challenging. In the current Cloudera Data Platform, the backend storage is a data lake that auto-scales, so we don't have to add more storage. In terms of security, we used to use Sentry  in traditional CDH, but in Cloudera Data Platform, Ranger provides more granular level of security, allowing us to manage who can access data at different levels, maybe at a tabular level or column level.

Streaming services are provided by NiFi, which is one of the best ecosystems for streaming and ETL support.

Cloudera Data Platform has positively impacted our organization by reducing overall manual intervention, requiring fewer efforts and resources to build a big data cluster compared to traditional methods. It is also cost-effective and more stable than the traditional ways of handling big data workload.

In terms of resources, we have reduced from ten resources to four or five resources, making it an effective reduction in manual effort. Regarding cost saving, since we are in the cloud, we are saving significant money compared to maintaining infrastructure on-premises.

Cloudera Data Platform could improve by innovating more in terms of full-fledged support for AI workloads, enriching machine learning or LLM, as there haven't been updates in that aspect over the last one and a half years.

I have been using Cloudera Data Platform for almost four years.

Cloudera Data Platform's scalability is very good.

Customer support is good. However, having a common chat channel between firms and service providers would make communication faster and more efficient.

Neutral

My advice to others looking into using Cloudera Data Platform is that if they are looking for big data workloads on the cloud where they can do analysis and achieve cost savings and resource reductions, it is definitely a good use case. It can vary based on business needs, but it is a good option for big data workloads.

I rated Cloudera Data Platform a six out of ten because I wish that it would keep up with market trends and release AI technology and AI-enabled workloads. Sometimes we struggle to get support, and having a common chat channel between firms and service providers would make communication and support more effective, especially in production.

My main use case for Cloudera Data Platform  is data analytics and AI.

For data analytics and AI in my day-to-day work, we have a multi-source system where the data keeps coming from different source systems, from RDBMS , in tabular format, or semi-structured, or streaming data from Kafka. We process and store data in the backend ADLS, then apply business rule logic to create a golden table which is published for business or end users who consume the data for analytics. Some AI engineers develop or run that code, Python code, or LLM against those data to gain insights.

The most unique feature I love about Cloudera Data Platform  is its integration with Ranger services. Ranger is more flexible compared to Cloudera's previous data distribution component, Sentry , making it more reliable and allowing for access control at a more granular level.

The Ranger integration makes it more flexible and reliable for me by allowing control over data access, specifying who can access at what level, such as table level, masking, or data layer level. This is crucial for managing all data inside the farm.

In terms of integration, it is very easy with Cloudera Data Platform. We just hook it up since it comes with the package when we install the CDP runtime, allowing us to select the ecosystem we want in our farm depending on our use cases. It is not a standalone installation requirement; it is an easy job. Scalability and flexibility are very good.

From a holistic view in the market, I have not seen enough innovation in Cloudera Data Platform, particularly in support for machine learning. It supports it, but not to a robust extent compared to other tech providers, such as Databricks , which are more flexible and in tune with current trends in AI and machine learning. I wish Cloudera would innovate and keep pace with market demands.

Regarding the user interface of Cloudera Data Platform, I have not faced any challenges, though we definitely look forward to innovation to support varied data models and scalability.

I have been using Cloudera Data Platform for almost four years.

Cloudera Data Platform is generally stable; however, we occasionally face minor network connectivity issues as confirmed by the vendor. Sometimes a node goes down, but it automatically returns to a healthy state.

Cloudera Data Platform has positively impacted my organization by eliminating challenges we faced with CDH, which had not been supported for a cloud journey. When adding scalability, such as horizontal scalability to our existing cluster, the process was time-consuming and required upfront costs for procuring servers. In contrast, CDP allows for easy, mostly automated scalability where I can schedule job workflows, fine-tune system resource metrics, and add nodes with just a click.

Customer support depends on the case severity, but from my experience, it is great. Cloudera support is timely and responsive, adhering to the SLAs they provide.

Neutral

Previously, we used Cloudera Data Distribution, known as CDH, which was on-premises and required more manual efforts among multiple teams, taking almost a month to set up a cluster. We switched primarily for cost-effectiveness, flexibility, and the reduced time required for setup.

Cloudera Data Platform has positively impacted my organization by eliminating challenges we faced with CDH, which had not been supported for a cloud journey. When adding scalability, such as horizontal scalability to our existing cluster, the process was time-consuming and required upfront costs for procuring servers. In contrast, CDP allows for easy, mostly automated scalability where I can schedule job workflows, fine-tune system resource metrics, and add nodes with just a click.

A solution architect from the vendor helps us resolve any ongoing issues such as bugs or vulnerabilities, and we appreciate the flexibility of the cloud journey.

In terms of return on investment, I see great changes in operational effectiveness measured by RTO when comparing on-premises solutions with cloud solutions. The difference is notable.

I have not been involved overall in cost negotiation, but we find Cloudera Data Platform to be cost-effective. We work with the Cloudera vendor to secure one or two-year licenses upfront for discounts.

We evaluated Databricks  three years ago, but it was not up to market standards in feature support at that time, particularly lacking an account console, which was introduced afterward. We have seen clients migrating from Cloudera to Databricks since the rollout of that console.

My advice for those considering Cloudera Data Platform is to evaluate their business use case and budget, as these two factors are crucial. If the organization does not need advanced features such as LLM or machine learning, Cloudera Data Platform may be suitable. However, based on the current market, if rating between Databricks and Cloudera, I would give Databricks a one and Cloudera a two.

There are lots of challenges I face while using Cloudera Data Platform. Sometimes, vulnerabilities depend on which version of CDP runtime I am using, so we work with the Cloudera vendor side to remediate any vulnerabilities based on that version. Along with that, we use it for data audit purposes, gathering all inflow data such as how data is being used, who has access, and how many times.

In terms of cost savings with Cloudera Data Platform, moving from on-premises to cloud is very cost-effective. We can use bare metal servers or on-spot servers, which makes it economical. In performance, it is superior to previous versions since multiple Spark versions are added to the CDP runtime, improving data distribution, handling, and fault tolerance, requiring no code fine-tuning.

I rate Cloudera Data Platform six out of ten.

Handling and processing big volumes of data is my main use case for Cloudera Data Platform .

We get the instrument data from various providers, and we process them, do reconciliation, and use Cloudera Data Platform  to process it and ingest it in a structured manner which is then used by our downstream consumers.

One unique aspect about my main use case with Cloudera Data Platform involves multiple application teams building their workflows on the platform. I don't have all the insights into other aspects.

The best features Cloudera Data Platform offers are the processing power with Spark and the distributed data storage, HDFS, which helps us handle massive volumes of data.

Cloudera Data Platform has positively impacted my organization by making it easier to handle such a massive scale of data onto our existing data warehouse systems, allowing us to store heterogeneous data sources.

Cloudera Data Platform can be improved by addressing the feasibility of using it in the cloud; there are some complexities around the components used in cloud by Cloudera Data Platform that are not really convenient. If those can be resolved, it could be widely adopted, similar to Databricks .

Cloudera Data Platform is stable functionality-wise, but it needs some bug fixes for security, which we are expecting Cloudera to provide.

The scalability of Cloudera Data Platform could be enhanced.

I have been using Cloudera Data Platform for around 10 years.

Cloudera Data Platform is stable functionality-wise, but it needs some bug fixes for security, which we are expecting Cloudera to provide.

The scalability of Cloudera Data Platform could be enhanced.

The customer support for Cloudera Data Platform is good.

Neutral

I don't have any specific advice for others looking into using Cloudera Data Platform as I don't see any negatives coming to my mind.

On a scale of one to ten, I rate Cloudera Data Platform a seven out of ten.

Neutral

We heavily use Cloudera Data Platform  for data science activities. Various departments in the company utilize it as a sandbox for data discovery. We have multiple data pipelines running on a daily and hourly basis, along with some real-time data pipelines.

Cloudera Data Platform  has significantly improved our data management. Distributed computing with Spark has enabled many processing types that were not possible before. By using the Hadoop  File System for distributed storage, we have 1.5 petabytes of physical storage with 500 terabytes of effective storage due to a replication factor of three.

There are challenges with upgrading or updating various services like Spark, Impala, and Hive  on on-premise and bare metal solutions. We aim to address these issues with a Kubernetes-based platform that will simplify the task of upgrading services. We also wish to implement lakehouse capabilities with Iceberg or Delta Lake frameworks.

I have been using Cloudera Data Platform since 2021. We began with a project a year prior, but it has been in production since then.

I would rate the stability of Cloudera Data Platform as seven out of ten.

For scalability, I rate Cloudera Data Platform at an eight out of ten as it is an on-premise solution.

I would rate the technical support from Cloudera as seven out of ten. Their support is helpful.

Neutral

Before Cloudera, we did not work with other big data platforms. This is our first big data platform, and we also have a classical data warehouse.

We employed local vendors for the implementation, and from our company's side, around ten to twenty people were involved, including engineers, data scientists, and business personnel.

The pricing model for Cloudera Data Platform is complex and has increased significantly compared to CDH. Initially, CDH had a straightforward pricing model based on nodes, but CDP includes factors like processors, cores, terabytes, and drives, making it difficult to calculate costs.

For on-premise use, I would not recommend Cloudera Data Platform as it is expensive and complicated to upgrade. However, for cloud usage, I am uncertain as I do not use it on the cloud. Currently, around thirty to forty people use Cloudera Data Platform in our organization. My final rating for Cloudera Data Platform is seven out of ten.