Cloudera on AWS

Neutral

We heavily use Cloudera Data Platform  for data science activities. Various departments in the company utilize it as a sandbox for data discovery. We have multiple data pipelines running on a daily and hourly basis, along with some real-time data pipelines.

Cloudera Data Platform  has significantly improved our data management. Distributed computing with Spark has enabled many processing types that were not possible before. By using the Hadoop  File System for distributed storage, we have 1.5 petabytes of physical storage with 500 terabytes of effective storage due to a replication factor of three.

There are challenges with upgrading or updating various services like Spark, Impala, and Hive  on on-premise and bare metal solutions. We aim to address these issues with a Kubernetes-based platform that will simplify the task of upgrading services. We also wish to implement lakehouse capabilities with Iceberg or Delta Lake frameworks.

I have been using Cloudera Data Platform since 2021. We began with a project a year prior, but it has been in production since then.

I would rate the stability of Cloudera Data Platform as seven out of ten.

For scalability, I rate Cloudera Data Platform at an eight out of ten as it is an on-premise solution.

I would rate the technical support from Cloudera as seven out of ten. Their support is helpful.

Neutral

Before Cloudera, we did not work with other big data platforms. This is our first big data platform, and we also have a classical data warehouse.

We employed local vendors for the implementation, and from our company's side, around ten to twenty people were involved, including engineers, data scientists, and business personnel.

The pricing model for Cloudera Data Platform is complex and has increased significantly compared to CDH. Initially, CDH had a straightforward pricing model based on nodes, but CDP includes factors like processors, cores, terabytes, and drives, making it difficult to calculate costs.

For on-premise use, I would not recommend Cloudera Data Platform as it is expensive and complicated to upgrade. However, for cloud usage, I am uncertain as I do not use it on the cloud. Currently, around thirty to forty people use Cloudera Data Platform in our organization. My final rating for Cloudera Data Platform is seven out of ten.

We use it for multiple domains, including oil & gas, finance (Morgan Stanley), and healthcare. We process around 186 TB of data per day for analytics purposes.

Currently, we use it for healthcare domain. 

Distributed computing, secure containerization, and governance capabilities are the most valuable features.

Since Cloudera acquired HDP, it's been bundled with CBH and HDP. However, the biggest challenge is cloud storage integration with Azure, GCP, and AWS. These platforms offer competitive storage solutions like Gen2, Gen1, Bigtable, BigQuery, Lightstore, S3 buckets, etc., which pose a significant competition to HDP.

I have experience with this product. The short form is HDP 2.7. I have been using it since 2011. 

It was on-premises and hybrid for the first three months, then we migrated it to AWS and Azure.

In terms of storing data in different formats, it's been somewhat unstable. But when compared to Azure Gen2 and its support and features, it's much more advanced. The suitability depends on specific use cases, but overall, HDP seems more mature than it was in the past.

From my experience with both HDP and CDH, they are both scalable. Currently, most people in my company have shifted to Azure, so they are using Gen2 primarily and discarding Gen1.  

I have frequently contacted technical support for both Cloudera and Hortonworks.

We have an IT system to raise issues against their team. Issues usually get attended by someone at an L1, L2, or L3 support level. They connect with us directly.

Previously, we used Cloudera Data Platform (CDP), which turned out to be a cloud-based Azure infrastructure, and implemented metadata solutions like Hive and others.

The setup was very difficult on non-cloud platforms. We had to implement a version-based approach. However, it became simpler with the use of Docker. We used to do it HDP sandboxes and VM boxes and then created clusters in the ancient days. Now, on cloud platforms, it's much easier, just a matter of a few clicks. That's another approach we can take.

I haven't done a price analysis specifically for HDP. However, when it was first introduced as Hadoop 2.0, there were a few use cases where the price was quite high.

It was particularly expensive for Cloudera and Hortonworks Data Platform. Both options were quite resource-intensive.

So, seven, or even nine or ten years ago, it was quite expensive.

I recommend a mature decision-making model. Assess your specific needs and use cases. If HDP suits your requirements, use it. Otherwise, there are many advanced options available. Review and choose the best one for your use case.

Overall, I would rate the solution a nine out of ten. 

I simply love this technology when it comes to new developments. And I've been working with it for the past twelve to thirteen years. However, with the emergence of new technologies, there might be a chance that I would reduce one point because there's room for improvement.  

We use Hortonworks Data Platform for data management, significant data ingestion, and analytics.

Hortonworks Data Platform has a limited user community. I haven't seen much discussion about user experiences. More information could be there to simplify the process of running the product.

We have been using Hortonworks Data Platform for a couple of months.

I rate the product's stability an eight out of ten.

We have five Hortonworks Data Platform users in our organization. It is a scalable platform.

The initial setup could be more straightforward. It would help if you are technically inclined to follow the necessary steps. There could be easy ways to set it up. It takes 45 minutes to complete and requires a team of five people to execute the process.

We implement the product in-house.

Currently, we are using the product in a sandbox environment, and there is no licensing. We might choose a licensing option once we get the results.

I recommend Hortonworks Data Platform to others and rate it an eight out of ten.