Certified Apache NiFi - Calculated Systems

Apache NiFi  is mainly used for data ingestion only, functioning as an extraction tool to extract and load data. It is not particularly strong at the transformation part, but it excels at the ingestion part, so we use it as an ingestion tool only.

For real-time monitoring, we use it for ingestion in real-time. It can be used for either batch ingestion or real-time ingestion, and we use it for both types. It performs well in real-time ingestion scenarios.

The main improvement that Apache NiFi  should make is the UI/UX of this product and also the organization of the project. For example, we are a big team and we need to work with Apache NiFi as an ingestion tool, so every member of the team wants to make a single project in Apache NiFi. There is no privacy in this tool; everyone can see what the other parts of the team or their colleagues are doing, so privacy and organization need improvement.

It is easy to integrate Apache NiFi with some data sources, which is a powerful feature in Apache NiFi and it has many connectors to various data sources.

The main benefit of Apache NiFi is that it is an open-source tool, so we don't pay a license for it. It saves time because we don't have to build connections from the beginning, as it comes with built-in connections.

In Apache NiFi, the ingestion function is what I really like the most. It is very good at the ingestion part.

Adding transformation capabilities is crucial. Apache NiFi is very powerful at ingestion but it needs to be powerful also at the transformation part. We don't use it for the transformation part and we have to go with other solutions. If Apache NiFi were powerful in transformation, it would be a unique tool.

We have been working with Apache NiFi for almost two years.

For stability in Apache NiFi, I would rate it a ten.

For scalability, I would rate it a five out of ten. As I mentioned, it lacks organization, so when we scale out, we want a tool that is organized better than Apache NiFi.

We are using an open-source version, so we don't have support. There is no support for it, but there are versions supported by them that we don't use.

The last time I used TIBCO was in my previous company, which was nearly two years ago.

The setup process for Apache NiFi is simple.

There are many competitors of Apache NiFi in the market, including Apache Spark .

We currently do not work with TIBCO Jaspersoft . I can talk about Apache NiFi and I have experience with the product. We use a reporting solution called SAP BusinessObjects . We use solutions like IBM DataStage, Apache NiFi, Apache Kafka , and Apache Spark . I am not working with SkyWalking from Apache. From other vendors, I don't know if there is a competitor of Apache in this specific role. I would rate this review a seven overall.

Apache NiFi  is used to fetch data from different sources and ingest it into different destinations. The entire platform depends on Apache NiFi  for data transformation and data movement. Multiple sources such as flat files, Oracle, Postgres are connected, and the main database where data is stored is Postgres and Apache Druid . The UI connects directly to those databases to ensure the data is available at all times. This platform has been built for many enterprise platforms.

Sources include Oracle and files in S3 . Apache NiFi creates flows using NiFi processors to fetch data from particular sources such as Oracle. Business data and customer data are available there. Every day there are many incidents, changes, and logging into the system. Data must be fetched daily in two ways: batch streaming and near real-time streaming. Data is fetched twice every day as a batch job. The data is processed after fetching. When connected to the Oracle database, data is received in Avro format. It is then changed into JSON format, and a mapping is used. Whatever destination is required, fields are mapped accordingly. The schema in Postgres is mapped using evaluate JSON path. Whatever JSON field is received is mapped to the respective schema in the database. Records are then ingested into Postgres. Almost daily, 5 million records are received and multiple flows process them. Each flow has a different schedule, which can be every five minutes or twice daily.

Another process retrieves cost data from AWS  and Azure . Everyday billing files are received in S3  or Azure  blob if Azure is being used. The data is retrieved and transformed with modifications made based on the schema. The data is then sent into Apache Druid  using the REST API.

Apache NiFi is the heart of data ingestion for the entire platform. Whatever data movement is happening will happen through Apache NiFi, which is the single data ingestion tool in the platform. Once data lands into the respective destinations using Apache NiFi, multiple transformations, queries, or other operations are performed and projected to the UI. The main gateway for all data from whatever source can be is Apache NiFi. It is a no-code platform where multiple flows can be created, which reduces the time spent writing code and integrating into different connectors. Connectors are already available in Apache NiFi, so flows just need to be created, which makes the process easy and stable.

Apache NiFi offers multiple features. One feature is that it is a no-code platform, so specific code does not need to be written. Direct connectors are available for whatever integrations are needed. If it is Azure, AWS , GCP, or a normal RDBMS , or if connecting to fetch files from a server where files are landing, different types of connectors are available. Integration is simple, flows are created, and the process can run in minutes. Another feature is that if anything needs to be changed or if moving from one flow to a different environment or server is necessary, it is easy to do. Templates can be created, downloaded, and reused as many times as needed. Particular flows can be stored and used multiple times. Email notifications can be configured to trigger if any failures happen, which allows developer teams or the support team to take immediate action. One of the most important features is that if billions and billions of records are coming daily, Apache NiFi ensures that by increasing the configurations and thresholds, that data can be transported, which makes operations easier and allows more things to be accomplished while ensuring SLAs are met.

The most reliable features are the building functionality, the availability of connectors, and the scalability that Apache NiFi offers. These are the features that can be relied on. For example, if billions of records need to be sent from one source to another destination, such as Oracle to Postgres or from a file to an RDBMS , writing a simple Python code requires writing the code and changing the schema. If schema changes occur, it is not easy, as all the code must be reviewed and updated, which takes considerable time for debugging if issues arise. Parallel running of the Python program, multithreading, and all these concepts must be managed, and it is not feasible. The script must be run somewhere and cron must be set up. All these features are bundled and provided in Apache NiFi, which makes operations much easier.

Apache NiFi has tremendously reduced the effort of maintaining multiple different connectors. It makes the transformation and ingestion of billions of records much easier. It is the only platform that is easy for developers to maintain, which makes the entire data platform a very scalable and reliable tool. All data ingestions happen through a single outlet, so if any issues occur, Apache NiFi is the only thing that needs to be addressed, rather than having to go to multiple places and do multiple things or have debugging become a nightmare if Apache NiFi had not been used. It tremendously makes the platform reliable and useful for customers without crossing any SLAs. Timely ingestions are happening and everything is functioning properly.

Improvements can be made in the way of the UI. From the deployment perspective, Git  configurations are available in 2.6 versions and 2.0 and later versions of Apache NiFi. Before 2.0, templates had to be created and stored in Apache NiFi Registry, which is available. However, templates still need to be imported and exported manually if moving from one environment to another environment. Even in 2.0 versions, although GitHub  configurations are available, how it will function needs to be evaluated. Seamless CI/CD deployments are somewhat tricky and challenging when it comes to Apache NiFi with the proper approvals, moving that flow to another environment, and giving the proper RBAC controls. These are areas that could be improved.

Documentation is adequate, but the only pain point is the deployment aspect.

Apache NiFi has been utilized as the main data ingestion tool for around six years and continues to be used currently.

There are no stability issues.

There are no scalability issues.

The first benefit is the time saved by moving to Apache NiFi. Generally, writing a Python program and creating different connectors using a Python program may take three weeks to one week for a developer to create, debug, and deploy. When using Apache NiFi, the time was reduced by almost half. Flows can be created in two days a week, whereas with Python it took almost one to one and a half weeks to complete a single flow. In terms of debugging, Apache NiFi is the only platform where errors can be easily debugged. A bulletin board is available where the exact issue can be viewed and traced back to see where it went wrong. These tasks can be completed quickly, possibly in 30 to 50 minutes to debug an issue. Whereas with Python, if working with a single program level and creating multiple files and multiple jobs, it is very difficult to go back and see exactly where the issue is because tracing from the start to find the issue is necessary, and then every line of code must be debugged. With Apache NiFi, the issue location is immediately identified, and focus can be placed on that particular processor. SLAs have been improved because if an SLA requires completing a job within 30 minutes, using a native programming language or normal jobs requires refinement and optimization for very large transformations, such as ingesting 5 billion records every day, which is not feasible with a normal Python program. Apache NiFi provides the infrastructure where an unlimited number of records, such as billions of records, can be processed in minutes, which ensures that SLAs have never been missed or crossed. SLAs are continuously met, and if additional processing is needed, threads can be increased and little configurations in Apache NiFi can be adjusted, which ensures SLA compliance. These are the metrics that have been seen and benefited from.

If an organization is looking for a unified tool to ingest data at very large scale and wants both batch and real-time streaming with one unified data tool, Apache NiFi is recommended. If a code-free approach is preferred and spending much time writing code or debugging code should be avoided, Apache NiFi is an excellent choice where data flows can be created simply through clicks instead of using other particular tools. This review has been given a rating of 8 out of 10.

I use Apache NiFi  for data ingestion from multiple sources into our data lake.

Can't say many positive impacts apart from ease of use for simpler use cases. The rest is mostly tech debt and negative impacts.

What stood out the most about Apache NiFi  was its ease of use when it comes to smaller datasets or when you have multiple streaming data sets that need to be ingested in multiple steps. When you don't have a team with a lot of coding background, Apache NiFi comes in very handy.

The ease of use in Apache NiFi has helped my team because anyone can learn how to use it in a short amount of time, so we were able to get a lot of work done. You don't have to build anything from scratch as you do with Spark or any other tool sets because you already have blocks created within Apache NiFi, so you save time. When we talk about how much time Apache NiFi has saved in development, you save considerable time.

There are a lot of challenges with Apache NiFi. I believe it is very limited when it comes to scalability and version control. Various improvements are needed for Apache NiFi; the way the tool is structured needs to improve.

Having version control and metadata access is essential. If Apache NiFi compute could scale with workload through ephemeral scaling, that could help; but there is not much we can extract from the Apache NiFi API. It is a huge pain when it comes to reading Apache NiFi configuration metadata. Apache NiFi needs to be faster, and having version control, metadata access, and some improvements would be beneficial. However, it is a good solution with room for improvement.

I have been using Apache NiFi for most of my career, but over the past two years, I have used it quite a lot.

Apache NiFi is stable in most cases. Downtime with Apache NiFi is not unheard of, so we do run into downtimes every now and then, but for the most part, it is stable.

Scalability for Apache NiFi is a problem; it does not scale as well as other Spark solutions. Scalability is one of the reasons downtime is still a thing on Apache NiFi.

I have had to reach out for help with Apache NiFi, but since I get Apache NiFi from Cloudera, they have their premium support, so it is not that big an issue.

We used an older version of Apache NiFi before Cloudera; we chose to continue with Apache NiFi because we were coming from Apache NiFi.

Complex

We host it on AWS  and get it from Cloudera

I haven't seen a return on investment. There may be return on investment based on the technology and easily moving our workloads onto Apache NiFi from our previous system. However, the tech debt that is created from Apache NiFi is not a good addition. It saves time on development, but then you have to cater to some custom solutions.

It is expensive with little to show for it. But I personally don't deal with billing specifics

We evaluated using Spark for all of our ingestions before choosing Apache NiFi, and at that point, we had to build a framework entirely from scratch. That is why we didn't go there, and we went for Apache NiFi.

The process and cost consumption, I believe, is actually not that good when working with Apache NiFi. Apache NiFi is one of the most expensive tools we use. I do believe it is an obsolete technology and the world is moving away from it. I have given this review a rating of five out of ten.