Certified Apache NiFi - Calculated Systems

My main use case for Apache NiFi  is to use it for basic ETL pipelines before ingesting data into the data lake. For example, if we want to ingest anything into Snowflake , before actually moving it to Snowflake , we do basic data massaging and transformations through Apache NiFi . This may include consuming a file, converting that file from one format to another, breaking it into chunks, and then pushing it to Snowflake.

A specific example is that we consume all caller data through Apache NiFi by connecting to Kafka topics and consuming those messages. We then convert those messages into JSON format and push those messages to Snowflake by running Snowflake procedures, which eventually ingest the data directly into Snowflake by reading it through an S3  bucket. We also push the actual JSON messages to S3  buckets through Apache NiFi itself.

The best features Apache NiFi offers include the integration capability because we have use cases wherein we have been using Apache NiFi to integrate through and consume from multiple sources. It can connect to any APIs, NAS  drives, and databases, and it can consume streaming data. We can connect to Kafka topics and queues as well through Apache NiFi. First of all, there is flexibility in consuming from multiple sources. We can also easily transfer information from one block to another. We have command and control on the controller services that we can maintain separately so that we do not have to repeat connections and create multiple connections in different processor groups.

Apache NiFi has positively impacted our organization by significantly helping us streamline the processes. Earlier, each team was creating their own ETL pipelines with no standard being followed. Apache NiFi gave us the opportunity to streamline that process. Each team can request access to Apache NiFi and be onboarded separately based on their needs. We have specifically standardized the pipeline design so that no one can deviate from that. For example, everyone is supposed to use specific variables when enabling the alerting system. We have a different tool called Moogsoft  that everyone can onboard to using the InvokeHTTP processor of Apache NiFi to send their alerts to a centralized system which can generate incidents. No one is now working in silos, and there is a specific pattern being followed by everyone. Furthermore, everyone must have Apache NiFi configured so that they can consume from a specific source but eventually push the data to our strategic cloud partner, AWS , first into an AWS  bucket which is common for all, and from that bucket, subsequent processing will be done in Snowflake. The standard being followed now once Apache NiFi has been introduced has allowed others to copy-paste successfully created pipelines, saving a lot of time in our overall software development life cycle by reducing redundancy and efforts.

A couple of improvements for Apache NiFi would be better logging. Sometimes when looking at logs and events, they do not always make sense. A strong recommendation is that there has to be improved logging in Apache NiFi. Secondly, when looking at the file states, the history of processed files should be more readable so that not only the centralized teams managing Apache NiFi but also application folks who are new to the platform can read how a specific document is traversing through Apache NiFi.

I have been using Apache NiFi for almost four and a half years now.

Apache NiFi's scalability is great; we can easily scale into it without encountering any challenges.

Customer support for Apache NiFi has been excellent, with minimal response times whenever we raise cases that cannot be directly addressed by logs. The support team has consistently provided great assistance with processor failures and helped us create ad-hoc processors as needed.

Positive

We did not use any previous solution before Apache NiFi. Apache NiFi has significantly helped us improve our processes and streamline them, while we are also using SnapLogic  for integration purposes in parallel.

I am not aware of any other solutions that were explored before onboarding Apache NiFi. Being from the analytics team, we were earlier using Tableau Prep for ETL and transformations and Informatica as well, but I am unsure if anything else was explored in the organization prior to using Apache NiFi.

Apache NiFi provides huge relief for all teams with similar use cases for ETL purposes, and it supports not just ETL but also ELT, allowing us to save significant time.

I cannot comment on the pricing, setup cost, and licensing for Apache NiFi, but I can say there was significant time saved across the development life cycle due to reusable pipelines offered by Apache NiFi.

The reason I rate Apache NiFi a seven is that most development folks are afraid to start using Apache NiFi because, to begin with, it does not always directly make sense. For example, there are other integration tools such as SnapLogic  where you can simply search for a specific processor, but that is not the case with Apache NiFi. You need some basic understanding of which processors are there before you can fetch what you need. Better UI design should allow newcomers to search using relevant keywords, such as API, to retrieve appropriate processors, which currently is not happening, requiring some understanding of Apache NiFi. The other challenge I mentioned is having better logging, especially for processor-related logs to help newcomers navigate effectively.

I think Apache NiFi is a great tool that one should definitely explore. It is essential to perform basic checks regarding requirements, but if someone is looking for ETL and ELT functionalities needing to connect to CSVs, JSONs, Excels, and databases, they can onboard to Apache NiFi. It offers great connectivity for consuming or pushing data through queues and cloud workloads. Overall, it is an excellent product, especially for basic data massaging and processing before pushing data to Snowflake or creating reports. I rate Apache NiFi a seven out of ten.

Apache NiFi  is used mainly for ETL to get data from multiple sources and then load it into a data lake. For example, data from Ab Initio is extracted and then loaded to an S3  bucket. From the S3  bucket, that data is read again and then loaded into other data layers. Different data layers, such as raw and raw silver, all use Apache NiFi .

Apache NiFi has positively impacted the organization by making development really easy, allowing efficient design and development, and enabling code reuse which has reduced the development effort. Integration with Git  is also really good for sharing code across teams. A reduction in development effort of about 30% has been observed.

The best features Apache NiFi offers include the ability to connect to any type of sources, which is a big advantage since most connectors are already available and do not need to be created. In transformation, it has a wide variety of transformations that can be used across big data and any kind of data, including JSON formatted data.

The connectors used most often include connecting to the Oracle database as the main focus and then connecting to log data, which have greatly benefited the team.

Improvements in the user interface to make it easier to use would be beneficial, and adding more security features would make Apache NiFi more secure and robust. Documentation and support could also be enhanced, as most support is usually received from users rather than from the product owners.

Apache NiFi has been used for more than five years.

Apache NiFi is stable.

The performance of Apache NiFi is really good. Based on the workload, more nodes can be added to make a bigger cluster, which enhances the cluster whenever needed. Apache NiFi's scalability is good and it is auto-scalable, which is pretty impressive.

Initially, customer support was contacted more often, but after understanding Apache NiFi better, not many issues have been faced. The customer support is really good, and they are helpful whenever concerns are posted, responding immediately.

Before Apache NiFi, StreamSets  was used, and it was unsatisfactory as it consumed all the server resources and did not release them. After finding Apache NiFi, it became a preferred solution.

Apache NiFi provides a good return on investment, although specific cost details are not known since that is not an area of involvement. Feedback has been given to stakeholders that Apache NiFi is beneficial in development and does great work, indicating a positive return on investment.

Before choosing Apache NiFi, other options such as StreamSets , Teradata , Informatica Big Data version, and Glue were evaluated, but Glue was not chosen due to its high cost.

To train or onboard new team members to use Apache NiFi, the code has been modularized and processes have been documented really well, so when new team members are onboarded, they are asked to review that documentation to understand the processes and the modules that have been created. In a couple of days, once they go through all that material, they are up to speed.

In terms of flexibility and ease of use, Apache NiFi is more open compared to other ETL tools that have been used, such as Informatica and Teradata . It is open source with many contributors and can handle various data sources, including log data, structured, unstructured, and semi-structured data, unlike traditional ETL tools.

Tools such as Prometheus and Grafana  are sometimes used to keep an eye on Apache NiFi server, and DataDog is also used along with it. Scaling Apache NiFi workloads is managed through auto-scaling.

To handle data security and compliance when using Apache NiFi, LDAP authentication is utilized, all clusters and nodes are kerberized, and single sign-on is used to authenticate. In transit, SSL encryption is used, and at rest, AES encryption is used, which is more than enough for the needs.

Apache NiFi is kept up to date by keeping an eye on new features that have been released, discussing them internally to assess if they need to be incorporated into development. If there are any gaps in the current version, an upgrade to the new version will be attempted.

Apache NiFi is a pretty good tool that meets most ETL needs, and in terms of performance and security, it is really good. After using it for quite some time without any issues, it is recommended as the number one tool for ETL. The overall review rating for Apache NiFi is 8 out of 10.

Positive

I have been using Apache NiFi  virtually daily, as it is part of my main responsibility in my current role.

My main use case for Apache NiFi  involves integrating various data sources and performing transformations to load them into mostly our NoSQL database, Elasticsearch, but sometimes into other databases as well.

For integrating and transforming data, we receive a lot of logs generated with our AWS  services that the company wants to collect, particularly for our security team to review those logs and ensure they can conduct their security checks and reviews to confirm there is no abnormal behavior. We use Apache NiFi to capture those logs sent to many S3  buckets, collect those logs, decompress them with Apache NiFi, perform any necessary transformations, and send them to Elasticsearch so that end users, often from the network team or security team, can then use Elasticsearch and Kibana for data analysis.

My advice for others considering Apache NiFi is that if you are willing to, you can use it on-premises; it offers great customizability. While it is specifically designed for streaming data, it can also accommodate batch data. Moreover, it is useful for various out-of-the-box solutions, including unique uses such as email notifications, showcasing flexibility in data orchestration, ETL, and other applications.

Apache NiFi offers great flexibility in terms of whether you want to be a low-code user or a high-code user, especially if you are a Python or Java developer, thanks to the recent addition of custom-built processors in the latest versions of Apache NiFi where you can use Python or Java to create your own processors versus using the great selection of out-of-the-box processors already available in Apache NiFi to do almost anything. If you are willing to put together a complex web of processors, you can do almost any data transformation you want, but the customizability with making your own processors, again with Python or Java, has been a huge benefit for performing both what Apache NiFi is specifically made to do and some more out-of-the-box solutions, such as creating some kind of email notification system as well. This kind of use with Apache NiFi has existed even before the implementation of custom processors. You could create scripts, even putting them in Python in Apache NiFi using the execute script methods, and this has existed before, but now it has even better functionality with the latest version of Python rather than just a Jython type of hybrid. Those are some of the best things that it offers.

The flexibility of Apache NiFi has helped me in my daily work, especially because instead of utilizing a bunch of Apache NiFi processors, which we do use for most of our processes, it can be much easier to combine transformation logic within Python processors since the majority of our team prefers Python programming as our choice of language. This integration allows us to put it all in one place. We can integrate Apache NiFi with our Python processors that we host on a Git  repository, which integrates very well, and we can manage the same scripts and make changes efficiently. It is great coming from a Python developer mindset shared amongst the team.

Apache NiFi has positively impacted my organization as it continually improves functionality and throughput with each iteration over the past three years. One of the big tradeoffs with open source is that how well it functions is largely dependent on the user, but that means you can adapt it to whatever custom use case you have. We have been able to consolidate several different authentication methods through just Microsoft, and Apache NiFi has been helpful in facilitating that. Additionally, due to its many ways of extracting data from different sources, we can develop specific solutions ourselves, allowing us to integrate various data sources. Thanks to the open-source customizability, we can adapt Apache NiFi to our built cluster, which has numerous benefits, particularly since we are managing many of our processes. This approach saves us significant costs compared to moving to something more managed or on the cloud, as managing open-source technologies ourselves ultimately reduces expenses.

Regarding cost savings, I do not have a strict idea of how much we have saved since the company was already using Apache NiFi when I joined, but I am certain comparisons have been made against other ETL or data orchestration tools that are popular among different cloud providers such as AWS  or Azure . The cost savings must be significant, particularly given that we are handling terabytes and petabytes of data daily, trying to find software that allows this in an affordable manner. It is clear that substantial savings exist, as long as we manage our own clusters and bugs effectively. The tradeoff with managed services is that they handle much of this, ensuring uninterrupted service, but these come at a cost. Conversely, with open-source software management, we incur no costs as we handle everything ourselves.

I believe Apache NiFi could be improved with easier, out-of-the-box provided monitoring solutions. While Apache NiFi has an API that generates logs, it would be beneficial to have simpler access to that data saved historically. It would assist in easily retrieving data for historical analysis and storing it elsewhere without the hassle of setting up APIs and delving into documentation. Just having a more streamlined approach to collecting this data would be greatly advantageous.

I would suggest continuous improvements regarding the custom developer-built processors, as many times the errors that arise are not useful. We often seem to struggle with a combination of implementing our own error handling or analyzing logs, as the information does not always align or proves unhelpful. Continuous enhancement in this area would be wonderful, so we do not need to decipher which error is more accurate or which report gets us nearer to the actual problem. For instance, I encountered a situation where flow files would not process; they were retried but returned to the queue before the Python processor due to ambiguous errors. It eventually turned out that the issue was the flow files' size being too large for the Python processor, which we only discovered by splitting the flow files, at which point the issue resolved. The initial error did not indicate it was related to memory or size limitations but appeared as a parsing error or something similar.

I have been working in my current field for about three years.

Apache NiFi is now more stable than before.

Apache NiFi's scalability is good. You can scale it up as long as you have the machines and servers available. If you have room for more instances, scaling up is fairly straightforward, provided you manage configurations effectively.

Apache NiFi's customer support is good.

I have definitely seen a return on investment through time savings. Working with Apache NiFi allows us to manage it more efficiently, transitioning from spending hours or days resolving issues to requiring much less intervention now. Thanks to improvements on both our side in how we run processes and enhancements to Apache NiFi, we have reduced the time commitment to almost not needing to interact with Apache NiFi except for minor queue-clearance tasks, allowing it to run smoothly. At this point, we have certainly saved hundreds of hours.

The customizability of Apache NiFi helps even with unique use cases, as I mentioned before, given that Apache NiFi can be used in this capacity. While there are better applications or software options available, when you are trying to keep it simple and finding ways to utilize a couple of processors for a unique solution, you can do that in Apache NiFi. For example, we have several notification-type pipelines we have built in Apache NiFi, such as reading from a SQL database to identify users who have not completed training and then sending them an email reminder to complete that training. We have that running regularly, week by week. Another instance involves a processing data flow that scans for specific data found in logs, which triggers an email notification to the relevant team letting them know that a unique identifier has appeared, allowing them to handle the situation.

I encountered some odd cases such as increasing concurrent threads on a processor, which should work similarly to copying several processors, yet functional throughput varies. It seems that using a distributed processor yields better throughput than just increasing the concurrent threads on one processor, which has been odd but is a workaround we had to adopt to boost throughput. Resolving such quirks could elevate the rating further.

I rate Apache NiFi an eight out of ten. I choose eight because, as open-source software, there is always room for improvement, but the tradeoff between learning how to use the software and the savings it provides, along with its customizability, ranks it pretty high. It is effective for what it does and continues to improve, so it could score higher if there are significant enhancements in custom-built processors and ongoing improvements in functionality.