Standardized data pipelines have streamlined ETL workflows but still need clearer logs and UI
What is our primary use case?
My main use case for Apache NiFi is to use it for basic ETL pipelines before ingesting data into the data lake. For example, if we want to ingest anything into Snowflake, before actually moving it to Snowflake, we do basic data massaging and transformations through Apache NiFi. This may include consuming a file, converting that file from one format to another, breaking it into chunks, and then pushing it to Snowflake.
A specific example is that we consume all caller data through Apache NiFi by connecting to Kafka topics and consuming those messages. We then convert those messages into JSON format and push those messages to Snowflake by running Snowflake procedures, which eventually ingest the data directly into Snowflake by reading it through an S3 bucket. We also push the actual JSON messages to S3 buckets through Apache NiFi itself.
What is most valuable?
The best features Apache NiFi offers include the integration capability because we have use cases wherein we have been using Apache NiFi to integrate through and consume from multiple sources. It can connect to any APIs, NAS drives, and databases, and it can consume streaming data. We can connect to Kafka topics and queues as well through Apache NiFi. First of all, there is flexibility in consuming from multiple sources. We can also easily transfer information from one block to another. We have command and control on the controller services that we can maintain separately so that we do not have to repeat connections and create multiple connections in different processor groups.
Apache NiFi has positively impacted our organization by significantly helping us streamline the processes. Earlier, each team was creating their own ETL pipelines with no standard being followed. Apache NiFi gave us the opportunity to streamline that process. Each team can request access to Apache NiFi and be onboarded separately based on their needs. We have specifically standardized the pipeline design so that no one can deviate from that. For example, everyone is supposed to use specific variables when enabling the alerting system. We have a different tool called Moogsoft that everyone can onboard to using the InvokeHTTP processor of Apache NiFi to send their alerts to a centralized system which can generate incidents. No one is now working in silos, and there is a specific pattern being followed by everyone. Furthermore, everyone must have Apache NiFi configured so that they can consume from a specific source but eventually push the data to our strategic cloud partner, AWS, first into an AWS bucket which is common for all, and from that bucket, subsequent processing will be done in Snowflake. The standard being followed now once Apache NiFi has been introduced has allowed others to copy-paste successfully created pipelines, saving a lot of time in our overall software development life cycle by reducing redundancy and efforts.
What needs improvement?
A couple of improvements for Apache NiFi would be better logging. Sometimes when looking at logs and events, they do not always make sense. A strong recommendation is that there has to be improved logging in Apache NiFi. Secondly, when looking at the file states, the history of processed files should be more readable so that not only the centralized teams managing Apache NiFi but also application folks who are new to the platform can read how a specific document is traversing through Apache NiFi.
For how long have I used the solution?
I have been using Apache NiFi for almost four and a half years now.
What do I think about the scalability of the solution?
Apache NiFi's scalability is great; we can easily scale into it without encountering any challenges.
How are customer service and support?
Customer support for Apache NiFi has been excellent, with minimal response times whenever we raise cases that cannot be directly addressed by logs. The support team has consistently provided great assistance with processor failures and helped us create ad-hoc processors as needed.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
We did not use any previous solution before Apache NiFi. Apache NiFi has significantly helped us improve our processes and streamline them, while we are also using SnapLogic for integration purposes in parallel.
I am not aware of any other solutions that were explored before onboarding Apache NiFi. Being from the analytics team, we were earlier using Tableau Prep for ETL and transformations and Informatica as well, but I am unsure if anything else was explored in the organization prior to using Apache NiFi.
What was our ROI?
Apache NiFi provides huge relief for all teams with similar use cases for ETL purposes, and it supports not just ETL but also ELT, allowing us to save significant time.
What's my experience with pricing, setup cost, and licensing?
I cannot comment on the pricing, setup cost, and licensing for Apache NiFi, but I can say there was significant time saved across the development life cycle due to reusable pipelines offered by Apache NiFi.
What other advice do I have?
The reason I rate Apache NiFi a seven is that most development folks are afraid to start using Apache NiFi because, to begin with, it does not always directly make sense. For example, there are other integration tools such as SnapLogic where you can simply search for a specific processor, but that is not the case with Apache NiFi. You need some basic understanding of which processors are there before you can fetch what you need. Better UI design should allow newcomers to search using relevant keywords, such as API, to retrieve appropriate processors, which currently is not happening, requiring some understanding of Apache NiFi. The other challenge I mentioned is having better logging, especially for processor-related logs to help newcomers navigate effectively.
I think Apache NiFi is a great tool that one should definitely explore. It is essential to perform basic checks regarding requirements, but if someone is looking for ETL and ELT functionalities needing to connect to CSVs, JSONs, Excels, and databases, they can onboard to Apache NiFi. It offers great connectivity for consuming or pushing data through queues and cloud workloads. Overall, it is an excellent product, especially for basic data massaging and processing before pushing data to Snowflake or creating reports. I rate Apache NiFi a seven out of ten.
Which deployment model are you using for this solution?
On-premises
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Data flows have transformed and now support reusable ETL pipelines across diverse sources
What is our primary use case?
Apache NiFi is used mainly for ETL to get data from multiple sources and then load it into a data lake. For example, data from Ab Initio is extracted and then loaded to an S3 bucket. From the S3 bucket, that data is read again and then loaded into other data layers. Different data layers, such as raw and raw silver, all use Apache NiFi.
How has it helped my organization?
Apache NiFi has positively impacted the organization by making development really easy, allowing efficient design and development, and enabling code reuse which has reduced the development effort. Integration with Git is also really good for sharing code across teams. A reduction in development effort of about 30% has been observed.
What is most valuable?
The best features Apache NiFi offers include the ability to connect to any type of sources, which is a big advantage since most connectors are already available and do not need to be created. In transformation, it has a wide variety of transformations that can be used across big data and any kind of data, including JSON formatted data.
The connectors used most often include connecting to the Oracle database as the main focus and then connecting to log data, which have greatly benefited the team.
What needs improvement?
Improvements in the user interface to make it easier to use would be beneficial, and adding more security features would make Apache NiFi more secure and robust. Documentation and support could also be enhanced, as most support is usually received from users rather than from the product owners.
For how long have I used the solution?
Apache NiFi has been used for more than five years.
What do I think about the stability of the solution?
What do I think about the scalability of the solution?
The performance of Apache NiFi is really good. Based on the workload, more nodes can be added to make a bigger cluster, which enhances the cluster whenever needed. Apache NiFi's scalability is good and it is auto-scalable, which is pretty impressive.
How are customer service and support?
Initially, customer support was contacted more often, but after understanding Apache NiFi better, not many issues have been faced. The customer support is really good, and they are helpful whenever concerns are posted, responding immediately.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
Before Apache NiFi, StreamSets was used, and it was unsatisfactory as it consumed all the server resources and did not release them. After finding Apache NiFi, it became a preferred solution.
What was our ROI?
Apache NiFi provides a good return on investment, although specific cost details are not known since that is not an area of involvement. Feedback has been given to stakeholders that Apache NiFi is beneficial in development and does great work, indicating a positive return on investment.
Which other solutions did I evaluate?
Before choosing Apache NiFi, other options such as StreamSets, Teradata, Informatica Big Data version, and Glue were evaluated, but Glue was not chosen due to its high cost.
What other advice do I have?
To train or onboard new team members to use Apache NiFi, the code has been modularized and processes have been documented really well, so when new team members are onboarded, they are asked to review that documentation to understand the processes and the modules that have been created. In a couple of days, once they go through all that material, they are up to speed.
In terms of flexibility and ease of use, Apache NiFi is more open compared to other ETL tools that have been used, such as Informatica and Teradata. It is open source with many contributors and can handle various data sources, including log data, structured, unstructured, and semi-structured data, unlike traditional ETL tools.
Tools such as Prometheus and Grafana are sometimes used to keep an eye on Apache NiFi server, and DataDog is also used along with it. Scaling Apache NiFi workloads is managed through auto-scaling.
To handle data security and compliance when using Apache NiFi, LDAP authentication is utilized, all clusters and nodes are kerberized, and single sign-on is used to authenticate. In transit, SSL encryption is used, and at rest, AES encryption is used, which is more than enough for the needs.
Apache NiFi is kept up to date by keeping an eye on new features that have been released, discussing them internally to assess if they need to be incorporated into development. If there are any gaps in the current version, an upgrade to the new version will be attempted.
Apache NiFi is a pretty good tool that meets most ETL needs, and in terms of performance and security, it is really good. After using it for quite some time without any issues, it is recommended as the number one tool for ETL. The overall review rating for Apache NiFi is 8 out of 10.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Daily workflows have integrated diverse logs and have delivered flexible data orchestration
What is our primary use case?
I have been using Apache NiFi virtually daily, as it is part of my main responsibility in my current role.
My main use case for Apache NiFi involves integrating various data sources and performing transformations to load them into mostly our NoSQL database, Elasticsearch, but sometimes into other databases as well.
For integrating and transforming data, we receive a lot of logs generated with our AWS services that the company wants to collect, particularly for our security team to review those logs and ensure they can conduct their security checks and reviews to confirm there is no abnormal behavior. We use Apache NiFi to capture those logs sent to many S3 buckets, collect those logs, decompress them with Apache NiFi, perform any necessary transformations, and send them to Elasticsearch so that end users, often from the network team or security team, can then use Elasticsearch and Kibana for data analysis.
My advice for others considering Apache NiFi is that if you are willing to, you can use it on-premises; it offers great customizability. While it is specifically designed for streaming data, it can also accommodate batch data. Moreover, it is useful for various out-of-the-box solutions, including unique uses such as email notifications, showcasing flexibility in data orchestration, ETL, and other applications.
What is most valuable?
Apache NiFi offers great flexibility in terms of whether you want to be a low-code user or a high-code user, especially if you are a Python or Java developer, thanks to the recent addition of custom-built processors in the latest versions of Apache NiFi where you can use Python or Java to create your own processors versus using the great selection of out-of-the-box processors already available in Apache NiFi to do almost anything. If you are willing to put together a complex web of processors, you can do almost any data transformation you want, but the customizability with making your own processors, again with Python or Java, has been a huge benefit for performing both what Apache NiFi is specifically made to do and some more out-of-the-box solutions, such as creating some kind of email notification system as well. This kind of use with Apache NiFi has existed even before the implementation of custom processors. You could create scripts, even putting them in Python in Apache NiFi using the execute script methods, and this has existed before, but now it has even better functionality with the latest version of Python rather than just a Jython type of hybrid. Those are some of the best things that it offers.
The flexibility of Apache NiFi has helped me in my daily work, especially because instead of utilizing a bunch of Apache NiFi processors, which we do use for most of our processes, it can be much easier to combine transformation logic within Python processors since the majority of our team prefers Python programming as our choice of language. This integration allows us to put it all in one place. We can integrate Apache NiFi with our Python processors that we host on a Git repository, which integrates very well, and we can manage the same scripts and make changes efficiently. It is great coming from a Python developer mindset shared amongst the team.
Apache NiFi has positively impacted my organization as it continually improves functionality and throughput with each iteration over the past three years. One of the big tradeoffs with open source is that how well it functions is largely dependent on the user, but that means you can adapt it to whatever custom use case you have. We have been able to consolidate several different authentication methods through just Microsoft, and Apache NiFi has been helpful in facilitating that. Additionally, due to its many ways of extracting data from different sources, we can develop specific solutions ourselves, allowing us to integrate various data sources. Thanks to the open-source customizability, we can adapt Apache NiFi to our built cluster, which has numerous benefits, particularly since we are managing many of our processes. This approach saves us significant costs compared to moving to something more managed or on the cloud, as managing open-source technologies ourselves ultimately reduces expenses.
Regarding cost savings, I do not have a strict idea of how much we have saved since the company was already using Apache NiFi when I joined, but I am certain comparisons have been made against other ETL or data orchestration tools that are popular among different cloud providers such as AWS or Azure. The cost savings must be significant, particularly given that we are handling terabytes and petabytes of data daily, trying to find software that allows this in an affordable manner. It is clear that substantial savings exist, as long as we manage our own clusters and bugs effectively. The tradeoff with managed services is that they handle much of this, ensuring uninterrupted service, but these come at a cost. Conversely, with open-source software management, we incur no costs as we handle everything ourselves.
What needs improvement?
I believe Apache NiFi could be improved with easier, out-of-the-box provided monitoring solutions. While Apache NiFi has an API that generates logs, it would be beneficial to have simpler access to that data saved historically. It would assist in easily retrieving data for historical analysis and storing it elsewhere without the hassle of setting up APIs and delving into documentation. Just having a more streamlined approach to collecting this data would be greatly advantageous.
I would suggest continuous improvements regarding the custom developer-built processors, as many times the errors that arise are not useful. We often seem to struggle with a combination of implementing our own error handling or analyzing logs, as the information does not always align or proves unhelpful. Continuous enhancement in this area would be wonderful, so we do not need to decipher which error is more accurate or which report gets us nearer to the actual problem. For instance, I encountered a situation where flow files would not process; they were retried but returned to the queue before the Python processor due to ambiguous errors. It eventually turned out that the issue was the flow files' size being too large for the Python processor, which we only discovered by splitting the flow files, at which point the issue resolved. The initial error did not indicate it was related to memory or size limitations but appeared as a parsing error or something similar.
For how long have I used the solution?
I have been working in my current field for about three years.
What do I think about the stability of the solution?
Apache NiFi is now more stable than before.
What do I think about the scalability of the solution?
Apache NiFi's scalability is good. You can scale it up as long as you have the machines and servers available. If you have room for more instances, scaling up is fairly straightforward, provided you manage configurations effectively.
How are customer service and support?
Apache NiFi's customer support is good.
How would you rate customer service and support?
What was our ROI?
I have definitely seen a return on investment through time savings. Working with Apache NiFi allows us to manage it more efficiently, transitioning from spending hours or days resolving issues to requiring much less intervention now. Thanks to improvements on both our side in how we run processes and enhancements to Apache NiFi, we have reduced the time commitment to almost not needing to interact with Apache NiFi except for minor queue-clearance tasks, allowing it to run smoothly. At this point, we have certainly saved hundreds of hours.
What other advice do I have?
The customizability of Apache NiFi helps even with unique use cases, as I mentioned before, given that Apache NiFi can be used in this capacity. While there are better applications or software options available, when you are trying to keep it simple and finding ways to utilize a couple of processors for a unique solution, you can do that in Apache NiFi. For example, we have several notification-type pipelines we have built in Apache NiFi, such as reading from a SQL database to identify users who have not completed training and then sending them an email reminder to complete that training. We have that running regularly, week by week. Another instance involves a processing data flow that scans for specific data found in logs, which triggers an email notification to the relevant team letting them know that a unique identifier has appeared, allowing them to handle the situation.
I encountered some odd cases such as increasing concurrent threads on a processor, which should work similarly to copying several processors, yet functional throughput varies. It seems that using a distributed processor yields better throughput than just increasing the concurrent threads on one processor, which has been odd but is a workaround we had to adopt to boost throughput. Resolving such quirks could elevate the rating further.
I rate Apache NiFi an eight out of ten. I choose eight because, as open-source software, there is always room for improvement, but the tradeoff between learning how to use the software and the savings it provides, along with its customizability, ranks it pretty high. It is effective for what it does and continues to improve, so it could score higher if there are significant enhancements in custom-built processors and ongoing improvements in functionality.
Which deployment model are you using for this solution?
On-premises
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Plug-and-play data flows have transformed how our team builds and runs large daily ingestions
What is our primary use case?
I have been using Apache NiFi for a couple of years now. My experience with Apache NiFi is that I have used it for a customer of ours who used that system to orchestrate ingestion processes from different sources into AWS. My main use case for Apache NiFi is data ingestion, so connecting to sources like APIs or different data lakes to get the data into S3. I manage more than 50 ingestion flows currently; some of them are quite small, dealing with a few megabytes a day, while others are multiple gigabytes a day and those are run daily, so we have a daily refresh of the sources.
What is most valuable?
Apache NiFi's best features include its plug-and-play solution, which means you don't need a lot of insight or knowledge to use it. The simplicity and plug-and-play approach of Apache NiFi has helped our team by allowing our customer to scale and build different ingestion flows, which previously needed additional development effort because custom solutions were required; now we just plug and play.
The features of Apache NiFi, including the integration capabilities to connect to different sources and monitoring with all the observability features, work really well for our team in obtaining the needed information. Apache NiFi has positively impacted our organization by making us a lot more productive; I would say development has significantly improved.
Development has improved with a reduction in time spent being the main benefit; before we needed a matter of days to create the ingestion flows, but now it only takes a couple of hours to configure.
What needs improvement?
I don't have any frustrations about how Apache NiFi can be improved; I'm pretty happy. I don't think there are needed improvements for Apache NiFi, particularly regarding the user interface, documentation, or performance.
For how long have I used the solution?
I have been working in my current field for five years now.
What do I think about the stability of the solution?
Apache NiFi is indeed stable.
What do I think about the scalability of the solution?
Apache NiFi's scalability works for us.
How are customer service and support?
I haven't interacted with their customer support.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
I have evaluated other options before choosing Apache NiFi, and I have used other options as well.
How was the initial setup?
Apache NiFi is deployed in our organization using public cloud infrastructure. I purchased Apache NiFi through the AWS Marketplace. There are no improvements needed for Apache NiFi deployment on AWS, as the experience with that was pretty smooth.
What about the implementation team?
I don't manage pricing, setup cost, or licensing for that customer, but I think they are pretty satisfied with it as well.
What other advice do I have?
My advice for others looking into using Apache NiFi is that it's a good solution. I would rate this review 9 out of 10.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Low-code orchestration has bridged on-prem and cloud workflows and now streamlines data sharing
What is our primary use case?
My main use case for
Apache NiFi is orchestration; I kickstart the job and then pass on the handler to
Databricks or
AWS to run the ETL pipeline.
A specific example of a workflow where Apache NiFi plays a key role is when there is an on-premises Hadoop system and a cloud component. Because of the company's policy regarding firewalls, the data cannot be directly moved through services such as Kinesis or DMS integrating directly with on-premises resources. Apache NiFi works as an orchestrator and a middle tool to get the parameters to trigger the job and then pass on the handler to the cloud services. Because of the firewall, Apache NiFi comes into the picture. Another use case for Apache NiFi is once the data is created in S3; I can extract a subset of the data and send it as an SFTP for outside recipients.
I have another scenario regarding my main use case with Apache NiFi; there is a use case for synthetic data, and we are using synthetic data generative AI software to synthesize data in the cloud environment. Now for users who are not on the cloud and want to access the synthetic data, Apache NiFi is used to pull the data back from the cloud.
What is most valuable?
The best features Apache NiFi offers include drag and drop capabilities; I would not say it is no-code, but it is pretty low-code.
The drag-and-drop interface of Apache NiFi has helped my team by cutting down on development time, allowing us to focus more on the mapping part and testing part of it.
In terms of features, if a job requires a predetermined mapping for the metadata, Apache NiFi comes in handy to map for the metadata, which is needed before an external file can be validated and pushed onto the cloud for processing. This is better than traditional systems where we would have to sit down and map individual attributes and their data types to create that metadata interface.
Apache NiFi has positively impacted my organization by definitely bridging the gap between the on-premises and cloud interaction until we find a solution to open the firewall for cloud components to directly interact with on-premises services.
What needs improvement?
Regarding improvements in Apache NiFi, there is scope; it should gear more towards an AI mechanism, especially for metadata generation. If I want to integrate Python code or embed Python code within the Apache NiFi parameters or workflows, it should come with AI integration that can assist in generating code based on user requirements. Gearing more towards a no-code mechanism will really enhance Apache NiFi's productivity, as customers would want that.
About needed improvements, I think integration with other tools would really help in the age of AI. Apache NiFi should have APIs or connectors that can connect seamlessly to other external entities, whether in the cloud or on-premises, creating a plug-and-play mechanism.
For how long have I used the solution?
I have been using Apache NiFi for roughly one and a half years.
What do I think about the stability of the solution?
The biggest challenges I have faced while using Apache NiFi revolve around reliability; I have seen Apache NiFi crashing at times, which is one of the issues we have faced in production. It does not happen often, but there have been instances where the NiFi nodes crashed, and sometimes we encountered issues retrieving data where it was unable to establish connectivity.
To troubleshoot or resolve the Apache NiFi crashes and data retrieval issues, my team primarily replicates the same scenario in development to see where the issue lies. On one occasion, the failure was linked to an authentication mechanism change at the enterprise level.
In my experience, Apache NiFi is stable.
What do I think about the scalability of the solution?
Apache NiFi's scalability has not been an issue yet; I would say it is pretty stable. Depending on the workload we process, it remains stable since at the end of the day, it is just used as an orchestration tool that triggers the job while the heavy lifting is done on Spark servers.
How are customer service and support?
I have not needed to reach out for help regarding customer support for Apache NiFi.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
I previously used
Airflow before switching to Apache NiFi.
I switched from Airflow to Apache NiFi because Airflow is still not widely used within the organization; my experience with Airflow has been with other companies, not with Mastercard.
What about the implementation team?
We have a dedicated team that handles version upgrades and maintenance for Apache NiFi.
What was our ROI?
I would say there has been time saved to an extent, but not in terms of resources when evaluating return on investment.
Which other solutions did I evaluate?
I was not involved in evaluating other options before choosing Apache NiFi.
What other advice do I have?
On a scale of one to ten, I would rate Apache NiFi an eight. I chose eight out of ten because there is scope for improvement; in this age, folks want more automation, which is why I kept it at eight. My advice for others considering using Apache NiFi is that it is a good tool, but there are also other options to consider before making a decision. I gave this review an overall rating of eight out of ten.
Data workflows have accelerated project delivery and reduce costs for analytics teams
What is our primary use case?
Apache NiFi is used for real-time and batch ingestion on data warehouse platforms. For example, Apache NiFi ingests all analytics from the e-commerce website into the data warehouse in the AWS Redshift database.
How has it helped my organization?
Speeding up projects with Apache NiFi has helped the organization by resulting in cost savings. A 30% reduction in cost was noticed as a specific metric regarding those savings.
What is most valuable?
The best feature of Apache NiFi is the simplicity of the tools because it is a drag-and-drop tool. The simplicity of Apache NiFi's tools helps by speeding up all the implementation process. Apache NiFi is also used to speed up projects in order to gain more projects in less time.
What needs improvement?
Apache NiFi is a good product as it is currently.
For how long have I used the solution?
Apache NiFi has been used for a long time, five years in different projects.
What do I think about the stability of the solution?
What do I think about the scalability of the solution?
The scalability of Apache NiFi is good because it is simple to scale up the resources.
How are customer service and support?
The customer support for Apache NiFi is fine. I would rate the customer support of Apache NiFi a 10 on a scale of 1 to 10.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
A custom solution implemented with Python was previously used before switching to Apache NiFi. The decision was made to switch from the custom Python solution to Apache NiFi to simplify all the deployment.
How was the initial setup?
Apache NiFi was purchased through the AWS Marketplace.
What was our ROI?
A return on investment has not been observed, and it is not possible to share these metrics.
What's my experience with pricing, setup cost, and licensing?
The experience with pricing, setup cost, and licensing was fine, as the integration with the AWS Marketplace was very good. The pricing in Italy is considered a little bit high, but the product is worth it.
Which other solutions did I evaluate?
Other options were not evaluated before choosing Apache NiFi.
What other advice do I have?
Apache NiFi receives a rating of 9 out of 10. This rating of 9 out of 10 for Apache NiFi was chosen because of the documentation and the support of the product. The advice for others looking into using Apache NiFi is to test the solution with a POC and then go to production in a quick way.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Visual workflow offers clarity and boosts data pipeline construction
What is our primary use case?
I am implementing the ETL workflow using
Apache NiFi to prepare data and upload it to the cloud. Our use case involves importing data from on-premise and private servers to build a data hub and data mart. The data mart is then published on the cloud.
How has it helped my organization?
We primarily use
Apache NiFi for data preparation tasks.
What is most valuable?
The visual workflow aspect of Apache NiFi is an invaluable feature as it operates on a no-code platform that allows for easy drag-and-drop pipeline construction. Compared to
Airflow, which requires programming before visual representation, Apache NiFi offers clarity in pipeline activities. This feature greatly aids in understanding what the pipeline is doing.
What needs improvement?
The logging system of Apache NiFi needs improvement. It is difficult to debug compared to
Airflow, where task details and issues are clear. With Apache NiFi, I have encountered processes that die without any traceable error, which might relate to the inadequate logging system.
For how long have I used the solution?
I have been working with Apache NiFi for about six months.
What do I think about the stability of the solution?
Sometimes, when I run Apache NiFi, processes crash without any clue, which might relate to the logging system. The process can die, and the logs do not show any detail to identify the problem, impacting stability.
What do I think about the scalability of the solution?
For scalability, I would rate it an eight. We can run parallel pipelines simultaneously without issues unless memory is full. Scarcity of memory is the only constraint, but processing capabilities allow us to handle much simultaneously.
How are customer service and support?
The technical support from the official Apache team is rated a three out of ten. Issues often require self-resolution or community help, as the support isn't effectively managed.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
I have used Airflow before, which required programming first and then visual representation of the workflow.
What about the implementation team?
There is another team responsible for setting up Apache NiFi, so I'm not involved in the deployment process.
What's my experience with pricing, setup cost, and licensing?
Apache NiFi is open-source and free. Its integration with systems like Cloudera can be expensive, but Apache NiFi itself presents the best pricing as a standalone tool.
Which other solutions did I evaluate?
Prior to Apache NiFi, I used Airflow, which differed mainly in its approach to programming and workflow visualization.
What other advice do I have?
Overall, I rate Apache NiFi an eight out of ten. I am quite happy with it.
