Mostly, we use it for the data warehousing side of use cases, where you have, like, a huge amount of data, and you are required to do reporting in terms of data science, data warehousing, or ad hoc reporting. The use cases we have used are, for example, data coming from MedTech devices, mostly sensor data, which we need to load in Snowflake and do data analytics. We have been using the tool for a couple of MedTech clients.
 

The most important part of the tool is that computing and storage are totally separated, and it keeps on evolving every two weeks, with the tool having releases. New features are coming up in the tool. With respect to AI, the tool is also progressing well. The scalability and performance are quite good. If you have data, like in CSV or any other format, you can load it very quickly and then do your analysis. Columnar database performance, scalability, and the addition of new features are a few useful features of the tool.

I think people do not want to create pipelines for many customers now. Normally, we have this layer architecture, like layer one, layer two, layer three, or layer four, where we have raw data, integrations, business data, and then semantic data, so we have to create various pipelines. People don't have to create or maintain pipelines since, in the future, if there are any changes in the source data, it should be very easy to configure and create the pipeline rather than the developer doing that for them. Though it may not be possible to make improvements based on the expectations of the people, considering the AI market, code generation can be simplified a little bit by using streams. People want to be able to develop the pipeline without involving many developers by doing some configurations and creating the pipeline. The customer expectation is that they don't want to create tables for each report, but what happens currently is that if you don't create that, then you have to run the query every time. Suppose I have created raw data, and I want to do some aggregation. In that case, if I don't create a materialized view or a table, I have to run those aggregate queries again and again, which will cost me the cost attached to Snowflake usage. From an improvement perspective, Snowflake can evolve in terms of writing costly, expensive queries with less cost and try to see if pipeline development can be made a little easier.

I have been using Snowflake for a year and a half.

There were use cases where there were only 10 to 15 users. There was one requirement where the customer asked for 3,000 concurrent users to try to get a real-time report from the tool, but then our company suggested that Snowflake was not the right choice for them because it is more kind of a data warehouse, and they were looking more into transactional reporting. For Snowflake-based projects where we have worked, it is more concerning a smaller number of users, like around 20 users. However, if a huge number of users are required, Snowflake is not the right choice.

My company has partnered with Snowflake. Normally, we reach out to the account manager or regional manager, and sometimes we get support. Most of the time, we ask for support from the architecture and solutions part of it to review it or for some workarounds. Right now, we have not gone for low-level technical support from Snowflake. Whatever we have worked on, we are able to manage.

I have been working all my life on databases, so I have almost twenty five years of experience in databases starting from SQL, Oracle8i, Oracle 9i to MySQL, SQL Server and Redshift. I have also used Solr and Elasticsearch, which are not databases but all data-related things I have worked on, including PostgreSQL.

The main thing about Snowflake is that it is totally outside the customer's cloud. If I am an AWS customer, even if Snowflake is hosting on AWS, it is on a separate account right now. If somebody has some critical data that cannot be shared outside the cloud, then such customers or people are a little hesitant to use Snowflake. Recently, there were some breaches or password issues, so security concerns like that are there. There is also the costing part attached to the tool. Now, people are looking into tools that are available at a lower cost and offer more user-friendliness. The tool is a good data cloud product, but it is a little bit outside the customer's environment, which makes it difficult to convince the customer to use it.

Speaking about the product's initial setup phase, I would say that the product is used just from the cloud. We have not installed it in any environment. I work with the tool's SaaS version.

The tool does add some value to the company. When it comes to pipeline development work, though customers expect it to be faster, I think if you have simple files, you can load them in a day and analyze the data. Productivity-wise, it is definitely much better compared to Redshift. Redshift Spectrum is catching up with Snowflake, but I have not explored it. To be very frank, I am not very familiar with Azure Data Warehouse, so I am not sure how it is different from Snowflake, but from what I have seen, it has been good in terms of productivity.

The pricing part is based on the computing and storage. The costs are different and then there are services costs as well. I have heard that Snowflake is costlier than Redshift or GCP BigQuery. A small customer may not go for Snowflake.

Speaking of how Snowflake enhances our company's AI-driven projects or analytics, I would say that the tool has features like Document AI and Snowflake Cortex. AI can be used if the tool is for very basic use cases, like anomaly detection or prediction. With simple use cases, you don't have to set up a big infrastructure. You just load data and use the tool's services. I have not used the tool for complex AI projects. I am not an AI person. Rather, I can be described as a data engineer or data architect. In our use cases, we have explored the AI feature of Snowflake more from document processing and doing a simple exploration of the feature. For customers, I have not used Snowflake's AI feature.

Speaking about how Snowflake's scalability feature impacted our data processing and analytics tasks, I would say that the tool has a virtual warehouse, so it really helps. You can scale based on your needs. You can change the warehouse sizing, which will help with the scalability. You can just increase the warehouse size, and it gets your work done.

There are various ways to integrate the tool. I think the tool has connectors also, but the external table is one way to load your data in Snowflake and start analyzing it quickly. Now, the tool also works with Apache Iceberg format, though I have not explored that. With respect to Snowpipe, getting data from CSV to Snowpipe are things we use, and they are all quite easy to use. In terms of native connectors to various data sources, though I have not explored them, I see the tool has support for various connectors. I believe that will be good. For most of the use cases, data is loaded onto S3, and then we use Snowpipe along with external tables and Snowpark ML to process the data.

Snowflake has something called Snowflake Horizon, which has bundled various features of data security, data governance, and compliance together, and they have come up with the package. The tool has very good data security in terms of masking data. You can have different roles and assign policies in terms of who you want to be able to see data of a particular department, so you can assign based on department ID that only certain people can see the data. I found good features in my various other cloud databases, and compared to them, Snowflake data security and data governance are quite capable.

I don't think it is difficult to maintain. As the organization grows, maintaining policies, user roles, and data masking policies might become a little tricky in Snowflake. In AWS, we have a well-architectured framework where you have a defined framework or pattern, and you try to reuse it and modify it as needed. I don't see such kind of information or patterns largely available in Snowflake. I think as an architect, if we have a well-architectured framework for Snowflake, it will be useful. In terms of maintenance, I think the performance and all is okay in the tool. Data governance and policy management are a little bit tedious for the tool.

I recommend the tool to others. People should only be okay with the product's cost.

I rate the tool an eight out of ten.

The platform's most valuable features include its ability to effectively summarize and manage large datasets, allowing multiple teams to analyze and generate insights. Its integration with data lakes for business impact analysis, performance metrics, and KPIs is particularly important.

Improvement is needed in integrating external tools, such as data catalogs, which can be complicated due to differing formats and usage across departments. The goal is to enhance collaboration and streamline workflows.

The product's scalability is crucial for managing petabyte-scale data generated daily across various regions, allowing for efficient data validation and handling.

The primary challenges during the initial setup were the high pricing and uncertainties regarding future costs associated with data usage. 

The deployment involved consultation among managers, agreement on on-site requirements, scale calculations, and collaboration with engineers for setup approval.

I rate the process a seven out of ten. 

Snowflake is integrated through a complex workflow that involves collecting data on the publisher side, using tools like Airflow and Kafka for batch jobs, and frequently importing data into the product from various sources, including S3 and Data Lakes. It creates a smooth data pipeline.

I rate it a seven out of ten.