Now, Snowflake is providing built-in declarative data pipelines through its dynamic tables. What\u2019s a declarative data pipeline? It\u2019s a type of pipeline that reduces development time and increases performance by letting you focus on a desired outcome rather than the underlying mechanics.<\/p>\n
Snowflake has also announced it will combine transactional and analytical support through its hybrid tables and offer open-format interoperability through its Iceberg Table<\/a> support. Iceberg Tables will let you store large datasets used outside of your daily workloads, offering a cost-effective data retention option and the ability to securely share with non-Snowflake users.<\/p>\n These are only some of the enhanced features Snowflake has teased in the last year. We\u2019re here to share the latest and greatest in Snowflake offerings and dig into the details all data engineers should know.<\/strong> Use this explainer to bring team members up to speed on Snowflake\u2019s new iteration \u2014 and follow us for future updates.<\/p>\n Snowflake has expanded the platform so it can handle both day-to-day tasks and big data analysis using the same system. They also improved how the system looks and feels for users. Moreover, with the introduction of Dynamic, Hybrid and Iceberg Tables, Snowflake provides tools and tactics to simplify and streamline daily data management<\/a>.<\/p>\n The Iceberg and Hybrid Tables are currently in the private preview stage, meaning customers need to specifically request access from Snowflake. The Snowflake Summit held on June 26, 2023, in Las Vegas, announced the preview for dynamic tables. Here\u2019s a run-down of all the remarkable things these tables can do.<\/strong><\/p>\n In Snowflake, a Dynamic Table is a table that automatically updates itself \u2014 including your logic \u2014 with the latest data as new data or updates flow into its source tables. It\u2019s similar to a View but supports more complex combinations of underlying data than a view, with lower code complexity than a stream. This offering fixes a major pain point \u2014 Snowflake\u2019s old mechanisms for merging and updating incremental data changes.<\/p>\n Today, most businesses have a real need for incremental data updates that use flexible pipelines. However, a common challenge for data engineers is knowing how to process incremental data without compromising the load times and credits of the underlying warehouse.<\/p>\n Yes, the old version of Snowflake provided streams and tasks to handle delta changes. However, those capabilities required extensive time and SQL knowledge to:<\/strong><\/p>\n On top of that, task scheduling sometimes caused process lags due to the refresh interval not being well-defined.<\/p>\n Data engineers would often work around these issues and process delta data by performing a full data refresh or an incremental refresh. A full refresh would truncate or drop and recreate the tables whenever you needed to consume the complete data from the source. An incremental refresh would save costs and eliminate the need to deal with high-volume data.<\/p>\n However, neither approach could solve for latency, maintenance issues or the costs of warehousing data. Full data refreshes also tended to hinder the scalability, time efficiency, and cost of warehousing while consuming a lot of data.<\/strong><\/p>\n Users needed a smart, tech-driven process to identify delta changes without manual intervention. Thankfully, Snowflake\u2019s developers delivered. Dynamic tables meet this need by providing flexibility and scalability to identify incremental data and process data streams, and you won\u2019t have to write a single line of code.<\/p>\n Dynamic table features include:<\/p>\n The bottom line: In the past, to process delta changes, data engineers had to use streams to capture change data and tasks to automate the job. With the introduction of dynamic tables, all incremental changes are handled in an automated way, eliminating the need for streams and tasks altogether.<\/p>\n Syntax:<\/strong> Where:<\/strong><\/p>\n Snowflake shares a simpler way to transform data without having to manage streams and tasks. https:\/\/www.snowflake.com\/blog\/dynamic-tables-delivering-declarative-streaming-data-pipelines\/<\/p><\/div>\n A hybrid table is a new Snowflake structure that stores the same data for both transactional and analytical purposes at the same time, seamlessly transitioning between modes depending on the operation. Customers can now simplify their data platform, reducing pipelines and eliminating synchronization issues.<\/strong><\/p>\n Consider, for example, an insurance company that uses both an insurance management system and an Oracle Siebel CRM application (in the claims module) to manage their data. The insurance an Online Analytical Processing (OLAP) system in Snowflake is designed to process high volumes of data with no latency. Data analysts at the insurance company use the OLAP System to derive analytical insights and make business decisions.<\/p>\n The Siebel insurance module within the system facilitates this process by ingesting great amounts of transactional data in real time. It takes in claims, auto policy and life insurance policy data, to name a few. The module ingests these data through quota management via issuance and maintenance.<\/strong><\/p>\n To extend their business forecasting ability, the insurance company decides to define a model that will pull their data from the Siebel i.e., Online Transaction Processing (OLTP) system. The model will also need to collaborate with the OLAP insurance system. Altogether, the model will help to provide powerful insights about the company\u2019s projected growth.<\/p>\n But the company has a problem: the data redundancy that comes with moving transactional and analytical data, stored separately, across their systems. Moreover, customers want a single place or system to store their entire data \u2014 one they can easily access.<\/p>\n Hybrid tables (aka Unistore) solve this problem. They provide a new workload that brings both transactional and analytical data together in a single platform. Hybrid tables are exactly what the insurance company in our example needs to streamline its current infrastructure and derive effective business insights. This updated architecture provides seamless data integration and simplified data joins.<\/p>\n Hybrid table features include:<\/strong><\/p>\n In addition, with hybrid tables, Snowflake enforces the uniqueness of the primary index and referential integrity constraints for relationships between tables using primary keys and foreign keys. You can now join both hybrid tables and normal tables without changing any query structure.<\/p>\n https:\/\/www.snowflake.com\/thankyou\/getting-started-with-transactional-and-analytical-data-in-snowflake\/<\/p><\/div>\n With this architecture, the storage layer brings both row and columnar storage under one umbrella. Since new features support row locking, you can perform updates more efficiently.<\/strong> In the background, this data synchronizes to columnar storage, which is specific to historical tables.<\/p>\n The biggest advantage is that it hides complexity from users. And, Snowflake lets you issue the query without worrying about how the data will be kept in the storage layer.<\/p>\n Syntax:<\/strong> To create a hybrid table, use HYBRID in CREATE syntax and define the primary or foreign key constraints like other relational tables. Snowflake enforces these constraints in the case of hybrid tables.<\/p>\n Let\u2019s face it: You\u2019ll need to handle large volumes of data if you want to offer effective internal analytics, gain insights into business growth potential, follow a proactive approach to customer retention, and stay competitive in general. At times, you may need to share these data with partners and research organizations. However, storing large volumes of sensitive and confidential information over the cloud \u2014 and potentially sharing it \u2014 can pose a real security challenge<\/a>.<\/strong><\/p>\n With Iceberg Tables, organizations can store big data on a cloud of their choice, like Amazon S3 or Azure Blob Storage, in the form of Parquet files. Since Iceberg supports dynamic data masking and row-level masking (all Snowflake native table features), it provides controlled access to and privacy for sensitive information.<\/p>\n Data is stored in customer-supplied storage<\/a>, meaning multiple applications or tools outside Snowflake itself can use it. In fact, organizations no longer pay Snowflake for storage costs. Instead, the cloud providers bill for storage.<\/p>\n Iceberg Tables mostly work like Snowflake native tables, but there are a couple of key exceptions:<\/p>\n Syntax:<\/strong> Want to know more? Take a look at Snowflake\u2019s documentation of these new features:<\/p>\n To conclude, each table has a unique role in business, streamlining data pipelines and automating changes with Dynamic tables. Snowflake is introducing Hybrid or Unistore tables for improved management of both transactional and analytical data. If you want to use Snowflake features while keeping data outside Snowflake, consider Iceberg Tables for tasks like Masking, Access Policies, Cloning, and more.<\/p>\n\n The Latest and Greatest Snowflake Updates<\/h2>\n
Save Time with Dynamic Tables<\/h3>\n
\n
\n
\nCREATE [ OR REPLACE] DYNAMIC TABLE
\nLAG = ' {seconds | minutes | hours | days}'
\nWAREHOUSE =
\nAS SELECT
\n<\/code><\/p>\n\n
![\"Centric](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2023\/09\/Snowflake-tables-chart.jpg\")
<\/a>Reduce Redundancy with Hybrid Tables<\/h3>\n
\n
![\"Centric](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2023\/09\/Snowflake-workflows.jpg\")
<\/a>
\nCREATE HYBRID TABLE Customers
\nCust_no number (25) PRIMARY KEY,
\nCust_name varchar (100),
\nOutstanding_Amt number (38,0),
\nCust_Phone number(20)<\/code><\/p>\nCreate the Invoice table with foreign key referencing the customer table<\/code><\/p>\nCREATE OR REPLACE HYBRID TABLE Invoices<\/code><\/p>\nInvoice_No number (25) PRIMARY KEY,
\nCust_no number (25),
\nInvoice_Amount number (38,0),
\nInvoice_Status varchar (20),
\nInvoice_Date timestamp_ntz,<\/code><\/p>\nCONSTRAINT fk_custkey FOREIGN KEY (Cust_no) REFERENCES Customers (Cust_no),<\/code><\/p>\nINDEX ind_date (Invoice_Date));<\/code><\/p>\nBoost Volume with Iceberg Tables<\/h3>\n
\n
Know the Difference: Iceberg Tables vs. Native Tables<\/h2>\n
![\"Centric](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2023\/09\/Blog_DA_Snowflake_Table_09062023.png\")
<\/a><\/p>\n
\nCreate external volume pointing to the S3 bucket<\/code>
\nCreate or replace external volume iceberg_int<\/code><\/p>\nstorage_locations =
\n(<\/code><\/p>\n(<\/code><\/p>\nname = 'my-s3-us-east-2'<\/code><\/p>\nstorage_provider = 'S3'<\/code><\/p>\nstorage_base_url = 's3:\/\/icebergpracticebucket1\/'<\/code><\/p>\nstorage_aws_role_arn = 'arn:aws:iam::913267004595: role\/icebergrole'<\/code><\/p>\n)<\/code>
\n);<\/code><\/p>\nCreate an Iceberg Table using ICEBERG in create statement and point to external volume<\/code><\/p>\nCreate or replace Iceberg Table customer_detail with external_volume='iceberg_int'; as Select <<col1>>, <<col2>><<coln>> from Table<\/code><\/p>\nTry these Snowflake Updates for Yourself<\/h2>\n
\n