Unlock Trapped Data in Your Data Lakes: Introducing Zero Copy File Federation in Data Cloud

Unlock Trapped Data in Your Data Lakes: Introducing Zero Copy File Federation in Data Cloud

Share article

In today's dynamic data landscape, organizations are grappling with a wealth of information spread across various data lakes and warehouses. This data, while holding immense potential, often remains siloed and difficult to access in real-time - hindering agility, inflating costs, and limiting the impact of crucial initiatives like agentic AI, automation, and advanced analytics. The challenge isn't just about having data; it's about achieving data fluidity - the seamless movement and activation of your information, wherever it resides.

Last year, we took a significant step towards this vision with the introduction of data federations in Salesforce Data Cloud. Today, we're thrilled to announce the Beta launch of Zero Copy File Federation, a powerful evolution that makes it even better to shatter data silos and unlock the full value of your enterprise data. This groundbreaking capability represents a pivotal advancement in our Zero Copy approach, offering a distinct and optimized pathway to access the vast datasets typically residing in data lakes and lakehouses, complementing our existing query federation capabilities. File Federation accesses data directly from Iceberg tables at storage level without compute overhead and hence is better suited for large-scale, high-volume datasets where performance and cost efficiency are crucial.

Unlocking Trapped Data for Limitless Possibilities

Imagine a world where the rich, structured, and unstructured data within your data lakes becomes instantly accessible for a multitude of critical use cases without the need for cumbersome data movement or duplication. Zero Copy File Federation makes this a reality. By providing a direct, Zero Copy connection to platforms like AWS Lake Formation, Databricks, and Snowflake (leveraging the open-source Apache Iceberg table format), we're unlocking previously trapped data for:

• Rich Customer 360 degree view: Build the real time 360 degree view of your customers by bringing data together from within the salesforce and outside enabling richer insights and timely decisions.
• Next-Generation Analytics: Gain deeper, more comprehensive insights by analyzing your entire data estate in place, without the latency and cost associated with traditional ETL processes.
• Transformative Agentic AI: Supercharge your AI-powered agents with direct access to vast amounts of historical and real-time data, enabling richer context, more personalized interactions, and truly intelligent automation.
• Intelligent Automations: Fuel your automation workflows with a holistic view of your data, triggering more sophisticated and effective actions based on insights derived from your data lakes.

Zero Copy: The Key to Data Fluidity and Activation:

Zero Copy File Federation plays a crucial role in our vision of Data Cloud as the central layer for activating applications and experiences. By eliminating the need to physically move or copy data by connecting to data at storage level, we're addressing key challenges:

• Cost Efficiency: Reduce storage duplication costs and minimize the compute resources required for data pipelines.
• High Scale: Connect to data with near native performance at scale with in-region access.
• Enhanced Agility: Access and analyze data in real-time, accelerating time-to-insight and enabling faster response to evolving business needs.

Simplified Operations: Eliminate the complexity of building and maintaining intricate data pipelines, freeing up valuable data engineering resources.

File Federation vs. Query Federation: Choosing the Right Path:

While both File and Query Federation offer unique value to the customers, File Federation offers several advantages that further enhance the Zero Copy experience for the customer. File Federation is based on the Iceberg tables, where Data Cloud scans the data in the customer's lake and mounts it for access. It offers the ability to access large volumes of data at near native latencies without utilizing the customer's compute. Furthermore, File Federation offers the ability to leverage the richness and sophistication of Iceberg table format with features such as time travel and schema evolution. Lastly, File Federation unlocks the ability to use CDC based features in Data Cloud without needing to cache your data. With the availability of UniForm, customers can seamlessly use their existing Delta tables for File Federation, saving them the effort to convert them to Iceberg tables.

Unlock the Full Value of Your Data, Across Your Enterprise:

The Beta launch of File Federation in Data Cloud is a significant leap forward in our commitment to data fluidity. We believe that by providing seamless, zero-copy access to your data lakes, we're empowering you to finally unlock the full value of your data across the entire enterprise for your customer 360, analytics, AI, and automation initiatives.

Beta Availability and Getting Started:

We're excited to offer select customers the opportunity to participate in the File Federation beta program. If you're leveraging data lakes like AWS Lake Formation, Databricks, Snowflake, or any Iceberg data lake with Iceberg REST catalog and are eager to unlock the power of this data for your Customer 360 and Agentforce initiatives, we encourage you to get started now.

A Retail Organization's journey to leverage the power of Zero Copy to make their data actionable in Data Cloud

To demonstrate the power of Zero copy and, in particular, File Federation, let us cTo demonstrate the power of Zero copy and, in particular, File Federation, let us consider the use-case of a retail organization that wants to use their customer loyalty data available in their Databricks Lakehouse and combine it with the customer profile and marketing data in CRM to create a 360 view of the customer. This unified view will then be used to achieve two goals

1. Automate a process to send a welcome email to customers who have joined their loyalty program
2. Feed their agents with customer loyalty information for self service
3. Enrich CRM with the customer data from Databricks

In this section, we will look at how we can achieve these use-case using the customer's 360 view.

Securely connect to external data via Zero Copy

The journey begins with the data specialist creating a secure connection between Databricks and Data Cloud. Leveraging credential vending for Unity catalog, the data specialist is able to create a connection to Databricks using only the catalog endpoint and personal access token. Credential Vending for Unity Catalog securely connects by providing temporary access, avoiding long-term token management.

Upon establishing the connection, the data specialist creates the data stream where they choose the desired objects and the associated fields from the source. This culminates in the creation of a Data Lake Object (DLO) or raw data from your lake.

Harmonizing the data in Data Cloud

With the connection and data stream established, the data specialist now maps the newly created external Data Model Object (DLO) to a standard or customer Data Model Object (DMO) or standardized data in Data Cloud. In this case, the data specialist has created their own custom Data Model Object (DMO) and maps the aforementioned Data Lake Object (DLO) to it.

Process automation to trigger email generation

The data specialist now creates a flow to trigger a welcome message to any customer who enrolls in the loyalty program. With File Federation, the data specialist can trigger data action without needing to cache the pertinent data in Data Cloud. Hence with Zero Copy, you can automate actions using lake data directly, without duplication.

Agentic experience using Zero Copy Data

The organization receives several questions from their customers regarding their loyalty rewards data and benefits, leading to their service team being overwhelmed. To alleviate this issue, the data specialist leverages agents in Data Cloud to address these customer queries, based on the data from their Databricks Lakehouse. AI agents access real-time lake data during customer interactions.

Enrich CRM with Unified Data from Databricks

The service tier and benefits for customers varies based on their loyalty tier. To ensure that the service team has complete visibility into the customer's profile, the data specialist uses copy field enrichment to enhance their CRM object, thus displaying the customer's loyalty program details along with other customer data in CRM. Using the copy field enrichment feature, the customer's data is now embedded in their contact record.

This retail organization's journey showcases the transformative power of Zero Copy File Federation. By securely and seamlessly connecting to their Databricks data lake, they've eliminated the complexities of traditional data movement, empowered their agentic AI with real-time loyalty insights, and ultimately delivered more personalized and efficient customer experiences - all while maximizing the value of their existing data investments

The same capabilities are available to connect to Iceberg based data lakes created in Snowflake or AWS.

Unlock the Potential of Your Data Platforms for Agentic AI Today:

The beta launch of File Federation marks a significant milestone in our commitment to empowering you with seamless access to all your data, regardless of where it resides. By bridging the gap between your data lakes and the intelligent capabilities of Agentforce, we're ushering in a new era of data-driven customer experiences. Join us in shaping the future of agentic AI by exploring the power of File Federation.

Unlock the future of data-driven customer engagement with the revolutionary Zero Copy integration between Salesforce and Databricks. Say goodbye to the complexity of traditional ETL processes-this secure, seamless, and flexible bidirectional integration empowers your organization to move faster, work smarter, and deliver standout customer experiences. With a real-time, 360-degree view of every customer powered by Salesforce Data Cloud Zero Copy, your teams gain the insights they need to personalize every interaction, maximize value at every touchpoint, and drive transformative business success.

Learn More:

Salesforce Data Cloud Documentation - File Federation

Share article