Unleashing the Power of your data and AI with bi-directional Zero Copy File Federation with Databricks

Unleashing the Power of your data and AI with bi-directional Zero Copy File Federation with Databricks

Share article

Overview

As the data ecosystem continues to expand and diversify, organizations are challenged by the complexity of managing information dispersed across multiple data lakes and warehouses. While this data holds immense value, it often remains fragmented and out of reach in real time-impeding innovation, escalating costs, and slowing progress on critical initiatives like agentic AI, automation, and advanced analytics. The core issue isn't the lack of data-it's the lack of data fluidity: the seamless ability to access, mobilize, and activate data wherever it resides.

Today, we are excited to announce the General Availability of bi-directional Zero Copy File Federation with Databricks. Data Cloud leads the Industry in innovating on zero copy data integration and is now expanding capabilities of Zero Copy Partner Network to File Federation partners supporting Iceberg as open standard and with a broader theme to support high scale and performance. It is not just about bringing the data from Databricks into the Salesforce data cloud. The strategic integration is fostering collaboration, enabling near real time analytics and driving the decision making process much faster by accessing the Salesforce data through file sharing functionality in Databricks.

Salesforce Data Cloud and Databricks integration is now Generally Available in both directions:

With Zero Copy File Federation, customers can now access billions of rows of data directly from their external Data Lake and activate their data without needing to copy it to Data Cloud. This cutting-edge feature marks a major leap forward in our Zero Copy strategy, providing a streamlined and purpose-built method to tap into the extensive datasets commonly found in data lakes and lakehouse environments. Unlike query-based methods, File Federation retrieves data directly from Iceberg tables at the storage layer, eliminating compute overhead on the source-making it ideal for handling massive data volumes where speed and cost optimization are essential.

With Zero Copy File Sharing, customers can also share their Data Cloud data into Databricks Unity Catalog. This integration lets you query Salesforce Data Cloud Objects directly from the Databricks Data Intelligence Platform, so you can run analytics without building pipelines or maintaining duplicate data. This enables you to use your Data Cloud customer 360 assets in place while Databricks handles processing and analysis in real time using Databricks SQL and MosaicAI for high performance and lower costs. See the Public Preview blog for how File Sharing works with Databricks.

In summary, with Zero Copy, you can further enhance unlocking your trapped data and powering many use-cases such as marketing, customer 360, automation and agents.

From Data to Decisions: A Retailer's Journey with Zero Copy in Data Cloud

The following use case demonstrates how customers harness the power of Zero Copy to drive meaningful outcomes across their organization. Northern Trail Outfitters, for example, stores customer transactions as Databricks Delta tables in the Databricks Data Intelligence Platform and is consumed by Iceberg readers by enabling UniForm. They will combine this with customer profile and email marketing data in Data Cloud to achieve the following goals:

• Display the last 10 customer transactions directly on the customer's Salesforce account page
  
• Enable service agents to efficiently handle transactional and warranty-related inquiries
  
• Trigger targeted marketing emails based on customers' monthly spending
• Share the unified customer data back to Databricks
  

In the next section, we'll explore how these outcomes can be realized by leveraging a unified 360-degree view of the customer.

Securely connect to data in Databricks via Zero Copy

The first step involves the data specialist establishing a secure connection between Databricks and Data Cloud. By leveraging Credential Vending for Unity Catalog, the connection is set up using just the catalog endpoint and a personal access token. This approach ensures secure, temporary access without the burden of managing long-term credentials, enabling streamlined and secure integration.

File Federation with Databricks currently supports AWS S3/Lake Formation and Azure based storage layers. Upon establishing the connection,

The data specialist creates a data stream, where they choose the desired object, the desired field, the primary keys and other details. The data stream acts as the conduit between Databricks and Data Cloud using the metadata.

Upon completing the creation of the data stream, an external Data Lake Object (DLO) is created that will then be mapped to the Data Model Object

Harmonize your data in Data Cloud

With the connection and data stream in place, the data specialist maps the newly created external Data Lake Object (DLO) to either a standard Data Model Object or custom Data Model Object. In this scenario, the data specialist has defined a custom DMO and maps the DLO directly to it.

Data Unification to create an unified individual

Unifying the data coming from all the internal and external sources is critical to creating a 360 view of the associated customer. Using Identity Resolution, Northern Trail Outfitters are able to create a total of 207 unified profiles from the 559 source profiles that were accessed from the different data sources.

Enrich CRM with Customer transactions from Databricks

Customer service tiers and benefits are determined by total transactional spend. To give the service team full visibility into each customer's profile, the data specialist uses copy field enrichment to augment the CRM object-bringing transactional details alongside existing customer data. With this enrichment, the customer's transactional information is now seamlessly integrated into their contact record.

Agentic Interactions Powered by Zero Copy Data

Northern Trail Outfitters receives frequent customer inquiries about transactions and warranties, putting a strain on their service team. To ease this burden, the data specialist deploys AI agents in Data Cloud that can handle these queries using data from their Databricks Data Intelligence Platform. During customer interactions, the agents access real-time lakehouse data to provide accurate and timely responses.

Trigger targeted marketing emails based on customers' monthly spending

The data specialist designs a flow to automatically trigger a marketing message for any customer whose monthly spend surpasses a defined threshold. Leveraging File Federation, this action is initiated without the need to cache relevant data in Data Cloud. With Zero Copy, actions can be executed directly on lake data-eliminating the need for duplication and streamlining automation.

This Northern Trail Outfitters use case highlights the transformative impact of Zero Copy File Federation. By securely connecting to their Databricks data lake, the organization removed the challenges of traditional data movement, enabled their agentic AI with real-time transactional insights, and delivered more personalized, efficient customer experiences-while fully maximizing the value of their existing data investments.

Share the unified customer insights back into Databricks for extended analysis

Lastly, the organization now wants to share the unified customer insights from Data Cloud back into Databricks for extended analytics and dashboards. To begin, they will set up a Data share target with the authentication details to get the connection established with Databricks.

With the Data share created in data cloud and a data share target, they begin next step by sharing the pertinent objects from Data Cloud to Databricks using the Link/Unlink capability, thus eliminating the need to maintain multiple copies for data and ensuring access to the most updated information from Data Cloud

The objects that were shared in the workspace enabled for Unity Catalog are viewable by Databricks persona.

It is not just that they are available in workspace and it is also available in Notebook for Data scientists to run machine learning models.

Unleash the Power of Your Data Platforms for Agentic AI

The launch of File Federation represents a pivotal step in our commitment to giving you seamless access to all your data-no matter where it resides. By connecting your data lakes with the intelligence of Agentforce, we're opening the door to a new era of data-driven customer experiences. Discover the potential of File Federation and help shape the future of agentic AI.

Unlock the next generation of customer engagement with the groundbreaking Zero Copy integration between Salesforce and Databricks. This secure, flexible, and bidirectional connection eliminates the need for complex ETL processes, enabling your teams to move faster, operate smarter, and deliver more impactful customer experiences. With Salesforce Data Cloud Zero Copy, you gain a real-time, 360-degree view of every customer-empowering your organization to personalize interactions, maximize value at every touchpoint, and drive transformative business outcomes.

Learn More:

Salesforce Data Cloud Documentation - File Federation

Share article