Coalesce 2.0: The Agentic Data Engineering Platform

We built Coalesce to be the most efficient way for enterprises to transform data with best practices built in by default. When we first introduced the platform in 2022, code was expensive and time-consuming.

Since then, nearly everyone has adopted AI for writing code. Code is no longer the bottleneck it used to be, and data teams build pipelines much faster, with fewer people, and less human review. But the core challenge to deliver reliable data pipelines at scale still takes human judgment. Agreeing on modeling standards, designing the architecture, deciding what governance applies, and knowing whether the numbers are actually right are still hard problems to solve. The 2026 State of Data Engineering Survey highlights this, with 82% of practitioners saying AI helps most with writing code, while only 13% say data modeling and 11% say governance and quality checks.

Anthropic recently shared how they're seeing this play out in their AI-Native SDLC playbook , as companies transform their development lifecycle and compress the time spent on coding.

Click to expand: With agentic coding compressing the build cycle, the emphasis shifts to where human judgment still matters most: planning, design, and maintenance. Source: Anthropic

The future DataOps lifecycle will need to do three things in this new era.

First, we believe a platform should serve the full development lifecycle, both the 80% of code a data engineer generates quickly with Claude Code, Cortex, or similar tools, and the 20% that still needs a human to review, debug, and iterate on.

Second, in an agentic world, governance is more important than ever. As more pipelines are written entirely by agents without human review, guardrails and standards should be baked into the development workflow.

Finally, transformation, cataloging, and quality tooling should share a context layer, so they're managed and deployed together rather than as separate parts.

The Future DataOps Lifecycle

For most data teams, even if the build phase has been shortened, the data development lifecycle hasn't changed significantly. Typically, a requirement lands in a document or a Slack thread, a data engineer builds it, sped up by agentic coding tools, a review is added, the code is merged, and in some cases, monitoring and context are added manually in the quality tool and catalog.

The best data teams are already adopting a different approach, and we expect most to catch up soon.

Instead of the sequenced approach, the agent takes on larger parts of the data development loop. Humans focus more on questions that still need a human answer, such as the nuances of a specific business problem, what the company-wide governance and quality policies should be, whether the numbers are intuitively right, and which data challenges matter most to the business.

In the future DataOps lifecycle, there are four main stages:

This changes the role the humans play. 80% of the build and governance work happens automatically based on standards the team set up centrally. The other 20% that's specific to the spec is where the data engineer's attention goes.

A typical workflow could look like this. Marketing has launched a set of new campaigns that need metrics for tracking. The marketing team and the data engineer set up a call. The initial spec is mostly automated from the call transcript, with the data engineer only needing to decide the data's purpose and which metrics should be available. The model has standards such as PII tagging, data quality checks and monitors, and descriptions added automatically. Once the engineer has sign-off from marketing on the initial spec, it's pushed to production, and the data product is available for everyone to find, with real-time monitoring already in place.

Click to expand: An entire pipeline of marketing metrics is built from a spec. See a live demo of spec-to-pipeline (30 minutes)

This only works if transform, catalog, and quality tools share the same context layer. If a production incident never makes it back to the definitions the agent builds from, the next update may repeat the same mistake.

Over the last year, we've rebuilt the Coalesce platform around these ideas. Here's what that looks like in practice.

Coalesce Platform 2.0: Built for Agentic Data Engineering

A local workflow optimized for agents

In our view, the best way to build data pipelines is to combine agentic coding with context and guardrails that enforce standards.

We're introducing a set of new tools to the Coalesce platform to make it best-in-class for agentic development, while still enforcing standards such as naming conventions, node types, and SQL templates.

We designed our local architecture with four core components:

• Coalesce CLI (coa) - authors, validates, and runs pipelines locally out of your Git repository. It gives your agent a deterministic execution engine to test DDL and DML directly against your data warehouse without routing code through the cloud.
• Coalesce MCP - connects your local agent to your Coalesce workspace, giving it read and administrative access to inspect run histories, monitor job status, add tests, and query deployed environments.
• Coalesce platform skills - a pre-packaged collection of skills available as a Claude Code plugin that teaches your local AI agent best practices for developing and governing data pipelines.
• Coalesce Desktop - a visual interface reading directly from your local workspace files with real-time, bidirectional sync. Developers run their agent in the terminal on one side of the screen and watch live lineage and node updates render visually on the other.

This creates a fast development loop. The best teams typically spin up the initial 70% to 90% of the code with AI coding tools, and then fine-tune the remaining details directly in the Desktop UI for the fastest possible iteration. You can still use the Coalesce cloud platform for the full development lifecycle, but for many teams it will serve more as a governance layer that code is promoted to once it passes all relevant checks locally.

Built-in governance for agents and humans

As AI agents make building data pipelines faster, governance has to move into the development workflow itself. We've spent the last year integrating our enterprise data catalog (formerly CastorDoc) and data observability platform (formerly SYNQ) into Coalesce. Both were mature standalone products with dozens of enterprise customers, so this is not a set of bolt-on features but two full platforms now running on the same context layer as Transform.

For quality, we're giving you the tools to catch issues through both data quality checks and sophisticated anomaly monitors. First, we're adding dozens of built-in tests from the Quality platform directly in Transform, supporting existing circuit-breaking capabilities to stop downstream nodes from updating on errors. We're also introducing annotations in Transform, letting you define which tables should have anomaly monitors enabled, which Coalesce Quality automatically provisions upon deployment.

For governance, critical metadata (such as column descriptions, asset ownership, and priority tiers) is managed directly across the platform with bidirectional synchronization. A central policy layer makes this possible, where rules like testing and monitoring requirements, PII masking, ownership assignment, documentation, and modeling patterns are checked automatically in the local CLI loop before any code reaches production. The same checks apply whether a person or an agent authored the change.

Take an insurance company, for example. Claims and regulatory audits often require point-in-time policy history, such as determining whether a policy was active, lapsed, reinstated, or canceled on a specific date of loss. That makes Slowly Changing Dimension (SCD) Type 2 a mandatory pattern for any table tracking policy or coverage status, and it's encoded as a standard. If an agent generates a simple overwrite or upsert instead, the check fails at build time, and the violation never reaches a downstream consumer.

These standards are managed centrally as policies in the Coalesce Platform, letting everyone from governance teams to business stakeholders add rules that are then applied as annotations and tags during development. You can decide which rules should be circuit breakers and which should only raise warnings, and route them through our new alerting engine so that both technical and non-technical stakeholders are notified.

One context layer across transform, catalog, and quality

Keeping transformation, catalog, and data quality in separate tools creates bottlenecks when you introduce AI agents. An agent can write SQL code fast, but without the right context, it may create code that ignores business definitions, breaks dependencies, or has data quality issues that are not caught.

Transform, Catalog, and Quality now run on one context layer. This means transformation logic, ownership, lineage, tests, and issue history are all stored in one place and accessible at any time. Your local agent reads that layer through the Coalesce MCP, and Git versions it alongside your code.

For example, a data engineer asks a local agent to build a new financial pipeline. Instead of guessing column logic, the agent pulls the ARR calculation, metric definitions, and PII masking rules directly from Catalog and writes the transformation logic against those definitions, attaching the required tests to the nodes. If a Quality monitor later flags an anomaly in production, that incident is linked to the asset in Catalog, so the next change to that pipeline starts from that history.

Data catalogs used to compete on connector counts. We believe the new benchmark is how much context is captured and how fast it compounds over time. When an agent builds a new model, it starts from the existing definitions and standards instead of making them up from scratch. When it looks at a data issue, it has access to recent definition changes or earlier incidents on the same asset.

Six months later, you can still see exactly which business logic or catalog definitions the agent used when it built the pipeline. Connecting context, transformations, and quality into one context layer makes it possible to enforce and scale governance as you build.

Scaling access to Coalesce product experts

After working with hundreds of data teams on their most complex projects, we've seen the same patterns lead to successful projects again and again. Whether you are migrating off a legacy warehouse, building out an enterprise-wide pipeline, or setting up governance across thousands of assets, you shouldn't have to start from scratch.

We are formalizing the most common tasks and entire projects into an opinionated layer that combines pre-packaged AI skills with hands-on expertise:

• Platform skills and playbooks - we encode proven standards directly into skills your AI agents can run, available standalone and as a Claude Code plugin. This spans data modeling, governance rollouts, testing and monitoring, and industry-specific templates.
• Expert enablement and solutions - for high-stakes initiatives, our team works directly alongside yours. We deliver end-to-end projects on the Coalesce platform, whether that means executing legacy data migrations, running data quality and governance reviews, or designing custom domain templates.

Resources

• Agentic Data Engineering: A Practitioner's Guide - the full framework: what an MCP is, how to write prompts that hold up, and the three maturity levels from ad hoc prompts to skills to production playbooks.
• Hands-On With Coalesce MCPs: Transform, Catalog, and Quality - what each of the three MCP servers exposes and how they divide the work: Catalog for lineage and impact, Transform to make the change, Quality for root cause.
• Hands-On With Coalesce's MCPs Part 2: Building Data Engineering Agents With Skills - turning a workflow you repeat into a skill the whole team runs the same way, including how to write a description that triggers reliably.
• Hands-On With Coalesce MCPs Part 3: Built-In Agents and Production-Grade Playbooks - putting workflows into production: issue triage, governance rollout, and when to reach for a built-in agent instead.
• Spec to pipeline - spec to pipeline in action in a 30-minute recorded demo.
• Local AI Development - the four components of the local workflow and how they split: the CLI to build, the MCP to observe.
• Command Line Interface (coa) - install, authentication, and the local build loop. Version 7.33 and above is the one with local development and SQL authoring against your warehouse.
• Coalesce Desktop - use the Coalesce UI for local development.
• Coalesce Copilot - the assistant inside the Transform workspace, turning natural language into governed transformations with a read/write toggle.
• Coalesce AI Documentation Assistant - generating and maintaining pipeline documentation, which is what keeps the catalog useful as agent context rather than a stale artifact.