Daily Briefing — Wednesday, November 18, 2026
Ignite week pulls the platforms toward agent-ready plumbing: CDC and ingestion fold into the lakehouse, retrieval and reranking land in warehouse SQL, and catalogs start governing agent skills and MCP access alongside the data itself.
⇣ Jump To
Click any section or topic below to jump to it.
Streaming & Messaging · CDC · ELT/ETL Ingestion · Transformation Frameworks
Cloud Data Warehouses · Table Formats · Architectural Patterns · Vector & Specialty Stores
AI-Driven Consumption · Semantic Layers & Retrieval · Enterprise RAG & Retrieval
⚡ Quick Takes
| Story | Signal |
|---|---|
| ↗ Confluent Cloud adds full-record backup and restore to its Azure Blob Storage connectors | Topic backup to object storage is becoming a first-class connector mode, not a DIY job |
| ↗ Client-side field-level encryption with Azure Key Vault now works over Private Link | Field-level encryption on streams is being fitted to private-network-only enterprise estates |
| ↗ Databricks puts an integrated PostgreSQL CDC pipeline into Beta | Lakehouse vendors keep absorbing CDC into the platform, squeezing standalone replication tools |
| ↗ Snowflake Openflow deployments go GA in Azure Government regions | Native ingestion is following warehouses into sovereign and government clouds |
| ↗ Snowflake ships a GA Spark Monitoring UI for Snowpark Connect workloads | Snowflake is building the operator tooling needed to win Spark workloads, not just run them |
| ↗ BigQuery adds a GA REST API for cross-region dataset replication and failover reservations | Warehouse disaster recovery is moving into infrastructure-as-code |
| ↗ Cortex AI Function Studio and AI Function Optimization reach general availability | In-warehouse AI is gaining the eval and cost-tuning tooling that production pipelines need |
| ↗ BigQuery previews Apache Iceberg views through its Lakehouse runtime catalog REST endpoint | Iceberg interoperability is moving up from tables to views |
| ↗ Microsoft launches Azure HorizonDB GA and previews a serverless Fabric HorizonDB at Ignite | Translytical Postgres inside the analytics platform is now table stakes for the big three |
| ↗ LanceDB raises $57.5M for its multimodal data lakehouse | Multimodal AI training data is creating demand for a table format beyond Parquet-era assumptions |
| ↗ Dell puts a single gateway between enterprise AI agents and the data they need | MCP gateways are becoming the default control point for agent-to-data access |
| ↗ Fabric IQ goes GA in Copilot, with Excel access in preview | Semantic models are becoming the agent context layer, not just a BI artifact |
| ↗ BigQuery makes AI.RANK and AI.SEARCH reranking GA, plus an optimized mode for AI.IF and AI.CLASSIFY | Retrieval and reranking are collapsing into warehouse SQL |
| ↗ Unity Catalog agent skills can now be shared through OpenSharing and Databricks Marketplace | Catalogs are starting to govern agent capabilities as shareable assets |
| ↗ Governed tag propagation extends to classic compute | Tag-based access control only works if propagation covers every compute path |
| ↗ Ataccama named sole Visionary in the 2026 Gartner Magic Quadrant for D&A Governance Platforms | The new governance MQ is out; agentic stewardship is the headline pitch |
| ↗ Snowflake makes SHOW GRANTS ... INCLUDE HIERARCHY generally available | Effective-privilege visibility is becoming a native query, not a script |
| ↗ Snowflake previews SCIM provisioning at the organization level | Identity governance is moving up to the organization tier in multi-account platforms |
| ↗ Microsoft Ignite Day 1: MCP Server for Enterprise GA and Purview network DLP | Security and compliance metadata is being exposed to agents through MCP by default |
Confluent Cloud Release Notes — November 2026
The Azure Blob Storage Sink and Source connectors gain a BACKUP_FULL_RECORD / RESTORE_BACKUP_FULL_RECORD mode that preserves the whole Kafka record rather than just the value payload. That turns object storage into a credible topic-level backup and replay target on Azure, which matters for DR runbooks and for rehydrating topics into new clusters without custom tooling.
✍️ Confluent · Read article →
Confluent Cloud Release Notes — November 2026
Confluent Cloud now supports CSFLE using Azure Key Vault reached over Azure Private Link, in Limited Availability and opt-in through the account team. For regulated Azure shops, it closes the gap where field-level encryption required a public KMS endpoint, keeping key traffic on the private network end to end.
✍️ Confluent · Read article →
Databricks Release Notes — November 2026
Lakeflow Connect can now ingest PostgreSQL change data capture without deploying a separate ingestion gateway, per the November 15 release note. Removing the gateway hop simplifies the operational footprint for Postgres-to-lakehouse replication and puts Databricks in more direct competition with Debezium-based and Fivetran HVR-style pipelines.
✍️ Databricks · Read article →
Snowflake Release Notes — November 2026
Openflow, Snowflake's managed NiFi-based ingestion service, is now generally available for Snowflake-hosted deployments in Azure Government. Public-sector teams that were blocked on FedRAMP-style region requirements can now run ingestion inside the platform rather than staging data through third-party ELT tools.
✍️ Snowflake · Read article →
Snowflake Release Notes — November 2026
Snowsight now offers a generally available monitoring UI for Spark batch jobs and sessions running through Snowpark Connect for Spark. Teams migrating PySpark code onto Snowflake compute get Spark-style job visibility in place, which removes one of the practical objections to lifting Spark transforms off dedicated clusters.
✍️ Snowflake · Read article →
BigQuery Release Notes — November 2026
Cross-region dataset replication and failover reservations can now be managed through datasets.insert, patch, update and get in the BigQuery REST API, now generally available. That makes warehouse DR configurable as code through Terraform or internal platform tooling instead of console-driven setup.
✍️ Google Cloud · Read article →
Snowflake Release Notes — November 2026
Snowflake made Cortex AI Function Studio GA for creating, evaluating and optimizing AI functions, alongside a separate GA AI Function Optimization feature aimed at more efficient execution. Together they give platform teams an eval-and-tune loop for SQL-callable LLM functions, which is where token cost and output quality are actually controlled once AI runs inside pipelines.
✍️ Snowflake · Read article →
BigQuery Release Notes — November 2026
Iceberg views can now be created in Google's Lakehouse runtime catalog via the Iceberg REST endpoint and queried from Spark and Trino, in Preview. Views are the piece of the Iceberg spec that most catalogs still handle poorly, so engine-portable view definitions are a meaningful step toward a genuinely shared semantic surface across engines.
✍️ Google Cloud · Read article →
Microsoft Tech Community — November 2026
Azure HorizonDB, Microsoft's PostgreSQL-compatible database with built-in vector search, is now GA, while Fabric HorizonDB enters preview as a serverless variant that provisions in seconds, pauses after five minutes idle, and pushes analytical queries down to Fabric Data Warehouse. Microsoft also made the Cosmos DB semantic reranker GA and previewed automatic embedding generation, reinforcing the push to keep operational, analytical and vector workloads on one data estate.
✍️ Shireesh Thota, Microsoft · Read article →
SiliconANGLE — November 2026
LanceDB raised $57.5 million led by Bessemer Venture Partners and Innovation Endeavors, bringing total funding to $98.5 million. Built on the open-source Lance format and querying data in place in customer cloud storage, it reports tables of 10 to 20 PB in large training workflows, with Runway, Netflix and Microsoft named as users.
✍️ Kyt Dotson, SiliconANGLE · Read article →
SiliconANGLE — November 2026
Dell introduced an MCP-backed Agentic Data Gateway as part of its AI Data Platform, claiming agent answer accuracy rose from roughly 60% to about 94% in its own testing. The figure is vendor-reported from a Dell-sponsored theCUBE segment, but the pattern of a governed MCP layer brokering agent access to enterprise data is spreading beyond the cloud data platforms.
✍️ Ryan Stevens, SiliconANGLE (Dell-sponsored segment) · Read article →
Microsoft Ignite Book of News — November 2026
Microsoft made Fabric IQ, its business-semantics layer over OneLake, generally available inside Copilot, with Excel integration in preview; the Book of News also lists Fabric apps and the Fabric Database Hub as GA. For architects, the significance is that the ontology layer defined in Fabric now becomes the context agents consume across the Microsoft surface, raising the stakes on getting that model right.
✍️ Microsoft · Read article →
BigQuery Release Notes — November 2026
A new GA AI.RANK function reorders rows by how well a text column answers a query, and AI.SEARCH gains a GA rerank option, putting a full retrieve-then-rerank pattern into SQL. An optimized mode for AI.IF and AI.CLASSIFY, also GA, cuts LLM token use and latency and adds multi-label classification.
✍️ Google Cloud · Read article →
Databricks Release Notes — November 2026
In Beta, data providers can add Unity Catalog agent skills to shares and Marketplace listings alongside tables and models. That extends the catalog's governed-sharing model from data assets to agent capabilities, meaning lineage and access policy questions now apply to the skills an agent loads, not only the data it reads.
✍️ Databricks · Read article →
Databricks Release Notes — November 2026
Unity Catalog governed tag propagation now works on classic compute running Runtime 19 and above, not just serverless, in Beta. Since tag-driven ABAC policies only hold if tags follow data through derived tables, closing the classic-compute gap removes a common hole in classification coverage for mixed estates.
✍️ Databricks · Read article →
GlobeNewswire — November 2026
Ataccama announced its placement in Gartner's 2026 Magic Quadrant for Data and Analytics Governance Platforms, published November 12, highlighting its ONE AI steward agent, Data Trust Index and MCP Server. The vendor release is a signal that the governance MQ is out, and that agentic stewardship and MCP exposure are now part of how vendors pitch against the catalog incumbents.
✍️ Ataccama · Read article →
Snowflake Release Notes — November 2026
SHOW GRANTS TO can now return the full inherited role hierarchy in one call, GA as of November 16. Effective-privilege resolution has long required recursive scripts against ACCOUNT_USAGE, so this makes access reviews and entitlement audits considerably easier to automate.
✍️ Snowflake · Read article →
Snowflake Release Notes — November 2026
Organization users and organization user groups can now be provisioned via SCIM in private preview. For multi-account Snowflake estates, it moves identity lifecycle from per-account integrations to a single org-level feed from the IdP, reducing orphaned-user drift across accounts.
✍️ Snowflake · Read article →
Gethyn Ellis (GRE Solutions) — November 2026
A practitioner roundup of Ignite Day 1 reports that Microsoft's MCP Server for Enterprise, which exposes Entra, Purview and Intune data to AI tools, is GA, along with network DLP through Global Secure Access. It also flags Fabric IQ previews in ChatGPT and Claude and the retirement of standalone Power BI Copilot by end of January 2027.
✍️ Gethyn Ellis · Read article →