Daily Briefing — Friday, November 20, 2026
Platforms spent the week before re:Invent pulling more of the stack inside the engine — dead-letter handling in the load path, retrieval and embeddings as SQL, and agent audit as a native governance view — while Postgres 19 and open Iceberg tooling kept the open side moving.
⇣ Jump To
Click any section or topic below to jump to it.
Streaming & Messaging · CDC · ELT/ETL Ingestion · Stream Processing · In-Process Compute
Cloud Data Warehouses · Lakehouses · Table Formats · Vector & Specialty Stores
AI-Driven Consumption · Enterprise RAG & Retrieval
Orchestration & Workflow · Data Observability · Governance, Security & Compliance · FinOps for Data
⚡ Quick Takes
| Story | Signal |
|---|---|
| ↗ Azure Blob Storage connectors now back up and restore complete Kafka records | Object storage becomes a faithful Kafka backup tier on Azure, not just a sink. |
| ↗ Iceberg Kafka Connect CDC: coordinator converts equality deletes to deletion vectors before commit | Native CDC into Iceberg is maturing, but random keys still carry a heavy commit-time tax. |
| ↗ COPY INTO and Snowpipe gain error logging and CHECK constraint enforcement (GA) | Dead-letter queues move inside the warehouse load path. |
| ↗ Schema detection and evolution for structured types is generally available | Nested data lands typed, not as VARIANT blobs. |
| ↗ Lakebase streaming sink is generally available | Lakehouse-to-Postgres serving becomes a first-class streaming sink. |
| ↗ Flink, Tableflow Iceberg REST Catalog and Lightning Tables now reachable over AWS PNI | Streaming-to-Iceberg path goes fully private on AWS. |
| ↗ Apache Arrow Java 20.0.0 release candidate goes to vote | Arrow Java's release cadence is about to become more predictable for JVM engines. |
| ↗ Dynamic tables: AUTO refresh mode now resolves to ADAPTIVE | A default changed under your pipelines — pin refresh modes in templates. |
| ↗ Hitachi Vantara VSP One Object adds in-place querying of Iceberg and Parquet | On-prem object storage is turning into a lakehouse query tier. |
| ↗ Apache Polaris 1.9.0 released, plus Iceberg materialized-view spec heads to vote | Open catalogs harden while Iceberg finally standardizes materialized views. |
| ↗ BigQuery previews CREATE AI INDEX and AI.SEARCH for natural-language search over columns | Embedding management is being absorbed into the warehouse DDL. |
| ↗ PostgreSQL 19 released | Postgres finally gets built-in, non-blocking repack and planner hints. |
| ↗ Cortex Agent SDK reaches general availability | Warehouse-native agents get a stable 1.0 SDK contract. |
| ↗ Genie Code CLI is generally available | Platform coding agents move into the terminal under a single budget. |
| ↗ CORTEX_SEARCH_RETRIEVE table function is generally available | Retrieval becomes a SQL table function, joinable like any other. |
| ↗ Airbyte adds semantic search to its agent context layer | ELT vendors are turning into agent retrieval layers. |
| ↗ BigQuery Studio Git repositories are generally available | Warehouse IDEs are converging on Git-native workflows. |
| ↗ Agent observability for BigQuery's Data Engineering Agent is GA | Agents that build pipelines now get traced like the pipelines themselves. |
| ↗ Row- and column-level security with dynamic masking on lakehouse Iceberg tables (GA) | Fine-grained policy now follows Iceberg tables, not just native ones. |
| ↗ Snowflake previews an account-wide AI Governance page | Agent activity audit becomes a first-class governance view. |
| ↗ Service policies can now mask or deny sensitive data in embedding inputs (Beta) | PII controls move upstream to the embedding call. |
| ↗ GCP-2026-084: path traversal in Dataform, BigQuery Studio and Colab Enterprise Git handling | Git integration in data IDEs widens the attack surface — review connections. |
| ↗ Configurable retention for system tables is generally available | Ten-year billing and audit history is now a setting, with a price. |
Confluent Cloud release notes — November 2026
Confluent's fully managed Azure Blob Storage Sink and Source connectors can now round-trip entire Kafka records — keys, values, headers and metadata — via the sink mode BACKUP_FULL_RECORD and source mode RESTORE_BACKUP_FULL_RECORD. Until now, object-storage sinks were lossy for replay purposes. For platform teams on Azure, this turns Blob into a credible topic-level backup and disaster-recovery target without custom serializers.
✍️ Confluent · Read article →
Apache Iceberg dev list — November 2026
A proposal for the Iceberg Kafka Connect sink adds opt-in CDC mode in which the commit coordinator resolves equality deletes into deletion vectors before committing, keeping append-only as the default. Benchmarks in PR 18003 show under 0.4 s per commit with monotonically increasing keys but 2–42 s with random UUID keys on a 10,000-file table. Anyone streaming Debezium-style changes into Iceberg should note how much key design now drives commit latency.
✍️ Apache Iceberg community (via Alex Merced's weekly digest) · Read article →
Snowflake release notes — November 2026
COPY INTO and Snowpipe now enforce CHECK constraints and route rejected CSV, JSON and Parquet rows — with the error and usually the source record — into the table's error table. Enable it with ALTER TABLE … SET ERROR_LOGGING = TRUE plus ON_ERROR = CONTINUE; bad rows can be fixed in SQL and re-inserted. This pulls dead-letter handling, previously a bolt-on, into the load path itself.
✍️ Snowflake · Read article →
Snowflake release notes — November 2026
Schema inference and evolution now produce typed OBJECT, ARRAY and MAP columns when loading Parquet, JSON, Avro and ORC through COPY INTO or Snowpipe, instead of collapsing nested data into VARIANT. Typed nested columns give better pruning and stricter contracts downstream. It's a quiet but meaningful step toward schema-on-load that doesn't sacrifice structure.
✍️ Snowflake · Read article →
Databricks release notes — November 2026
Structured Streaming can now write directly into Lakebase (Databricks' managed Postgres) as a GA sink, including Trigger.AvailableNow() on serverless; DBR 19+ adds merge strategies and mergeSchema. Writing to external PostgreSQL through a Unity Catalog connection is in Public Preview. This closes the loop from lakehouse pipelines into an OLTP serving store without a separate reverse-ETL hop.
✍️ Databricks · Read article →
Confluent Cloud release notes — November 2026
Confluent Cloud for Apache Flink, the Tableflow Iceberg REST Catalog and Lightning Tables now support private connectivity over Private Network Interface on AWS. Enterprise and Freight clusters already on PNI no longer need a separate PrivateLink setup to reach these services. For regulated shops, the stream-to-table path can now stay entirely on private networking.
✍️ Confluent · Read article →
Apache Arrow dev list — November 2026
The Arrow Java 20.0.0 RC0 vote opened November 17 and collected a run of binding +1s, with Andy Grove validating it through an Apache DataFusion Comet upgrade. A parallel proposal would move Arrow Java to quarterly-or-on-demand releases. JVM-side engines (Spark, Flink, Iceberg) depend on this artifact, so cadence changes ripple into their upgrade planning.
✍️ Apache Arrow community · Read article →
Snowflake release notes — November 2026
With the default REFRESH_MODE = AUTO, newly created incrementalizable dynamic tables now resolve to ADAPTIVE rather than INCREMENTAL; non-incrementalizable definitions still fall back to FULL, and existing tables keep their resolved mode. Teams that want strict INCREMENTAL without adaptive reinitialization must now set it explicitly. Worth auditing pipeline templates, since cost and refresh behavior of new tables changes silently.
✍️ Snowflake · Read article →
Blocks & Files — November 2026
Hitachi Vantara expanded VSP One Object to query Iceberg, Parquet, JSON and CSV directly on object storage, claiming up to 5x faster analytical queries with 80% fewer CPUs, plus a new four-node starting configuration. Storage vendors keep pushing compute toward the bucket. For on-prem lakehouse architects, it's another option for keeping Iceberg data local while serving SQL.
✍️ Chris Mellor, Blocks & Files · Read article →
DEV Community — November 2026
Polaris 1.9.0 shipped on its second release candidate, bringing external principal support (credential-mode=external), an STS-only S3 credential-vending field, and a fix so commit-outcome-unknown no longer deletes metadata. Separately, the Iceberg community called a vote to add materialized views to the view spec, with refresh state tracked in the storage table's snapshot summary. Both matter for anyone standardizing on an open REST catalog.
✍️ Alex Merced, DEV Community · Read article →
BigQuery release notes — November 2026
A new CREATE AI INDEX statement and AI.SEARCH function let BigQuery run natural-language search over table columns without managing embeddings manually (Preview). The same day added gemini-4-neon to generative AI functions (GA). Warehouses are absorbing the embedding pipeline that teams used to run alongside them in a separate vector store.
✍️ Google Cloud · Read article →
PostgreSQL Global Development Group — November 2026
PostgreSQL 19 went GA on November 19 with a new REPACK command (including a non-blocking CONCURRENTLY mode) to reclaim space and reorganize tables, plus pg_plan_advice and pg_stash_advice extensions to steer planner decisions. Performance work includes auto-scaling I/O workers and faster foreign-key checks. With Postgres now underpinning Lakebase, Neon-style agent databases and pgvector estates, plan-advice controls are especially relevant for stabilizing vector-query plans.
✍️ PostgreSQL Global Development Group · Read article →
Snowflake release notes — November 2026
Snowflake's Cortex Agent SDK is GA at version 1.0.0 for Python and TypeScript, giving developers a supported way to build and invoke Cortex Agents from application code. Same-day releases added prompt caching for Cortex AI Functions and a log mode for prompt-injection guardrails. For platform teams, this is the stable contract for embedding governed agents into apps.
✍️ Snowflake · Read article →
Databricks release notes — November 2026
The Genie Code CLI is GA and is administered like Genie Code in the workspace, with a single budget shared across CLI and workspace usage. Bringing the coding agent to the terminal puts it where data engineers already run bundles and tests. The shared budget gives admins one cost control for both surfaces.
✍️ Databricks · Read article →
Snowflake release notes — November 2026
The CORTEX_SEARCH_RETRIEVE table function, previewed in October, is now GA: it runs low-latency Cortex Search queries from SQL, either single-shot or one query per input row via a LATERAL join. Retrieval becomes a composable relational operator rather than an API call outside the warehouse. That makes batch RAG enrichment and retrieval evaluation expressible as plain SQL pipelines.
✍️ Snowflake · Read article →
Blocks & Files — November 2026
Airbyte added semantic search to its context layer so AI agents can search customer conversations, documents and tickets and combine the results with structured data in Snowflake and BigQuery, exposed through the Airbyte MCP server. No GA status was given. Ingestion vendors are repositioning as retrieval providers for agents, not just pipeline builders.
✍️ Chris Mellor, Blocks & Files · Read article →
BigQuery release notes — November 2026
BigQuery Studio's folder-based integration with remote Git repositories is GA, letting teams version SQL, notebooks and pipeline code against their existing Git hosts. Note the companion security bulletin below covering Git integration in the same surfaces. Proper Git workflows are a prerequisite for treating warehouse code as production software.
✍️ Google Cloud · Read article →
BigQuery release notes — November 2026
BigQuery's Data Engineering Agent now emits Cloud Monitoring metrics and Cloud Trace spans, generally available. Teams can monitor what the agent did, how long it took and where it failed using the same tooling as the rest of their stack. Observability for AI pipeline builders is becoming table stakes before they're trusted in production.
✍️ Google Cloud · Read article →
BigQuery release notes — November 2026
BigQuery now enforces row-level and column-level security with dynamic data masking on Iceberg tables in the Lakehouse runtime catalog, generally available; a new INFORMATION_SCHEMA.ROW_ACCESS_POLICIES view also went GA. Fine-grained policy on open-format tables is what makes a single governed Iceberg copy viable. Expect parity questions for other engines reading the same tables.
✍️ Google Cloud · Read article →
Snowflake release notes — November 2026
A new Snowsight page under Governance & security surfaces AI usage metrics for Cortex Code and Cortex Agents, audits agent activity and data exposure, and keeps a changelog of agent changes (Preview). It gives governance teams one place to see what agents touched. Agent audit is becoming a native platform feature rather than something stitched from query history.
✍️ Snowflake · Read article →
Databricks release notes — November 2026
Databricks service policies can now mask or block sensitive data in single or batch embedding inputs and reject unsafe content before the model runs. A companion beta makes output policies evaluate streaming responses incrementally by default — with the caveat that already-released text can't be recalled. PII controls are extending to the vectorization step, which is where much sensitive data quietly leaks into indexes.
✍️ Databricks · Read article →
Google Cloud security bulletin — November 2026
Google disclosed a path-traversal issue in workspace directory handling and Git integration across Dataform, BigQuery Studio and Colab Enterprise. It lands the day after Studio's Git repositories went GA. Platform security teams should review the bulletin's remediation guidance and any repository connections their projects allow.
✍️ Google Cloud · Read article →
Databricks release notes — November 2026
Account admins can now set account-wide retention of 30 to 3,650 days for supported Databricks system tables; the first 395 days are free, with storage charges beyond that. Longer billing and audit history enables year-over-year cost analysis and longer compliance lookbacks without exporting to a separate store. The trade-off is now an explicit storage cost decision.
✍️ Databricks · Read article →