Daily Briefing — Friday, November 20, 2026

Data & AI Platforms Briefing

Platforms spent the week before re:Invent pulling more of the stack inside the engine — dead-letter handling in the load path, retrieval and embeddings as SQL, and agent audit as a native governance view — while Postgres 19 and open Iceberg tooling kept the open side moving.

⇣ Jump To

Click any section or topic below to jump to it.

Move & Transform

Streaming & Messaging ·  CDC ·  ELT/ETL Ingestion ·  Stream Processing ·  In-Process Compute

Store & Architect

Cloud Data Warehouses ·  Lakehouses ·  Table Formats ·  Vector & Specialty Stores

Consume & Activate

AI-Driven Consumption ·  Enterprise RAG & Retrieval

Govern & Operate

Orchestration & Workflow ·  Data Observability ·  Governance, Security & Compliance ·  FinOps for Data

⚡ Quick Takes

Story Signal
↗  Azure Blob Storage connectors now back up and restore complete Kafka records Object storage becomes a faithful Kafka backup tier on Azure, not just a sink.
↗  Iceberg Kafka Connect CDC: coordinator converts equality deletes to deletion vectors before commit Native CDC into Iceberg is maturing, but random keys still carry a heavy commit-time tax.
↗  COPY INTO and Snowpipe gain error logging and CHECK constraint enforcement (GA) Dead-letter queues move inside the warehouse load path.
↗  Schema detection and evolution for structured types is generally available Nested data lands typed, not as VARIANT blobs.
↗  Lakebase streaming sink is generally available Lakehouse-to-Postgres serving becomes a first-class streaming sink.
↗  Flink, Tableflow Iceberg REST Catalog and Lightning Tables now reachable over AWS PNI Streaming-to-Iceberg path goes fully private on AWS.
↗  Apache Arrow Java 20.0.0 release candidate goes to vote Arrow Java's release cadence is about to become more predictable for JVM engines.
↗  Dynamic tables: AUTO refresh mode now resolves to ADAPTIVE A default changed under your pipelines — pin refresh modes in templates.
↗  Hitachi Vantara VSP One Object adds in-place querying of Iceberg and Parquet On-prem object storage is turning into a lakehouse query tier.
↗  Apache Polaris 1.9.0 released, plus Iceberg materialized-view spec heads to vote Open catalogs harden while Iceberg finally standardizes materialized views.
↗  BigQuery previews CREATE AI INDEX and AI.SEARCH for natural-language search over columns Embedding management is being absorbed into the warehouse DDL.
↗  PostgreSQL 19 released Postgres finally gets built-in, non-blocking repack and planner hints.
↗  Cortex Agent SDK reaches general availability Warehouse-native agents get a stable 1.0 SDK contract.
↗  Genie Code CLI is generally available Platform coding agents move into the terminal under a single budget.
↗  CORTEX_SEARCH_RETRIEVE table function is generally available Retrieval becomes a SQL table function, joinable like any other.
↗  Airbyte adds semantic search to its agent context layer ELT vendors are turning into agent retrieval layers.
↗  BigQuery Studio Git repositories are generally available Warehouse IDEs are converging on Git-native workflows.
↗  Agent observability for BigQuery's Data Engineering Agent is GA Agents that build pipelines now get traced like the pipelines themselves.
↗  Row- and column-level security with dynamic masking on lakehouse Iceberg tables (GA) Fine-grained policy now follows Iceberg tables, not just native ones.
↗  Snowflake previews an account-wide AI Governance page Agent activity audit becomes a first-class governance view.
↗  Service policies can now mask or deny sensitive data in embedding inputs (Beta) PII controls move upstream to the embedding call.
↗  GCP-2026-084: path traversal in Dataform, BigQuery Studio and Colab Enterprise Git handling Git integration in data IDEs widens the attack surface — review connections.
↗  Configurable retention for system tables is generally available Ten-year billing and audit history is now a setting, with a price.

Move & Transform

› Streaming & Messaging

Confluent Cloud release notes — November 2026

Azure Blob Storage connectors now back up and restore complete Kafka records

Confluent's fully managed Azure Blob Storage Sink and Source connectors can now round-trip entire Kafka records — keys, values, headers and metadata — via the sink mode BACKUP_FULL_RECORD and source mode RESTORE_BACKUP_FULL_RECORD. Until now, object-storage sinks were lossy for replay purposes. For platform teams on Azure, this turns Blob into a credible topic-level backup and disaster-recovery target without custom serializers.

✍️ Confluent · Read article →

› CDC

Apache Iceberg dev list — November 2026

Iceberg Kafka Connect CDC: coordinator converts equality deletes to deletion vectors before commit

A proposal for the Iceberg Kafka Connect sink adds opt-in CDC mode in which the commit coordinator resolves equality deletes into deletion vectors before committing, keeping append-only as the default. Benchmarks in PR 18003 show under 0.4 s per commit with monotonically increasing keys but 2–42 s with random UUID keys on a 10,000-file table. Anyone streaming Debezium-style changes into Iceberg should note how much key design now drives commit latency.

✍️ Apache Iceberg community (via Alex Merced's weekly digest) · Read article →

› ELT/ETL Ingestion

Snowflake release notes — November 2026

COPY INTO and Snowpipe gain error logging and CHECK constraint enforcement (GA)

COPY INTO and Snowpipe now enforce CHECK constraints and route rejected CSV, JSON and Parquet rows — with the error and usually the source record — into the table's error table. Enable it with ALTER TABLE … SET ERROR_LOGGING = TRUE plus ON_ERROR = CONTINUE; bad rows can be fixed in SQL and re-inserted. This pulls dead-letter handling, previously a bolt-on, into the load path itself.

✍️ Snowflake · Read article →

Snowflake release notes — November 2026

Schema detection and evolution for structured types is generally available

Schema inference and evolution now produce typed OBJECT, ARRAY and MAP columns when loading Parquet, JSON, Avro and ORC through COPY INTO or Snowpipe, instead of collapsing nested data into VARIANT. Typed nested columns give better pruning and stricter contracts downstream. It's a quiet but meaningful step toward schema-on-load that doesn't sacrifice structure.

✍️ Snowflake · Read article →

› Stream Processing

Databricks release notes — November 2026

Lakebase streaming sink is generally available

Structured Streaming can now write directly into Lakebase (Databricks' managed Postgres) as a GA sink, including Trigger.AvailableNow() on serverless; DBR 19+ adds merge strategies and mergeSchema. Writing to external PostgreSQL through a Unity Catalog connection is in Public Preview. This closes the loop from lakehouse pipelines into an OLTP serving store without a separate reverse-ETL hop.

✍️ Databricks · Read article →

Confluent Cloud release notes — November 2026

Flink, Tableflow Iceberg REST Catalog and Lightning Tables now reachable over AWS PNI

Confluent Cloud for Apache Flink, the Tableflow Iceberg REST Catalog and Lightning Tables now support private connectivity over Private Network Interface on AWS. Enterprise and Freight clusters already on PNI no longer need a separate PrivateLink setup to reach these services. For regulated shops, the stream-to-table path can now stay entirely on private networking.

✍️ Confluent · Read article →

› In-Process Compute

Apache Arrow dev list — November 2026

Apache Arrow Java 20.0.0 release candidate goes to vote

The Arrow Java 20.0.0 RC0 vote opened November 17 and collected a run of binding +1s, with Andy Grove validating it through an Apache DataFusion Comet upgrade. A parallel proposal would move Arrow Java to quarterly-or-on-demand releases. JVM-side engines (Spark, Flink, Iceberg) depend on this artifact, so cadence changes ripple into their upgrade planning.

✍️ Apache Arrow community · Read article →

↑ Top

 

Store & Architect

› Cloud Data Warehouses

Snowflake release notes — November 2026

Dynamic tables: AUTO refresh mode now resolves to ADAPTIVE

With the default REFRESH_MODE = AUTO, newly created incrementalizable dynamic tables now resolve to ADAPTIVE rather than INCREMENTAL; non-incrementalizable definitions still fall back to FULL, and existing tables keep their resolved mode. Teams that want strict INCREMENTAL without adaptive reinitialization must now set it explicitly. Worth auditing pipeline templates, since cost and refresh behavior of new tables changes silently.

✍️ Snowflake · Read article →

› Lakehouses

Blocks & Files — November 2026

Hitachi Vantara VSP One Object adds in-place querying of Iceberg and Parquet

Hitachi Vantara expanded VSP One Object to query Iceberg, Parquet, JSON and CSV directly on object storage, claiming up to 5x faster analytical queries with 80% fewer CPUs, plus a new four-node starting configuration. Storage vendors keep pushing compute toward the bucket. For on-prem lakehouse architects, it's another option for keeping Iceberg data local while serving SQL.

✍️ Chris Mellor, Blocks & Files · Read article →

› Table Formats

DEV Community — November 2026

Apache Polaris 1.9.0 released, plus Iceberg materialized-view spec heads to vote

Polaris 1.9.0 shipped on its second release candidate, bringing external principal support (credential-mode=external), an STS-only S3 credential-vending field, and a fix so commit-outcome-unknown no longer deletes metadata. Separately, the Iceberg community called a vote to add materialized views to the view spec, with refresh state tracked in the storage table's snapshot summary. Both matter for anyone standardizing on an open REST catalog.

✍️ Alex Merced, DEV Community · Read article →

› Vector & Specialty Stores

BigQuery release notes — November 2026

BigQuery previews CREATE AI INDEX and AI.SEARCH for natural-language search over columns

A new CREATE AI INDEX statement and AI.SEARCH function let BigQuery run natural-language search over table columns without managing embeddings manually (Preview). The same day added gemini-4-neon to generative AI functions (GA). Warehouses are absorbing the embedding pipeline that teams used to run alongside them in a separate vector store.

✍️ Google Cloud · Read article →

PostgreSQL Global Development Group — November 2026

PostgreSQL 19 released

PostgreSQL 19 went GA on November 19 with a new REPACK command (including a non-blocking CONCURRENTLY mode) to reclaim space and reorganize tables, plus pg_plan_advice and pg_stash_advice extensions to steer planner decisions. Performance work includes auto-scaling I/O workers and faster foreign-key checks. With Postgres now underpinning Lakebase, Neon-style agent databases and pgvector estates, plan-advice controls are especially relevant for stabilizing vector-query plans.

✍️ PostgreSQL Global Development Group · Read article →

↑ Top

 

Consume & Activate

› AI-Driven Consumption

Snowflake release notes — November 2026

Cortex Agent SDK reaches general availability

Snowflake's Cortex Agent SDK is GA at version 1.0.0 for Python and TypeScript, giving developers a supported way to build and invoke Cortex Agents from application code. Same-day releases added prompt caching for Cortex AI Functions and a log mode for prompt-injection guardrails. For platform teams, this is the stable contract for embedding governed agents into apps.

✍️ Snowflake · Read article →

Databricks release notes — November 2026

Genie Code CLI is generally available

The Genie Code CLI is GA and is administered like Genie Code in the workspace, with a single budget shared across CLI and workspace usage. Bringing the coding agent to the terminal puts it where data engineers already run bundles and tests. The shared budget gives admins one cost control for both surfaces.

✍️ Databricks · Read article →

› Enterprise RAG & Retrieval

Snowflake release notes — November 2026

CORTEX_SEARCH_RETRIEVE table function is generally available

The CORTEX_SEARCH_RETRIEVE table function, previewed in October, is now GA: it runs low-latency Cortex Search queries from SQL, either single-shot or one query per input row via a LATERAL join. Retrieval becomes a composable relational operator rather than an API call outside the warehouse. That makes batch RAG enrichment and retrieval evaluation expressible as plain SQL pipelines.

✍️ Snowflake · Read article →

Blocks & Files — November 2026

Airbyte adds semantic search to its agent context layer

Airbyte added semantic search to its context layer so AI agents can search customer conversations, documents and tickets and combine the results with structured data in Snowflake and BigQuery, exposed through the Airbyte MCP server. No GA status was given. Ingestion vendors are repositioning as retrieval providers for agents, not just pipeline builders.

✍️ Chris Mellor, Blocks & Files · Read article →

↑ Top

 

Govern & Operate

› Orchestration & Workflow

BigQuery release notes — November 2026

BigQuery Studio Git repositories are generally available

BigQuery Studio's folder-based integration with remote Git repositories is GA, letting teams version SQL, notebooks and pipeline code against their existing Git hosts. Note the companion security bulletin below covering Git integration in the same surfaces. Proper Git workflows are a prerequisite for treating warehouse code as production software.

✍️ Google Cloud · Read article →

› Data Observability

BigQuery release notes — November 2026

Agent observability for BigQuery's Data Engineering Agent is GA

BigQuery's Data Engineering Agent now emits Cloud Monitoring metrics and Cloud Trace spans, generally available. Teams can monitor what the agent did, how long it took and where it failed using the same tooling as the rest of their stack. Observability for AI pipeline builders is becoming table stakes before they're trusted in production.

✍️ Google Cloud · Read article →

› Governance, Security & Compliance

BigQuery release notes — November 2026

Row- and column-level security with dynamic masking on lakehouse Iceberg tables (GA)

BigQuery now enforces row-level and column-level security with dynamic data masking on Iceberg tables in the Lakehouse runtime catalog, generally available; a new INFORMATION_SCHEMA.ROW_ACCESS_POLICIES view also went GA. Fine-grained policy on open-format tables is what makes a single governed Iceberg copy viable. Expect parity questions for other engines reading the same tables.

✍️ Google Cloud · Read article →

Snowflake release notes — November 2026

Snowflake previews an account-wide AI Governance page

A new Snowsight page under Governance & security surfaces AI usage metrics for Cortex Code and Cortex Agents, audits agent activity and data exposure, and keeps a changelog of agent changes (Preview). It gives governance teams one place to see what agents touched. Agent audit is becoming a native platform feature rather than something stitched from query history.

✍️ Snowflake · Read article →

Databricks release notes — November 2026

Service policies can now mask or deny sensitive data in embedding inputs (Beta)

Databricks service policies can now mask or block sensitive data in single or batch embedding inputs and reject unsafe content before the model runs. A companion beta makes output policies evaluate streaming responses incrementally by default — with the caveat that already-released text can't be recalled. PII controls are extending to the vectorization step, which is where much sensitive data quietly leaks into indexes.

✍️ Databricks · Read article →

Google Cloud security bulletin — November 2026

GCP-2026-084: path traversal in Dataform, BigQuery Studio and Colab Enterprise Git handling

Google disclosed a path-traversal issue in workspace directory handling and Git integration across Dataform, BigQuery Studio and Colab Enterprise. It lands the day after Studio's Git repositories went GA. Platform security teams should review the bulletin's remediation guidance and any repository connections their projects allow.

✍️ Google Cloud · Read article →

› FinOps for Data

Databricks release notes — November 2026

Configurable retention for system tables is generally available

Account admins can now set account-wide retention of 30 to 3,650 days for supported Databricks system tables; the first 395 days are free, with storage charges beyond that. Longer billing and audit history enables year-over-year cost analysis and longer compliance lookbacks without exporting to a separate store. The trade-off is now an explicit storage cost decision.

✍️ Databricks · Read article →

↑ Top

Compiled by Rainvil Labs — Friday, November 20, 2026
Sources verified via live web research on November 20, 2026, drawing on Snowflake, Databricks, Google Cloud BigQuery and Confluent Cloud release notes, the PostgreSQL project, Apache Iceberg and Arrow developer lists, DEV Community, and Blocks & Files. Several items are vendor release notes rather than independent coverage. This briefing is for informational purposes only and does not constitute legal, regulatory, or investment advice.