Daily Briefing — Monday, November 16, 2026

Data & AI Platforms Briefing

Agents are moving from the chat window into the plumbing — repairing ETL records, writing back from Slack, and earning their own permission models — while AWS and Databricks keep hardening open tables as the transactional ground they stand on.

⇣ Jump To

Click any section or topic below to jump to it.

Move & Transform

Streaming & Messaging ·  CDC ·  ELT/ETL Ingestion ·  Transformation Frameworks

Store & Architect

Cloud Data Warehouses ·  Lakehouses ·  Table Formats ·  Architectural Patterns ·  Query Engines ·  Vector & Specialty Stores

Consume & Activate

AI-Driven Consumption ·  Enterprise RAG & Retrieval ·  BI & Analytics

Govern & Operate

Orchestration & Workflow ·  Governance, Security & Compliance

⚡ Quick Takes

Story Signal
↗  AWS maps five streaming paths into Iceberg Managed “streaming tables” trade upserts for cost — fine for events, wrong for CDC
↗  Confluent moves Salesforce CDC creds to secret managers Credential hygiene is becoming table stakes for managed CDC connectors
↗  AWS reference: agents that repair failed ETL records Self-healing pipelines arrive with confidence thresholds and PRs as the control surface
↗  Fivetran: 39% lose a quarter of AI budget to data fixes Integration debt is now being priced as an AI cost, not an IT cost
↗  Glue 6.0: native Arrow UDFs ~4.4x faster than Python UDFs Spark 4's arrow_udf is the new default for unavoidable Python logic
↗  Databricks bundles can now declare AI Function tasks LLM transforms are being folded into infrastructure-as-code
↗  Redshift adds multi-statement transactions on Iceberg Warehouse-grade DML semantics keep migrating onto open tables — one table at a time
↗  BigQuery federates to MySQL, Oracle, Postgres Federation keeps widening as warehouses court operational sources
↗  Databricks pushes Lakehouse//RT against real-time OLAP Agent-driven query concurrency is the new justification for lakehouse serving tiers
↗  AWS publishes a Hudi-to-Iceberg migration playbook Hudi estates are now being treated as migration candidates
↗  Databricks argues LTAP is tiering, not replication OLTP-on-the-lake is moving from pitch to architecture debate
↗  JSONBench grows to 27 engines; ClickHouse still on top Native JSON performance is becoming a first-class benchmark axis
↗  Pinecone adds Gemini Embedding 2 server-side Vector stores are absorbing the embedding step — and the model choice
↗  Genie in Slack goes GA with per-user UC permissions Chat-surface agents now write back — approval gates become a platform concern
↗  MotherDuck case study: agentic analysis, no data team Agents lower the SQL bar; domain validation stays the bottleneck
↗  Databricks frames OpenAI tools on top of Genie Ontology The semantic/context layer is becoming the portable asset across AI frontends
↗  Snowflake's CoCo converts Tableau workbooks to Streamlit AI-assisted BI migration shifts dashboard ownership to platform teams
↗  Snowflake DCM separates plan from apply Declarative schema management is adopting Terraform-style least privilege
↗  Unity Catalog makes pipeline flows securable Permissions now attach to the pipelines, not just the tables
↗  Databricks GA: act as a service principal without secrets Secretless impersonation cuts credential sprawl — audit the manager role
↗  Snowflake clean rooms can now persist results Clean rooms are evolving from one-shot queries to stateful workflows
↗  Palo Alto: most orgs can't reconstruct agent actions Agent auditability is the governance gap of the moment
↗  JetBrains CEO: bake agent governance into the workflow Policy documents are giving way to enforced, logged agent controls

Move & Transform

› Streaming & Messaging

AWS Big Data Blog — November 2026

Choose the right streaming ingestion path for Apache Iceberg tables on AWS

AWS lays out five ways to stream into Iceberg — native streaming tables fed by MSK Express brokers or Kinesis On-Demand, MSK Connect with the Iceberg sink, Data Firehose, Managed Flink, and Spark Structured Streaming on EMR — and scores them on eight criteria including upserts, schema evolution, table maintenance and fan-out. The new streaming tables are append-only with fixed hourly partitioning and 5–15 minute freshness, but AWS claims delivery costs up to 60% below a self-managed pipeline. Useful as a decision matrix: the cheapest path gives up upserts and partition control, which rules it out for CDC-shaped workloads.

✍️ Mazrim Mehrtens, Ayan Majumder (AWS) · Read article →

› CDC

Confluent Cloud Release Notes — November 2026

Secret manager integration extended to fully managed Salesforce connectors

Confluent Cloud's secret manager integration now covers eight fully managed Salesforce connectors, including the Salesforce CDC Source, Platform Event, PushTopic and SObject Sink connectors, for both CLIENT_CREDENTIALS and PASSWORD grant types. Connector credentials can now live in an external secret store rather than in connector config. A small change, but it closes a common audit finding on SaaS CDC pipelines where OAuth secrets sat in plain connector definitions.

✍️ Confluent · Read article →

› ELT/ETL Ingestion

AWS Big Data Blog — November 2026

Build a self-healing ETL pipeline with Amazon Bedrock AgentCore

A reference build in which an AWS Glue job validates records against 15 schema types and routes failures to a four-agent system on Bedrock AgentCore (Strands Agents SDK). The agents propose corrections using a Bedrock Knowledge Base and pre-computed S3 Vectors strategies, write fixes back to the lake and can open pull requests; below a 70% confidence threshold the orchestrator falls back to an SNS alert. It is a concrete template for the “agent repairs the pipeline” pattern — and a reminder that the confidence gate and the PR trail are where the governance lives.

✍️ Kevin Tinn, Utsab Roy (AWS) · Read article →

Fivetran — November 2026

The Cost of AI Without Connected Data: Nearly 4 in 10 Companies Spend 25% or More of AI Budgets Fixing Data Problems

Fivetran's survey of 500 senior IT, data and AI/ML leaders across the US, Europe and APAC finds 39% spend at least a quarter of their AI budget fixing data problems — what it calls the “AI tax.” Firms in the highest data-cost group are 5x as likely to have a quarter or more of AI projects delayed or stalled, and 66% cite data quality or integration as a leading source of unexpected work. Vendor-sponsored and self-reported, but it gives platform teams a usable number when arguing that ingestion and integration debt is an AI budget line.

✍️ Fivetran (press release) · Read article →

› Transformation Frameworks

AWS Big Data Blog — November 2026

Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

AWS benchmarks five UDF strategies on Glue 6.0 (Spark 4.1.1, 10 G.1X workers) at 10M–100M rows: standard Python, Arrow-optimized Python, vectorized pandas, the new native arrow_udf, and plain Spark SQL. Spark SQL wins every case, but native Arrow UDFs are the fastest UDF option — 3.13s vs 13.82s for a standard Python UDF on a math-heavy 100M-row workload, about 4.4x. The practical takeaway for Spark 4 migrations: rewrite to SQL where you can, and move remaining Python UDFs to arrow_udf rather than pandas UDFs.

✍️ Iván Fernández Perea (AWS) · Read article →

Databricks Release Notes — November 2026

AI Function task support in Declarative Automation Bundles (Beta)

Databricks' Declarative Automation Bundles can now define AI Function tasks, in Beta. That means LLM-powered transforms — classification, extraction, summarization via AI Functions — can be declared, versioned and deployed alongside the rest of a job's code. It pulls AI-in-SQL steps into the same CI/CD path as conventional transforms instead of leaving them as ad hoc notebook cells.

✍️ Databricks · Read article →

↑ Top

 

Store & Architect

› Cloud Data Warehouses

AWS Big Data Blog — November 2026

Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

Redshift (Patch 206+) now supports BEGIN…COMMIT blocks of INSERT, UPDATE, DELETE and MERGE against an Iceberg table, published as a single atomic snapshot; a rollback publishes nothing, though all writes in a transaction must target the same table. AWS reports batching 25 INSERTs per commit cut runtime by 14.77%, and at 32 concurrent writers four-statement transactions reached 10.37x the single-writer rate versus 2.89x with auto-commit. Single-table scope is the catch — cross-table atomicity on Iceberg still needs catalog-level transactions.

✍️ Shiyu Xu et al. (AWS) · Read article →

BigQuery Release Notes — November 2026

BigQuery connections to MySQL, Oracle, and PostgreSQL (Preview)

BigQuery can now create connections to MySQL, Oracle and PostgreSQL and run federated queries against them, in Preview. Extending federation beyond Cloud SQL and Spanner to Oracle in particular lets teams join operational data in place during migrations or for low-volume lookups. Treat it as a bridge rather than an ingestion strategy: federated scans push load onto the source database.

✍️ Google Cloud · Read article →

› Lakehouses

Databricks Blog — November 2026

Lakehouse//RT: Scale real-time analytics, not your bill

Databricks pitches Lakehouse//RT, now in beta, as a replacement for separate real-time serving stacks, aimed at query volume driven increasingly by AI agents. It cites sub-100 ms latency at 12,000 QPS from the launch benchmark, and customer-reported 6–10x lower infrastructure cost; one customer moved from several clusters at 200 QPS to a single small warehouse above 250 QPS. If the numbers hold, it is a direct challenge to ClickHouse, Pinot and Druid deployments that exist only to serve high-concurrency queries off the lake.

✍️ JooHo Yeo, Kaitlin Baumgardner (Databricks) · Read article →

› Table Formats

AWS Big Data Blog — November 2026

Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

A step-by-step guide for moving Hudi Copy-on-Write and Merge-on-Read tables to Iceberg on S3, via in-place paths (Iceberg add_files or Apache XTable) or a full CTAS rewrite, with S3 Tables as a target and validation in Athena. It flags a sharp edge: Iceberg 1.9.0–1.10.0 omit column statistics during add_files/snapshot, fixed in 1.10.1, so AWS recommends Glue 6.0+ or EMR 7.14+ for production. That AWS is publishing Hudi exit playbooks says a lot about where the table-format contest has settled.

✍️ Nitin Kumar et al. (AWS) · Read article →

› Architectural Patterns

Databricks Blog — November 2026

Dispelling LTAP myths

Databricks engineer Sergey Melnik defends the Lakebase LTAP design, in which each table keeps one durable copy in the lake as Delta or Iceberg, with Postgres caches and pageservers in front for latency. He argues this is tiering, not replication — no layer has its own catalog or can be failed over to as a separate database — and rebuts claims that one copy means slow point lookups or that open table formats imply eventual consistency. Worth reading for anyone weighing whether operational Postgres and the analytical lake can really share storage.

✍️ Sergey Melnik (Databricks) · Read article →

› Query Engines

ClickHouse Blog — November 2026

JSONBench is back, and ClickHouse still leads

ClickHouse's JSONBench now compares 27 engines (up from five at its January 2025 debut) on a billion Bluesky JSON documents on identical hardware. ClickHouse says it has shipped 178 changes to its native JSON type since, and returned a cold count-by-event-type query in 661 ms versus 32 minutes 26 seconds for MongoDB. Vendor-run, so read the methodology, but semi-structured analytics without upfront flattening is now a real engine-selection criterion.

✍️ Tom Schreiber, Mark Needham (ClickHouse) · Read article →

› Vector & Specialty Stores

Pinecone Blog — November 2026

Pinecone integrated embedding now supports Google's Gemini Embedding 2

Pinecone's integrated embedding on the Documents API now supports Gemini Embedding 2, so clients send plain text and Pinecone embeds both writes and queries server-side. It is limited to serverless Documents API indexes (not BYOC, pod-based or Records API), with Python SDK 10.3.0 support and Node.js/Go pending; the Builder plan includes 10M tokens a month, doubled to 20M through December 31. Hosted embedding simplifies pipelines but couples your vectors to the store's model menu — a lock-in question worth asking before re-indexing.

✍️ Gavin Johnson, Manish Talreja (Pinecone) · Read article →

↑ Top

 

Consume & Activate

› AI-Driven Consumption

Databricks Blog — November 2026

Genie in Slack is now Generally Available: Governed AI cowork, right where work happens

Genie in Slack is GA: users DM Genie or @mention it in channels, and every answer runs under the asker's Unity Catalog and source-system permissions. Through MCP connectors Genie can also update documents, comment on tickets and change records, asking for approval before each write, and it can deliver scheduled analyses to a DM. For platform teams the point is that identity propagation, not the chat UI, is what makes this safe to roll out.

✍️ Toussaint Webb, Sydney Sundell (Databricks) · Read article →

MotherDuck Blog — November 2026

Business Knowledge Is Still King: Agentic Data Analysis Without a Data Team

Simon Späti walks a non-SQL domain expert through building a pipeline and live dashboard from Swiss weather and Swissgrid electricity data using Claude and MotherDuck. The domain insight mattered: a consumption-weighted temperature across 22 stations improved fit (R² 0.924 to 0.936) and moved the heating threshold to about 14 °C, while the workflow caught a date-parsing bug that would have dropped 61% of 2025 rows. A grounded case study for how far agentic analysis gets without a data team — and where validation still has to come from someone who knows the data.

✍️ Simon Späti (MotherDuck) · Read article →

› Enterprise RAG & Retrieval

Databricks Blog — November 2026

OpenAI and Databricks: Enterprise AI on your governed data

Databricks outlines three patterns from its multi-year OpenAI partnership (signed September 2025): ChatGPT users querying via the Genie plugin, Codex coding through Unity Gateway, and custom GPT-based agents in Agent Bricks or Databricks Apps. Genie Ontology is positioned as the context layer that hands ChatGPT the company's metric definitions. The architecture story is that the governed context layer, not the model, is the part enterprises keep when they swap frontends.

✍️ Harish Gaur et al. (Databricks) · Read article →

› BI & Analytics

Snowflake Release Notes — November 2026

Convert Tableau workbooks to Streamlit in Snowflake apps with CoCo (Preview)

A new Cortex Code skill, in Preview, converts a Tableau workbook into a multipage Streamlit in Snowflake app with one page per dashboard. The generated app is first reviewed against synthetic data, and CoCo asks for approval before touching real tables; it ships with CoCo CLI 1.2.33+, CoCo Desktop 1.23.1+ and CoCo in Snowsight. It is a migration play aimed squarely at BI licence spend — and moves dashboard logic into code the platform team will end up owning.

✍️ Snowflake · Read article →

↑ Top

 

Govern & Operate

› Orchestration & Workflow

Snowflake Release Notes — November 2026

DCM Projects PLAN privilege (Preview)

Snowflake's Database Change Management projects gain a PLAN privilege, in Preview, letting a role run EXECUTE DCM PROJECT … PLAN without holding OWNERSHIP. That separates “show me the diff” from “apply the change,” so CI jobs and reviewers can preview declarative schema changes with least privilege. It is the plan/apply split Terraform users expect, now in the warehouse's native change tooling.

✍️ Snowflake · Read article →

› Governance, Security & Compliance

Databricks Release Notes — November 2026

Flows are securable objects in Unity Catalog (Beta)

Pipeline flows become first-class Unity Catalog securables in Beta, with READ FLOW, REFRESH FLOW, WRITE FLOW, APPLY TAG and MANAGE privileges plus CREATE FLOW at the schema level (enrolment via the account team). Teams can now grant someone the right to refresh a flow without granting write access to the tables behind it. Governance is moving from data objects down to the pipeline units that produce them.

✍️ Databricks · Read article →

Databricks Release Notes — November 2026

OAuth on-behalf-of for service principals is generally available

Users holding the Service Principal Manager role can now use the standard U2M OAuth flow to act as a service principal without an OAuth secret; the capability is GA and on by default. It removes a common reason to mint and distribute long-lived service-principal secrets for testing and automation. Security teams should note the default-on status and review who holds the manager role.

✍️ Databricks · Read article →

Snowflake Release Notes — November 2026

Snowflake Data Clean Rooms updates (API 19.4)

Clean Rooms API 19.4 lets a SQL analysis template save its output to a table inside the clean room via save_mode and name in the analysis spec, with saved results not visible to other collaborators. VIEW_UPDATE_REQUESTS now also shows the specific reason the automated security scan rejected a code spec. Persisted results make multi-step clean-room workflows possible; the scan-reason detail makes the review loop far less opaque.

✍️ Snowflake · Read article →

SiliconANGLE — November 2026

Palo Alto Networks report finds organizations struggle to track and secure AI agents in the cloud

Palo Alto Networks' State of Cloud Security Report 2026, from 2,800 security leaders and practitioners in 10 countries, finds AI agents nearly universal in development pipelines while many organizations cannot reconstruct what an agent actually did. On average 19% of high and critical application security issues reached production over 90 days, with fixes taking 18.4 days. For data platforms the lesson is direct: agent actions on data need the same lineage and audit trail as human ones.

✍️ Duncan Riley (SiliconANGLE) · Read article →

InfoWorld — November 2026

How AI agents are changing the rules of software governance

JetBrains CEO Kirill Skrygan argues governance has to move from written policy into controls built into the workflow, because coding agents now work across repositories, run tests and open pull requests. He calls for audit trails recording what an agent changed, what instructions it received and which tests it ran, plus scoped permissions and explicit human approval for critical systems. An opinion piece, but the same blueprint applies to agents touching pipelines and production data.

✍️ Kirill Skrygan (InfoWorld) · Read article →

↑ Top

Compiled by Rainvil Labs — Monday, November 16, 2026
Sources verified via live web research on November 16, 2026, covering items published November 12–16 (prior briefing: November 13). Outlets used: AWS Big Data Blog, Databricks Blog and Release Notes, Snowflake Release Notes, BigQuery Release Notes, Confluent Cloud Release Notes, Fivetran, ClickHouse, Pinecone, MotherDuck, SiliconANGLE, InfoWorld. Vendor benchmarks and survey figures are as reported by the publisher. This briefing is for informational purposes only and does not constitute legal, regulatory, or investment advice.