Daily Briefing — Friday, November 13, 2026

Data & AI Platforms Briefing

Agents are crossing from data consumers to platform operators: Genie now runs multistep jobs and ZeroOps proposes pipeline fixes, while deny policies on AI securables, native lineage, slot caps and budget webhooks race to keep that autonomy bounded.

⇣ Jump To

Click any section or topic below to jump to it.

Move & Transform

Streaming & Messaging ·  Transformation Frameworks ·  In-Process Compute

Store & Architect

Cloud Data Warehouses ·  Lakehouses ·  Table Formats ·  Query Engines

Consume & Activate

AI-Driven Consumption ·  Semantic Layers & Retrieval

Govern & Operate

Orchestration & Workflow ·  Data Observability ·  Data Contracts & Lineage ·  Governance, Security & Compliance ·  FinOps for Data

⚡ Quick Takes

Story Signal
↗  Redis Kafka connectors add Microsoft Entra ID authentication for Azure Managed Redis Keyless connector auth keeps spreading across managed Kafka Connect
↗  Standalone pipelines on serverless general compute are now generally available Declarative pipelines shrink to notebook-scale units
↗  AI Functions Asynchronous API enters Beta LLM extraction becomes a batch ETL primitive
↗  BigQuery JSON_FLATTEN function is generally available Less UNNEST boilerplate for semi-structured ELT
↗  ClickHouse release 26.10 Spill-to-disk closes another OOM gap in ClickHouse transforms
↗  DuckDB 2.0: Upgrade on Purpose, Before pip Does It for You Pin DuckDB versions now; 1.4 support ends Nov 17
↗  Vote opens on an arrow.parquet.file canonical extension type Arrow and Parquet negotiate who owns file-reference types
↗  BigQuery generative AI functions add gemini-4-hydrogen (GA) Warehouse inference now follows model releases within days
↗  BigQuery graph measures reach GA with multi-fact-table GRAPH_EXPAND Graph analytics folds into the warehouse
↗  Databricks to invest ~$3 billion across Europe and the Middle East Lakehouse vendors compete on regional presence and sovereignty
↗  Iceberg v4 Default Value Expressions vote passes Iceberg v4 takes on more schema semantics
↗  Edelbyte reels in $70M for its multipurpose data analytics platform Federated, vector-aware engines draw big early money
↗  Databricks' new AI agent tells CEO Ali Ghodsi the unvarnished truth Agentic BI depends on ontology plus gateway infrastructure
↗  Genie One connects to ChatGPT, Claude and Cursor via MCP plugins Governed data access shows up inside third-party assistants
↗  Semantic view materializations are generally available Semantic layers absorb aggregate management
↗  Many-to-many relationships in semantic views (Preview) Warehouse semantic models handle messier real-world joins
↗  Apache Airflow 3.4.0: Task Loops, Dags in Any Language, and AI Agents Orchestrators add bounded loops for agentic workflows
↗  Agents that build their own databases are forcing a data foundation rethink Ephemeral agent-built data stores outrun lineage tooling
↗  Genie ZeroOps is available in Beta Platform-native agents move into incident remediation
↗  Table-level lineage in Knowledge Catalog for BigQuery pipelines (GA) Native lineage becomes table stakes in managed pipelines
↗  ABAC DENY policies extend to AI securables and individual objects (Beta) Attribute-based policy now covers agents and MCP endpoints
↗  Budget alerts can now trigger webhooks (Beta) Budget signals become automation triggers
↗  BigQuery project caps per reservation are generally available Per-project slot caps contain noisy-neighbor and agent spikes

Move & Transform

› Streaming & Messaging

Confluent Documentation — November 2026

Redis Kafka connectors add Microsoft Entra ID authentication for Azure Managed Redis

Confluent Cloud's fully managed Redis Kafka Sink and Source connectors can now authenticate to Azure Managed Redis with a Microsoft Entra ID application, configured through an Azure Provider Integration instead of a shared access key. It is a small release-note entry, but it removes one more long-lived secret from streaming pipelines that feed or drain Redis caches. Teams standardizing on Entra for workload identity can bring these connectors into the same rotation and audit model as the rest of their Azure estate.

✍️ Confluent · Read article →

› Transformation Frameworks

Databricks Documentation — November 2026

Standalone pipelines on serverless general compute are now generally available

Databricks now lets engineers create and refresh standalone materialized views and streaming tables straight from a notebook on serverless general compute, defined in SQL via spark.sql() or with the Python pipelines decorators. The feature is GA as of November 11. It lowers the ceremony for declarative pipelines: an incremental table no longer needs its own pipeline object and dedicated compute to exist.

✍️ Databricks · Read article →

Databricks Documentation — November 2026

AI Functions Asynchronous API enters Beta

A new asynchronous API runs ai_parse_document, ai_extract or ai_classify over up to 50,000 requests supplied as a JSONL file in a Unity Catalog volume, writing one result per request back to a volume. It is in Beta behind a workspace-level preview toggle. For document-heavy ingestion, this turns LLM extraction into a batch job with file-in, file-out semantics rather than a long-running synchronous SQL query.

✍️ Databricks · Read article →

Google Cloud Documentation — November 2026

BigQuery JSON_FLATTEN function is generally available

JSON_FLATTEN extracts every non-array value that sits directly in a JSON input or under one or more consecutively nested arrays, returning them as flattened values. The function went GA on November 11. It replaces a common pile of UNNEST and JSON_QUERY_ARRAY boilerplate when normalizing semi-structured event payloads in SQL-first transformation layers.

✍️ Google Cloud · Read article →

› In-Process Compute

ClickHouse Blog — November 2026

ClickHouse release 26.10

ClickHouse 26.10 ships 104 new features, 225 performance optimizations and 739 bug fixes, headlined by a function composition operator (|) and a written_rows quota counter. IN subqueries can now spill to disk via max_bytes_before_external_set; in the post's own test, peak memory fell to 203 MiB under a 512 MiB limit. Spill-to-disk for set building matters for transform workloads that previously failed on large semi-joins.

✍️ ClickHouse · Read article →

The Data Guy George (Substack) — November 2026

DuckDB 2.0: Upgrade on Purpose, Before pip Does It for You

A practitioner's upgrade guide to DuckDB 2.0.0, an LTS release supported until at least October 2027. New database files use storage version 69, which no 1.x build can open, and DuckDB 1.4 loses community support on November 17, 2026. The author also notes that the Quack client-server mode is still beta and that MotherDuck had not yet supported 2.0 as of November 5, so pin versions before an unplanned pip upgrade breaks shared files.

✍️ George Yates · Read article →

Apache Arrow dev list — November 2026

Vote opens on an arrow.parquet.file canonical extension type

Arrow developers have opened a vote on a canonical FILE extension type, arrow.parquet.file, for referencing Parquet files from within Arrow data. The thread includes debate over whether Parquet-specific types should be owned by the Parquet project rather than Arrow. The outcome shapes how engines pass file-level references (for example, manifests or multimodal assets) through Arrow-native pipelines without custom metadata conventions.

✍️ Apache Arrow community · Read article →

↑ Top

 

Store & Architect

› Cloud Data Warehouses

Google Cloud Documentation — November 2026

BigQuery generative AI functions add gemini-4-hydrogen (GA)

BigQuery's generative AI functions now support the gemini-4-hydrogen model, GA as of November 12, two days after gemini-4-carbon landed on November 10. Databricks added the same Gemini 4 models to Unity Gateway in the same window. The cadence shows warehouse-resident inference tracking frontier model releases within days, which makes model version pinning a pipeline concern.

✍️ Google Cloud · Read article →

Google Cloud Documentation — November 2026

BigQuery graph measures reach GA with multi-fact-table GRAPH_EXPAND

Graph measures in BigQuery are now generally available, and GRAPH_EXPAND gains support for multiple fact tables. The release lets teams define aggregations over property-graph schemas directly in the warehouse instead of exporting to a separate graph engine. It is another sign that graph modeling is being absorbed into the SQL warehouse rather than living in a specialty store.

✍️ Google Cloud · Read article →

› Lakehouses

Databricks Newsroom — November 2026

Databricks to invest ~$3 billion across Europe and the Middle East

Databricks plans about $3 billion of investment across EMEA over three years, its largest regional commitment outside North America, aimed at Lakebase, Genie and Unity Gateway. The company says it has more than 2,000 employees in the region, expects that team to grow about 50%, and plans to train roughly 500,000 people in data and AI skills by 2030. Paired with its recent sovereignty commitments, it signals regional lakehouse capacity becoming a selling point for EU data-residency buyers.

✍️ Databricks · Read article →

› Table Formats

Apache Iceberg dev list — November 2026

Iceberg v4 Default Value Expressions vote passes

The Iceberg community closed its vote on default value expressions for the v4 spec with 11 binding and 7 non-binding +1 votes. The change extends column defaults beyond literals, so schema evolution can declare computed defaults in the table format itself. In the same week, the Iceberg materialized view spec was declared ready for broader review, per Alex Merced's Apache Data Lakehouse Weekly roundup.

✍️ Apache Iceberg community · Read article →

› Query Engines

SiliconANGLE — November 2026

Edelbyte reels in $70M for its multipurpose data analytics platform

Tel Aviv-based Edelbyte, founded in 2025 by former Azure Data Explorer executives, launched with $70 million across two rounds led by Sequoia and Quiet Capital, at a reported $500 million valuation. It pitches itself as a Snowflake and Databricks alternative that stores records and vectors side by side at hundreds of petabytes and federates queries to data where it already sits. The company claims multipetabyte queries under 100 milliseconds; that figure is its own and unverified.

✍️ Maria Deutscher · Read article →

↑ Top

 

Consume & Activate

› AI-Driven Consumption

TechCrunch — November 2026

Databricks' new AI agent tells CEO Ali Ghodsi the unvarnished truth

Databricks has updated Genie to run multistep jobs, grounding it in an ontology built from customer data, and CEO Ali Ghodsi says he uses it to surface problems in a 12,000-person company that leaders rarely hear about. Cost control runs through Unity Gateway, which applies usage limits and can route to cheaper open-weight models. For platform teams, the takeaway is that the ontology and the gateway, not the chat UI, are the parts they will end up owning.

✍️ Julie Bort · Read article →

Databricks Documentation — November 2026

Genie One connects to ChatGPT, Claude and Cursor via MCP plugins

New Databricks Genie plugins for ChatGPT and Grok Bot/Cursor, plus a Genie connector for Claude, expose the Genie One MCP server so users can ask governed data questions from those clients. A workspace or team admin installs the integration, and each user signs in with their own Databricks identity. Data access now arrives from third-party assistants, so per-user auth and Unity Catalog permissions carry the governance load.

✍️ Databricks · Read article →

› Semantic Layers & Retrieval

Snowflake Documentation — November 2026

Semantic view materializations are generally available

Snowflake semantic view materializations, which pre-aggregate frequently queried dimensions and metrics, are now GA and can include semi-additive metrics. A bundled CoCo skill recommends materializations, and SYSTEM$CHECK_HYPOTHETICAL_SEMANTIC_VIEW_MATERIALIZATION_REWRITE tests whether a planned one would be used before you build it. This gives the semantic layer its own aggregate-awareness, a job that BI tools or hand-built rollup tables used to do.

✍️ Snowflake · Read article →

Snowflake Documentation — November 2026

Many-to-many relationships in semantic views (Preview)

Semantic views can now model many-to-many relationships, either through a bridge table with two many-to-one links or directly on non-unique columns, for example orders(promo_code) REFERENCES promotions(promo_code). The feature is in Preview, and queries that traverse these relationships cannot use LOD metrics or multi-path USING metrics. It removes a common reason teams kept semantic logic in BI tools instead of the warehouse.

✍️ Snowflake · Read article →

↑ Top

 

Govern & Operate

› Orchestration & Workflow

Apache Airflow Blog — November 2026

Apache Airflow 3.4.0: Task Loops, Dags in Any Language, and AI Agents

Airflow 3.4.0 adds Task Loops (AIP-111), where .loop() requires a max_iterations cap and takes an optional until predicate, and lets whole Dags be written in TypeScript, Java or Go through the AIP-85 Dag Importer, with all three SDKs GA. The common.ai provider reaches GA at 1.1.0, while durable step-level replay, sandbox toolsets and code mode stay experimental. Bounded loops and polyglot Dags move Airflow toward agent-style iterative workflows without dropping its scheduling guarantees.

✍️ Rahul Vats · Read article →

› Data Observability

SiliconANGLE — November 2026

Agents that build their own databases are forcing a data foundation rethink

Speaking on theCUBE at KubeCon + CloudNativeCon NA, SanjMo analyst Sanjeev Mohan argued that observability becomes a prerequisite once AI agents spin up short-lived databases and route around governed pipelines. His example is a loan agent that pulls census and social-media data before rejecting an applicant, which leaves an audit trail no traditional lineage tool would capture. TheCUBE is a paid media partner of the event.

✍️ Jonathan Anthony · Read article →

Databricks Documentation — November 2026

Genie ZeroOps is available in Beta

Genie ZeroOps monitors jobs, pipelines and tables in the background, detects failures and data quality issues, investigates root cause and proposes a fix. It entered Beta on November 12. Databricks is putting autonomous remediation inside the platform, which puts pressure on standalone observability vendors whose differentiation has been root-cause analysis.

✍️ Databricks · Read article →

› Data Contracts & Lineage

Google Cloud Documentation — November 2026

Table-level lineage in Knowledge Catalog for BigQuery pipelines (GA)

Knowledge Catalog now tracks table-level data lineage for BigQuery pipelines, generally available as of November 12. Lineage is captured as a side effect of running pipelines rather than through a separate scanner or OpenLineage integration. Column-level coverage is not mentioned in the entry, so teams with fine-grained impact analysis needs should check before retiring other tooling.

✍️ Google Cloud · Read article →

› Governance, Security & Compliance

Databricks Documentation — November 2026

ABAC DENY policies extend to AI securables and individual objects (Beta)

Unity Catalog ABAC DENY policies can now target model services, model provider services, MCP services, agent services and skills, and can attach directly to individual tables, volumes, functions, models and secrets. The capability is in Beta. Putting agents and MCP endpoints under the same attribute-based deny model as tables gives governance teams one policy plane across data and AI assets.

✍️ Databricks · Read article →

› FinOps for Data

Databricks Documentation — November 2026

Budget alerts can now trigger webhooks (Beta)

Databricks budgets can now fire webhooks when spend thresholds are exhausted, so budget events can flow into incident, ticketing or automation tools. The feature is in Beta, and the release note gives no limits or payload details. It gives FinOps teams a hook to automate responses, such as pausing jobs, instead of relying on email alerts someone may not read.

✍️ Databricks · Read article →

Google Cloud Documentation — November 2026

BigQuery project caps per reservation are generally available

BigQuery project caps, also called scheduling policies, limit the maximum slots and concurrency any single project can consume within a shared reservation. They are GA for Enterprise and Enterprise Plus editions. This gives platform teams a guardrail against one runaway workload, increasingly an agent-driven one, starving everyone else on a pooled reservation.

✍️ Google Cloud · Read article →

↑ Top

Compiled by Rainvil Labs — Friday, November 13, 2026
Sources verified via live web research on November 13, 2026, covering items published November 6–13, 2026 that were not in prior briefings. Outlets used: Databricks, Snowflake, Google Cloud and Confluent documentation, Databricks Newsroom, TechCrunch, SiliconANGLE, the ClickHouse and Apache Airflow blogs, The Data Guy George (Substack), and the Apache Arrow and Iceberg mailing lists. Vendor-reported figures are noted as such. This briefing is for informational purposes only and does not constitute legal, regulatory, or investment advice.