Daily Briefing — Wednesday, September 9, 2026
Agent workloads are forcing the platform to grow new load-bearing parts — backup for Iceberg tables, a policy plane above every vendor's agent gateway, context layers bolted onto operational databases — while the engines underneath quietly trade determinism and bespoke code for speed and upstream defaults.
⇣ Jump To
Click any section or topic below to jump to it.
Streaming & Messaging · ELT/ETL Ingestion · Stream Processing · In-Process Compute
Cloud Data Warehouses · Table Formats · Query Engines
Orchestration & Workflow · Data Observability · Catalogs & Metadata · Governance, Security & Compliance · FinOps for Data
⚡ Quick Takes
| Story | Signal |
|---|---|
| ↗ Data Streaming Landscape Q3 2026: Who Controls Your Streams | Sovereignty and consolidation now outrank features in streaming platform choice. |
| ↗ Reverse ETL to Attio | Reverse ETL shifts from connector catalogs to agent-generated custom destinations. |
| ↗ Fivetran Release Notes — September 2026 | Custom-connector debugging becomes a first-class managed-ELT feature. |
| ↗ Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs | Upstream autoscaling is now good enough to retire bespoke platform code. |
| ↗ Polars 2.0 pre-release comes with a 5x speed boost — but it could change row order | Default-streaming dataframes trade determinism for speed — audit your joins. |
| ↗ Amazon Redshift rg.large instances now support single-node clusters | Warehouse and lake converge on one engine; entry price drops. |
| ↗ Simplify pipelines with new BigQuery identity columns | Cloud warehouses backfilling classic RDBMS ergonomics for modelling work. |
| ↗ Rubrik extends data protection to Apache Iceberg tables on AWS | Iceberg is now production-critical enough to need real backup tooling. |
| ↗ Measuring real-time performance per dollar under continuous load: CostBench’s first end-to-end results | Benchmarks shifting from raw latency to cost under sustained load. |
| ↗ Introducing chDB Postgres extension: high-performance imports from cloud storage | Embedded OLAP engines are eating the lightweight ingest tier. |
| ↗ RavenDB Debuts Quill to Connect AI Agents with Enterprise SQL Data | Context layers arrive as retrofit middleware, not migration projects. |
| ↗ How MCP Toolbox turns agent text into ClickHouse vectors | Vector search keeps collapsing into general-purpose analytical engines. |
| ↗ Kestra 2.0: a new engine, workers anywhere, and flows as agent tools | Orchestrators repositioning as the governed control plane for agent workloads. |
| ↗ Apica Gives Customers AI-Assisted Control Across Telemetry Infrastructure | Agents inflate telemetry volume; pipeline control turns conversational. |
| ↗ DataHub: Context Engineering Leaders Are 4 Times More Likely to Lead in AI | Metadata catalogs rebranding as enterprise context management infrastructure. |
| ↗ Trust3 AI Extends Agent Security Controls to Databricks Unity AI Gateway | Agent policy fragments per platform; a control plane above gateways emerges. |
| ↗ VAST Data and CrowdStrike Take Aim at a Growing AI Security Problem | Endpoint security vendors are moving into the data pipeline layer. |
| ↗ Quantum Launches Autonomous Data Management to Cut Storage Costs and Risk | Classification becomes a cost-optimisation lever, not just a compliance one. |
Kai Waehner — September 2026
Waehner redraws the streaming market map on two axes — operational versus analytical workload, and who controls where the platform actually runs — arguing deployment model is no longer a vendor property and sovereignty has become a structural buying criterion. IBM closed the Confluent acquisition in March and now owns Confluent Cloud, Platform, Private Cloud, WarpStream and Red Hat’s Strimzi-based Kafka, with Jay Kreps stepping back in August; Databricks Zerobus and Snowflake Datastream appear for the first time by speaking the Kafka wire protocol. He also pushes back on “the topic becomes the table” — zero-copy is rare, and storing twice is usually the better pattern.
✍️ Kai Waehner · Read article →
dltHub — September 2026
dltHub walks a three-stage go-to-market pipeline — REST ingestion into DuckDB, People Data Labs enrichment, then reverse ETL into Attio — entirely scaffolded by its AI Harness agent rather than hand-written. The interesting part is destination semantics: Attio’s domains field is unique so companies upsert via PUT, but people have no unique key, so the agent generated a PATCH-or-POST fallback on LinkedIn URL plus a six-line guard that reads prior DuckDB output to avoid re-spending enrichment credits. The case for @dlt.destination is precisely the irregular destination that no connector catalog covers.
✍️ Aman Gupta · Read article →
Fivetran — September 2026
The September changelog ships connector-SDK 2.12.0 with crash reports in fivetran debug — capturing exception, handler, config and SDK version — plus per-connector memory logging, a real step up in custom-connector observability. S3-Compatible Storage goes GA, Workday RaaS gains a WQL query mode for paginated incremental syncs of oversized reports, and BigQuery large-row handling now syncs rows under roughly 100 MB untruncated instead of applying a blanket 8 MB per-column cap. Row filtering extends to Float, Double and BigDecimal.
✍️ Fivetran · Read article →
InfoQ — September 2026
Netflix is retiring its homegrown cluster-level Flink autoscaler in favour of the Apache community’s operator-level autoscaler across more than 30,000 streaming jobs in multiple AWS regions, because the in-house version could not reason about complex stateful pipelines. One team reported a 58% cut in annualised Flink compute spend, roughly $1.1M a year. Because Netflix runs its own control plane rather than the Flink Kubernetes Operator, the community carved the autoscaler’s core logic into a standalone library that Netflix wraps in a Spring Boot service orchestrated by Temporal.
✍️ Leela Kumili · Read article →
The New Stack — September 2026
The first Polars 2.0 release candidate makes the streaming engine the default for collect() on any LazyFrame — “easily 5x faster” in aggregate with large memory savings — but that engine gives no row-order guarantee for join, group_by and unpivot, so pipelines quietly relying on incidental ordering will break. Mitigations are explicit sorts, maintain_order=True, or pinning engine affinity back to in-memory. The RC also previews a new IO-plugin design, a faster S3 reader, a cost-based planner with join reordering, and wider SQL coverage.
✍️ Meredith Shubel · Read article →
AWS What’s New — September 2026
Redshift’s Graviton-based rg.large instances can now run as single-node clusters on patch P204 or later, giving proofs of concept and test workloads a cheap footprint without paying for HA they don’t need. AWS claims the RG generation is up to 2.4x faster than previous-generation RA3 at 30% lower price per vCPU, and RG nodes carry Redshift’s vectorised data lake query engine that reads Iceberg and Parquet on the cluster nodes themselves. Available in 29 regions including both GovCloud partitions.
✍️ Amazon Web Services · Read article →
Google Cloud Blog — September 2026
BigQuery now supports identity columns that auto-generate sequential 64-bit integers, restoring a relational primitive that warehouse teams have been hand-rolling for years via row_number windows, UUID hashing or external sequence services. It is a small feature with a disproportionate effect on dimensional-modelling code, and it removes a class of key-collision bug that shows up under concurrent loads.
✍️ Wawrzek Hyska · Read article →
SiliconANGLE — September 2026
Rubrik announced Apache Iceberg Protection — backup and recovery for tables registered in AWS Glue Data Catalog and Amazon S3 Tables, with immutable copies held in the customer’s own AWS account. The failure mode it names is the right one: Iceberg snapshots are metadata pointers, not copies, so a table deleted, ransomwared or overwritten by a misfiring agent loses its snapshot history along with it. After an initial full copy Rubrik captures only the latest compacted snapshots to keep backup windows viable at petabyte scale, and restores can land on an isolated branch for validation before promotion. GA later this month.
✍️ Duncan Riley · Read article →
ClickHouse — September 2026
ClickHouse publishes first end-to-end results from CostBench, a benchmark built to measure real-time analytics performance per dollar under sustained concurrent load rather than the cold single-query timings most vendor benchmarks report. The framing is the contribution: continuous ingest plus concurrent querying is the actual shape of production real-time workloads, and price-performance under that load is what lands on the invoice. Vendor-authored, so read the methodology before quoting the numbers.
✍️ Tom Schreiber and Lionel Palacin · Read article →
ClickHouse — September 2026
ClickHouse released a Postgres extension built on chDB, its in-process engine, letting Postgres read and import directly from cloud object storage at columnar-engine speed with no external ETL hop. It mirrors DuckDB’s embedded-analytics play but with ClickHouse execution, aimed at the common case where an operational Postgres needs to pull large Parquet or object-store datasets. Worth a look for anyone maintaining a staging pipeline whose only job is moving files into Postgres.
✍️ David Wheeler · Read article →
BigDATAwire — September 2026
RavenDB launched Quill, a governed context layer that sits on top of existing PostgreSQL, SQL Server or MySQL systems, stays synchronised via change data capture, and ships the whole retrieval stack — search, retrieval, agent runtime — without migrating the system of record or standing up a separate vector database. Agent scopes are enforced in the layer independently of source-database permissions, so what an agent can see becomes a configuration choice rather than a custom security project. RavenDB cites Gartner findings that only 28% of AI initiatives report positive ROI and pins the failure on integration and governance rather than model quality.
✍️ RavenDB · Read article →
ClickHouse — September 2026
A walkthrough of using MCP Toolbox to take raw agent-generated text, embed it, and land the vectors directly in ClickHouse so semantic retrieval runs against the same store already holding the analytical data. It is a concrete instance of the “no separate vector database” pattern — vector columns and similarity search as features of the OLAP engine rather than a bolted-on specialty store. For platform teams the win is one fewer sync path between warehouse and vector store.
✍️ Pete Hampton · Read article →
Kestra — September 2026
Kestra 2.0 rebuilds the execution engine to decouple workers from the database, letting them run on any network, and adds governance primitives — Policies, Cases, Promote and flow Drafts — plus MCP tool triggers that expose orchestrated flows directly to AI agents. Kestra reports sustained throughput doubled to roughly 4,000 executions per minute on the same Postgres with a flat p99, and a new Log Data Store can route logs to JDBC, Elasticsearch, Datadog or Splunk separately from the rest of the metadata. Still Apache 2.0.
✍️ Kestra · Read article →
BigDATAwire — September 2026
Apica Ascent 3.0 adds Venn, a persistent AI assistant across Flow, Observe and Fleet that lets teams author pipeline rules, manage fleet agents and configure alerts in natural language, plus a built-in MCP server so customers can drive the same pipelines from their own AI tooling. Any change that would delete or materially alter configuration still requires administrator approval. A new flow-only mode skips Lake storage overhead and moves telemetry up to 10x faster in Apica’s benchmarks — framed against agent workloads pushing telemetry volumes as much as 9.5x higher.
✍️ Apica · Read article →
BigDATAwire — September 2026
A BARC study of 285 data, AI, IT and business stakeholders, sponsored by DataHub, found organisations with mature context engineering programmes are four times more likely to qualify as AI leaders — 49% of “context leaders” versus 12% of everyone else. BARC scores six elements: data integration, workflow orchestration, retrieval methods, federated metadata, prompt engineering and the semantic layer; data quality and preparation led the challenge list at 49%, ahead of model limitations at 29% and governance gaps at 25%. Notably 44% still manage context inside a single agent, team or platform rather than across the enterprise.
✍️ DataHub / BARC · Read article →
BigDATAwire — September 2026
Trust3 AI, founded by the creators of Apache Ranger, extended its agent security control plane into the Databricks Unity AI Gateway with purpose-based access control, agent discovery and profiling, and a continuously assessed Trust Score that gates what an agent may do at runtime. The same policy graph already covers Azure AI Gateway, Snowflake Cortex AI Gateway and Bedrock AgentCore Gateway, with enforcement pushed down to each platform’s own gateway via plugins. The premise: every data platform now ships its own agent gateway, so enterprises need one policy layer above them rather than four rulebooks.
✍️ Trust3 AI · Read article →
BigDATAwire — September 2026
VAST Data and CrowdStrike deepened their partnership to secure the proprietary data feeding models and agents, not just the infrastructure underneath, with native Falcon sensor support on the VAST AI Operating System certified and generally available now. Integrations with Falcon Next-Gen SIEM and Falcon Guardian — CrowdStrike’s new AI detection and response product — are in private preview. The stated goal explicitly includes tracking who or what touches enterprise data as it moves through the pipelines preparing it for AI.
✍️ BigDATAwire · Read article →
BigDATAwire — September 2026
Quantum released Autonomous Data Management, which profiles inactive, duplicated, outdated, orphaned and risk-sensitive data across environments and then continuously automates placement — what stays on primary storage, what is archived or deleted, what needs extra protection or governance. The framing is cost and risk together: organisations scaling AI and analytics are paying to store and protect data they have no visibility into. Claimed outcomes are reclaimed primary capacity, lower backup and power cost, and less cyber exposure from stale data.
✍️ Quantum Corporation · Read article →