Daily Briefing — Friday, August 28, 2026
The data infrastructure layer is being bought, standardized, and governed all at once — AWS moves to acquire the company behind DuckDB, Iceberg hardens V4 from roadmap into wire format, and catalogs quietly rebrand themselves as context layers for agents.
⇣ Jump To
Click any section or topic below to jump to it.
Streaming & Messaging · ELT/ETL Ingestion · Stream Processing · In-Process Compute
Lakehouses · Table Formats · Query Engines · Specialty Platforms
AI-Driven Consumption · Enterprise RAG & Retrieval · Reverse ETL & Activation
Orchestration & Workflow · Data Observability · Catalogs & Metadata · Governance, Security & Compliance
⚡ Quick Takes
| Story | Signal |
|---|---|
| ↗ The Apache Software Foundation Announces New Top-Level Projects | Rust log brokers graduate from experiment to governed infrastructure. |
| ↗ Fabric August 2026 Feature Summary | Fabric quietly fills its operational-technology ingestion gap. |
| ↗ How Netflix Scaled Its Real-Time Service Map | Backpressure to Kafka beats load-shedding for correctness-critical streams. |
| ↗ DuckLabs to Join AWS, Projects to Remain Open Source | Hyperscalers are now buying the in-process compute layer itself. |
| ↗ Announcing Polars 1.44 | Single-node engines close the lakehouse write-path gap. |
| ↗ How DuckDB Runs Recursive CTEs Faster | Keyed recursion makes in-SQL graph traversal viable at scale. |
| ↗ Building for the AI Era: Lakebase, Streaming, and Lakehouse Innovations at VLDB 2026 | Automated layout beats hand-picked clustering keys 95% of the time. |
| ↗ Build with geospatial and variant types in Iceberg v3 on AWS Glue 6.0 | Iceberg v3 types kill the lat/long-and-JSON-string workaround. |
| ↗ Apache Data Lakehouse Weekly: August 19 to 26, 2026 | Iceberg V4 stops being roadmap and becomes wire format. |
| ↗ How Suprema Gaming made its data platform agent-ready with ClickHouse Cloud | Agents need canonical real-time data or they invent numbers. |
| ↗ August 2026 Announcements (Foundry) | Palantir's Ontology moves into source control as TypeScript. |
| ↗ Introhive Makes Relationship Intelligence Available to Enterprise AI Through MCP | Model-agnostic graph access beats per-assistant custom integrations. |
| ↗ Why basic RAG fails at multi-hop reasoning (and how GraphRAG fixes it) | Retrieval quality is a data modeling problem, not embeddings. |
| ↗ Dun & Bradstreet Brings the D&B Commercial Graph to Perplexity | Data vendors ship MCP connectors instead of bulk feeds. |
| ↗ AWS Introduces Specification Driven Composition for Flexible Data Workflows | Declarative specs cut dataset onboarding from weeks to days. |
| ↗ Grafana Labs Crosses 10,000-Customer Milestone as AI Adoption Accelerates Growth | Open telemetry stack encroaches on dedicated data observability. |
| ↗ Announcing OpenMetadata 2.0: The Open Context Layer for AI Agents | Catalogs rebrand as context layers with MCP and graphs. |
| ↗ Nasuni Acquires DryvIQ to Expand Unstructured Data Intelligence and Governance | Storage vendors annex unstructured data classification and policy. |
| ↗ What’s new in Microsoft Security: August 2026 | Purview auto-labeling throughput jumps fivefold to 500k files daily. |
AIwire — August 2026
The ASF graduated Apache Iggy — a Rust-based persistent message streaming platform — to Top-Level Project status alongside Apache Sourcelume. Iggy runs a thread-per-core, shared-nothing design on Linux io_uring, claims millions of messages per second and hundreds of TB/day from a single node at single-digit-millisecond tail latency, and speaks TCP, QUIC, HTTP and WebSockets natively rather than one binary protocol. It is now a governance-credible option for latency-sensitive event transport where a full Kafka footprint is overkill.
✍️ The Apache Software Foundation · Read article →
Microsoft Fabric Community — August 2026
Microsoft's monthly Fabric drop spans OneLake, Data Factory and Real-Time Intelligence, with the ingestion-relevant items being a redesigned Eventstream editor with “Always Publish” and GA-readiness for the Eventstream MQTT connector supporting TLS, mTLS and custom CAs managed through Azure Key Vault. The MQTT path closes an OT/IoT ingestion gap that previously demanded a bridge service. Worth attention if you are sizing up Fabric as the ingestion tier rather than only the warehouse.
✍️ Microsoft Fabric team · Read article →
InfoQ — August 2026
Netflix rebuilt the streaming pipeline behind Service Topology into three stages — batching, intermediary resolution, then enrichment and persistence — on Apache Pekko Streams, so backpressure propagates upstream until the Kafka consumer pauses rather than dropping records. They also swapped gRPC between stages for server-sent events once serialization and connection-pool overhead turned expensive at volume. The explicit trade is delayed freshness under load over an incomplete map — the right default for any pipeline feeding incident tooling.
✍️ InfoQ · Read article →
DuckDB Blog — August 2026
DuckLabs, the Amsterdam company behind DuckDB, is being acquired by AWS with the deal expected to close in early September; both founders stay on to lead technical direction. The DuckDB project itself is not part of the transaction — it remains MIT-licensed under the independent DuckDB Foundation. Anyone who has pushed DuckDB into transform layers, local dev, or embedded pipeline compute now has a vendor-alignment question to answer even though the license does not change.
✍️ Hannes Mühleisen and Mark Raasveldt, DuckLabs · Read article →
Polars Blog — August 2026
Polars 1.44 lands Iceberg schema evolution on sink, an adaptive rate limiter for cloud object-store I/O, and correlated subqueries in the SQL layer. The Iceberg sink work matters most: schema drift no longer forces a manual table rewrite when Polars is the writer in a lakehouse transform step. Paired with the rate limiter, this positions Polars as a credible single-node transform engine against S3/GCS rather than a pandas replacement.
✍️ Polars · Read article →
DuckDB Blog — August 2026
A deep dive on DuckDB's USING KEY extension for recursive CTEs, which treats the union table as keyed state rather than an append-only working set, letting rows be replaced in place across iterations. That turns graph and hierarchy traversals — shortest path, bill-of-materials, lineage walks — from repeated full-set materialization into bounded state updates. Directly relevant if you are pushing lineage or dependency resolution into SQL instead of an external graph service.
✍️ DuckDB · Read article →
Databricks — August 2026
Databricks previews its four VLDB 2026 papers plus the Enzyme demo and a sponsor talk on LakehouseRT, Lakebase and LTAP (Lake Transactional Analytical Processing). The numbers are unusually specific: AutoLiquid's CLUSTER BY AUTO beat customer-chosen clustering keys on over 95% of evaluated workloads, Ultron's history-based optimizer cut median join latency 25%, and Structured Streaming microbatch pipelining lifted throughput up to 3x. This is the engineering substrate behind the “one copy of lake storage for OLTP and OLAP” claim.
✍️ Indrajit Roy and Ippokratis Pandis · Read article →
AWS Big Data Blog — August 2026
A worked fleet-telemetry build exercising four Iceberg v3 column features in a single table on Glue 6.0 with Spark 4.1: GEOMETRY(4326) with ST_Intersects geofence predicates, TIMESTAMP_NTZ(9) nanosecond ordering, VARIANT with variant_get typed extraction, and DEFAULT column values, all over merge-on-read deletion vectors. Two sample events one nanosecond apart are correctly sequenced where microsecond precision would tie them. Geospatial needs an explicit flag; the rest works unconfigured and stays readable by any v3-capable engine.
✍️ Shoukat Ghouse, Shrey Malpani and Kartik, AWS · Read article →
DEV Community — August 2026
A dev-list digest of Iceberg, Polaris, Parquet, Arrow and DataFusion activity, with V4 moving from principles to wire format. Concrete items: the apache/iceberg-verification conformance-fixture repo passed its vote with five binding +1s, the equality-delete deprecation vote closed forbidding new equality-delete writes in V4, and the IRC thread is converging on a versioned loadTable endpoint rather than stuffing V4 fields into the v1 response. Teams running streaming upserts into Iceberg need an exit plan from equality deletes; catalog operators need to plan a second loadTable endpoint.
✍️ Alex Merced · Read article →
ClickHouse — August 2026
A Snowflake-to-ClickHouse Cloud migration written up as an architecture case study for agent-facing analytics: warehouse spend down roughly 62%, mart reads measured 27x faster, and freshness moved from a four-hour dbt rebuild cycle to roughly one-minute p95 lag via ClickPipes CDC plus Debezium and Kafka. Every migrated domain was validated to the cent against the reference source before cutover, with the validation rules captured as version-controlled dbt tests. The governance angle is the interesting one — agents are treated as ordinary governed consumers with RBAC and query-time PII masking, not a side channel.
✍️ ClickHouse · Read article →
Palantir — August 2026
Foundry's August release notes push the Ontology toward a code-managed artifact: new Foundry CLI capabilities let you declare object types and links as TypeScript, preview an embedded Ontology against a local server, and auto-regenerate Ontology SDK bindings whenever definitions change. Separately, DeepSeek V3.2 with its 164k-token context, Z.ai's GLM 5 and Moonshot models are now reachable in AIP through AWS Bedrock. Ontology-as-code is the notable shift — the semantic layer becomes a reviewable, diffable repo artifact rather than console state.
✍️ Palantir Technologies · Read article →
PR Newswire — August 2026
Introhive moved its MCP Server to general availability for all customers, expanding beyond the legal-sector commercial preview launched in April 2026. Connected once at the firm level, it exposes relationship strength, interaction history and network connections to Claude, Harvey, ChatGPT or Copilot without surfacing the raw underlying records and without custom integration work. The deliberately model-agnostic framing is the architectural bet: the relationship graph is the durable asset, the assistant querying it is interchangeable.
✍️ Introhive · Read article →
The New Stack — August 2026
A hands-on teardown of why vector-only RAG collapses on multi-hop questions: the intermediate concepts linking A to C rarely coexist in the same chunk, so top-k similarity retrieval never surfaces the bridge. The piece walks through replacing flat chunk retrieval with a Neo4j-backed graph traversal in Python, including the entity-and-relationship extraction step that turns documents into a queryable knowledge graph. It reframes retrieval as a data modeling problem rather than an embedding-tuning problem — which changes who owns the RAG pipeline.
✍️ Emmanuel Akita · Read article →
BigDATAwire — August 2026
D&B is exposing its Commercial Graph and Risk Analytics to Perplexity and the Perplexity Computer agent platform through MCP servers, published as marketplace connectors rather than a bulk data feed. The graph covers more than 650 million global business entities anchored on the D-U-N-S identifier and is validated by over 100 billion monthly data quality checks. A concrete signal that third-party data vendors now treat MCP as the distribution channel: activation is shifting from syncing rows into SaaS tools toward serving governed entities to agents at query time.
✍️ Dun & Bradstreet · Read article →
InfoQ — August 2026
AWS lays out a specification-driven pattern that separates workflow intent from processing logic, using declarative specs, reusable processing capabilities and pre-execution validation instead of hand-built pipelines per dataset. AWS reports dataset onboarding dropping from weeks to days while keeping traceability, versioning, data classification and governance attached to the spec itself. It is a concrete answer to the “every new source needs a new DAG” problem that quietly consumes data engineering capacity.
✍️ Leela Kumili · Read article →
Business Wire — August 2026
Grafana Labs passed 10,000 customers, up from the 7,000 customers and $400M ARR reported in September 2025, crediting AI-era infrastructure workloads standardizing on Grafana Cloud. Named adopters include Anthropic, Lovable, 7AI, TeamSystem and Zama. The signal for data platform teams is consolidation pressure — as AI workloads generate telemetry spanning pipelines, models and agents, the open-standards observability stack is absorbing territory dedicated data-observability vendors have been defending.
✍️ Grafana Labs · Read article →
OpenMetadata Blog — August 2026
OpenMetadata 2.0 formally drops the “data catalog” label and repositions as an open context layer for data and AI, aimed at serving agents as first-class metadata consumers alongside humans. The groundwork shipped in 1.13.0 in June: MCP services exposing the platform as callable LLM tools, and an RDF-backed knowledge graph restructuring the metadata store into something an agent can traverse, with stored memories sitting alongside lineage and quality signals. Collate's commercial layer adds a Context Center and agent memory on top.
✍️ Shawn Gordon, OpenMetadata · Read article →
BigDATAwire — August 2026
Nasuni acquired DryvIQ to fold AI-powered classification and policy enforcement directly into its File Data Platform, targeting the unstructured-data blind spot most catalogs ignore. DryvIQ's models scan 550+ file formats, 100+ entity types and 1,000+ business document types, detecting PII, PHI and PCI across 175+ languages, with configurable rules that quarantine high-risk content, apply retention and sensitivity labels, and support defensible deletion across 40+ repositories mapped to GDPR, HIPAA and PCI-DSS. This is the storage layer absorbing governance — classification moving in-place next to the files rather than into a separate control plane.
✍️ Nasuni · Read article →
Microsoft Security Blog — August 2026
Microsoft's monthly security roundup carries a Purview throughput change that matters for anyone running classification at enterprise scale: auto-labeling policies now process up to 500,000 SharePoint and OneDrive files per day, up from 100,000. For organizations that have been backlogged on initial labeling sweeps, a fivefold throughput lift changes the timeline for reaching full coverage — and therefore the timeline for enforcing label-dependent DLP and Copilot access policies.
✍️ Microsoft · Read article →