Daily Briefing — Monday, September 28, 2026
The agent era is being operationalized from both ends: platforms are adding log semantics, CDC hardening and incremental refresh underneath, while MCP credentials, runtime kill switches and decision-level observability become the governance problems on top.
⇣ Jump To
Click any section or topic below to jump to it.
Streaming & Messaging · CDC · Transformation Frameworks
Cloud Data Warehouses · Architectural Patterns · Query Engines · Vector & Specialty Stores
AI-Driven Consumption · Semantic Layers & Retrieval · Enterprise RAG & Retrieval
Data Observability · Catalogs & Metadata · Governance, Security & Compliance
⚡ Quick Takes
| Story | Signal |
|---|---|
| ↗ Amazon EventBridge relaunches event buses for enterprise scale | AWS’s serverless router is acquiring log semantics — ordering, retention, replay — and blurring the bus-vs-log line. |
| ↗ Amazon Kinesis Data Streams adds service-managed partition keys | Managed streams are absorbing partitioning decisions for workloads that don’t need per-key ordering. |
| ↗ Snowflake Openflow runtime updates harden Oracle, PostgreSQL and SQL Server CDC | Built-in warehouse CDC is maturing through unglamorous correctness fixes rather than headline launches. |
| ↗ Amazon EMR on EKS adds interactive sessions via Spark Connect | Spark Connect is becoming the default interface to Spark in managed services. |
| ↗ Amazon EMR introduces Long Term Support, starting with Spark 4.1 | Hyperscalers are pairing long support windows with AI upgrade agents to cut the cost of Spark version moves. |
| ↗ CHECK constraints on hybrid tables (GA) | Snowflake keeps adding the OLTP integrity guarantees operational workloads expect, one feature at a time. |
| ↗ Snowpipe Streaming: event table monitoring | Streaming-ingest observability is moving into the platform itself. |
| ↗ Dynamic tables: incremental refresh now supports the VALUES clause | Declarative pipelines get cheaper as the incremental-refresh envelope widens. |
| ↗ Data Dialogue: How S&P Global Energy made its structured data estate conversational with Genie Agents and MCP | A reference pattern for serving a governed data estate to agents: domain-owned, mesh-style, MCP-composed. |
| ↗ ClickHouse Release 26.9 | Real-time OLAP engines keep absorbing warehouse-style incremental processing. |
| ↗ Perplexity replaces DynamoDB with home-built CobbleDB | Retrieval at AI scale is breaking per-byte-priced managed KV stores, pushing builders to split serving from durable storage. |
| ↗ Dataiku Expands Cobuild to Turn Business Questions Into Governed AI Work | Data platforms are becoming governed backends that external coding agents drive, not destinations users log into. |
| ↗ Ekai raises $1.7M to give enterprise AI agents verified business context | A counter-position to auto-mining semantics from query logs: define first, validate, then generate. |
| ↗ Persistent AI memory adds another layer to enterprise data | Retrieved context now includes derived, stateful memory that needs lineage, authority rules, and a lifecycle. |
| ↗ AWS CloudWatch Omni goes after the hardest question in agentic AI: Why did the agent do that? | Observability is shifting from “is it running?” to “was the decision right?” — and hyperscalers want the evaluation layer. |
| ↗ Collibra brings runtime governance to enterprise AI agents | Catalog vendors are moving from design-time policy to runtime enforcement in the agent path. |
| ↗ Hardcoded credentials in public MCP files open door to cyberattacks | MCP config files are the new leaked .env; agent credentials need joiner-mover-leaver discipline. |
| ↗ Pulling the plug: Why the AI kill switch might be a dead end | Containment design — credential TTLs, reversibility tiers, dependency maps — is becoming a platform-engineering requirement. |
| ↗ New Okta for AI Agents innovations increase visibility, secure connections at runtime, and enforce continuous agent governance | Identity providers are positioning as the agent control plane between agents and data. |
AWS What’s New — September 2026
A new “enhanced” custom event bus adds strict ordering, content-based deduplication, CloudEvents publishing with schema preserved, 24 hours of built-in retention (extendable to a year for replay), and cross-account sharing via AWS RAM; pricing shifts from per-event to per-data-transferred, and the old bus becomes “classic.” Ordering plus replay moves EventBridge into territory teams previously reserved for Kafka, Kinesis, or SNS+SQS fan-out, which reopens the “do we need a log here?” question for event-driven integration.
✍️ Amazon Web Services · Read article →
AWS What’s New — September 2026
On-demand streams (Standard and Advantage) can now accept records without a partition key; the service distributes them across shards based on warm capacity, eliminating hot-key throttling. The usual workaround — random UUID keys — still produced uneven shard load. Requires the latest SDK or KPL, available in all commercial regions at no extra cost.
✍️ Amazon Web Services · Read article →
Snowflake Documentation — September 2026
The September 24 runtime (2026.9.24.2) adds client-side ROWID chunking for Oracle CDC on read-only databases and very large tablespaces, removes a scheduling lag on PostgreSQL CDC merges, and fixes schema evolution when a removed column is renamed; the September 22 runtime added cursor-based SQL Server capture for sources without Change Tracking or CDC and fixed a Change Tracking bug that could permanently skip changes. These edge cases decide whether Openflow can credibly displace GoldenGate, Qlik Replicate, or self-run Debezium estates.
✍️ Snowflake · Read article →
AWS What’s New — September 2026
Interactive Spark sessions can now be launched from SageMaker Unified Studio notebooks or local IDEs (Jupyter, VS Code) against existing EKS clusters, running as pods under an IAM execution role and tagged by project and user. Supported on EMR 7.14 (Spark 3.5) and emr-spark-8.1.0 (Spark 4.1). It turns Spark Connect’s thin-client split into a production pattern on customer-run Kubernetes.
✍️ Amazon Web Services · Read article →
AWS What’s New — September 2026
emr-spark-8.1.0 is EMR’s first LTS release, with 36 months of security, bug, and data-corruption fixes, full Iceberg v3 support, catalog-by-name references (including cross-account and S3 Tables) with automatic Iceberg/Delta/Hudi detection, and broader fine-grained access control. AWS pairs it with its Apache Spark Upgrade Agent to migrate older jobs — a three-year Spark baseline plus AI-assisted migration materially changes upgrade planning for platform teams.
✍️ Amazon Web Services · Read article →
Snowflake Release Notes — September 2026
Hybrid (Unistore) tables now enforce CHECK constraints — inline or multi-column — on every INSERT, UPDATE, MERGE, and CTAS, moving integrity rules out of application code. The limits matter: constraints can only be declared at CREATE time (no ALTER ADD), and COPY INTO fails against a constrained hybrid table, so bulk-load paths need rework.
✍️ Snowflake · Read article →
Snowflake Release Notes — September 2026
Snowpipe Streaming now emits ingestion telemetry — row counts, server-side latency, row and channel errors, channel activity — to the default event table, enabled per schema with LOG_EVENT_LEVEL=INFO and queryable in SQL alongside SDK-side Prometheus metrics. Combined with last week’s partitioned-Iceberg streaming GA, streaming into Snowflake-managed Iceberg is becoming something you can operate and alert on without bespoke tooling.
✍️ Snowflake · Read article →
Snowflake Release Notes — September 2026
Incremental refresh (INCREMENTAL, AUTO, and ADAPTIVE modes) now handles VALUES clauses such as inline lookup tables; a companion Sept 23 note extends incremental support to approximate aggregates (APPROX_COUNT_DISTINCT, APPROX_PERCENTILE, APPROX_TOP_K) in the SELECT list. Fewer dynamic tables will silently fall back to full refresh — a direct compute saving on declarative pipelines.
✍️ Snowflake · Read article →
Databricks Blog — September 2026
S&P Global Energy has subject-matter experts curate one narrow Genie Agent per dataset group (LNG cargo, outages, netbacks) over native and federated tables; each becomes a Unity Catalog-governed MCP server, and a FastMCP proxy composes them into namespaced per-commodity endpoints — also exposed to external customers’ agents. Lessons: keep agents narrow and compose at the MCP layer, federate before building pipelines, and benchmark accuracy continuously.
✍️ Debaprasad Satapathy & Eric Henderson, Databricks · Read article →
ClickHouse Blog — September 2026
The release ships 56 features, 135 performance optimizations, and 464 bug fixes, headlined by incremental refreshes for append-only materialized views, disk spilling for high-cardinality DISTINCT, conditional LIMIT boundaries, time-limited access tokens, and broader PromQL support. Incremental refresh and spill-to-disk remove two common reasons to run a separate transform layer or oversize nodes for real-time rollups.
✍️ ClickHouse · Read article →
InfoQ — September 2026
Perplexity split storage into Pillar (durable store on YTsaurus), Lorry (partition-aligned batch writes to S3), and CobbleDB, a Rust/RocksDB key-value store on local NVMe that serves batched reads of passages and embeddings using async replicas, same-zone routing, and hedged reads. p50 latency fell from 31.4 ms to 5.6 ms, p99 from 123 ms to 24.2 ms, with storage cost down at least 20%; Perplexity plans to open-source it.
✍️ Olimpiu Pop, InfoQ · Read article →
BigDATAwire — September 2026
Cobuild Insights answers plain-language questions from governed data, while Dataiku Headless lets Claude Code, Codex, and Cursor build and modify pipelines, models, and agents inside a live Dataiku environment — including migrating Excel or Alteryx logic into governed projects. The AI Catalog now lists semantic models, so assembled answers and pipelines draw on vetted assets with lineage.
✍️ Dataiku (press release) · Read article →
SiliconANGLE — September 2026
Ekai “forward-engineers” semantic models and transformation code from domain-expert definitions, validating everything against the warehouse before it ships, and explicitly rejects inferring semantics from dashboards, query history, or dbt projects. It claims semantic modeling that took 3–6 months done in about 6 hours; it runs on Snowflake with support for Databricks, BigQuery, Redshift, Postgres, ClickHouse, and DuckDB.
✍️ Duncan Riley, SiliconANGLE · Read article →
TechTarget — September 2026
Agent memory is becoming a distributed data layer — Writer’s team-level Agent Memory, AWS AgentCore Memory’s expiration and relevance-decay lifecycle, Couchbase’s AI Data Plane, Gemini Enterprise Memory Bank. The piece proposes treating memory as an overlay on the data map and checking it against the authoritative system before any consequential action, correcting or expiring stale memory on conflict.
✍️ James Alan Miller, TechTarget · Read article →
SiliconANGLE — September 2026
Built on OpenTelemetry/OpenInference, CloudWatch Omni ships 17 built-in evaluators (faithfulness, routing correctness, and others) that run continuously on live traffic, storing agent traces alongside app and infrastructure telemetry. Kerravala warns that chatty agent spans can push ingestion costs past the AI budget itself, and argues investigation history should be treated as governance evidence.
✍️ Zeus Kerravala, ZK Research · Read article →
SiliconANGLE — September 2026
CEO Felix Van de Maele details how Agent Contracts are compiled and deployed into existing AI gateways, with Guardian Agents blocking non-compliant actions before execution. Live Map pre-builds reusable context from unstructured documents for token efficiency, and Maestro steward agents claim to automate up to 80% of governance work — the operational detail behind last week’s launch.
✍️ Victoria Gayton, SiliconANGLE · Read article →
TechTarget — September 2026
Hush Security analyzed roughly 82,000 public MCP config files: 12% of credential slots held a hardcoded secret, and 24% of those were both non-expiring and broad-scope, including model-provider API keys. Remediation: rotate at the provider (scrubbing Git history isn’t enough), use environment variables or secret managers, adopt an MCP gateway, and move to the 2026-07-28 MCP spec.
✍️ Alissa Irei, TechTarget · Read article →
TechTarget — September 2026
Proposed kill-switch legislation collides with technical reality: agents spawn sub-agents and queue jobs, so killing the parent can leave work running. Security leaders argue for “positive control” instead — short-lived scoped credentials treated as “a lease, not a license,” with human approval gated by how reversible an action is.
✍️ Richard Livingston, TechTarget · Read article →
Okta Newsroom — September 2026
Agent SSO (Cross App Access replacing non-expiring keys with short-lived tokens), agent-to-agent connections, and resource access certifications are GA; Agent Gateway — policy enforcement and logging on every tool call — plus shadow-agent discovery are slated next, with a gateway-level kill switch that revokes all active tokens and in-flight sessions planned for Q4. For data platform teams, this puts the IdP directly in the MCP tool-call path.
✍️ Okta · Read article →