Prefect Buys Dagster 🤝, Netflix’s Dependency Graph 🎥, Expedia’s AI Guardrails 🤖

Prefect is acquiring Dagster while retaining its names, pricing, and roadmaps. Around 40 staff will join Prefect, pairing Dagster’s outcome tracking ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌  ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 

Sign Up |Advertise|View Online TLDR Together With TLDR Data 2026-07-16 Your AI isn't hallucinating. Your data can't agree with itself (Sponsor) Most AI projects don't fail on model quality. They fail because "revenue" means one thing in Finance's dashboard, another in the CRM, and something else in your AI assistant's answer. The model isn't wrong — it's repeating whichever definition it was handed. That's the context gap, and it widens with every AI tool you add. Timextender's whitepaper breaks down what production-grade AI actually requires. Gartner expects 40%+ of agentic AI projects cancelled by 2027 BCG: 74% can't scale value from AI The cause is the data underneath, not the model Want to know where your data stands? Take the 3-minute check for your score and top three gaps. Check your AI-readiness (3 min) → Get the whitepaper → 📱 Deep Dives Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned (20 minute read) Netflix built a three-layer service topology system using eBPF flow logs, application metrics, and distributed traces, feeding a three-stage streaming pipeline that aggregates, resolves load balancer edges, and persists a dependency graph queryable at sub-second latency. The hardest problem, a 100x load variance caused by power-law traffic concentrating on popular services, was solved with multi-stage redistribution. Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption (7 minute read) Grab migrated petabyte-scale Hive Parquet tables to Apache Iceberg, achieving a 10x query speedup on their dataset, a 95% reduction in daily S3 API costs on their operations table, and a 50% compute savings on ML pipelines. It also built a custom UnifiedSparkCatalog that abstracts Iceberg, Delta, Hudi, and Hive behind a single interface with fallback logic and Hive compatibility. Nobody Has Cracked Agent Memory (12 minute read) Build memory as an ontology pipeline: warehouse data, extract graph objects with an LLM, validate, deduplicate them, and serve standard queries and deep-search via MCP. MongoDB handles text, vector, and graph search. Use a graph database only for deep traversal or core graph logic. How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes (11 minute read) Razorpay cut data warehouse refresh time by 90% by replacing full table scans with incremental graph traversal: a silver layer deduplicates daily changes per primary key, secondary indexes store join columns and partition metadata to avoid full scans, and a graph traversal processes only changed rows while back-traversing dependencies to enrich updates. 🚀 Opinions & Advice What is ACID on a Data Lake? (14 minute read) Hudi, Iceberg, and Delta Lake add ACID to object storage through a metadata log and atomic publish step, so multi-file writes become visible only after a final atomic metadata operation publishes the commit and readers see a consistent snapshot. Guarantees apply per table, commits take seconds, and isolation is snapshot-based, not fully serializable. Cache Layer Architecture: A Practical Guide to Speed & Scale (9 minute read) Cache layers can fail at scale through stampedes when popular keys expire, hot keys overwhelming one node, stale data from invalidation races, and avalanches when an outage clears the cache, and the backend absorbs full cold load. Defenses include TTL jitter, request coalescing, key splitting, and consistent hashing with virtual nodes. The 90/90 rule for the dashboard dumpster (3 minute read) Dashboard sprawl is creating 3 major costs for data organizations: low trust, wasted warehouse compute, and dead artifacts, with one example showing hundreds of dashboards but only dozens opened in a quarter—roughly 90% unused. A practical mitigation is a 90/90 lifecycle rule: archive dashboards after 90 days of no views, then delete them after another 90 if no one responds. 💻 Launches & Tools Compliance controls in the wrong place cost more than bad hardware. (Sponsor) Compliance controls placed in the wrong spot can drop GPU utilization from 65% to 40% without changing a single piece of hardware. WhiteFiber breaks down where security overhead hides in the AI training stack and what to do about it. Read the breakdown Talk to WhiteFiber's engineering team Prefect just bought Dagster, another big Airflow rival — and it's not a data pipeline story (4 minute read) Prefect is acquiring Dagster while retaining its names, pricing, and roadmaps. Around 40 staff will join Prefect, pairing Dagster's outcome tracking with Prefect execution and FastMCP to strengthen open-source orchestration. Arroyo is joining Cloudflare (4 minute read) Cloudflare has acquired Arroyo, a Rust-based SQL stream processor, for its Developer Platform. It supports stateful joins and aggregations, remains Apache 2.0 open source, and will first add SQL processing to Cloudflare Pipelines. Benchmarking Single Node vs Distributed (5 minute read) On a 1TB TPC-H benchmark with matched aggregate specs (128 vCPUs, 512GB RAM), distributed Polars on 32 nodes slightly outperformed a single m8i.32xlarge, but only on I/O-bound queries where the cluster's burst network bandwidth reached 400 Gbps versus the single node's 50 Gbps sustained. Join-heavy queries ran faster on a single node because shuffle overhead across the network erased parallelization gains. 🎁 Miscellaneous How Expedia Group Builds AI That Lasts at Scale (8 minute read) Expedia Group's AI framework rests on key requirements: every model must tie to a measurable business outcome, be built on shared platform foundations rather than a bespoke stack, and have defined owners across business, product, AI, and operations G-Eval, Explained (5 minute read) G-Eval evaluates open-ended text by having an LLM generate its own scoring steps from a rubric via chain-of-thought, then computing a weighted expectation over token-level probabilities for each score rather than reading a printed digit, which makes the score more stable than a single sampled output. Two important practices for more reliable results are using a judge model from a different model family and calibrating rubrics against human-labeled examples. ⚡ Quick Links Greysight (Tool) Greysight is a free, open-source Snowflake cost observability tool that tracks warehouse, AI, and storage spend without storing customer data. Cloudflare as a Data Platform? (10 minute read) Early experimentation shows R2 + Iceberg can serve as a lightweight lakehouse, but workflow orchestration remains a gap. Love TLDR? Tell your friends and get rewards! Share your referral link below with friends to get free TLDR swag! https://sparklp.co/32815a84/11 Track your referrals here. Want to advertise in TLDR? 📰 If your company is interested in reaching an audience of data engineering professionals and decision makers, you may want to advertise with us. Want to work at TLDR? 💼 Apply here, create your own role or send a friend's resume to jobs@tldr.tech and get $1k if we hire them! TLDR is one of Inc.'s Best Bootstrapped businesses of 2025. If you have any comments or feedback, just respond to this email! Thanks for reading, Joel Van Veluwen, Tzu-Ruey Ching & Remi Turpaud Manage your subscriptions to our other newsletters on tech, startups, and programming. Or if TLDR Data isn't for you, please unsubscribe.