Code History as Data 🔍, Petabyte Kafka Replatforming 🔄, AI Velocity vs Quality 📉

Reddit migrated its petabyte-scale Apache Kafka fleet with over 500 brokers from Amazon EC2 to Kubernetes without any downtime or data loss ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌  ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 

Sign Up |Advertise|View Online TLDR TLDR Data 2026-03-19 📱 Deep Dives How Reddit Migrated Petabyte-Scale Kafka from EC2 to Kubernetes (9 minute read) Reddit migrated its petabyte-scale Apache Kafka fleet with over 500 brokers from Amazon EC2 to Kubernetes without any downtime, data loss, or client-side changes. It achieved this by introducing a DNS facade for seamless connection abstraction, doubling cluster size with higher-ID EC2 brokers to free low IDs, and forking Strimzi operator for mixed EC2-Kubernetes operation. Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta's Ads Ranking Innovation (6 minute read) Meta's Ranking Engineer Agent (REA) is an AI agent that accelerates ads ranking by managing the full ML lifecycle with almost no human input. REA uses the Confucius framework for persistent state and tool access. It generates high-quality hypotheses from historical insights with ML Research Agent by following a Three-Phase Planning Framework (Validation → Combination → Exploitation) within compute budgets and handling errors autonomously via runbooks. How we optimized Dash's relevance judge with DSPy (10 minute read) Using DSPy's optimizer, Dropbox defined a clear objective (minimize normalized mean squared error against human judgments), incorporated structured feedback from rating gaps and human rationales, enforced JSON validity guardrails, and ran iterative prompt refinement, enabling 10-100x more synthetic labeling at the same cost, and accelerating model switches from weeks to 1-2 days. 🚀 Opinions & Advice Claude Code isn't going to replace data engineers (yet) (8 minute read) Claude Code is a powerful productivity booster for rapid prototyping, iteration, and handling repetitive tasks in data engineering, but it lacks the judgment, trust, and reliability needed for unsupervised production work due to confident-yet-wrong outputs, silent data loss, and non-determinism, making it a strong "companion" tool today rather than a replacement. Dispatches from the Gartner Data & Analytics Summit 2026: The Noise, the Slop, and the Signal (10 minute read) AI adoption has accelerated dramatically, with agents now generating write-heavy, contextual "trajectory" data that strains traditional operational databases and blurs boundaries between infrastructure and feature stores. Real-world deployments, such as OpenAI's scaling of PostgreSQL to 800 million users and insurers leveraging 95% accurate speech-to-text for rapid care plan creation, highlight measurable ROI. AI governance has become urgent, requiring new guardrails atop existing data governance, while the hype around context graphs signals growing complexity and unresolved semantic integration challenges. 💻 Launches & Tools Your CEO has questions about the data. Can she answer them? (Sponsor) The way people ask questions has changed. With Starburst's AIDA, every employee from analyst to CEO can explore enterprise data in plain language, run complex analyses, and apply their own business rules in real time. No SQL. No tickets. No BI backlog. Meet AIDA, your AI data assistant pgit: What If Your Git History Was a SQL Database? (13 minute read) pgit is a Git-like CLI that stores repositories in PostgreSQL with built-in delta compression, making the entire commit history SQL-queryable while often matching or beating Git's best compression. It enables powerful codebase analysis, allowing developers and AI agents to instantly query history, detect patterns, and generate insights without custom tooling. GreenBoost — 3-Tier GPU Memory Extension for Linux (GitLab Repo) GreenBoost is an open-source Linux kernel module and CUDA shim that extends GPU VRAM using system RAM and NVMe, allowing large AI models to run on smaller GPUs without modifying inference code. It works by intercepting CUDA memory allocations and transparently offloading overflow to slower memory tiers, trading some performance for significantly increased capacity. TigerFS (Tool) TigerFS is a system that mounts a PostgreSQL database as a filesystem, letting agents and humans read and write shared data as files with full ACID transactions and version history. It replaces fragile file-based coordination by combining the simplicity of files with the reliability, structure, and concurrency of a database. Metabase (GitHub Repo) Metabase is an open-source business intelligence platform that lets teams query, visualize, and share data without heavy tooling. It connects to many databases, supports SQL and no-code exploration, and provides dashboards, alerts, and embedding. It offers a lightweight, self-hostable analytics layer that reduces dependency on complex BI stacks while enabling governed, accessible data exploration. 🎁 Miscellaneous Are AI agents actually slowing us down? (9 minute read) Heavy reliance on AI coding tools at tech giants like Anthropic, Amazon, Uber, and Meta is driving significant increases in code output (engineers classified as "power users" of AI produce 52% more pull requests), but is also leading to notable declines in software quality, increased outages, and growing technical debt. At Anthropic, 80%+ of production code is AI-generated, resulting in critical UX bugs impacting millions, while Amazon has seen a rise in SEVs and now mandates senior review of AI-assisted code. These trends highlight the need for stronger architectural oversight, code validation, and emphasis on quality assurance. GitGuardian Reports an 81% Surge of AI-Service Leaks as 29M Secrets Hit Public GitHub (4 minute read) AI-assisted coding drove a 34% year-over-year surge in leaked secrets in 2025, with Claude Code commits showing a 3.2% leak rate (over twice the 1.5% baseline) resulting in nearly 29 million exposed credentials detected on GitHub. Leaks involving AI service credentials jumped 81% year-over-year, and internal repositories were six times more likely to contain hardcoded secrets. Persistent remediation gaps (64% of valid secrets from 2022 still unrecalled) highlight critical governance and NHI management shortfalls. Snowflake Cortex AI Escapes Sandbox and Executes Malware (11 minute read) Modern AI agents across tools like Snowflake Cortex, Copilot, Claude, and Slack are vulnerable to prompt injection attacks that can bypass safeguards, execute malware, and exfiltrate sensitive data. A real example shows how an injected prompt tricked Cortex into running malicious code outside its sandbox without user approval, enabling attackers to steal or destroy data using the victim's credentials. ⚡ Quick Links Snap Decisions: How Open Libraries for Accelerated Data Processing Boost A/B Testing for Snapchat (4 minute read) Snap has migrated its A/B testing data pipelines for Snapchat, processing over 10 PB of data per day, from CPU-based Apache Spark to NVIDIA GPU-accelerated Spark on Google Kubernetes Engine, realizing 4x faster runtimes and 76% daily cost savings. AI blind spot debt threatens governance (3 minute read) Unmanaged models create hidden technical debt. Love TLDR? Tell your friends and get rewards! Share your referral link below with friends to get free TLDR swag! https://sparklp.co/32815a84/11 Track your referrals here. Want to advertise in TLDR? 📰 If your company is interested in reaching an audience of data engineering professionals and decision makers, you may want to advertise with us. Want to work at TLDR? 💼 Apply here, create your own role or send a friend's resume to jobs@tldr.tech and get $1k if we hire them! TLDR is one of Inc.'s Best Bootstrapped businesses of 2025. If you have any comments or feedback, just respond to this email! Thanks for reading, Joel Van Veluwen, Tzu-Ruey Ching & Remi Turpaud Manage your subscriptions to our other newsletters on tech, startups, and programming. Or if TLDR Data isn't for you, please unsubscribe.