# Evil Martians > Evil Martians is a developer tools consultancy that helps startups build AI agents, AI-native products, scale infrastructure, and reach product-market fit. Founded in 2006, we work with 40+ developer tools startups each year, many backed by Y Combinator. Our verticals are AI agents and AI-native development, developer tools, infrastructure, and cybersecurity. Our core stack spans Ruby on Rails, TypeScript, Python, Go, Rust, React, and Next.js. We are the leading Ruby on Rails consultancy and build AI Harnesses that make agentic coding more productive for Rails and TypeScript teams. - Founded: 2006 - Website: https://evilmartians.com - GitHub: https://github.com/evilmartians - Contact: https://evilmartians.com/contact-us Evil Martians is a product development consultancy for developer tools startups. We design and develop high-performance developer tools that scale. Our core verticals are AI agents and AI-native development, developer tools, infrastructure, and cybersecurity. We are the leading Ruby on Rails consultancy—we authored the book Layered Design for Ruby on Rails Applications, maintain dozens of Rails open source projects, and build AI Harnesses that encode architectural patterns into AI coding tools for more productive agentic coding. We work with 40+ developer tools startups each year, many backed by Y Combinator. Our core stack spans Ruby on Rails, TypeScript, Python, Go, Rust, React, and Next.js. We help open-source devtools companies build commercial products. 65% of our clients raise Series A or B within 1–3 years. 8 startups grew from seed to unicorn with our help. 25B+ total downloads of our open source projects. ## Offices - **New York**: 77 Sands St. Brooklyn, New York 11201 - **California**: 535 Mission St. San Francisco, California 94105 - **Portugal**: Rua Alexandre Oneill, 38, Porto 4400-008 - **Japan**: 19F Osaka Umeda Twin Towers North, 8-1 Kakuda-cho, Kita‑ku, Osaka 530-0017 ## Services ### Agent Experience URL: https://evilmartians.com/services/agent-experience Build Agent Experience for your product: discovery optimization, agent-friendly APIs and CLIs, and security guardrails against abuse. 80% of databases on Neon are created by agents, not humans. 30% of new Supabase signups come from AI builder platforms. 10% of Vercel signups arrive via ChatGPT. The next wave of developer tool growth is driven by AI agents that discover, provision, and operate infrastructure on behalf of developers. This isn't about building agents. It's about building _for_ agents. Agent Experience (AX)—a term coined by Matt Billmann at Netlify—extends Developer Experience into a world where agents increasingly act on developers' behalf. We help devtools, infrastructure, and security products become first-class citizens in agentic workflows. #### Discovery optimization Agents don't watch demo videos, respond to ads, or care about your brand. They search the web, read docs, scan repos—and pick whichever tool has the clearest technical content. If your docs are rendered client-side, gated behind auth, or buried in PDFs, you're invisible. Mintlify reports that nearly half of docs traffic now comes from AI agents. Discovery happens two ways. Your buyers—founders, engineering leads, developers—ask LLMs directly: "best WebSocket library for Rails," "who can help us scale Kubernetes." And agents acting on their behalf evaluate tools programmatically, parsing docs and benchmarks to make provisioning decisions. Both paths reward the same thing: clear, structured, machine-readable content. We start with your ICP: who's searching for tools like yours, what stacks do they use, and what will they ask? We research your target market—engineering blogs, job postings, open source activity, founder profiles—then align your content so LLMs surface your product for those queries. This means structured content formats (`llms.txt`, per-page `.md` endpoints, JSON-LD schema, `ai.txt`) but also the content strategy itself: tech stack mentions calibrated to your market, client stories with quantified outcomes, and descriptions that read naturally to humans but signal clearly to machines. We built this methodology for [evilmartians.com](/). We researched 90+ target companies across AI agents, developer tools, infrastructure, and cybersecurity. We aligned every page—services, client stories, blog posts—to the queries their buyers ask. Our `llms.txt` serves structured content that LLMs consume directly. Every page has a `.md` version. We wrote the playbook by shipping it ourselves. #### Agent-friendly CLIs and APIs If your product requires a browser to sign up, create resources, or check logs, agents will use a competitor that doesn't. Supabase, Railway, Vercel, Stripe—the companies winning agent-driven growth all share one pattern: programmatic access from signup to production. We build that layer. - **Programmatic signup and provisioning** — account creation, API key generation, resource provisioning, all via CLI. No browser tab required - **CLI-complete workflows** — we audit existing CLIs for gaps where agents hit a wall and fall back to "please visit the dashboard" - **Structured output and feedback loops** — `--json` flags, machine-readable error codes, webhooks and health checks that let agents verify and self-correct - **MCP server integrations** — expose your product's capabilities directly to agent workflows so they can invoke your tool without leaving the editor We proved this on our own projects. **AnyCable+ CLI**, built with [Terminalwire](https://terminalwire.com/), lets users sign up, create [AnyCable](https://anycable.io/) instances, manage credentials, and check status—all from the terminal. #### Security guardrails Everything that makes a product agent-friendly is also what bad actors exploit at machine speed. Heroku, GitLab, Travis CI, and Fly.io all killed or restricted free tiers after automated abuse overwhelmed them. Railway spent a third to half of all engineering cycles fighting fraud. Companies that skip fraud _prevention_ end up fighting it reactively. The reactive response damages their own product. We build the guardrails that prevent this. - **Agent-aware authentication** — token-based auth where every agent action is traceable to a human identity, without browser redirects or CAPTCHAs - **Least-privilege authorization** — agent permissions scoped to the minimum required for each operation, enforced as code - **Abuse detection and blocking** — WAF configuration, rate limiting, and bot detection tuned for agent traffic patterns - **Free tier verification** — identity verification before granting compute access, keeping the acquisition funnel open while filtering out abuse #### AI Harnesses AI coding tools produce generic code by default. An AI harness encodes your architecture, conventions, and constraints so agent output matches what your team would actually write. Here's what that looks like in practice. - **[Layered Rails skills](https://github.com/palkan/skills)** — architecture-aware rules that enforce service objects, query objects, and policy patterns - **[Inertia Rails skills](https://github.com/inertia-rails/skills)** — integration-specific constraints for Inertia.js and Rails - **SAST rules tuned for LLM patterns** — catching the mistakes agents make most: SQL injection via string interpolation, path traversal, SSRF, hardcoded secrets - **CI guardrails with [TestProf](https://test-prof.evilmartians.io/)** — agents generate tests prolifically. Without constraints, they bloat your suite and CI time. TestProf keeps it fast - **Dev Containers for isolated execution** — agents build, test, and iterate in sandboxed environments, not on your machine or in production - **`CLAUDE.md` and `AGENTS.md` rules** — project-level instructions that shape every AI interaction The harness approach works because it meets agents where they are: define constraints upfront, let the agent operate within them. At the [SF Bay Area Ruby Meetup](/events/the-silicon-heel-sf-ruby-meetup), Vladimir Dementyev showed that an agent with a proper harness changed 33 files with zero unprotected actions. Without one, it changed 52 files and left 7 unprotected. #### Agent observability When agents act on behalf of your users, you need visibility into what they're doing. Traditional APM wasn't designed for multi-step agent workflows where one user action triggers dozens of LLM calls and branching decisions. We built [AgentPrism](/chronicles/debug-ai-fast-agent-prism-open-source-library-visualize-agent-traces), an open source React component library for visualizing AI agent traces. Debug agent behavior without reading raw logs. For real-time AI, we built [AnyCable](https://anycable.io/) support for [LLM streaming](/chronicles/anycable-rails-and-the-pitfalls-of-llm-streaming), handling token-by-token delivery over WebSockets at scale. ### AI harness URL: https://evilmartians.com/services/ai-harness Your AI coding agents can do far more. Evil Martians build the harness that unlocks your team's full productivity potential. Point Cursor, Claude Code, or Codex at your codebase and it simply reaches for two patterns: the most common one from its training data, and the one already most common in your repo, usually legacy code. An AI harness fixes that. [GitHub](https://github.blog/ai-and-ml/github-copilot/how-github-copilot-is-getting-better-at-understanding-your-code/) describes how coding agents read your open files to match the surrounding code. #### What you're aiming for - Agent output following the architecture your team agreed on, keeping the codebase consistent as it grows. - Every engineer shipping at the level of your strongest one, with hard-won conventions encoded and shared across the team. - AI spend you can account for: clarity on which tools and workflows pay off, and tokens going where they earn their keep. - One playbook for the whole team, so engineers, designers, and writers pull from the same source of truth. #### The AI harnesses Evil Martians build An AI harness encodes your conventions, architecture, and guardrails as deterministic tools, skills, evals, and telemetry. We build it for your stack, whether you ship Rails, TypeScript, Go, or Python, so agents produce code your senior engineers would approve. - **Skills and a shared marketplace**: your team's best practices and unique expertise, encoded once and reused by everyone. Designers encode brand guidelines, writers distribute tone of voice, and business folks share ICP and research skills with the whole team. - **Guardrails that hold as you grow**: when you ship a new design system or architectural pattern, agents default to the legacy version because there's more of it in the code. We ship deterministic linters and hooks that catch the divergence and tell the agent how to follow the new convention. - **Evals**: tests that confirm skills behave correctly and fire on the right requests. - **Observability**: telemetry and a dashboard into skill usage and gaps, built on AgentPrism, our open source agent-trace visualization library. - **Cost control**: telemetry on where your AI tokens go, with expensive AI calls swapped for cheaper deterministic checks where the task allows. Every rule you can encode as a tool is [a rule the LLM cannot forget](/chronicles/stop-writing-rules-in-agents-md-use-agent-hooks-and-nano-staged-instead). #### What we've already shipped this way Evil Martians [cut the Rails test suite and CI time in half](/chronicles/the-whop-chop-how-we-cut-a-rails-test-suite-and-ci-time-in-half) for the Rails unicorn [Whop](/clients/whop), then scoped a full AI harness for their codebase: automated linting on skills, evals that confirmed skills behaved correctly and fired on the right requests, telemetry into skill usage and gaps, and gating that caught regressions before they shipped. In February 2026, Tether invested $200 million in [Whop](/clients/whop), driving the valuation to a whopping $1.6 billion. Evil Martians added LLM safeguards to [Playbook](/clients/playbook), cutting engineering managers' PR review time by filtering routine issues before a human reads the pull request. For the [fastest-growing AI support startup](/clients/ai-startup-under-nda), Evil Martians run forward-deployed engineering at scale. Internally, Evil Martians run the same harness architecture across our 50-person distributed team. The skills, hooks, conventions, and dashboards we use every day are the same ones we bring into client codebases. #### What it takes to start Over 20 years of consulting startups, we've found that a 2-week engagement is the right way to start. It's enough time for us to ship something valuable, and enough time for you to get a real sense of what Martians can deliver. A 2-week sprint is $14,000. A senior Martian engineer works alongside your team for two weeks. We read your Git history, strategize with your principal engineers, and ship the first working version of the harness. **From you**: access to your codebase and Git history, plus time with your principal engineers to set direction. **From us**: a senior Martian for two weeks at $14,000. Extendable at $7,000 per week. ### Product development URL: https://evilmartians.com/services/product-development Forward-deployed engineers who ship your product from PoC to PMF: embedded in your repo, your stack, your cadence. We embed directly with your team—your repo, your Slack, your standups. Not consultants who hand off a deck, but forward-deployed engineers who ship production code from day one. We've built products across AI, developer tools, cybersecurity, fintech, healthcare, and e-commerce. We come when needed and disconnect when done, so you hire full-time at your own pace. #### AI products In our long-term partnership with [StackBlitz](/clients/stackblitz), we helped scale [bolt.new](https://bolt.new/) from zero to $40M+ ARR in five months. We've been [StackBlitz](/clients/stackblitz)'s engineering partner since 2021, building the Ruby on Rails backend, enterprise features, and performance infrastructure that enabled bolt.new to become one of the fastest-growing AI products ever launched. For [Recraft](/clients/recraft), an AI image generation platform with $42M in funding, we built a custom Discord bot as an acquisition channel. For [BetterUp](/clients/betterup), we built their React frontend and scalable AWS backend for the AI coaching platform. For [GMI Cloud](/clients/gmicloud), we delivered their marketing site in weeks—$82M+ in Series A funding; they needed to move fast. We also build autonomous AI-powered workflows: LLM integrations, agent infrastructure, and the evaluation tooling that keeps AI systems reliable in production. #### Developer tools and infrastructure For [HTTPie](/clients/httpie), the API testing tool, we led the product design that hit 4th Product of the Day on Product Hunt. For [Steel](/clients/steel), an open source browser API, we built backend services within their TypeScript monorepo. For [Playbook](/clients/playbook), a cloud platform for creatives with 700K+ users, we built serverless TypeScript features, ML pipelines, and GraphQL API optimizations on Google Cloud. For [NATS](/clients/nats), we made their Ruby SDK 3x faster. For [Akeero](/clients/akeero), an AWS security audit platform, we built real-time multi-user editing with our open source sync library [Logux](/products/logux). For [Rootly](/clients/rootly), we redesigned the incident management platform frontend and backend. #### Cybersecurity [Teleport](/clients/teleport) is a Go-based privileged access management platform. We built access control plugins and transformed their documentation into a customer acquisition channel. They reached unicorn status. [Tines](/clients/tines) automates security workflows. We designed and built the product with 100x performance improvements in target areas. They reached unicorn status. For [Wallarm](/clients/wallarm), an API security platform protecting 20,000+ apps, we optimized their Go-based event processing pipeline for high-throughput security data. For [Certora](/clients/certora), a blockchain security startup backed by Tiger Global, we built a [VS Code extension](/chronicles/smartly-contracted-ui-design-and-frontend-for-certora) for smart contract verification in two months—from design sprint to marketplace. For [Ghost Security](/clients/ghost-security), we designed and built the enterprise security platform MVP. #### Ruby on Rails We are the leading Ruby on Rails consultancy. Our team created [TestProf](https://test-prof.evilmartians.io/), [AnyCable](https://anycable.io/), [Action Policy](https://actionpolicy.evilmartians.io/), and dozens of other open source tools the Rails ecosystem depends on. Our Head of Backend, Vladimir Dementyev, is the author of [Layered Design for Ruby on Rails Applications](https://www.packtpub.com/en-us/product/layered-design-for-ruby-on-rails-applications-9781801813785)—now in its second edition. We run the [SF Bay Area Ruby Meetup](/events/the-silicon-heel-sf-ruby-meetup). Here's who runs Rails with us. [bolt.new](/clients/stackblitz)—$40M+ ARR, the Rails backend behind one of the fastest-growing AI products ever launched. [Doximity](/clients/doximity)—used by 80% of US doctors. [Tines](/clients/tines)—security automation unicorn, $90M+ raised. [BetterUp](/clients/betterup)—AI coaching platform, $570M+ in funding. [Whop](/clients/whop)—$150M+ in yearly payments on a Rails monolith. [ClickFunnels](/clients/clickfunnels)—the sales funnel platform, AnyCable at scale. [Rootly](/clients/rootly)—YC-backed incident management on Hotwire. [Fountain](/clients/fountain)—YC-backed HR tech. [Chowbus](/clients/chowbus)—food delivery platform. [Healthie](/clients/healthie)—virtual-first healthcare. Lago—open source billing, $22M+ raised. We know Rails inside out because we build the tools the ecosystem runs on. #### Fullstack TypeScript We build TypeScript frontends and backends across React, Next.js, Node.js, and Electron. For [Steel](/clients/steel), an open source browser API, we built backend services within their TypeScript monorepo. For [Playbook](/clients/playbook), a cloud platform for creatives with 700K+ users, we built serverless TypeScript features, ML pipelines, and GraphQL API optimizations on Google Cloud. For [HTTPie](/clients/httpie), we built the desktop app with React, TypeScript, and Electron. For [Aptos](/clients/aptos), a Layer-1 blockchain with $350M+ in funding, we built the developer portal and AI assistant with React and Next.js. For [Certora](/clients/certora), we built a [VS Code extension](/chronicles/smartly-contracted-ui-design-and-frontend-for-certora) with TypeScript and Svelte. For [Blockscout](/clients/blockscout), the open source blockchain explorer, we designed and built the React frontend. We also maintain open source TypeScript projects used by millions: [Nano ID](https://github.com/ai/nanoid) (111M weekly npm downloads), [PostCSS](https://postcss.org/), [Nano Stores](https://github.com/nanostores/nanostores), and [size-limit](https://github.com/ai/size-limit). #### Fintech For [Aptos](/clients/aptos), a Layer-1 blockchain, we built the core developer portal and AI assistant. For [Fund That Flip](/clients/fund-that-flip), a real estate financing platform, we shipped full-stack features across their investment pipeline. For [Tesote](/clients/tesote), a treasury management platform for CFOs, we built bank integrations and monitoring systems. For [Blockscout](/clients/blockscout), the open source blockchain explorer, we designed and built the frontend. #### Healthcare and edtech For [Healthie](/clients/healthie), a virtual-first healthcare SaaS, we migrated their real-time features to [AnyCable](https://anycable.io/)—our own open source WebSocket infrastructure. For [Doximity](/clients/doximity), a digital platform for medical professionals, we [integrated Go-powered real-time features](/chronicles/growing-pains-and-a-dose-of-go-real-time-features-for-this-rails-app) into their Rails app. For [2U](/clients/2u), a large edtech platform, we executed a GraphQL migration with zero downtime. #### E-commerce and marketplaces For [Whop](/clients/whop), we [cut their Rails test suite and CI time in half](/chronicles/the-whop-chop-how-we-cut-a-rails-test-suite-and-ci-time-in-half) and [migrated from PostgreSQL to PlanetScale with zero downtime](/chronicles/migrating-whop-from-postgresql-to-planetscale-zero-downtime). For [Zipline](/clients/zipline), a retail communication platform, we improved infrastructure and built key product features with large-scale performance optimization. For [Groupon](/clients/groupon), we spent six years as part of the core tech team—scaling, shipping features, and defending against DDoS attacks. #### When to engage There are several points where working with us makes the most sense. **Right after a funding round**, when there's pressure to ship and hiring takes months. **At the scaling stage**, when you need specific expertise—AI integration, real-time infrastructure, performance optimization—that doesn't justify a full-time hire. **Pre-PMF**, when you need to validate fast: our design sprints have launched products in as little as two weeks. Many of our clients—including several backed by Y Combinator—started with us at this stage. We work with TypeScript, Python, Go, Ruby on Rails, Rust, and React. We charge $7,000/week per engineer. We run 2–3 week sprints with clear deliverables, focused on what matters for the business. ### Design for devtools URL: https://evilmartians.com/services/design-developer-tool CLI, dashboard, SDK, and AI product interfaces that developers actually love: we design UX for developer tools, not consumer apps. Most consumer app design patterns break when applied to developer tools. Developers are expert users with different needs: information density over whitespace, keyboard shortcuts over click targets, customizable layouts over guided flows. We've been designing interfaces for this audience since 2006. #### What we've designed *Read also Evil Martians' article*: https://evilmartians.com/chronicles/ui-design-for-httpie-macos-vibes-for-the-api-testing-client [HTTPie](/clients/httpie)—product design for the API testing client, hitting 4th Product of the Day on Product Hunt. [Tines](/clients/tines)—a drag-and-drop workflow builder for security teams that helped them reach unicorn status. [Ghost Security](/clients/ghost-security)—a noise-free security dashboard that groups and prioritizes events visually instead of burying them in cluttered tables. [Tegon](/clients/tegon)—a full [UI/UX redesign](/chronicles/designing-tegon-a-signature-vision-for-the-ai-driven-issue-tracker) for this YC-backed AI-first issue tracker. [Quotient](/clients/quotient)—a [prompt evaluation sandbox](/chronicles/ui-for-quotient-ai-the-developer-tool-for-prompt-engineers) for testing LLMs. [Rootly](/clients/rootly)—product redesign for a YC-backed incident management platform. [Medplum](/clients/medplum)—designing a healthcare developer platform from a demo app. #### How we think about developer UX Developer tools have no "happy path." Users jump between panels and features based on their current task. We design for three contexts: immediate (debugging a specific error), intermediate (building a feature), and broad (shipping to production). Each demands different [layout patterns](/chronicles/devtool-layout-rule) and [information density](/chronicles/keep-it-together-5-essential-design-patterns-for-dev-tool-uis). *Read also Evil Martians' article*: https://evilmartians.com/chronicles/devs-in-mind-how-to-design-interfaces-for-developer-tools We design with radio buttons over dropdowns, sliders with precise text inputs, resizable panels, and persistent interface states. These patterns are for people who live in their tools eight hours a day—whether that's a dashboard, a CLI, or an AI-powered interface. #### Documentation and onboarding The first experience with your product isn't the dashboard—it's the docs. We design documentation as a product surface: information architecture that matches how developers actually search, code examples that copy-paste cleanly, and onboarding flows that get users to their first success in minutes. For [Teleport](/clients/teleport), we transformed their documentation into a [customer acquisition channel](/chronicles/next-level-docs-with-nextjs-teleports-new-customer-aquisition-channel) with Next.js—docs that don't just explain the product but sell it. #### Design sprints for devtools Many of our client relationships start with a design sprint: two weeks from first conversation to validated product design. We've kicked off engagements this way for Tines, Ghost Security, HTTPie, Tegon, and Quotient. For pre-PMF startups, this is the fastest path from idea to something you can test with real developers. ### Frontend & real-time URL: https://evilmartians.com/services/real-time Dev tool interfaces, small libraries, design systems, real-time collaboration, LLM streaming, and voice AI: the frontend infrastructure for products developers use daily. AI products live or die by how fast they feel. When an agent generates code, the user needs to see tokens streaming in real time. When teammates collaborate on a document, edits need to appear instantly. When a dashboard monitors production systems, data can't be seconds stale. The frontend infrastructure behind these interactions is a specialty—not something you bolt on at the end. #### Zero-latency interfaces Every spinner, every loading state, every moment your product feels sluggish—that's a user losing flow state. For professional tools where people spend hours per day, latency isn't a UX issue. It's a retention issue. We build zero-latency interfaces using local-first principles: data lives on the device first, syncs in the background, and works offline by default. [Optimistic UI patterns](/chronicles/optimistic-ui-in-rails-with-optimism-and-inertia) make server-rendered apps feel instant. WebAssembly runs [heavy computation in the browser](/chronicles/ruby-on-rails-on-webassembly-a-guide-to-full-stack-in-browser-action) without server round-trips. [bolt.new](/clients/stackblitz)—$40M+ ARR, the fastest-growing AI product ever launched—runs a full Node.js environment inside the browser via WebAssembly. No server round-trips for builds, installs, or file operations. #### Dev tool interfaces Developer tools demand interfaces that handle high information density, complex interactive workflows, and keyboard-first navigation. We build the frontend engineering behind these products. For [Tines](/clients/tines), we built a React frontend with interactive node diagrams and real-time state management across hundreds of connected actions, delivering [100x performance improvements](/chronicles/product-design-that-sells-the-smart-ux-for-tines) in target areas. Tines reached unicorn status. For [Tegon](/clients/tegon), we engineered the React and Next.js frontend for a YC-backed AI-first issue tracker. For [HTTPie](/clients/httpie), we built the desktop app with React, TypeScript, and Electron. We maintain [KeyUX](https://github.com/ai/keyux), an open source library that adds keyboard navigation to web apps: hotkey support via `aria-keyshortcuts`, arrow navigation in menus, and focus management. These are the interaction patterns professional tools need but most teams skip. #### Small libraries The modern frontend ecosystem ships megabytes of JavaScript for routine tasks. We started a small libraries movement to push back: purpose-built packages that solve one problem in as few bytes as possible. [Nano ID](https://github.com/ai/nanoid) generates unique string IDs in 116 bytes—130 times smaller than uuid. It has 111M weekly npm downloads, more than React itself. [Nano Stores](https://github.com/nanostores/nanostores) is a tiny state manager (334 bytes) that works across React, Vue, Svelte, and vanilla JS. [size-limit](https://github.com/ai/size-limit) enforces bundle size budgets in CI. If a dependency pushes your bundle past the limit, the build fails before the PR merges. We also maintain [PostCSS](https://postcss.org/) (158M weekly downloads) and [Browserslist](https://github.com/browserslist/browserslist) (118M weekly downloads)—both bigger than React. This isn't just open source advocacy. When we build frontends for clients, we reach for the smallest tool that solves the problem. Fewer bytes mean faster loads, less complexity, and lower infrastructure costs. #### Design systems for agentic development When AI coding tools generate UI, they're only as good as the component system they work with. A well-structured design system with clear component APIs, typed props, and documented patterns produces consistent agent output. A messy one produces slop. We build component architectures that work for both humans and agents. In Rails, our [view_component-contrib](https://github.com/palkan/view_component-contrib) gem standardizes [ViewComponent](/chronicles/viewcomponent-in-the-wild-embracing-tailwindcss-classes-and-html-attributes) interfaces across the application. In React and TypeScript, we build typed component libraries with explicit prop APIs. The component catalog doubles as a specification that agents follow—the same principle behind [AI Harnesses](/services/agent-experience) applied to the view layer. #### Real-time collaboration *Read also Evil Martians' article*: https://evilmartians.com/chronicles/collaborative-real-time-security-logux-for-akeero For [Akeero](/clients/akeero), an AWS security audit platform, we built real-time multi-user editing with [Logux](/products/logux)—our open source library for local-first sync. Edits appear instantly for all participants, with conflict resolution handled automatically. For [Healthie](/clients/healthie), a virtual-first healthcare SaaS, we migrated their real-time features to [AnyCable](https://anycable.io/) for better performance and reliability. For [ClickFunnels](/clients/clickfunnels), we stabilized their AnyCable infrastructure at scale. We've solved WebSocket scaling, [connection avalanches](/chronicles/connection-avalanche-safety-tips-and-prepping-for-realtime-applications), and delivery guarantees at production scale. #### LLM streaming and generative UI Streaming AI responses over HTTP works until it doesn't. [Token-by-token delivery](/chronicles/anycable-rails-and-the-pitfalls-of-llm-streaming) over WebSockets solves the problems HTTP streaming creates: dropped connections, backpressure handling, multiplexing multiple streams, and delivering structured UI updates alongside raw tokens. We built this infrastructure for AI products and open sourced the patterns through [AnyCable](https://anycable.io/). #### Voice AI Voice AI needs real-time infrastructure with stricter latency requirements than chat or collaboration—bidirectional audio streaming, speech-to-text processing, and sub-second response times. We've built this stack with AnyCable. [Doximity](/clients/doximity), used by 80% of US doctors, ships voice features powered by AnyCable—including "Hold for Me," where a digital assistant handles phone holds and alerts the doctor when a real person picks up. We built a [voice assistant connecting Twilio with OpenAI's Realtime API](/chronicles/anycable-speaking-needing-help-with-a-twilio-openai-connection) through AnyCable Go—bidirectional audio, speech-to-text, and tool calling over WebSockets. We've also [connected Twilio streams with offline speech recognition](/chronicles/anycable-goes-off-rails-connecting-twilio-streams-with-hanami) for real-time call transcription. [LiveVoice](https://blog.anycable.io/p/livevoice-shares-how-they-use-anycable), an ultra-low-latency audio streaming platform for simultaneous interpretation, runs on AnyCable Pro. #### When you need this Zero-latency interfaces, devtool dashboards, interactive workflow builders, real-time collaboration, LLM streaming, voice AI, design systems, component architectures. If your product needs a frontend that's fast, interactive, and built for daily professional use, we've done it before. ### Ruby on Rails consulting URL: https://evilmartians.com/services/ruby-on-rails-consulting The leading Ruby on Rails consultancy for startups: 100+ open source projects, 25B+ downloads. Shipping production code at bolt.new, Whop, Lago, and ClickFunnels. Evil Martians have been building Rails applications since 2006, we maintain 100+ open source projects with 25 billion+ downloads, and we wrote the acclaimed book on Layered Ruby on Rails architecture. We're the team that helps you build and grow your Ruby and Rails products, while improving team's developer productivity. Based in New York and San Francisco, we work with startups globally. #### Your Rails app needs to ship faster You just raised a round and need to ship before the runway gets short. Or your team is small and hiring senior Rails engineers takes months. We embed directly with your team—your repo, your Slack, your standups. Not consultants who hand off a deck, but engineers who ship production code from day one. We build MVPs, APIs, billing systems, admin dashboards. We build on Rails 8 and help teams upgrade. We _teach_ your agents to generate maintainable code. We come when needed and disconnect when done, helping you hire internal team at your own pace. We helped [StackBlitz](/clients/stackblitz) build bolt.new's Rails backend from early architecture through $40M+ ARR. We've been their engineering partner since 2021. For [Lago](/clients/lago)—the YC-backed open source billing platform with $22M in funding—we are shipping product features while maintaining their high quality bar. For [BetterUp](/clients/betterup)—the AI coaching platform with $570M+ in funding—we built the React frontend and scalable AWS backend. For [Fountain](/clients/fountain), a YC-backed startup, we shipped features and scaled infrastructure. Many of our clients are backed by Y Combinator. #### Your Rails app needs to scale The app that got you to product-market fit isn't the app that handles 10x the traffic. Slow queries, background jobs backing up, test suites that take 30 minutes. We've fixed all of these. For [Whop](/clients/whop)—$150M+ in yearly payments on a Rails monolith—we [cut CI time in half](/chronicles/the-whop-chop-how-we-cut-a-rails-test-suite-and-ci-time-in-half) and [prepared the zero-downtime migration of 750GB from PostgreSQL to PlanetScale](/chronicles/migrating-whop-from-postgresql-to-planetscale-zero-downtime). For [ClickFunnels](/clients/clickfunnels), we [eliminated flaky tests](/chronicles/flaky-tests-be-gone-long-lasting-relief-chronic-ci-retry-irritation) and improved CI reliability from ~80% to near-100% across 9K+ unit tests. For PowerHRG, we optimized their CI pipeline. For [Tines](/clients/tines), a security automation unicorn with $90M+ raised, we delivered 100x performance improvements in target areas. For [Doximity](/clients/doximity)—used by 80% of US doctors—we [integrated Go-powered real-time features](/chronicles/growing-pains-and-a-dose-of-go-real-time-features-for-this-rails-app) into their Rails app. We built the tools Rails teams reach for when things get slow. [TestProf](https://test-prof.evilmartians.io/) diagnoses and fixes slow test suites. Thousands of teams use it daily. [AnyCable](https://anycable.io/) replaces Action Cable with a Go-based WebSocket server that uses 3x less memory. It powers real-time at [ClickFunnels](/clients/clickfunnels), [CoinGecko](https://www.coingecko.com/), and [Doximity](/clients/doximity). Scaling Rails doesn't mean rewriting it—it means [upgrading the cookware while keeping the recipes](/chronicles/baking-with-rails-at-scale-recipes-in-ruby-cookware-from-go-c-rust). You keep business logic in Ruby and delegate computationally intensive work to specialized tools in Go, C, and Rust: [imgproxy](https://imgproxy.net/) for image processing, AnyCable for WebSockets, [serverless functions for file processing](/chronicles/how-to-make-serverless-file-processing-a-piece-of-cake). For [Playbook](/clients/playbook)—a cloud platform for creatives with 700K+ users—we built serverless file processing pipelines on Google Cloud alongside their Rails app. #### Your Rails architecture is holding you back The monolith that worked at 3 engineers doesn't work at 30. Domain boundaries are blurred, callbacks are everywhere, and new features keep breaking old ones. Our Head of Backend, Vladimir Dementyev, wrote the highly-rated [Layered Design for Ruby on Rails Applications](/products/layered-design-book)—now in its second edition. The patterns in that book—faceless service objects replaced by form objects, workflows and operations, authorization extracted to [Action Policy](https://actionpolicy.evilmartians.io/), notifications unified through [Active Delivery](https://github.com/palkan/active_delivery), configuration made type-safe with [Anyway Config](https://github.com/palkan/anyway_config)—are the patterns we apply on every engagement. For the frontend layer, we use [Inertia Rails](https://inertia-rails.dev/) with [inertia_rails-contrib](https://github.com/skryukov/inertia_rails-contrib) (built by us)—React or Vue frontends powered by Rails, no separate API. We've refactored codebases at [Whop](/clients/whop), [Tines](/clients/tines), and [ClickFunnels](/clients/clickfunnels) without risky rewrites. For [Monograph](/clients/monograph)—$30M in funding from Index Ventures and Tiger—we migrated the backend to GraphQL and moved the frontend from the monolith to Next.js. #### Your Rails app needs real-time features Chats, notifications, typing indicators, presence, cursors, collaboration, live dashboards—most Rails apps need real-time eventually. Action Cable and Solid Cable are not delivering that instant and reliable real-time UX. We built [AnyCable](https://anycable.io/) to solve exactly this. It keeps the familiar Action Cable API while handling connections in Go—3x less memory, delivery guarantees, resumable sessions. For Hotwire-only apps, setup is simple: swap the adapter. For [ClickFunnels](/clients/clickfunnels), we run AnyCable at scale. For [Healthie](/clients/healthie), we migrated their real-time features to AnyCable. For [Doximity](/clients/doximity), we built Go-powered real-time features alongside their Rails app. #### You're adding autonomous AI workflows to your Rails product Rails is a great backend for AI products—bolt.new proved it at $40M+ ARR. We build autonomous agent pipelines in Ruby that triage issues, plan fixes, write code, self-review, open PRs, and loop on CI failures and review feedback until done. No human in the loop. Our internal agent systems run on Ruby, Roda, Sequel, and PostgreSQL with RubyLLM, checkpoint-recovery, per-task budget tracking, and CI-driven refinement. We integrate LLM APIs, build agent infrastructure, and add AI-powered features to existing Rails applications. For real-time AI responses, we built [AnyCable's LLM streaming support](/chronicles/anycable-rails-and-the-pitfalls-of-llm-streaming)—streaming LLM output over WebSockets to your frontend. We build AI Harnesses—open source skill sets that encode architectural patterns into AI coding tools like Claude Code and Codex, so agentic coding produces better Rails code. If you're building autonomous AI workflows on Ruby, this is our territory. #### Your Ruby app needs concurrency done right Threads, Fibers, Ractors, connection pools—Ruby's concurrency model has matured, but getting it right is still hard. Race conditions, non-atomic database transactions, and thread-safety bugs are invisible until they hit production under load. We maintain [rubocop-thread_safety](https://github.com/rubocop/rubocop-thread_safety) (21M+ downloads) to catch concurrency bugs statically, and [isolator](https://github.com/palkan/isolator) to detect non-atomic interactions within database transactions at runtime. We made the [NATS](/clients/nats) Ruby SDK 3x faster. NATS is a high-performance messaging system where concurrency correctness is everything. Our team speaks on this topic at conferences around the world: [testing multithreaded code](/events/dont-be-thread-ened-euruko) (Julia Egorova, EuRuKo 2025), [threads and execution context in Ruby](/events/ruby-threads-callbacks-osakarubykaigi) (Vladimir Dementyev, RubyKaigi and RubyConf AU). #### Why Evil Martians for Ruby on Rails We don't just use Rails—we build the tools the ecosystem runs on. 25 billion+ combined downloads across 100+ open source projects: [TestProf](https://test-prof.evilmartians.io/), [AnyCable](https://anycable.io/), [Action Policy](https://actionpolicy.evilmartians.io/), [Lefthook](https://github.com/evilmartians/lefthook) (1.2M weekly npm downloads), [Yabeda](https://github.com/yabeda-rb/yabeda), [Logidze](https://github.com/palkan/logidze), [isolator](https://github.com/palkan/isolator), [anyway_config](https://github.com/palkan/anyway_config), [Active Delivery](https://github.com/palkan/active_delivery), and dozens more. We keynote at all major Ruby and Rails conferences globally. We organize the [SF Ruby Conference](https://sfruby.com)—450+ attendees, 40+ speakers, demos from companies like bolt.new, Chime, and Gusto—and the [SF Bay Area Ruby Meetup](/events/the-silicon-heel-sf-ruby-meetup). We write books on writing maintainable sofwtare and receive awards for our open source projects. Eight of our clients grew from seed to unicorn. Here's who trusts us with their Rails code: [bolt.new](/clients/stackblitz) ($40M+ ARR), [Whop](/clients/whop) ($150M+ yearly payments), [Tines](/clients/tines) (unicorn, $90M+ raised), [Doximity](/clients/doximity) (80% of US doctors), [BetterUp](/clients/betterup) ($570M+ in funding), [ClickFunnels](/clients/clickfunnels), [Lago](/clients/lago) ($22M+ raised, open source billing), [Playbook](/clients/playbook) ($22M+ raised, 700K+ users), [Monograph](/clients/monograph) ($30M, Index Ventures), [Rootly](/clients/rootly) (YC-backed), [Fountain](/clients/fountain) (YC-backed), [Keygen](/clients/keygen), [Synonym](/clients/synonym) (a16z-backed), [Chowbus](/clients/chowbus), [Healthie](/clients/healthie), PowerHRG, and [NATS](/clients/nats). #### When to engage There are several points where working with us makes the most sense. **Right after a funding round**, when there's pressure to ship and hiring takes months. **At the scaling stage**, when your Rails app needs specific expertise—performance optimization, real-time infrastructure, database scaling—that doesn't justify a full-time hire. **Pre-PMF**, when you need to validate fast: our design sprints have launched products in as little as two weeks. We charge $7,000/week per engineer. We run 2–3 week sprints with clear deliverables. 65% of our clients raise their next round within 1–3 years. ### Platform engineering URL: https://evilmartians.com/services/sre Deploy anywhere and scale to millions: Kubernetes, Terraform, Go infrastructure, and cloud platforms for developer tools. Your product needs to deploy to Kubernetes, scale across cloud providers, and ship updates multiple times per day without breaking production. We've built this infrastructure for AI products, developer tools, and security platforms. We bring open source tooling we created specifically for this work. #### What we've built We've been [StackBlitz](/clients/stackblitz)'s infrastructure partner since 2021—building the backend and performance systems that support 3M+ monthly developers and bolt.new's explosive growth to $40M+ ARR. For [Whop](/clients/whop), we [cut their Rails test suite and CI time in half](/chronicles/the-whop-chop-how-we-cut-a-rails-test-suite-and-ci-time-in-half) and executed a [zero-downtime migration from PostgreSQL to PlanetScale](/chronicles/migrating-whop-from-postgresql-to-planetscale-zero-downtime). [Teleport](/clients/teleport) is a privileged access management platform used by 100+ enterprise customers. We built their Go-based plugin ecosystem and deployment infrastructure. [NATS](/clients/nats) is an open source messaging system for distributed systems. We improved their Ruby SDK performance by 3x. [BetterUp](/clients/betterup) needed a scalable AWS backend for their AI coaching platform ($570M+ in funding). We built it. #### Kubernetes and deployment Our [Martian Kubernetes Kit](/chronicles/martian-kubernetes-kit-running-apps-and-running-them-well) is a battle-tested toolkit for running production workloads on Kubernetes—covering deployment, monitoring, autoscaling, and security out of the box. We use it with our clients and ship it as open source. For teams not ready for Kubernetes, we work with [Kamal](/chronicles/mrsk-hot-deployment-tool-or-total-game-changer) for zero-downtime deployments, Docker-based development environments, and managed cloud platforms on AWS and GCP. #### Multi-tenancy and self-hosted deployments Enterprise customers want your product on their infrastructure. That means multi-tenant isolation, per-tenant resource limits, data residency compliance, and deployment automation that works across AWS, GCP, and on-prem Kubernetes clusters. We've built this for [bolt.new](/clients/stackblitz)—on-prem Kubernetes deployments for enterprise customers—and we bring that experience to every product that needs to deploy anywhere without the multi-tenancy pain. We handle [fair job prioritization](/chronicles/fair-multi-tenant-prioritization-of-sidekiq-jobs-and-our-gem-for-it) so one tenant's bulk import doesn't starve the rest, and the deployment tooling to ship updates across all environments simultaneously. #### CI/CD and developer productivity Slow CI kills momentum. We profile test suites with [TestProf](https://test-prof.evilmartians.io/) (our open source tool), eliminate [flaky tests](/chronicles/flaky-tests-be-gone-long-lasting-relief-chronic-ci-retry-irritation), optimize build pipelines, and set up monitoring that tells you what's wrong before users notice. The goal: your team deploys confidently, multiple times per day—with alerting, distributed tracing, and infrastructure as code via Terraform. ### Performance & scale URL: https://evilmartians.com/services/performance-scalability Handle explosive growth without re-architecting: database scaling, background jobs, and the profiling to know where it hurts. Your product is growing faster than your infrastructure can handle. The answer isn't a rewrite—it's knowing exactly where it hurts and fixing that. We profile, measure, and fix. We've done this for products handling millions of users, and we bring open source tools we built specifically for this work. #### What we've scaled We helped [StackBlitz](/clients/stackblitz) scale bolt.new through explosive growth—from zero to $40M+ ARR in five months. We've been their engineering partner since 2021, building the performance infrastructure that kept the product standing while usage went vertical. For [Whop](/clients/whop), we [cut their Rails test suite and CI time in half](/chronicles/the-whop-chop-how-we-cut-a-rails-test-suite-and-ci-time-in-half) and [migrated 750GB from PostgreSQL to PlanetScale with zero downtime](/chronicles/migrating-whop-from-postgresql-to-planetscale-zero-downtime)—dual-database mode, schema dialect differences, all while serving millions of users. For [Tines](/clients/tines), a security automation platform, we delivered 100x performance improvements in target areas. For [Wallarm](/clients/wallarm), we optimized their Go-based event processing pipeline for high-throughput security data. For [NATS](/clients/nats), we made their Ruby SDK 3x faster. #### Database architecture We design PostgreSQL schemas that scale without rewrites. Our [partitioning strategies](/chronicles/flying-free-a-tale-of-partitioning-with-canaries-in-postgresql) handle tables with tens of millions of rows and hundreds of gigabytes of data. We've executed zero-downtime database migrations for high-traffic products. We know when to partition, when to shard, when to add read replicas, and when the real problem is your query patterns—not your database. #### Background jobs and resilience We build [fair background job prioritization](/chronicles/fair-multi-tenant-prioritization-of-sidekiq-jobs-and-our-gem-for-it) for multi-tenant Sidekiq, so one customer's bulk import doesn't starve everyone else. We add [circuit breakers](/chronicles/circuit-breakers-and-ruby-in-2025-dont-break-your-launch) for production resilience—so a failing third-party API doesn't cascade into a full outage. We build [API-first architectures](/chronicles/contract-shock-therapy-the-way-to-api-first-documentation-bliss) with contract-driven development, so your services stay reliable as the team grows. #### Multi-language performance Most products outgrow a single language. We know when to keep Rails for auth, billing, and CRUD—and when to reach for [Go, C, or Rust](/chronicles/baking-with-rails-at-scale-recipes-in-ruby-cookware-from-go-c-rust) for data pipelines, agent runtimes, or high-throughput processing. For [Doximity](/clients/doximity), we [integrated Go-powered real-time features](/chronicles/growing-pains-and-a-dose-of-go-real-time-features-for-this-rails-app) into their Rails app. For [Steel](/clients/steel), we built backend services within their TypeScript monorepo. #### Profiling and test performance We built [TestProf](https://test-prof.evilmartians.io/), the open source toolkit for [diagnosing and fixing slow Ruby test suites](/chronicles/testprof-a-good-doctor-for-slow-ruby-tests)—used by thousands of teams worldwide. For [ClickFunnels](/clients/clickfunnels), we improved CI reliability from ~80% to near-100% across 9K+ unit tests and 1K+ feature tests. Slow tests mean slow deploys, which means slow shipping. In a world where AI writes code and tests verify it, fast tests are the bottleneck. We bring the same profiling discipline to application performance: flamegraphs, allocation tracking, query analysis, and the experience to know which numbers actually matter. Our Head of Backend, Vladimir Dementyev, wrote [Layered Design for Ruby on Rails Applications](https://www.packtpub.com/en-us/product/layered-design-for-ruby-on-rails-applications-9781801813785)—now in its second edition. The patterns in that book—service objects, query objects, bounded contexts—are also performance patterns. Clean architecture is fast architecture. #### Why Rails teams call us We're one of the leading Ruby on Rails consultancies—and performance is where it starts. We created the tools Rails teams reach for when things get slow: TestProf, [AnyCable](https://anycable.io/), [Action Policy](https://actionpolicy.evilmartians.io/), [Yabeda](https://github.com/yabeda-rb/yabeda). We run the [SF Bay Area Ruby Meetup](/events/the-silicon-heel-sf-ruby-meetup). If your Rails app is the bottleneck, we've fixed it before. ### Design engineering URL: https://evilmartians.com/services/design-engineer Ship features in weeks, not months. Our designers eliminate the design-to-dev handoff using AI-powered workflows and direct code implementation. #### 2-week design engineering sprint—$14,000 One Martian designer who codes. AI-powered prototyping with bolt.new, Claude, and no-code tools. Direct code fixes by the designer who created it—zero handoff. Production-ready output your team integrates immediately. Need more scope? Add weeks at $7,000 each. --- *Book a free 30-minute consultation to define exactly what we can ship for you in a 2-week design engineering sprint.* [Contact Evil Martians](https://evilmartians.com/contact-us) --- #### What we've shipped this way [GMI Cloud](/clients/gmicloud)—a conversion-driven website for this $82M+ GPU cloud infrastructure provider, [delivered in 3.5 weeks](/chronicles/gmi-cloud-meets-webflow-and-speeds-ahead-in-the-ai-compute-race) with Webflow and Spline.

🚀🌟 A huge shoutout to our friends at @evilmartians for helping us bring the new @gmi_cloud website to life in just 3.5 weeks! Your design, animation, and development magic made it happen! 🙌💻✨ #Webflow #SplineTool #GMICloud https://t.co/6UshrGo4K7

— GMI Cloud (@gmi_cloud) July 24, 2024
[Aptos Network](/chronicles/super-speed-super-quality-lessons-from-the-aptos-network-site-launch)—a full website launch in one month using Astro, React, and Tailwind CSS with AI-assisted development. #### How it works Our designers use developer tools daily and understand code. When they spot a UI issue, they fix it directly—no ticket, no handoff, no waiting. They prototype in bolt.new, validate with real users, then hand off clean code to your engineers. One designer [compressed weeks of work into days](/chronicles/weeks-to-days-a-case-for-expert-led-ai-driven-design-engineering) using AI-assisted workflows—exploring dozens of chart types in Google Colab with Gemini, automating Figma layouts with custom scripts, and building interactive prototypes directly in bolt.new. #### Tools we've built this way [Harmonizer](https://harmonizer.evilmartians.com/)—a [Figma plugin](https://www.figma.com/community/plugin/1483474069475958506/harmonizer) for generating accessible, consistent color palettes. Prototyped in bolt.new, validated fast, then polished by an engineer. *Image: Harmonizer: a tool for generating accessible, consistent color palettes for user interfaces* [Code to markup](https://psd-coder.github.io/code-to-markup/)—an open source tool for converting code snippets into HTML for static sites. Built by a single engineer using Claude, from problem to shipped solution in days. *Image: Convert your code snippets into beautiful HTML markup for your static websites* ### Security product design URL: https://evilmartians.com/services/security-product-design Design and build security products that technical teams trust: threat dashboards, workflow automation, and access control interfaces for cybersecurity startups. #### Why security product design is different Security products serve users making high-stakes decisions under time pressure—incident response, vulnerability triage, access approvals. The tolerance for confusion is zero. Dense threat data needs clear hierarchy. Complex workflows need visual clarity. And the users are often security analysts and ops teams, not developers—a different design paradigm than most developer tools. We've designed security products that reached unicorn status. Twice. #### What we've designed and built [Tines](/clients/tines) is a security automation platform. We designed and built the core product, including the visual workflow builder that lets security teams automate complex processes without code. We covered UI design, React/GraphQL frontend, and Ruby on Rails backend, delivering [100x performance improvements](/chronicles/product-design-that-sells-the-smart-ux-for-tines) in target areas. Tines has raised $90M+ and reached unicorn status. [Ghost Security](/clients/ghost-security) helps enterprises find application vulnerabilities. We designed their MVP and built the Vue.js/TypeScript frontend—the threat visualization dashboard, vulnerability detail views, and the information architecture for complex security data. They raised $15M. Their CTO called us "truly in an elite class of modern design experts." [Teleport](/clients/teleport) is a privileged access management platform used by 100+ enterprise customers. We built Go-based access control plugins for Slack, Jira, PagerDuty, GitLab, and Mattermost—creating an ecosystem that made Teleport indispensable to ops teams. We transformed their documentation into a [customer acquisition channel](/chronicles/next-level-docs-with-nextjs-teleports-new-customer-aquisition-channel) with Next.js. Teleport has raised $170M and reached unicorn status. *Read also Evil Martians' article*: https://evilmartians.com/chronicles/smartly-contracted-ui-design-and-frontend-for-certora [Certora](/clients/certora) builds smart contract verification tools. We designed and built a [VS Code extension](/chronicles/smartly-contracted-ui-design-and-frontend-for-certora) for formal verification in two months, from design sprint to marketplace. The interface visualizes complex proof results in a way that blockchain security engineers can act on immediately. #### Security infrastructure Security products typically run on Go, TypeScript, and Python. We work across all three. For [Wallarm](/clients/wallarm), an API security platform protecting 20,000+ apps and ranked in the Top 150 Cybersecurity Companies, we optimized their Go-based event processing pipeline for high-throughput security data. We understand the architecture patterns behind threat detection, access control, and security data pipelines. #### AI and agentic security The next wave of security products protects AI systems—agent-to-agent communication, MCP tool integrations, LLM-powered APIs. We bring both security product design experience (Tines, Teleport, Wallarm) and AI infrastructure expertise (bolt.new, [AgentPrism](/chronicles/debug-ai-fast-agent-prism-open-source-library-visualize-agent-traces), [AI Harnesses](https://github.com/palkan/skills)) to teams building in this space. ### SDKs, extensions & plugins URL: https://evilmartians.com/services/plugin-and-extension-creation Build VS Code extensions, Figma plugins, browser extensions, and SDKs that make your developer tool indispensable. Your developer tool's ecosystem — extensions, plugins, SDKs — is what turns users into advocates. We've built all of these, for our clients and for our own open source projects. #### IDE extensions For [Certora](/clients/certora), a Series B blockchain security startup, we built a [VS Code extension](/chronicles/smartly-contracted-ui-design-and-frontend-for-certora) for smart contract verification in just two months — from design sprint to marketplace. We've also built [IntelliJ IDEA language support](/chronicles/the-scenic-route-lessons-building-intellij-idea-language-support) and documented the hard lessons of IDE plugin development across both platforms. #### Browser extensions and Figma plugins For [Playbook](/clients/playbook), a Series A storage platform for creatives, we built [Chrome extensions](/chronicles/how-to-make-complex-chrome-plugins-a-zero-gravity-guide), [converted them to Safari](/chronicles/how-to-quickly-and-weightlessly-convert-chrome-extensions-to-safari), and built a Figma plugin — all part of a multi-platform ecosystem strategy. *Image: Viewing images in the Gallery* *Image: Searching images* We build our own Figma plugins too: [Harmonizer](https://www.figma.com/community/plugin/1483474069475958506/harmonizer) for accessible color palettes, built with [advanced Figma API patterns](/chronicles/how-to-make-next-level-figma-plugins-auth-routing-storage-and-more) we've documented extensively. #### SDK development and optimization Developer tools [live or die by their SDKs](/chronicles/the-sdk-imperative-why-developer-tools-live-or-die-by-their-sdks). We optimized the Ruby SDK for [NATS](/clients/nats), an open source messaging system, achieving 3x performance improvement for high-load cases. This was a major enhancement to the client so I appreciate the practical solution that was implemented to boost the client's performance and scalability. #### Plugin ecosystems For [Teleport](/clients/teleport), the privileged access management platform, we built and documented five plugins — Slack, Jira, Mattermost, GitLab, and PagerDuty — creating an ecosystem that made Teleport indispensable to ops teams. ### Devtools startup advisory URL: https://evilmartians.com/services/advisory Refine your strategy and accelerate product-market fit for your devtools startup through a 6-month advisory program with our CEO. #### 6-month advisory program Six focused 1-hour sessions with Evil Martians CEO [Irina Nazarova](/martians/irina-nazarova). Irina advises devtools founders on growth strategy, product analytics, and business forecasting. She runs the [SF Ruby](https://sfruby.com/) community, speaks internationally about the business of developer tools, and meets hundreds of devtools founders every year. This is not generic startup coaching. Irina works exclusively with founders of technical products targeting software developers—the market she’s operated in since 2006. #### What we cover - **Positioning and pricing**—[value-based pricing models](/chronicles/value-pricing-for-dev-tools-a-strategy-beneficial-for-both-sides) that align with how developers evaluate and buy tools - **Fundraising strategy**—who invests in developer tools, and how to [approach them effectively](/chronicles/how-to-cultivate-capital-6-tips-from-developer-tools-founders-and-vcs) - **Product-market fit**—signals that matter for devtools: adoption curves, community engagement, and usage patterns - **Go-to-market**—open source strategy, developer community building, launch execution, and [what founders making $100M+ ARR have learned](/chronicles/what-devtools-founders-making-100m-arr-told-in-san-francisco) #### Why this works Evil Martians works with 40+ developer tools startups every year. Eight grew from seed to unicorn. 65% raise Series A or B within one to three years. Irina brings pattern recognition from this portfolio—[what to validate before building](/chronicles/founders-if-you-want-a-plus-startup-success-do-the-homework-first), when to invest in platform engineering, and where the devtools market is heading. ### Developer marketing URL: https://evilmartians.com/services/tech-marketing Convert technical audiences with landing pages, documentation, launch strategies, and content that developers actually trust. #### Landing pages that convert We [studied 100 developer tool landing pages](/chronicles/we-studied-100-devtool-landing-pages-here-is-what-actually-works-in-2025) and documented what actually works: specific metrics over vague claims, working code examples over screenshots, and CTAs that respect developers' intelligence. Our [conversion optimization](/chronicles/how-to-kill-conversions-on-your-developer-tool-landing-page) work shows how common mistakes — vague hero text, invisible CTAs, corporate speak — kill developer trust. #### Documentation as acquisition For [Teleport](/clients/teleport), we transformed their documentation into a [customer acquisition channel](/chronicles/next-level-docs-with-nextjs-teleports-new-customer-aquisition-channel) with Next.js — not just reference docs, but a designed experience that brought developers in and kept them. We've also built [API-first documentation workflows](/chronicles/contract-shock-therapy-the-way-to-api-first-documentation-bliss) for contract-driven development. #### Content and launch strategy Engineer-written content [generated $700K in revenue from 6 articles](/chronicles/creating-a-winning-sales-flow-with-authentic-writing-from-engineers) for one of our clients. We help developer tool teams [run launch weeks](/chronicles/how-to-do-launch-weeks-for-developer-tools-startups-and-small-teams) that drive adoption — our own AnyCable launch week achieved 5x trial signups. We've documented how AI startups [use changelogs to build developer trust](/chronicles/how-ai-startups-use-changelogs-to-win-developer-trust). We know what works because we do it ourselves — 100+ open source projects, a technical blog read by millions, and conference talks at RailsConf, RubyConf, and dozens of developer events worldwide. ### Hiring & training URL: https://evilmartians.com/services/training Build and upskill engineering teams with hands-on training from engineers who ship production systems at scale. #### Knowledge transfer that sticks When we embed with a client team, we don't just ship code — we transfer skills. For [Doximity](/clients/doximity), we ran training sessions so their engineers could maintain their AnyCable real-time infrastructure independently. For [Iterative](/clients/iterative), we [designed and built the DVC Studio UI](/chronicles/designing-machine-learning-for-dvc-studio), then enabled their frontend team to own it going forward. *Image: In a project for Iterative, an open source developer tools company, we [trained their frontend team](/chronicles/designing-machine-learning-for-dvc-studio) to work independently on the DVC Studio product.* We've documented the full process: our guide to [transferring projects to another team](/chronicles/the-sword-is-yours-transferring-projects-to-another-team-unscathed) covers architecture documentation, onboarding playbooks, and the handover practices we've refined over hundreds of engagements. #### Hiring process design [Our own hiring process](/chronicles/hire-better-10x-faster-how-we-screen-50-engineers-weekly-with-no-hr) screens 50+ engineers weekly without HR, focusing on fundamentals over frameworks — semantic HTML, accessibility, performance optimization, and problem-solving approach. We help developer tools companies design hiring funnels that find the right engineers faster while respecting candidates' time. #### Team mentorship We embed with your engineers for intensive pair programming, code reviews, and architecture discussions. Teams learn by shipping real features alongside experienced engineers — not through toy examples or slide decks. ## Clients ### AI concierge tool for enterprises URL: https://evilmartians.com/clients/ai-startup-under-nda Evil Martians partnered with a renowned agent support platform to streamline AI-driven development, building a scalable design system and efficient developer workflows. We delivered secure, enterprise-ready infrastructure, from DevContainers to production VoIP integrations. Industries: Developer Tools Key facts: $250M raised; $3B+ total valuation ### Bolt.new URL: https://evilmartians.com/clients/stackblitz Evil Martians and StackBlitz: tech partners since 2021. Our engineers helped scale their WebContainers platform from pioneering browser IDE to bolt.new—the AI-powered tool that hit $20M+ ARR in just 2 months and disrupted the industry of vibe coding. We've built core infrastructure powering 3M+ monthly developers, from backend Rails to enterprise features and performance optimization. Together, we're transforming how developers build worldwide. Industries: Developer Tools, AI Key facts: $113M total funding; $700M valuation ### Tines URL: https://evilmartians.com/clients/tines Evil Martians helped an advanced security orchestration and automation platform, Tines, with the core product, covering UI design, frontend, and backend. Our team designed a pro-oriented interface easy-to-use even for newcomers. Industries: Cyber Security, Developer Tools Key facts: $271M total funding; $1.125B valuation ### Teleport URL: https://evilmartians.com/clients/teleport Evil Martians has partnered with Teleport since 2020, engineering enterprise-ready features for their open-source infrastructure access platform. Our engineering and design teams work across the platform – from architecting the plugin ecosystem and deployment strategies to transforming their documentation into a powerful growth driver with Next.js. Combining deep Go expertise with product thinking, we continue as their technical partner driving R&D initiatives as well as building core features that support their growth.. Industries: Cloud Infrastructure, Developer Tools, Cyber Security, Y Combinator Key facts: $169M total funding; $1.1B valuation ### Daylight Security URL: https://evilmartians.com/clients/daylight-security Daylight Security is a cybersecurity platform that combines autonomous AI workflows with expert-led SOC/MDR services. As the platform scaled toward enterprise adoption, Evil Martians joined as a design partner to elevate the product experience. Industries: Developer Tools, Cyber Security, AI Key facts: $40M total funding ### NATS URL: https://evilmartians.com/clients/nats NATS by Synadia is an open-source messaging system, a secure connective technology for modern distributed systems at the edge and in the cloud. They reached out to Evil Martians to improve the performance of NATS's Ruby SDK, and we ended up speeding it up by more than three times for high-load cases. Industries: Developer Tools Key facts: $25M total funding ### Gusto URL: https://evilmartians.com/clients/gusto Gusto runs payroll for hundreds of thousands of American businesses, and a large share of that work happens asynchronously, in the background, on a deadline. When Gusto wanted deeper expertise in that layer, it brought in Evil Martians to work alongside its own engineering team. Industries: HR-tech, FinTech, Y Combinator Key facts: $9.3B+ valuation; $1B annual revenue; 500K+ businesses on payroll ### Lago URL: https://evilmartians.com/clients/lago Lago is an open-source metering and billing system that partnered with Evil Martians to scale their engineering team, keep product development moving fast, and maintain high quality. Industries: Developer Tools Key facts: $22M total funding ### Wallarm URL: https://evilmartians.com/clients/wallarm We helped Wallarm, an API security platform, redesign and optimize their Go-based event processing pipeline that handles critical security data through NATS messaging and ClickHouse storage, and guided their team through architectural improvements and urgent feature implementations. Industries: Developer Tools, Cyber Security Key facts: $75M total funding; 20K+ Protected apps and APIs ### Buildkite URL: https://evilmartians.com/clients/buildkite Buildkite is a CI/CD platform used by Airbnb, Canva, Pinterest, and Uber. Evil Martians are building a unified design system and migrating legacy components across all product surfaces. Industries: Developer Tools, Design Key facts: A$59M total funding; 60K+ developers; 1K+ organizations ### Lingo.dev URL: https://evilmartians.com/clients/lingo-dev Lingo.dev (YC F24) is a fast-growing localization engineering platform. Evil Martians rewrote their open-source bundler plugin from prototype to production in four weeks. Industries: Developer Tools, AI Key facts: $4.2M funding; 4 weeks prototype to production ### AppSignal URL: https://evilmartians.com/clients/appsignal AppSignal is an app monitoring tool for teams of 10 to 100 engineers, supporting Ruby, Elixir, Node.js, JavaScript, Python, PHP, Java, Rust, Go, and more. Evil Martians partnered with them to rebuild their website with a bold creative direction, shipping designs directly into the codebase. Industries: Developer Tools Key facts: $22M total funding ### BetterUp URL: https://evilmartians.com/clients/betterup BetterUp, the leading coaching and professional development platform, enhanced their cutting-edge AI capabilities with Evil Martians’ support. We collaborated to create an intuitive interface with sophisticated React frontend and scalable AWS backend that maintained BetterUp’s commitment to enterprise-grade security. All while pushing the boundaries of AI-powered coaching. Industries: Business Tools, AI Key facts: $570M+ total funding; 4,000+ ICF-certified coaches ### Factorial URL: https://evilmartians.com/clients/factorial Factorial is an AI-first workforce operations platform and one of the EU’s top 20 most valuable scale-ups. Evil Martians are helping its engineering team optimize a large Ruby monolith, improve the GraphQL layer, and prepare the database for global scale. Industries: HR-tech, Business Tools Key facts: $300M+ total funding; $2.5B valuation ### Playbook URL: https://evilmartians.com/clients/playbook Since 2021, Evil Martians has collaborated with Playbook to build a next-generation cloud platform for creatives that serves 700,000+ users worldwide. Our engineering teams have developed their core platform features – from ML-powered visual search and AI-assisted file management to Figma plugin and third-party integrations. Our relationship deepened when Playbook acquired CanFY, an AI assistant platform we incubated, bringing advanced image recognition and natural language processing capabilities to their creative ecosystem. Industries: Business Tools, AI Key facts: $22M+ total funding ### Recraft URL: https://evilmartians.com/clients/recraft Recraft is shaping the future of AI-driven creativity. Our collaboration introduced a custom Discord bot, streamlining client acquisition and engagement. Plus, our tailored analytics enabled data-driven decisions, allowing for monitoring of the success of this customer acquisition channel. Industries: Business Tools, AI Key facts: $42M total funding; 1M+ users ### Thicket URL: https://evilmartians.com/clients/thicket Thicket is an educational marketplace where PhD-level experts share knowledge with curious learners. Evil Martians built a course marketplace in two weeks to validate the idea; now we're about to open it for sign-ups so anyone can enroll in a live course they're interested in. Industries: AI, EdTech, Design Key facts: 2 weeks to a live homepage ### Devpost URL: https://evilmartians.com/clients/devpost Devpost is the leading platform for hackathons and developer communities. Evil Martians migrated their infrastructure from EngineYard to Railway and strengthened their security and performance posture. Industries: Developer Tools Key facts: 1,200+ hackathons per year; $10.5M total funding ### Rootly URL: https://evilmartians.com/clients/rootly Rootly is a Y Combinator startup developing an incident management platform. The team needed an urgent design makeover for their core marketing assets—and they asked Evil Martians to implement it on the frontend and backend, add new design elements, and systematize the UI components. Industries: Cyber Security, Developer Tools, Y Combinator, AI Key facts: $15M+ total funding; Backed by Y Combinator ### Whop URL: https://evilmartians.com/clients/whop Evil Martians worked with Whop, a creator marketplace platform, on resolving critical performance challenges. We implemented data denormalization strategies, developed data seeding solutions with evil-seed, and enhanced their developer experience to support their rapid growth. Industries: eCommerce, Business Tools, Retail Key facts: $273M total funding; $1.6B valuation ## Open Source Projects - [Rails Startup Stack](https://evilmartians.com/opensource/rails-startup-stack-link): Our selection of services, gems, guides, and materials to build and grow fast-paced businesses with Ruby on Rails. - [AgentPrism](https://evilmartians.com/opensource/agent-prism): Open course React component library that turn AI agent traces into clear, visual diagrams for debugging. (https://github.com/evilmartians/agent-prism) - [Overmind](https://evilmartians.com/opensource/overmind): Advanced Procfile-based process manager for web application development. With it, you can efficiently run several processes from your Procfile in a single terminal. (https://github.com/DarthSim/overmind) - [Agent Skills](https://evilmartians.com/opensource/agent-skills): Our engineering practice packaged as agent skills: install one and your AI coding agent builds the way we do. (https://github.com/evilmartians/agent-skills) - [Hivemind](https://evilmartians.com/opensource/hivemind): Advanced Procfile-based process manager to run several processes from your Procfile in a single terminal. (https://github.com/DarthSim/hivemind) - [Autoprefixer](https://evilmartians.com/opensource/autoprefixer): Autoprefixer lets you write your CSS rules without vendor prefixes. (https://github.com/postcss/autoprefixer) - [Browserslist](https://evilmartians.com/opensource/browserslist): Defines and shares the list of target browsers between various frontend build tools. (https://github.com/browserslist/browserslist) - [Gon](https://evilmartians.com/opensource/gon): A Ruby gem that provides a straightforward way to pass Ruby variables to the Rails frontend code. (https://github.com/gazay/gon) - [Lefthook](https://evilmartians.com/opensource/lefthook): Git hooks manager for Node.js, Ruby, or any other type of projects. (https://github.com/evilmartians/lefthook) - [Ruby Next](https://evilmartians.com/opensource/ruby-next): A transpiler and a collection of polyfills for supporting new and upcoming Ruby features in older versions and alternative implementations. (https://github.com/ruby-next/ruby-next) - [Size Limit](https://evilmartians.com/opensource/size-limit): Linter for your JavaScript application that makes the Web lighter by preventing JS library bloat. (https://github.com/ai/size-limit) - [TestProf](https://evilmartians.com/opensource/testprof): A Ruby test profiling toolkit with a collection of different tools to analyze your test suite performance. (https://github.com/test-prof/test-prof) - [Action Policy](https://evilmartians.com/opensource/action-policy): An authorization framework for Ruby on Rails applications. (https://github.com/palkan/action_policy) - [Mock Suey](https://evilmartians.com/opensource/mock-suey): A collection of tools to keep Ruby mocks in line with real objects. (https://github.com/test-prof/mock-suey) - [OKLCH Color Picker & Converter](https://evilmartians.com/opensource/oklch-color-picker): OKLCH Color Picker & Converter helps designers and web developers pick and convert colors in OKLCH and LCH color space, enhance a11y in their projects, and prepare for a future with a vibrant P3 color space. (https://github.com/evilmartians/oklch-picker) - [Polychrom](https://evilmartians.com/opensource/polychrom): A Figma plugin that ensures UI text is readable by leveraging the new APCA algorithm. (https://github.com/evilmartians/figma-polychrom) - [Harmony](https://evilmartians.com/opensource/harmony): Accessible UI Color Palette (https://github.com/evilmartians/harmony) - [Harmonizer](https://evilmartians.com/opensource/harmonizer): Accessible Color Palettes Generator (https://github.com/evilmartians/harmonizer) - [RunRuby.dev](https://evilmartians.com/opensource/runruby): Brings Ruby programming into your browser, streamlining the process of writing and running Ruby code. (https://github.com/skryukov/runruby.dev) - [apcach](https://evilmartians.com/opensource/apcach): JavaScript color calculator for composing colors with consistent APCA contrast ratio. (https://github.com/antiflasher/apcach) - [Yabeda](https://evilmartians.com/opensource/yabeda): Make Ruby and Rails application monitoring as easy as possible. (https://github.com/yabeda-rb/yabeda) - [Ossert](https://evilmartians.com/opensource/ossert): An open source SaaS that provides maturity maintenance certification for Ruby libraries. (https://github.com/ossert/ossert) - [Nano ID](https://evilmartians.com/opensource/nanoid): A nano-sized unique string ID generator for JavaScript. (https://github.com/ai/nanoid) - [Storeon](https://evilmartians.com/opensource/storeon): An event-based, Redux-like state manager for React, Preact, Angular, Vue, and Svelte. (https://github.com/storeon/storeon) - [Color Name](https://evilmartians.com/opensource/color-name): A Figma plugin that answers the question “How should I name this color?” once and for all. (https://github.com/romashamin/figma-color-name) - [PostCSS-modules](https://evilmartians.com/opensource/postcss-modules): A PostCSS plugin to use CSS Modules everywhere: not only at the client side. (https://github.com/css-modules/postcss-modules) - [State Machine](https://evilmartians.com/opensource/state-machine): A Sketch plugin that allows you to use ⌃⌘S to switch an active tab on a tab bar you’re designing, an active navigation section in a menu, or a page number in a pagination control. (https://github.com/romashamin/statemachine-sketch) - [Compo](https://evilmartians.com/opensource/compo): A Sketch plugin that makes it easier to work with interface components. (https://github.com/romashamin/compo-sketch) - [Sort Me](https://evilmartians.com/opensource/sort-me): A Sketch plugin that makes sorting artboards and layers dead simple. (https://github.com/romashamin/sort-me-sketch) - [Size Marks](https://evilmartians.com/opensource/size-marks): A Photoshop™ script for web designers and frontend engineers. (https://github.com/romashamin/Size-Marks-PS) ## Products - [imgproxy](https://evilmartians.com/products/imgproxy): imgproxy is a blazing fast and secure image processing tool. Built with a focus on developer productivity, imgproxy saves time and money on developing hand-rolled image processing pipelines. Our Pro version provides priority support, advanced image adjustments, ML features, watermarking, and video support. (https://imgproxy.net/) - [Layered Design for Ruby on Rails Applications](https://evilmartians.com/products/layered-design-book): Discover practical design patterns for maintainable web applications. Transition from the traditional Rails MVC trio to an abstraction model and evolve your application codebase with synchronized components, boosting flexibility and maintainability. (https://www.amazon.com/Layered-Design-Ruby-Rails-Applications/dp/1801813787) - [AnyCable](https://evilmartians.com/products/anycable): Building realtime features at scale is easy with the dedicated realtime server, AnyCable. It runs next to your main app (or, in our cloud) and handles realtime load, while ensuring the deliverability of messages and reliability of connections. Use powerful high-level abstractions to build chats, collaboration, IoT data features, gen-AI streaming, etc. (https://anycable.io/) - [PostCSS](https://evilmartians.com/products/postcss): One of the most popular and most depended-on npm libraries, PostCSS transforms CSS using an extensible plugins API. With more than 200 plugins, developers can lint CSS, support variables and mixins, transpile future CSS syntax, and more. (https://postcss.org/) - [Martian Grotesk](https://evilmartians.com/products/martian-grotesk): An open typeface family with a strong personality designed for web and digital projects. It contains a variable font and 63 styles. Martian Grotesk appears legible on most platforms, even when being rendered in small sizes. (https://github.com/evilmartians/grotesk) - [Martian Mono](https://evilmartians.com/products/martian-mono): Martian Mono is a monospaced version of the Martian Grotesk for code style design. It's the first Evil Martians open source typeface on the market. (https://github.com/evilmartians/mono) - [Logux](https://evilmartians.com/products/logux): A new way to connect clients and server. Instead of sending HTTP requests (AJAX/REST), it synchronizes the log of operations between client, server, and other clients through WebSockets. (https://logux.org) - [LaunchKit](https://evilmartians.com/products/launchkit): Free template for devtool landing pages (https://launchkit.evilmartians.io) ## Agent Skills - [llms-visibility](https://github.com/evilmartians/agent-skills): Make a site readable to ChatGPT, Claude, Perplexity, and the coding agents that fetch URLs: Markdown routes, Accept content negotiation, llms.txt, and crawler signals in robots.txt. Pushes back on the AI-SEO advice we tested and found useless. - [storybook-workbench](https://github.com/strongeron/storybook-workbench): Eleven skills that turn Storybook into an audit layer for vibe-coded React and Vite apps: tell real components from dead ones, map routes and flows, check design-system health, bridge Figma both ways, and write CSF3 stories for the states that matter. - [layered-rails](https://github.com/palkan/skills): Write, refactor, and review Rails code by the principles from the Layered Design for Ruby on Rails Applications book: service, form, policy, and query objects, plus a codebase analysis that scores callbacks, finds god objects by churn and complexity, and flags layer violations in a diff. - [good-readme](https://github.com/evilmartians/agent-skills): Write or rewrite an open source README the way PostCSS, Nano ID, imgproxy, and AnyCable did it: lead with what the project does and how it differs, drop the badges, and never invent a benchmark the repo can't back up. - [inertia-rails-skills](https://github.com/inertia-rails/skills): Fixes what LLMs get wrong about Inertia.js on Rails, across React, Vue 3, and Svelte: controllers and prop types, forms without react-hook-form, pages and navigation, shared-prop typing, shadcn/ui, and Alba serializers with auto-generated types. - [skills-visibility](https://github.com/evilmartians/agent-skills): Publish a catalog of agent skills and make it discoverable to AI coding agents: the .well-known/agent-skills/index.json index, integrity digests, single-file vs multi-file vs bundle packaging, and every install method. Pushes back on pointing the index at raw GitHub URLs instead of a domain you control. - [secure-npm-package](https://github.com/evilmartians/agent-skills): Release an npm package with no token left to steal: Trusted Publishing, provenance, and staged publishes from a single GitHub Actions workflow, plus click-by-click npmjs.com and GitHub settings with every URL resolved from your own package.json. - [rails-boot-profiling](https://github.com/palkan/require-profiler): Profile Rails boot time with require-profiler: find the slowest requires and initializers, catch boot-time YAML parsing and HTTP calls, deep-dive into stubborn files with Stackprof, and turn the findings into fixes that speed up your Rails app startup. - [tailwind-best-practices](https://github.com/evilmartians/agent-skills): Writes and audits Tailwind CSS with the five practices that keep a fast-to-write codebase maintainable: design tokens over magic values, short and consistently ordered class lists, semantically grouped tokens, and fixed variants for design-system components instead of arbitrary className props. ## Guides ### Evil Martians: Product Development Consulting for Devtools Startups URL: https://evilmartians.com/devtools Evil Martians transform growth-stage startups into unicorns, build developer tools, and create open source products. # Product Development Consulting for Dev Tools Startups ## Our services - [Ruby on Rails consulting](https://evilmartians.com/services/ruby-on-rails-consulting) - [Frontend & real-time](https://evilmartians.com/services/real-time) - [Design for devtools](https://evilmartians.com/services/design-developer-tool) - [Platform engineering](https://evilmartians.com/services/sre) - [Developer marketing](https://evilmartians.com/services/tech-marketing) ## Our clients - [HTTPie](https://evilmartians.com/clients/httpie) - [Teleport](https://evilmartians.com/clients/teleport) - [StackBlitz](https://evilmartians.com/clients/stackblitz) - [Certora](https://evilmartians.com/clients/certora) - [Tines](https://evilmartians.com/clients/tines) - [Iterative](https://evilmartians.com/clients/iterative) ## Open source we developed - **autoprefixer** (21+K GitHub stars): Parse CSS and add vendor prefixes to rules by Can I Use - **postcss** (28+K GitHub stars): Transforming styles with JS plugins - **nanoid** (22+K GitHub stars): A tiny (130 bytes), secure, URL-friendly, unique string ID generator for javaScript - **imgproxy** (7+K GitHub stars): Fast and secure standalone server for resizing and converting remote images - **150,000** stars on GitHub across all projects built by Martians - **25 Billion** downloads of our projects ### Let’s see how we can help ## UX Design - UX Design - Web, Mobile and Desktop - Rapid Experimentation - User Interviews - Analytics Let’s start with a design sprint. - Price: $14K - Duration: 2 weeks ## Real-time and Collaborative UI - Collaboration mode for your users - Scalable real-time updates - Real-time IoT data processing - Action Cable and Hotwire at scale Let’s start with a proof of concept solution. - Price: $21K - Duration: 3 weeks Powered by AnyCable, a scalable real-time framework built by Evil Martians ## Performance Bottlenecks - Backend scalability - Frontend perfomance - Complex upgrades - Refactoring legacy applications Let’s start with a performance audit. - Price: $7K - Duration: 1 week ## Infrastructure Optimization - CI/CD setup - Auto-scaling - Observability - Cloud migration - Cloud cost optimization Let’s fix one problem in your current setup. - Price: $7K - Duration: 1 week ## Technical Marketing - Strategy - Technical articles - Content distribution - Documentation Let’s start with 2–3 articles. - Price: $7K - Duration: 2 weeks ### One more thing… ### Evil Martians: Product Development Consulting for Startups URL: https://evilmartians.com/for-startups Evil Martians get startups to product-market fit fast. # Product Development Consulting for Startups ## Our services - [Ruby on Rails consulting](https://evilmartians.com/services/ruby-on-rails-consulting) - [Frontend & real-time](https://evilmartians.com/services/real-time) - [Design for devtools](https://evilmartians.com/services/design-developer-tool) - [Platform engineering](https://evilmartians.com/services/sre) - [Developer marketing](https://evilmartians.com/services/tech-marketing) ## Our clients - [HTTPie](https://evilmartians.com/clients/httpie) - [Teleport](https://evilmartians.com/clients/teleport) - [StackBlitz](https://evilmartians.com/clients/stackblitz) - [Certora](https://evilmartians.com/clients/certora) - [Tines](https://evilmartians.com/clients/tines) - [Iterative](https://evilmartians.com/clients/iterative) ## Open source we developed - **autoprefixer** (21+K GitHub stars): Parse CSS and add vendor prefixes to rules by Can I Use - **postcss** (28+K GitHub stars): Transforming styles with JS plugins - **nanoid** (22+K GitHub stars): A tiny (130 bytes), secure, URL-friendly, unique string ID generator for javaScript - **imgproxy** (7+K GitHub stars): Fast and secure standalone server for resizing and converting remote images - **150,000** stars on GitHub across all projects built by Martians - **25 Billion** downloads of our projects ### Let’s see how we can help ## UX Design - UX Design - Web, Mobile and Desktop - Rapid Experimentation - User Interviews - Analytics Let’s start with a design sprint. - Price: $6K per week - Duration: 2-4 weeks ## Real-time and Collaborative UI - Collaboration mode for your users - Scalable real-time updates - Real-time IoT data processing - Action Cable and Hotwire at scale Let’s start with a proof of concept solution. - Price: $6K per week - Duration: 2–4 weeks Powered by AnyCable, a scalable real-time framework built by Evil Martians ## Performance Bottlenecks - Backend scalability - Frontend perfomance - Complex upgrades - Refactoring legacy applications Let’s start with a performance audit. - Price: $6K - Duration: 1 week ## Infrastructure Optimization - CI/CD setup - Auto-scaling - Observability - Cloud migration - Cloud cost optimization Let’s fix one problem in your current setup. - Price: $6K per week - Duration: 1 week ## Technical Marketing - Strategy - Technical articles - Content distribution - Documentation Let’s start with 2–3 articles. - Price: $6K - Duration: 2 weeks ### Evil Martians: Ruby on Rails Startup Stack URL: https://evilmartians.com/rails-startup-stack Why Ruby on Rails is the best startup stack in 2026, including for AI products: architecture, essential gems, CI, deployment, and the open source power-ups that make Rails startups ship faster. # Ruby on Rails Startup Stack Ruby on Rails is the best startup stack. Garry Tan, YC's president, is building and evangelizing Ruby on Rails and Claude Code. Intercom's AI agent Fin resolves 1M+ tickets per week on a 2M-line Rails monolith. bolt.new hit $40M+ ARR on a Rails backend. Chime, Figma, and Omada Health, all built with Ruby, had IPOs in 2025. [NexHealth](https://www.nexhealth.com/) hit a $1B valuation on Rails. [Thatch](https://www.thatch.ai/)—founded by ex-Stripe engineers—raised $84.5M building on Rails. This is our opinionated stack for building on Rails today. It comes from 20 years of shipping Rails applications, maintaining 100+ open source projects with 25 billion+ downloads, and organizing the [SF Ruby Conference](https://sfruby.com) (450+ attendees, 40+ speakers) and the [SF Bay Area Ruby Meetup](/events/the-silicon-heel-sf-ruby-meetup). Every recommendation here is what we use on client projects, in our own internal tools, and in our public-facing products like [AnyCable Plus](https://plus.anycable.io/) (managed AnyCable). It's what we'd use if we were starting a company tomorrow. --- *Evil Martians is the leading Ruby on Rails consultancy. We embed with your team and ship production code from day one. Let's talk about what you're building.* [Contact Evil Martians](https://evilmartians.com/contact-us) --- - [Rails for AI startups](#rails-for-ai-startups) - [Architecture](#architecture) - [Essential gems](#essential-gems) - [Database](#database) - [Frontend](#frontend) - [Real-time](#real-time) - [Concurrency](#concurrency) - [Observability](#observability) - [Image processing](#image-processing) - [Testing](#testing) - [Developer tools](#developer-tools) - [Deployment and clouds](#deployment-and-clouds) - [Agentic coding with layered architecture](#agentic-coding-with-layered-architecture) ## Rails for AI startups The biggest misconception in 2026: "AI products need Python backends." They don't. bolt.new—the fastest-growing AI product ever launched—runs on Rails. Garry Tan, YC's president, is building and evangelizing Ruby on Rails and Claude Code. He open-sourced "gstack"—6 Claude Code skills for planning, review, and shipping. 12 out of 25 startups that demoed at [SF Ruby Conference 2025](https://sfruby.com) were AI-native companies. Ruby is actually faster than Python in benchmarks. 2.9x faster in loops, 1.9x faster in Levenshtein distance, 2.8x faster in n-body simulation. Default Rails (43K req/s) beats optimized Django (33K req/s) in TechEmpower benchmarks. And Ruby is 15–25% more token-efficient than Python. That matters when AI coding tools generate your code. Rails conventions function as context engineering for AI. Twenty years of training data creates tighter predictions. The framework structure eliminates the need for extensive project documentation—your AI coding assistant already knows how a Rails app works. Both Anthropic and OpenAI now ship official Ruby SDKs. Rails gives AI startups what they actually need: fast iteration on product, battle-tested auth and billing, background jobs for async LLM calls, WebSocket streaming for real-time AI responses, and a mature ecosystem that lets a small team move fast. You don't write your model inference in Ruby—you write your product in Ruby and call the model APIs. Use [RubyLLM](https://github.com/crmne/ruby_llm) for a unified interface to OpenAI, Anthropic, Google, and other LLM providers. It handles streaming, tool use, and embeddings with a clean Ruby API. For LLM response streaming over WebSockets, see [AnyCable, Rails, and the pitfalls of LLM-streaming](/chronicles/anycable-rails-and-the-pitfalls-of-llm-streaming). For building AI features the Rails way, read [Exploring Active Agent](/chronicles/exploring-active-agent-or-can-we-build-ai-features-the-rails-way). ## Architecture This follows Vladimir Dementyev's highly-rated [Layered Design for Ruby on Rails Applications](/products/layered-design-book)—now in its second edition. The core idea: instead of dumping logic into service objects, extend Rails conventions with purpose-built abstraction layers. Domain logic lives in models. Complex operations are namespaced under the model they belong to. Controllers stay thin. Every other layer—policies, deliveries, notifiers, configs—has a clear purpose. ``` app/ ├── models/ # Domain logic lives HERE │ ├── mission.rb │ └── mission/ │ └── report.rb # Domain services namespaced under the model ├── controllers/ # Thin—HTTP concerns only |-- forms/ # Handling complex UI forms, wizards ├── policies/ # action_policy ├── deliveries/ # active_delivery + abstract_notifier ├── presenters/ # SimpleDelegator wrappers ├── components/ # ViewComponent ├── configs/ # anyway_config classes ├── jobs/ # Coordinate workflow, models execute └── clients/ # External API wrappers ``` No catch-all `app/services/` directory. The book calls services "a waiting room for abstractions that have yet to reveal themselves"—so we extract to purpose-built layers instead. Jobs coordinate workflow; models execute business logic. Policies handle authorization. Deliveries unify notifications. Configs are typed. For modular monoliths, watch [Between Monoliths and Microservices](/events/between-monoliths-and-microservices). For legacy rescue, see [Terraforming Legacy Rails Applications](/events/terraforming-legacy-rails-applications). ## Essential gems These are the gems we reach for on every new Rails project. Most of them we built. | Gem | Role | Replaces | |-----|------|----------| | [anyway_config](https://github.com/palkan/anyway_config) | Type-safe config from credentials/ENV/YAML | `.env` files, hand-rolled config | | [action_policy](https://actionpolicy.evilmartians.io/) | Authorization with scoping + params filtering | Pundit, CanCanCan | | [active_delivery](https://github.com/palkan/active_delivery) | Single notification entry point → mailer/Slack/SMS | Scattered `deliver_later` calls | | [abstract_notifier](https://github.com/palkan/abstract_notifier) | Non-email notification channels | Custom Slack wrappers | | [view_component](https://viewcomponent.org) | Testable, encapsulated UI components | Deep partial trees | | [n_plus_one_control](https://github.com/palkan/n_plus_one_control) | Catches N+1s in tests automatically | Manual includes auditing | | [isolator](https://github.com/palkan/isolator) | Detects non-atomic interactions within DB transactions | Manual transaction auditing | | [after_commit_everywhere](https://github.com/Envek/after_commit_everywhere) | Transactional callbacks outside of models | Inline `after_commit` hacks | | [freezolite](https://github.com/palkan/freezolite) | Auto-freezes all string literals | `# frozen_string_literal: true` in every file | | [ruby_llm](https://github.com/crmne/ruby_llm) | Unified LLM interface for Ruby | Multiple API client gems | | [herb](https://herb-tools.dev/) | HTML-aware ERB linting, formatting, and LSP | No tooling (ERB had none) | For our full gem recommendations, read [Gemfile of Dreams: libraries we use to build Rails apps](/chronicles/gemfile-of-dreams-libraries-we-use-to-build-rails-apps)—our curated list of production-tested gems for every layer of a Rails app. For configuration, see [Anyway Config: keep your Ruby configuration sane](/chronicles/anyway-config-keep-your-ruby-configuration-sane). For notifications, see [Crafting user notifications in Rails with Active Delivery](/chronicles/crafting-user-notifications-in-rails-with-active-delivery). Try the [Action Policy interactive tutorial](https://tutorial.actionpolicy.evilmartians.io/). **Feature flags:** Use [Flipper](https://www.flippercloud.io/) for feature flags. We use it on every project—read [how we extended it the Martian way](/chronicles/flipper-gem-is-amazing-we-extended-it-the-martian-way) with actor IDs, team-wide flags, percentage rollouts, and analytics. **Resilience:** Add [circuit breakers](/chronicles/circuit-breakers-and-ruby-in-2025-dont-break-your-launch) so a failing third-party API doesn't cascade into a full outage. **Event-driven processing:** For Kafka workloads alongside Rails, use [Karafka](https://karafka.io)—an efficient Kafka processing framework built for Ruby. **API documentation:** Use [Skooma](/opensource/skooma) (built by us) for validating API implementations against your OpenAPI spec. Read our two-part guide: [A documentation-first approach to Rails API development](/chronicles/let-there-be-docs-a-documentation-first-approach-to-rails-api-development) and [Generating an OpenAPI schema across the Rails stack](/chronicles/let-there-be-docs-generating-openapi-schema-across-rails-stack). ## Database **Starting out?** SQLite is great for internal tools and first MVPs. Pair it with [Litestream](https://litestream.io/) for continuous replication to S3—you get backups and disaster recovery without running a separate database server. **Growing?** PostgreSQL. GitHub runs on it. Chime runs on it. This is the database for production Rails. We've been building on PostgreSQL for nearly 20 years: - [PostgreSQL and Rails, sitting in a tree](/chronicles/postgresql-and-rails-sitting-in-a-tree)—the foundation - [Table partitioning in PostgreSQL](/chronicles/a-slice-of-life-table-partitioning-in-postgresql-databases)—when your tables outgrow a single partition - [Partitioning with canaries](/chronicles/flying-free-a-tale-of-partitioning-with-canaries-in-postgresql)—safe rollout strategies - [Soft deletion with PostgreSQL](/chronicles/soft-deletion-with-postgresql-but-with-logic-on-the-database)—keeping logic in the database - [Migrating Whop from PostgreSQL to PlanetScale with zero downtime](/chronicles/migrating-whop-from-postgresql-to-planetscale-zero-downtime)—when the time comes Use [Logidze](/opensource/logidze) for audit trails with PostgreSQL triggers. Use [store_model](https://github.com/DmitryTsepelev/store_model) for ActiveModel-like validation on JSON columns. Both built by us. Essential reading: ["High Performance PostgreSQL for Rails"](https://pragprog.com/titles/aapsql/high-performance-postgresql-for-rails/). ## Frontend Rails gives you several paths for the frontend. All of them work at scale—Intercom's 2M-line monolith proves the full-stack Rails path works, and bolt.new proves the API-backed React path works too. The modern Rails frontend is a [layered toolkit, not a single tool](/chronicles/keeping-rails-cool-the-modern-frontend-toolkit)—you pick the right approach for each part of your app. Also check our classic [Frontendless Rails frontend](/events/frontendless-rails-frontend-railsconf) talk. ### Hotwire-first The Rails default. Use **Hotwire/Turbo** with **importmaps** for zero-build simplicity. Add [Herb](https://herb-tools.dev/) for HTML-aware ERB linting and smart devtools—built on a fast C parser with bindings for Ruby, Node.js, Rust, and WASM, it understands your templates at the syntax level, not just as strings. [Herb v0.9](https://herb-tools.dev/blog/whats-new-in-herb-v0-9) adds Action View tag helper support, 24 linter rules, and a ~2.3x faster CLI. Start with the [official Turbo docs](https://turbo.hotwired.dev). Then read our guide [The future of full-stack Rails: Turbo Morph Drive](/chronicles/the-future-of-full-stack-rails-turbo-morph-drive). Use [ViewComponent](https://viewcomponent.org) for reusable, testable UI with [view_component-contrib](https://github.com/palkan/view_component-contrib) (built by us) for extensions and dev tools—our 3-part guide [ViewComponent in the Wild](/chronicles/viewcomponent-in-the-wild-building-modern-rails-frontends) covers beginner to advanced. ### Vite Ruby for modern builds Whether you need a build step for React/Vue/Svelte components, or experience issues with importmaps, [Vite Ruby](https://vite-ruby.netlify.app/) is here to save the day. We use it across our projects and bring it to customer toolboxes regularly. Read [how we Vite-lize Rails](/chronicles/vite-lizing-rails-get-live-reload-and-hot-replacement-with-vite-ruby). For eliminating the need of addding Node.js runtime to your stack, we use [bundlebun](https://github.com/yaroslav/bundlebun)—Bun packed in a Ruby gem, so `bundle install` is enough to have an all-in-one JavaScript runtime, package manager, and build tool. ### Adding React, Vue, or Svelte gradually **Drop in a React/Vue/Svelte component into a Hotwire app?** Use [Turbo Mount](https://github.com/skryukov/turbo-mount) (built by us). No need to rebuild everything. Read [The art of Turbo Mount: Hotwire meets modern JS frameworks](/chronicles/the-art-of-turbo-mount-hotwire-meets-modern-js-frameworks). **Reaching for React or Vue, but not a fan of APIs?** Use [Inertia Rails](https://inertia-rails.dev/) with [inertia_rails-contrib](https://github.com/skryukov/inertia_rails-contrib) (built by us) for generators and dev tools. Use [Typelizer](https://github.com/skryukov/typelizer) to generate TypeScript types from Ruby serializers (and make your React devs happy). We also built [Inertia Rails Skills](https://github.com/inertia-rails/skills)—AI coding skills for Claude Code and Codex that understand Inertia Rails patterns and produce quality code. Read more: [Inertia.js in Rails: a new era of effortless integration](/chronicles/inertiajs-in-rails-a-new-era-of-effortless-integration). Add [Tailwind CSS](https://tailwindcss.com/docs/guides/ruby-on-rails)—it works neatly with Rails. ## Real-time Chats, notifications, typing indicators, presence, cursors, collaboration, live dashboards—most Rails apps need real-time eventually. Rails ships with Action Cable, and Solid Cable, but this tooling won't give you the truly instant and reliable real-time UX. [AnyCable](https://anycable.io/) provides 10x faster streaming—it replaces Action Cable with a Go-based WebSocket server that uses 3x less memory and provides delivery guarantees, resumable sessions, and reliable streams. It keeps the familiar Action Cable API—for Hotwire-only apps, setup is simple: swap the adapter and everything works. [Circle.so](https://circle.so/), [Jobber](https://getjobber.com/), and [Doximity](/clients/doximity) all run AnyCable in production. Also has a [managed version](https://plus.anycable.io/). The developer experience is minimal. Adding [real-time presence to a Hotwire app](/chronicles/simple-declarative-presence-for-hotwire-apps-with-anycable) takes a single HTML tag—``—and you get automatic WebSocket subscription, a live user counter, and Turbo Stream-driven DOM updates. No imperative JavaScript required. For scaling real-time, read [Connection avalanche safety tips](/chronicles/connection-avalanche-safety-tips-and-prepping-for-real-time-applications). For LLM streaming over WebSockets, see [AnyCable, Rails, and the pitfalls of LLM-streaming](/chronicles/anycable-rails-and-the-pitfalls-of-llm-streaming). ## Concurrency Ruby 3 is 3x faster than Ruby 2. YJIT adds ~15% on top. ZJIT—Shopify's next-generation JIT compiler—is coming in Ruby 3.5 with even bigger gains. M:N thread support enhances concurrency. Ruby's concurrency story has matured: Puma runs your app in threads, Sidekiq processes jobs in threads, and Ractors offer true parallelism. Understanding threads, fibers, and execution context matters as your app grows. We maintain [rubocop-thread_safety](https://github.com/rubocop/rubocop-thread_safety) (21M+ downloads) to catch concurrency bugs statically, and [isolator](https://github.com/palkan/isolator) to detect non-atomic interactions within database transactions at runtime. Our talks and writing on the topic: - [Don't Be "Thread"-ened: Testing Multithreaded Code with Confidence](/events/dont-be-thread-ened-euruko)—Julia Egorova at EuRuKo 2025 - [Threads, callbacks, and execution context in Ruby](/events/ruby-threads-callbacks-osakarubykaigi)—Vladimir Dementyev at Osaka RubyKaigi and RubyConf AU - [One row, many threads: How to avoid database duplicates in Rails](/chronicles/one-row-many-threads-how-to-avoid-database-duplicates-in-rails-applications) - [Cables, concurrency, and Ruby](/events/cables-concurrency-and-ruby)—Vladimir Dementyev on Ruby Rogues We also made the [NATS](/clients/nats) Ruby SDK 3x faster. NATS is a high-performance messaging system where concurrency correctness is everything. ## Observability You can't fix what you can't see. [Yabeda](https://github.com/yabeda-rb/yabeda) is a family of gems for Rails application monitoring—Sidekiq, Puma, Prometheus, GraphQL, and more. Yabeda is built by Evil Martians and notably used by 37signals (Basecamp). It gives you metrics out of the box without building custom instrumentation. Read [Meeting Yabeda](/chronicles/meet-yabeda-modular-framework-for-instrumenting-ruby-applications) to get started. Use [Logidze](/opensource/logidze) for tracking Active Record changes over time. ## Image processing Every app with user-generated content needs fast image processing. Don't do it in Ruby. [imgproxy](https://imgproxy.net/) (32M+ Docker pulls) is a Go server for on-the-fly image processing—fast, secure, and widely adopted. Use [imgproxy-rails](https://github.com/imgproxy/imgproxy-rails) to plug it into ActiveStorage with no code changes. Built by us. ## Testing Intercom runs 100K tests per PR in 6 minutes. That's the bar. Here's how to get there. **Capybara + Cuprite** for system tests. Cuprite uses Chrome DevTools Protocol directly—faster and more reliable than Selenium. Follow our guide: [System of a Test: setting up end-to-end Rails testing](/chronicles/system-of-a-test-setting-up-end-to-end-rails-testing). **webmock** for HTTP stubbing of external APIs. **[n_plus_one_control](https://github.com/palkan/n_plus_one_control)** catches N+1 queries automatically in your tests. No more manual auditing. **[TestProf](https://test-prof.evilmartians.io/)** for test suite profiling. It finds the exact bottlenecks—factory cascades, unnecessary database hits, slow before hooks. We used it to [cut CI time in half at Whop](/chronicles/the-whop-chop-how-we-cut-a-rails-test-suite-and-ci-time-in-half) and [get 5x faster tests](/chronicles/railing-against-time-tools-and-techniques-that-got-us-5x-faster-results) for another client. The trilogy: [TestProf I](/chronicles/testprof-a-good-doctor-for-slow-ruby-tests) (diagnosis), [TestProf II](/chronicles/testprof-2-factory-therapy-for-your-ruby-tests-rspec-minitest) (factory optimization), [TestProf III](/chronicles/test-prof-3-guided-and-automated-ruby-test-profiling) (automated profiling). Start with the [RailsConf workshop](https://www.youtube.com/watch?v=PvZw0CnZNPc). For flaky tests, read [Flaky tests, be gone: long-lasting relief for chronic CI retry irritation](/chronicles/flaky-tests-be-gone-long-lasting-relief-chronic-ci-retry-irritation)—based on our work at [ClickFunnels](/clients/clickfunnels). For load testing, use [k6](https://k6.io)—the modern framework for simulating real user traffic. Test your WebSocket load too: [AnyCable, k6 and Yabeda](/chronicles/real-time-stress-anycable-k6-websockets-and-yabeda). ## Developer tools **Ruby LSP.** The official language server for Ruby—autocompletion, go-to-definition, inline diagnostics, and formatting. Works with VS Code out of the box and integrates with other editors via LSP. Claude Code recently added Ruby LSP as an official plugin. This is the modern Ruby development experience. **[Herb](https://herb-tools.dev/).** A modern HTML+ERB toolchain built on a fast C parser with bindings for Ruby, Node.js, Rust, and WASM. It brings first-class linting, formatting, and language server intelligence to Rails view templates—something ERB has never had. Read about the last release [Herb v0.9](https://herb-tools.dev/blog/whats-new-in-herb-v0-9). [Lefthook](https://evilmartians.com/opensource/lefthook) (1.2M weekly npm downloads) manages Git hooks across your team. Read our guide on [5 ways to configure Lefthook](/chronicles/5-cool-and-surprising-ways-to-configure-lefthook-for-automation-joy). **RuboCop.** Use [RuboCop Gradual](https://github.com/skryukov/rubocop-gradual) (built by us) to adopt linting rules incrementally—no big-bang cleanup needed. We also maintain [rubocop-thread_safety](https://github.com/rubocop/rubocop-thread_safety) (21M+ downloads) for concurrency checks and [rubocop-md](https://github.com/rubocop/rubocop-md) for linting Ruby in Markdown docs. Need custom rules? Read [Writing custom RuboCop rules in 2026](/chronicles/writing-custom-rubocop-rules-in-2026). ## Deployment and clouds **Docker** is the foundation. Use [Ruby on Whales](/chronicles/ruby-on-whales-docker-for-ruby-rails-development)—our Docker setup for Rails development. Reproducible environments, easy onboarding for new teammates. **Puma** is the web server. Threaded, clustered, battle-tested. The Rails default for good reason. **[Kamal](https://kamal-deploy.org/)** for deployment. It's the Rails-native tool—deploy anywhere you can run Docker. Read our take: [Kamal: hot deployment tool or total game changer?](/chronicles/mrsk-hot-deployment-tool-or-total-game-changer) **Starting out:** [Render](https://render.com), [Fly.io](https://fly.io), or [Railway](https://railway.app). All three handle infrastructure so you can focus on product. **Scaling:** AWS ECS for container orchestration without Kubernetes complexity. **At scale:** AWS EKS or self-managed Kubernetes when you need it. Our [Martian Kubernetes Kit](/chronicles/martian-kubernetes-kit-a-smooth-sailing-toolkit-from-our-sre-team) is the toolkit we use internally. See also [running apps well on Kubernetes](/chronicles/martian-kubernetes-kit-running-apps-and-running-them-well). ## Agentic coding with layered architecture AI coding tools like Claude Code, Codex, and Copilot produce better Rails code when they understand your architecture. We build [Layered Rails skills](https://github.com/palkan/skills) for Claude Code—slash commands that analyze your codebase architecture, review code changes for layered design violations, detect god objects and low-value callbacks, and guide gradual adoption of new patterns. We also built [Inertia Rails Skills](https://github.com/inertia-rails/skills) for teams using Inertia—AI coding tools that understand Inertia patterns and produce idiomatic code. We set up **hooks for guardrails**: pre-commit checks that catch architectural violations before code gets merged. The AI writes code that follows your patterns, not generic Rails conventions. We open source these tools and help teams adopt them. The future of Rails development is AI-assisted. Our architecture patterns make that AI produce better code. ## Blog (latest 30 posts) For all posts, see individual .md links at https://evilmartians.com/llms.txt ### Get in, human: cut Rails boot time with require-profiler and this guide Date: 2026-08-19T00:00:00.000Z | Authors: Artur Petrov, Vladimir Dementyev, Travis Turner | Categories: Rails, DX, Performance URL: https://evilmartians.com/chronicles/get-in-human-cut-rails-boot-time-with-require-profiler-and-this-guide Every Rails app race starts the same: load code, read config, then the green flag. But the bigger the app, the more startup drags on deploys, CI, and developer patience. This is doubly so as AI agents boot your app far more often than humans do. Thus, meet [require-profiler](https://github.com/palkan/require-profiler), a new one-stop profiler for Ruby's code loading process! Grab it first when optimizing boot, leaving sampling profilers for the deep dives. We'll show the telemetry from real projects: one cut the dev boot of [Factorial](https://evilmartians.com/clients/factorial)'s 200-component monolith by 40%. And read till' the final lap: profiling an [AnyCable](https://anycable.io)-powered app's boot uncovered a surprise in Ruby itself! Now the usual objection: nobody has time to tune boot. It sounds like a week of work with no guaranteed payoff (though here the prize compounds with every boot, every day). But with the right tools, your first pass takes an afternoon, and most of the fixes in apps are one-liners. (Besides, the framework's creator is literally a [Le Mans class-winning driver](https://dhh.dk). Speed is practically canon.) We've poked at boot before on [Railsmas day 2](https://evilmartians.com/chronicles/railsmas-on-mars-12-days-of-mandatory-developer-joy-and-challenge#day-2-faster-rails-boot-time), and the other 11 days hold more goodies. #### Lap 0: Bootsnap Before entering the profiling race, make sure [Bootsnap](https://github.com/rails/bootsnap) is on and working. It's the easiest possible win at the start. Legacy apps occasionally run with it missing entirely (we've seen it multiple times), and that's where the huge wins hide. On Factorial's monolith, the same boot runs 2.6 times slower without it. It ships enabled in new Rails apps, so the checklist is really two items: keep it on and *updated*, and precompile it properly for production. We put *updated* in that checklist on purpose. The ecosystem's common curse is that gems ship speedups, and nobody collects them. We'll meet this pattern again several times in this post. In Bootsnap's case, recent versions roughly halve the load-path scan by cutting `stat` syscalls (an optimization also headed to Ruby 4.1 as a new `Dir.scan`). Bump the gem to at least `1.20.0` and win your first seconds. The full story of that optimization is in [Jean Boussier's write-up](https://byroot.github.io/ruby/performance/2026/04/18/faster-paths.html), along with the interesting `File.join` history and speedups uncovered by the same profiling effort. Now for production, don't forget [precompilation](https://github.com/rails/bootsnap?tab=readme-ov-file#precompilation). In development the cache warms up *automagically*. But a production image gets no warm-up laps, so "pre-warm" the cache during the Docker build: ```bash bundle exec bootsnap precompile --gemfile app/ lib/ ``` Include all the code directories (i.e., `config`, `engines`, `packs`, and other custom top-level directories). To verify nothing is missed, enable the instrumentation and boot: ```ruby # config/boot.rb Bootsnap.instrumentation = ->(event, path) { puts "#{event} #{path}" } ``` ```bash rm -rf tmp/cache/bootsnap/* bundle exec bootsnap precompile --gemfile app/ lib/ bundle exec rails runner 'puts :ok' | grep 'miss' ``` And watch your Docker setup. The cache lives in `tmp/`, and that location invites accidents: a multi-stage `COPY` can skip it, a cleanup step can wipe it, and a volume can shadow it at runtime. #### Lap 1: require-profiler For years, newbies started their boot profiling races with [Bumbler](https://github.com/nevir/Bumbler). It's a flat list of per-gem require times, plus a separate mode for initializers. But it never tells the whole story, though. Transitive requires get blurred, whatever a gem *does* while loading hides inside one opaque number, and the Ruby code that runs past the requires remains totally invisible. On Factorial's monolith (before the pit stop), it flagged just six gems, barely over a second combined, `rails` itself and `anycable-rails` among them (remember that last one for the final lap). Not much to act on when the stopwatch showed ~22 seconds with Bootsnap enabled. So, let's welcome a new contender to the race: [require-profiler](https://github.com/palkan/require-profiler), an easy-to-use one-stop shop for nearly every boot issue. It profiles Ruby's code loading itself: every `require`, `require_relative`, and `load`, as a tree, with timings. And a plugin system covers the costs requires alone can't see. The gem also has a fascinating origin story. It was built on [require-hooks](https://github.com/ruby-next/require-hooks), a universal way to intercept Ruby's code loading process. Despite Ruby's famous flexibility, overriding the loading is officially unsupported. Yet everyone seems to love patching it anyway (RubyGems, Zeitwerk, Bootsnap), and resolving the resulting conflicts is _hacky as hell_. So Vladimir Dementyev [made the case for a proper loader API in Ruby itself at RubyKaigi 2026](https://evilmartians.com/events/require-hooks-rubykaigi), with require-profiler premiering as the talk's bonus track. Another require-hooks-born trick is [Freezolite](https://github.com/ruby-next/freezolite): `frozen_string_literal` for all your files, no magic comments. The idea recently [landed in Bootsnap itself](https://github.com/rails/bootsnap/pull/535) and powers [the Rails move to frozen literals by default](https://github.com/rails/rails/pull/57252). Add it to the Gemfile and point it at your app: ```sh bundle exec ruby -r./config/boot -require-prof config/environment.rb ``` Puzzled how we added a new option straight to Ruby? The gem ships a file named `equire-prof.rb`, so the flag parses as `-r equire-prof` (same trick as Ruby's own [`-run`](https://github.com/ruby/ruby/blob/master/lib/un.rb)). Loading `config/boot.rb` first sets up Bundler and Bootsnap (remember the conflicting patches? Bootsnap must hook in first). Then everything starting from `config/environment.rb` gets profiled. (If the app still carries [Spring](https://github.com/rails/spring), disable it while profiling with `DISABLE_SPRING=1`, otherwise you're timing a fork, not a boot.) The output is an indented tree: one line per loaded file with its time in milliseconds, self plus children, and a few synthetic lines mixed in. The rough shape is this: ``` entry_file.rb — self + children time, in ms required_file.rb — ... parsed_file.yml — ... prefix:synthetic_line — ... (an HTTP call `http:` or a Rails phase `rails:`) ``` And here's a slice of Factorial's pre-pit-stop real profile (shortened): ``` config/application.rb — 6198.333ms components/integrations_google_users/lib/integrations_google_users.rb — 177.365ms gems/google-apis-admin_directory_v1-0.42.0/lib/google/apis/admin_directory_v1.rb — 125.054ms rails:initializer:load_config_initializers:gems/railties-7.2.2.2/lib/rails/engine.rb:640 — 791.744ms config/initializers/countries.rb — 1369.053ms gems/faker-2.19.0/lib/locales/ja/address.yml — 357.229ms gems/faker-2.19.0/lib/locales/es.yml — 6.947ms config/initializers/http_showcase.rb — 39.739ms http:GET:https://www.ruby-lang.org/ — 11.574ms rails:initializer:run_prepare_callbacks:gems/railties-7.2.2.2/lib/rails/application/finisher.rb:73 — 4872.904ms rails:to_prepare:gems/karafka-2.5.2/lib/karafka/railtie.rb:87 — 830.3ms karafka.rb — 830.111ms rails:load_hook:after_initialize:karafka.rb:20 — 705.965ms ``` The slice above shows all four kinds of boot cost. Plain requires sit at the top: here, the Google gem load. The `rails:` lines track the whole Rails initialization pipeline: initializers, then reload callbacks, finally load hooks. The YAML lines are files parsed during boot, better caught on a cold rerun since Bootsnap can make them too fast to notice. And the `http:` lines are network calls made along the way; ours is planted for the showcase, though real ones sneak into apps surprisingly often. Still overwhelmed? The real report goes on like this for thousands of lines, drowning real issues in harmless ones. You can filter out the noise: `REQUIRE_PROFILE_THRESHOLD` drops everything faster than N milliseconds, and `REQUIRE_PROFILE_FOCUS` zooms into a pattern, ancestors included, so chains stay readable: ```sh # Only files that took 100ms or more REQUIRE_PROFILE_THRESHOLD=100 bundle exec ruby -r./config/boot -require-prof config/environment.rb # Zoom into a single gem, full require chains included REQUIRE_PROFILE_FOCUS="aws" bundle exec ruby -r./config/boot -require-prof config/environment.rb ``` Radio channel clear: "_Okay, filters on and telemetry reads clean. One kind of line explained at a time, please._" #### The requires: the heaviest gems ``` components/integrations_google_users/lib/integrations_google_users.rb — 177.365ms gems/googleauth-1.5.2/lib/googleauth.rb — 51.428ms gems/google-apis-admin_directory_v1-0.42.0/lib/google/apis/admin_directory_v1.rb — 125.054ms gems/hexapdf-1.7.0/lib/hexapdf.rb — 62.873ms gems/caxlsx-3.4.1/lib/caxlsx.rb — 51.834ms gems/savon-2.15.0/lib/savon.rb — 92.329ms gems/elasticsearch-8.13.0/lib/elasticsearch.rb — 104.781ms gems/elasticsearch-api-8.13.0/lib/elasticsearch/api.rb — 98.503ms gems/karafka-web-0.11.4/lib/karafka/web.rb — 243.248ms gems/karafka-2.5.2/lib/karafka.rb — 142.431ms lib/frontend_navigation.rb — 75.717ms ``` You'll meet the same offenders in app after app. The heaviest are usually generated API gems (a favorite sin of cloud providers), heavy-processing libraries (PDF, XLSX, etc.), and big transport clients like `savon` (SOAP), `elasticsearch`, and `karafka` (Kafka). Luckily, the majority of them are rarely needed at boot, and some gems can be partially loaded. For internal dev tooling (admin UIs, `bullet`, console enhancements, stray test tooling, and the like), gem groups are a solution when the default environment can skip a dependency. The habit to build: keep an eye on heavy dependencies, and when only one corner of the app needs them, load them there instead of on every boot. Gems aren't the only suspects, though. Any plain app file can lose time the same way, because everything at class or module level runs *during* the require. The lineup's last line is a homegrown example: `lib/frontend_navigation.rb`, a generated map of every frontend route that lets backend code build typed links into the UI. Factorial pays for it in boot time by defining thousands of classes at load time. So, big DSLs are a high-risk tax to pay. Like muscle cars. Impressive, but _thirsty_. GraphQL, Grape, and Active Admin are the notorious ones that we've met each on different projects. These parts aren't always easy to delay or skip conditionally, but manual optimizations and caches are always available to racers. #### YAML lines: what the tire warmers can hide ``` # A warm run config/initializers/countries.rb — 447.26ms gems/faker-2.19.0/lib/locales/ja/address.yml — 182.118ms gems/faker-2.19.0/lib/locales/es.yml — 2.831ms # A cold rerun (`DISABLE_BOOTSNAP=1`) config/initializers/countries.rb — 1369.053ms gems/faker-2.19.0/lib/locales/ja/address.yml — 357.229ms gems/faker-2.19.0/lib/locales/es.yml — 6.947ms ``` The require tree comes from hooking Ruby's code loading, but not every boot cost is a require. Catching the rest is the gem plugin system's job. YAML is the built-in plugin: every file parsed during boot gets its own line in the tree, no setup needed. Bootsnap can mask these lines, though: it caches the parsed result and serves it back on warm boots. So profile twice, with the cache and without (`DISABLE_BOOTSNAP=1`). The block above is the same `countries.rb` line, warm and cold side by side. Warm, Bootsnap serves the already-parsed YAML from its cache, and the heaviest files still surface, since serving the cache isn't free either. Cold, every file gets parsed for real, a second slower in total, and the true source arrives: `faker`'s Japanese address book is not what anyone ordered (unless you're a big fan of _The Fast and the Furious: Tokyo Drift_). Here's what actually happened: `faker` appends every locale it ships to `I18n.load_path`, and we asked the `countries` gem to register a custom country before I18n was configured. That made I18n parse the entire load path: hundreds of files, all thrown away by the reloader before the first request. I18n is the repeat offender behind most of this. Usually it hides in less exotic places though: model validations love to eagerly resolve translation messages at class-load time, preloading the entire locale set. So watch anything that touches I18n before the app is fully configured: class-level `I18n.t` calls in constants or validation messages (use the lazy block version), and locale-hungry gem setup in early initializers. #### HTTP lines: off-track excursions ``` # A synthetic showcase config/initializers/http_showcase.rb — 39.739ms http:GET:https://www.ruby-lang.org/ — 11.574ms # Another project config/initializers/fragment.rb — 2064.081ms http:POST:https://api.fragment.dev/graphql — 512.632ms ``` YAML isn't the only plugin in the box: add the [sniffer](https://github.com/aderyabin/sniffer) gem to the Gemfile, and HTTP requests show up as `http:`-prefixed lines. The first block is our planted evidence: `http_showcase.rb` exists only to show the line shape, since Factorial's boot makes no network calls. In other projects, though, real ones get added and forgotten more often than expected. Here's how to hunt for them: ```sh REQUIRE_PROFILE_FOCUS="http:" bundle exec ruby -r./config/boot -require-prof config/environment.rb ``` Sniffer races in another series too: [Isolator](https://github.com/palkan/isolator) is built on it to catch non-atomic interactions within DB transactions, such as network requests. An HTTP call during boot sounds exotic until you go looking: at [Whop](https://evilmartians.com/clients/whop), it was added by a third-party dependency (the second block in the snippet). Each one adds latency to every single boot and a fun new failure mode when the network hiccups mid-deploy. #### Rails lines: the factory telemetry ``` rails:initializer:setup_main_autoloader:gems/railties-7.2.2.2/lib/rails/application/finisher.rb:18 — 1829.278ms rails:initializer:run_prepare_callbacks:gems/railties-7.2.2.2/lib/rails/application/finisher.rb:73 — 4872.904ms rails:to_prepare:gems/karafka-2.5.2/lib/karafka/railtie.rb:87 — 830.3ms karafka.rb — 830.111ms rails:initializer:finisher_hook:gems/railties-7.2.2.2/lib/rails/application/finisher.rb:93 — 2086.614ms rails:load_hook:after_initialize:karafka.rb:20 — 705.965ms ``` The newest trick grew straight out of profiling monoliths like Factorial: the profiler now captures the Rails initialization pipeline as first-class `rails:` lines. `initializer:` covers railtie initializers (one-time setup steps that gems and the app register for boot), `to_prepare:` the reload callbacks (code that reruns on every code reload in development), `load_hook:` the lazy load hooks (deferred setup that fires when a framework part first loads, like `ActiveSupport.on_load(:active_record)`). This closes the last thing the require tracking left open, because on a big app the requires are only half the story. At Factorial, the `initialize!` phase ate roughly two-thirds of the whole boot, and the `run_prepare_callbacks` line above claimed a fifth on its own. The first culprit we found at Factorial is relevant to lots of Rails apps: `Devise`. By default, it [forces all application routes to be reloaded](https://github.com/heartcombo/devise/commit/eba2527b8e730907c51da8eceaa9b996b865523a), effectively loading them twice. The cure is one config line in `config/initializers/devise.rb` (try it on your next race too). On another routes-heavy app we audited earlier, this line alone once cut the boot from ~31 to ~23 seconds: ```ruby # When false, Devise will not attempt to reload routes on eager load. # This can reduce the time taken to boot the app but if your application # requires the Devise mappings to be loaded during boot time the application # won't boot properly. config.reload_routes = false ``` Another culprit was `to_prepare` hooks carrying crunch work that runs on every boot *and every dev reload*. Factorial leans heavily on centralized registry code catalogs, and two of these hooks constantized almost every app class into lists on each pass, feeding some skip logic down the line. Only rare code flows ever read the result. Both hooks became lazy: they resolve on first use and stay memoized until the relevant code actually changes. Be careful with such changes, though: lazy work on a hot path can grow the tail latency. So always mind where a boot win lands: some fixes speed up development only (like the disabled global requires and the gem-group moves), `to_prepare` ones pay off on every reload, and some reach production boots too, for good or bad (like the I18n fixes). Initializers also love opening connections (eager database checks, Redis connections, etc.), and those stay under speed radar even to the HTTP plugin: `sniffer` can intercept HTTP clients, not raw TCP. If an initializer line is heavy with nothing beneath it, suspect the network first. Did you hear the announcement? Rails 8 now includes [global lazy routes](https://github.com/rails/rails/pull/52353) by default—large route sets finally stopped costing time on the grid. Another `rails:` find at Factorial had consequences far beyond its own line: Karafka. Its railtie dragged the whole consumer graph into every process just to keep the topic topology on hand for producers (rarely needed in other process types). Along the way, it cost the app its [lazy routes](https://github.com/amatsuda/routes_lazy_routes). Now Karafka loads only in its consumer processes, and the routes are lazy again for everyone else. Back in the profile slice, Zeitwerk shows up too: the `setup_main_autoloader` line, weighing almost two seconds. Part of the cure was upstream: Zeitwerk 2.7.4 and up ship a [validation optimization](https://github.com/fxn/zeitwerk/pull/322) that pays off exactly at this size (and it's [not the first time](https://github.com/fxn/zeitwerk/pull/308) a project this big has pushed the autoloader to its limits). The updating curse strikes again: bump the gem and collect the win. Years of postponed upgrades? Measure the gap in libyears, the laps you're down on the leader: *Read also Evil Martians' article*: https://evilmartians.com/chronicles/railsmas-on-mars-12-days-of-mandatory-developer-joy-and-challenge And one of the deepest cuts hides in the same autoloader corner: [`config.add_autoload_paths_to_load_path = false`](https://guides.rubyonrails.org/configuring.html#config-add-autoload-paths-to-load-path), the default since Rails 7.1 (Zeitwerk never consults `$LOAD_PATH` anyway). Same story as before: options like this sit uncollected for years. The final find again traces back to the monolith's size: the structure sends one more bill. Each of the example app's 200+ components is a Rails engine under the hood (with Packwerk guarding the boundaries), and sorting their initializers hit an O(n²) path in railties' `tsort` usage, costing seconds on every boot. [Rails 8.1 ships the fix](https://github.com/rails/rails/pull/53615). Until the upgrade hits, a guarded local patch holds the position. #### The whole boot on one screen Whoa, pump the brakes there! The project we're speeding through carries over 200 components? Yes, Factorial, like lots of big codebases, takes the component/pack architecture to heart. This has its evident drawbacks (several slowdowns above trace back to the deep filesystem hierarchy) and its perks. And one of the perks deserves a special showcase, but first we need to learn another, a more graphical way to read the report to understand it. The same profile exports as JSON and opens in [speedscope](https://www.speedscope.app/)'s neat UI: ```sh REQUIRE_PROFILE_PATH=tmp/require-profile.json bundle exec ruby -r./config/boot -require-prof config/environment.rb ``` Three views are in the spotlight here. Time Order replays the boot left to right: the requires first, then the `rails:` phases, in the order they really ran. Unlike a sampled flamegraph, every millisecond on this axis is an exact load duration: *Image: The whole boot on one screen: requires on the left, `rails:` phases on the right—and every millisecond on the axis actually happened* Left Heavy regroups the same data by weight: identical subtrees merge, and the widest offenders float to the left, regardless of how scattered their loads were in time: *Image: `run_prepare_callbacks` is the heaviest phase of the boot* Sandwich ranks every frame by self time; the files that are slow on their own, not through their children (it's also the simplest view to find the next cue): *Image: The top 10 slowdown contenders* This is where the promised perk of the component structure finally shines. The profiler emits every path segment as its own frame, so costs stack by directory: `components/#{name}/` blocks stand side by side on the screen, and every slow millisecond arrives with a code owner attached. Yet there is still a blind spot worth noting. It can mark some line in the report as heavy, but never explain *why*. And some costs never get a line at all: for example, a bloated `$LOAD_PATH` taxes every require a little, everywhere at once. The answers hide below the require machinery, in Ruby internals, native code, and syscalls. Reading that layer takes a sampling profiler—the co-driver with the pace notes. #### Lap 2: reading a sampling profiler without spinning out So if you've ever needed to look inside the engine bay, down at Ruby internals—and let's be honest, this is where many developers bail out and defer to the "mechanics"—here's a short (extreme) driving course for you. The sampling idea itself is simple: the profiler interrupts the program hundreds of times per second and records the current call stack. Time is attributed statistically; a method "taking N%" was on the stack in N% of samples. Nothing is traced exactly, which is why you should trust percentages over milliseconds on short boot runs. require-profiler integrates natively with [Stackprof](https://github.com/tmm1/stackprof): [Vernier](https://vernier.prof) works here too, with a richer UI: threads, allocations, and GC are also visible, though these parts are less relevant to boot. ```sh REQUIRE_PROFILE_STACKPROF=warden/cognito/test_helpers.rb bundle exec ruby -r./config/boot -require-prof config/environment.rb ``` That writes a speedscope-ready JSON next to the raw dump. Reading the result takes one skill: telling *total time* (a method plus everything it calls) from *self time* (the work done in the method's own body). A frame with huge total and near-zero self is a manager, not a worker. Drill into its children until self time shows up. The views read the same on sampled dumps: Left Heavy turns thousands of samples into wide, readable bars, and Sandwich surfaces the native frames immediately, C code and syscalls included. The first sampler find paid off quickly at Factorial: extra filesystem traversal, caused by deep directory nesting (components inside components), dragged the boot by over half a second. That's pure `Dir.glob` work, syscalls with no tree line of their own. Fixed-depth globs instead of `components/**` walks cured it. An extreme case at this size, granted, but checking for excessive work wins again, this time via sampling. And the second find was a brand-new catch. The require tree blamed a file in a local Warden fork—and that's all we could say. The sampler showed the rest: the time was sinking into OpenSSL key generation, native code invisible to any require-level view: {% image "stackprof-sandwich.png", caption: "OpenSSL key generation: 88% of the file's load, all self time", alt: "speedscope Sandwich view over the Stackprof dump" %} It costs up to half a second when the machine runs low on entropy. The same C boundary hides the classic native extension init (`grpc`, `sassc`, etc.): one `require` kicks off the whole library setup, and Ruby sees none of it. C boundaries are where samplers pay for themselves. So here's the final plan. Run require-profiler first: plain requires, initializers, YAML, and HTTP, all in one readable pass. Drop to the sampler only for the lines the tree can't explain. > With that, Factorial's pit stop is over: the boot went from roughly 23 to 13 seconds, 40% off, with no obvious single hotspot left behind. Production got about 20% off too (several fixes were development-only), and every deploy now clocks faster. Past that point you're not removing bolts, you're redesigning the car. One such possible redesign is *scoped boot*: each process type loads only its own components, so no web code in Sidekiq (another benefit of the component architecture). That's a story for another legendary ride though. #### One last ride: AnyCable's own boot race Remember `anycable-rails` on Bumbler's shortlist at the very start? Framework gems end up on every leaderboard sooner or later, and nothing keeps a toolmaker honest like pointing the tool at their own gem. And yes, the surprise promised at the start finally shows up here. So we profiled an [AnyCable](https://anycable.io)-powered app's boot again, this time a bare demo. `anycable-rails` topped the leaderboard (what a win!) at 112ms, ahead of `rails` itself at 78ms, before a single cable was even connected. require-profiler broke it down without drama. The first 10ms was dead weight: [Anyway Config](https://github.com/palkan/anyway_config)'s Doppler secrets loader required `net/http` at load time, pulling in the whole HTTP stack whether or not you use Doppler. Another 10ms came from testing-framework patching that lived in the production path. After our patches, it only kicks in when the Action Cable test classes are actually loaded. The gRPC stack added about 20ms more with nothing to trim inside. Need to trim the gRPC part anyway? Try [`anycable-rails-core`](https://github.com/anycable/anycable-rails): the same gem without Google's hefty bundled gRPC server. Bring a lighter Ruby integration instead, or skip the gRPC integration entirely. The remaining chunk in the report was refinements, the biggest surprise of the race. Rewriting the gem's refinement-based patching as plain class reopening saved 10ms more, which made no sense at first: refinements aren't supposed to cost anything at *definition* time. A quick test said otherwise: activating them late in the boot, with ~30,000 classes around, is wildly more expensive than early, at ~4,000. [An isolated benchmark](https://gist.github.com/ardecvz/fdd33501625199018fb698cfea8d8537) confirmed it universally: on Ruby 3.3, `using` overhead grows with the class count, adding 50% or more at a real app's scale. The cause is [Ruby issue #21201](https://bugs.ruby-lang.org/issues/21201): every refinement activation invalidates method caches through a full object-space scan. Ruby 4.0 fixed it with a dedicated refinement-cache table (unfortunately, no backports to older versions), and the same benchmark there shows the overhead more than ten times smaller. Verify from your own pit box: [the benchmark](https://gist.github.com/ardecvz/fdd33501625199018fb698cfea8d8537) is one file—generate N classes, require `anyway_config`, compare with `NOUSING=1` across Ruby versions. What to do if you're an AnyCable user? Upgrade Ruby regularly, to 4.0 and up in this case. Free optimizations help whether you know about them or not, and regular upgrades make each one easier. Stuck on an older Ruby? Then require refinement-heavy gems (`anyway_config`, `anycable-rails`) early in the Gemfile, right after `rails/all` (luckily, such gems are rare in the ecosystem). After the fixes, `anycable-rails` ducked back behind Rails in the standings ...the only race where the maintainer celebrates losing a position. #### Handing the wheel to your AI robot family Now we'll go from tutorial to future infrastructure. require-profiler ships its own instructions for AI agents: a [`SKILL.md`](https://github.com/palkan/require-profiler/blob/master/skills/rails-boot-profiling/SKILL.md) distributed via [Rails Hyperdrive](https://github.com/Bakaface/rails-hyperdrive), the gem that lets any Ruby library deliver skills and guidelines straight into your agent's context: ```sh bundle add rails-hyperdrive require-profiler --group development bin/rails hyperdrive:init create .mcp.json append .gitignore append Gemfile insert config/routes.rb create .claude/skills/rails-boot-profiling/SKILL.md create .hyperdrive/lock.yml done hyperdrive initialized ``` The skill packs everything handy from this ride: the baseline measurement run, threshold-then-focus narrowing, the YAML and HTTP checks, the Stackprof handoff, and a checklist of the common offenders above. So the next time someone types "my app started booting slowly" the agent loads the skill as relevant, runs the profiler, and drives you to a quicker land in full self-driving mode (beware: please keep your hands on the steering wheel, it may do the wrong thing at the worst time, and the driver is responsible for all collisions with production). *Image: Type * We're collecting more skills like this, across all stacks, in [our skills section](https://evilmartians.com/skills). Each one is a piece of engineering practice you can grab and install manually. *Download Evil Martians' agent skill*: https://github.com/palkan/require-profiler The pit board, in five lines: - **Lap 0:** keep Bootsnap enabled, updated, and precompiled for production. Remember it can mask YAML costs while you profile by serving pre-parsed caches, so add a cold rerun. - **Lap 1:** run require-profiler first. The require tree with its plugins covers nearly every boot issue, and most fixes are one-liners. Consult the common offenders in the article. - **Lap 2:** when the tree shows a file but not the reason, scope in with `REQUIRE_PROFILE_STACKPROF`. Native code is where samplers pay off. - **Final lap:** profile third-party gems and frameworks too, especially the ones you maintain. - **Victory lap:** install the skill via Rails Hyperdrive and let your agents help run the whole playbook. Every Rails app race still starts the same, but now AI agents run it hundreds of times a day, and every boot is on the clock. Reach for [require-profiler](https://github.com/palkan/require-profiler) first (almost one-stop, easy to use), and save the sampling profiler for the rare corners the tree can't explain. Teams that internalize profiling ship more than the ones still shopping for a bigger engine. And somewhere at Le Mans, perhaps someone salutes. ### Ship agent skills like packages: discovery index, digests, and install sources Date: 2026-08-12T00:00:00.000Z | Authors: Rita Klubochkina, Travis Turner | Categories: AI, DX URL: https://evilmartians.com/chronicles/publishing-agent-skills-discovery-index Two posts ago, an AI startup found us because Claude, which recommended [Evil Martians](/) when they asked for a senior dev agency. One post ago, we measured the traffic behind that. Over two months, **coding agents read [evilmartians.com](/) more than twice as often as people did**: 268,000 agent requests against 107,000 human pageviews. Both pointed to the same idea: site audiences are increasingly machines, so build for them. This post takes the next step. We stopped waiting for agents to read our pages and started handing them something to install: our own engineering practice, packaged as agent skills, published at [/agent-skills](/agent-skills), and discoverable through a `.well-known` index any agent can find. Here's how to publish yours. An [agent skill](https://agentskills.io/) is a folder of instructions that teaches an AI coding agent a repeatable practice: the conventions and checks for doing a specific job well, packaged so any agent can follow them. Someone authors a practice once, then everyone else's agents install it. This post is a full how-to for publishing your own skills and making them discoverable, using our catalog as the worked example. --- *Let’s run a 2-week LLM-optimization sprint with Evil Martians to improve your discoverability.* [Contact Evil Martians](https://evilmartians.com/contact-us) --- **TL;DR:** An agent skill is a `SKILL.md` file (plus optional extra files) that teaches a coding agent a repeatable practice. To make yours discoverable, publish a `.well-known/agent-skills/index.json` at your domain: a short list of `name`, `description`, `type`, `url`, and `sha256:` digest per skill. The [`npx skills` installer](https://github.com/vercel-labs/skills) reads it. Author each skill in a GitHub repo (installers and Claude's plugin marketplace resolve straight from the repo), but re-host the bytes on your own domain so the digest is computed over what *you* serve. Skills come in three shapes: single-file, multi-file, and bundle, each handled differently. Offer more than one install command, because your readers live in different tools. None of this is new machinery: it's what npm and RubyGems have always done, pointed at a folder with a `SKILL.md` in it. The playbook below is also packaged as a skill, the same way we did for [part one](/chronicles/how-to-make-your-website-visible-to-llms). Install it and your agent can publish your catalog for you, discovery index and digests included. Read on for what it does, and why each piece works the way it does. *Download Evil Martians' agent skill*: https://github.com/evilmartians/agent-skills #### What's an agent skill, and why open-source ours? An agent skill is the smallest useful unit of "how we do this." Anthropic [introduced the format](https://claude.com/blog/skills) for Claude in October 2025 and released it as an [open standard](https://agentskills.io/) that December. [Cursor](https://cursor.com/), [Codex](https://openai.com/codex/), [GitHub Copilot](https://github.com/features/copilot), and [Gemini CLI](https://geminicli.com/) read the same file now, each from a skills directory it scans on startup. A skill is a `SKILL.md` file: YAML front matter with a `name` and a `description` that tells the agent *when* to reach for it, then Markdown instructions for the *how*. Some skills ship extra files alongside: a script, a reference doc, a template. That's the entire format; so, no runtime, API, or server. An agent loads the file when the description matches the task, and follows it. We use a lot of these internally, so we published the ones useful beyond our own projects at [/agent-skills](/agent-skills). Here's a few of them: - **`good-readme`** — writes a README the way we'd review it, with the sections that actually get read. - **`layered-rails`** — our conventions for structuring a Rails app in layers, so the code stays legible as it grows. - **`secure-npm-package`** — the checks we run before publishing an npm package, so a supply-chain mistake gets caught before it ships. - **`storybook-workbench`** — eleven skills for building and auditing a [Storybook](https://storybook.js.org/), published as one bundle. - **`inertia-rails-skills`** — another eleven, covering [Inertia.js](https://inertiajs.com/) on Rails across React, Vue 3, and Svelte, published from the [Inertia Rails](https://github.com/inertia-rails/skills) project's own repo. - **`llms-visibility`** — makes a site readable to LLMs and the coding agents that fetch URLs: Markdown routes, `Accept` content negotiation, and `llms.txt`, plus the AI SEO anti-patterns to refuse. It's [part one](/chronicles/how-to-make-your-website-visible-to-llms) of this series, packaged as a skill. Why give them away? The same reason we open-source everything else! When a client's coding agent installs `layered-rails`, it structures their Rails app the way our engineers would, on a Tuesday afternoon, with no Martian in the room. Sharing a skill is the most compact form of distribution we've found for an opinion: it travels into codebases we'll never touch and does the work there. It's the same approach that turned [PostCSS](https://postcss.org/) and [imgproxy](https://imgproxy.net) from things we built for ourselves into tools other teams now run in production. That said, this only works if agents can *find* the skills. Which brings us to the actual subject of this post. #### The discovery index: one JSON file agents already look for Here's the mechanism. The [`npx skills`](https://github.com/vercel-labs/skills) installer (an open source tool from Vercel Labs, distributed on npm as `skills` and backed by the [skills.sh](https://skills.sh) registry) can install a skill straight from a hosted site, not only from a Git repo. Just point it at our skills page: ``` npx skills add https://evilmartians.com/agent-skills ``` Behind that command, it reads the discovery index we serve at `https://evilmartians.com/.well-known/agent-skills/index.json` and lists every skill we publish for you to pick from in an interactive prompt. This is the same idea as `llms.txt` from part one, `robots.txt`, or a sitemap: a well-known path where machines look for a machine-readable summary of what you offer. To skip the prompt, name what you want up front: `--skill ` installs one, `--skill '*'` takes the whole catalog. The full `https://` URL is what triggers discovery: hand `npx skills` a bare `owner/repo` or domain instead and it treats the source as a Git repo to clone, not a site to read an index from. *Read also Evil Martians' article*: https://evilmartians.com/chronicles/how-to-make-your-website-visible-to-llms The file is small, and let's note that the format isn't ours. The `.well-known/agent-skills/index.json` discovery document comes from an open [RFC published by Cloudflare](https://github.com/cloudflare/agent-skills-discovery-rfc). It builds on [RFC 8615](https://www.rfc-editor.org/rfc/rfc8615), the `.well-known/` convention that already gives us `robots.txt` and `security.txt`, and it points at the [`schemas.agentskills.io/discovery/0.2.0`](https://agentskills.io/) schema. The document is a list of entries: ```json { "$schema": "https://schemas.agentskills.io/discovery/0.2.0/schema.json", "skills": [ { "name": "llms-visibility", "description": "Make a site readable to ChatGPT, Claude, Perplexity, and the coding agents that fetch URLs…", "type": "skill-md", "url": "https://evilmartians.com/agent-skills/llms-visibility/SKILL.md", "digest": "sha256:9f2b…" }, { "name": "layered-rails", "description": "Write, refactor, and review Rails code using layered architecture principles…", "type": "archive", "url": "https://evilmartians.com/agent-skills/layered-rails.tar.gz", "digest": "sha256:1c4a…" } ] } ``` We have five fields per skill. `name` is the slug an installer uses. `description` is the same one-line "use this when…" the agent reads to decide relevance. `type` is either `skill-md` (the payload is a lone `SKILL.md`) or `archive` (it's a tarball with more than one file). `url` points at the bytes. `digest` is a SHA-256 hash of exactly those bytes, in `sha256:` form. The installer re-hashes whatever it downloads and refuses to install on a mismatch, and the RFC makes that mandatory rather than optional politeness: a conformant client MUST verify the digest and MUST NOT use content that fails. So the digest is a tamper check: it guarantees you got the skill the publisher meant to publish, not something rewritten in transit. We have two top-level keys, and that's the whole document. There's no `version`, `origin`, `bundles` array, and no per-skill `bundle` field; a bundle is represented only as its member skills, each a normal entry. Inventing a structure the schema doesn't define is the most common way to hand an installer something it ignores. The one field with rules of its own is `name`, which has to be a valid skill name (lowercase letters, digits, single dashes) or an installer won't take the entry. Our build checks that before writing the index and leaves a skill out with a warning rather than publishing an entry nothing can install. Who reads this format today? The `npx skills` installer is the reference consumer, but the RFC is vendor-neutral by design, so the convention isn't owned by one client. Cloudflare authored the discovery spec, Anthropic defined the underlying skill format, and the schema lives at agentskills.io. Any agent, registry, or installer can fetch the index and resolve a skill from it, the same way any crawler can read your `robots.txt`. The schema is versioned (we're on `0.2.0`), so it can evolve without breaking the installers pinned to a given version. If part one's lesson was "_ship clean Markdown at a well-known URL and tell the world it exists_," this is the same move applied to a different artifact. If you have skills, put a list of them where agents look. #### Single-file, multi-file, and bundle: three shapes, three install paths That `type` field is doing more work than a two-value enum suggests. Not every skill is shaped the same, and the shape decides how it's served and installed. There are three to look at: | Shape | `type` in the index | What `url` points at | What the installer does | | --- | --- | --- | --- | | Single-file | `skill-md` | the `SKILL.md` itself | copies one file | | Multi-file | `archive` | a flat `.tar.gz` | unpacks it into the skill's folder | | Bundle | one entry per member skill | each member's own payload | installs members one at a time, or takes the combined `-bundle.tar.gz` in one shot | **A single-file skill is just `SKILL.md`.** Nothing else in the folder. In the index, it is `type: skill-md`, and its `url` points straight at the Markdown file. This is the least work to install, because installing is copying one file to the right place. It's also the only shape short enough to paste directly into a running agent, so for these we offer a "_copy the whole `SKILL.md`_" action alongside the install command. **A multi-file skill ships `SKILL.md` plus extra files**—a script it runs, a reference table, a template it fills in. You can't install that by copying one file, so we pack it into a flat `.tar.gz` (`SKILL.md` and its friends at the archive root) and the index marks it `type: archive`, with the `url` pointing at the tarball. The installer unpacks the archive into the skill's folder, and the digest covers the whole tarball. One thing to get right while you're building that tarball: the archive and its hash have to be made together. A `.tar.gz` bakes in more than file contents—member order, file modes, embedded mtimes, and gzip's own timestamp all end up in the bytes—so the same skill archived twice can hash differently while nothing about it changed. We rebuild the archives and the index in one pass, so ours can't drift apart. If yours are built in separate steps, pin the variance (`tar --sort=name --owner=0 --group=0 --numeric-owner --mtime='UTC 2020-01-01'`, piped through `gzip -n` to drop the timestamp) or you'll advertise a digest for a file you no longer serve. **A bundle is one repo that publishes several skills at once.** Our `storybook-workbench` is one repo carrying eleven skills. Each of the eleven gets its *own* discovery entry, under its own name, so `npx skills … --skill ` installs exactly one. The bundle's own page has no `SKILL.md` to preview, because a bundle is a container for skills rather than a skill itself. What it *does* offer is a combined `-bundle.tar.gz` with every skill side by side as its own folder, so a single `curl … | tar -xz -C ~/.claude/skills` drops the whole set into place at once. That's usually the fastest way to adopt a bundle: one command, eleven skills. This specificity matters when you publish your own. A lone `SKILL.md` and a folder of files are handled differently at every layer: the archive we build, the `type` in the index, the digest target, and the install command a reader copies. If you get the shape right, the rest follows. But if you treat a multi-file skill like a single file, the install drops everything but `SKILL.md`. #### Author in a repo, serve from your own domain Every skill we publish is authored in a GitHub repo and served to installers from `evilmartians.com`. Those are two different jobs, and it pays to be deliberate about which one does what. The repo is non-negotiable, for three reasons: - **The authoring workflow lives there** — a skill is code-adjacent, it gets reviewed, versioned, and improved like anything else, and Git is where that happens. - **`gh skill install` reads `SKILL.md` straight from the repo** — a GitHub-native install path needs a GitHub-native home. - **Claude's plugin marketplace is repo-shaped** — a repo with a [`.claude-plugin/marketplace.json`](https://docs.claude.com/en/docs/claude-code/plugin-marketplaces) at its root is an installable marketplace, and the `name` in that manifest is the `@marketplace` half of `claude plugin install @`. Without a repo there's no manifest, and without a manifest there's no plugin to install. But look back at the `url` fields in that index: they point at `evilmartians.com`, not `raw.githubusercontent.com`. The RFC allows either—a `url` resolves against the index and can live at any origin—so this is a choice, and we make it for three reasons: - **Half the catalog has no raw URL to point at** — that flat `.tar.gz` every multi-file skill installs from doesn't exist in the repo; our build creates it. GitHub will hand you a tarball of an entire repo, wrapped in a prefix folder with everything else inside, which is not the file an installer needs. - **A payload on your own domain is a request you can see** — classified at your own edge, which is the only reason we can measure any of this. - **The digest is computed over the exact bytes we serve** — the file our build just wrote into `public/`. The hash can't disagree with what we serve, because it's taken over what we serve. That final one has a payoff worth elaborating upon. A raw URL tracks a branch perfectly well: point it at `main` and it always serves the latest. What it can't do is keep the digest true while it does. Publish a hash, let upstream push, and the bytes change underneath a digest you already advertised. Verification is a MUST, so every conformant installer now refuses the skill, and keeps refusing until you happen to rebuild. With raw URLs you pick your poison: stale-but-valid (pin a commit SHA) or fresh-but-broken (track the branch and hope). Re-hosting removes the choice, because the fetch, the write, and the hash all happen in one build, so **an upstream push goes live with a correct hash, automatically.** The bill for that comes due as latency. A raw URL is live the instant someone merges; ours is live on the next deploy, so a skill author pushing a fix waits for a site build to see it in the index. We think that's a fair trade for a digest that's never wrong, but it's a real cost, and it's the reason to keep the vendoring step boring enough that a rebuild is cheap. Be clear about what re-hosting does and doesn't buy, too. It guarantees *consistency*, meaning the bytes always match the published digest. It doesn't buy immunity from upstream. Push something bad to a skill's repo and we'll re-host it and hash it just as faithfully. The digest protects the trip from us to the installer; reviewing what goes into the repo is a separate job, and still yours. In general, as a rule, **serve the artifact and compute its integrity digest from the same place.** Author wherever you like, but publish from a location you control. #### Every install method, and why we shipped all of them Open any skill's card on [/agent-skills](/agent-skills) and you'll see a target selector (Claude Code, Cursor, Codex, GitHub Copilot, Gemini CLI) over a set of install methods. `SKILL.md` is a cross-agent standard, so one payload serves every agent, and only the destination directory differs. There are four: - **[`npx skills`](https://github.com/vercel-labs/skills)** — reads our discovery index and resolves the payload itself: `npx skills add https://evilmartians.com/agent-skills --skill good-readme -a claude-code -g`. It's the only method that exercises the whole chain: it finds the index, picks the `url` matching the skill's `type`, and verifies the digest before anything lands on disk. - **[`claude plugin`](https://docs.claude.com/en/docs/claude-code/plugins)** — Claude's marketplace path: `claude plugin marketplace add evilmartians/agent-skills`, then `claude plugin install good-readme@evilmartians`. That `@` half is the `name` from the repo's `.claude-plugin/marketplace.json`, not the repo's own name, so read it rather than guess it. Claude-only, and the *best* option for a bundle, because one plugin install pulls the whole set. - **[`gh skill`](https://cli.github.com/)** — installs one skill straight from the repo that authors it, no discovery index involved: `gh skill install evilmartians/agent-skills good-readme`. It finds a skill by its `SKILL.md`, so bundles are the one shape we don't offer it for. It's also the only path that doesn't ask you to trust us: it pulls from the repo you can sit and read, not from our domain and our digest. - **`curl`** — a self-contained one-liner that pulls the payload from our domain and drops it into the agent's skills directory. No installer to trust, nothing to learn: it's `curl` and `tar`. Four commands for one file might seem redundant until you see what each does that the others can't. `claude plugin` installs an eleven-skill bundle in a single step. `gh skill` skips us entirely and installs from the repo you just read. `npx skills` is the only one that verifies the digest before anything lands on disk. `curl` needs nothing installed first, which is the whole point inside a container build where adding a dependency is the cost you're avoiding. Covering all four costs us a few generated command strings. The alternative costs us readers since someone who can't run your one command rarely goes hunting for a workaround; they just close the tab. We'd rather ship four than lose them. #### Four checks before you ship yours Everything above is easy to get subtly wrong in ways that only crop up on someone else's laptop, so verify it the way an installer will: - **The index parses and lists what you think it lists.** `curl -fsSL https://yourdomain.com/.well-known/agent-skills/index.json` should return valid JSON with every skill you meant to publish, and nothing you didn't. - **One digest matches its payload.** Pick an entry, run `curl -fsSL | sha256sum`, and compare. That's the check a conformant installer runs before it writes anything to disk, and it's the one that breaks the moment you hash something other than what you serve. - **The archives are flat.** `tar -tzf .tar.gz` should list `SKILL.md` at the root, not `/SKILL.md`. A wrapping folder is the classic broken install: the files land one level too deep and the agent never sees the skill. - **A real install works end to end.** `npx skills add https://yourdomain.com/agent-skills --skill ` should drop the skill into the target agent's skills directory. Ask the agent something the skill covers, and watch whether it picks it up: that's the only test of whether your `description` is doing its job. #### Measuring what gets installed *Read also Evil Martians' article*: https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured [Part two](/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured) of this series had one governing rule: **you don't know if any of it works until you measure**. That rule applies here too, and the surface to measure is the install itself. We'll be honest up front: the catalog is new, so what we have is instrumentation, not results. So, nothing to brag about yet. Still, here's the instrumentation anyway, because the time to build it is _before_ the traffic arrives, not after. We can't see a `curl` that runs on someone's laptop, but we can see every interaction with the install UI. Each button on a skill card fires a tracked event: an install command copied from the button (tagged with the target agent and the method), a command selected and copied by hand, a `SKILL.md` body copied whole, an archive downloaded, and a click through to the source on GitHub. When a skill card is embedded inside a blog post, every event is suffixed with the post's slug, so a copy or a download can be attributed to the article that drove it. That's the segmentation [part two](/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured) showed you need before you trust any number: which skills get adopted, from which posts, into which agents. The same server-side technique from part two applies to the payload fetches themselves. Requests to `/.well-known/agent-skills/index.json` and to each skill's `url` are ordinary HTTP hits you can classify by `User-Agent` at the edge, which is where to look if you want to know which agents resolve your skills rather than which humans copy your commands. And as part two warned, split your user-agent buckets before you believe the totals: our own numbers only made sense once ChatGPT and Claude Code were counted separately, because they turned out to want opposite things. #### FAQ #### What is an agent skill? A `SKILL.md` file (plus optional extra files) that teaches an AI coding agent a repeatable practice. YAML front matter gives it a `name` and a `description` that says when to use it, and the Markdown body says how. Anthropic introduced the format for Claude, and most major agents (Cursor, Codex, GitHub Copilot, Gemini CLI) now read it too. It's a plain file, with no runtime. #### Do agent skills work outside Claude Code? Yes. `SKILL.md` is a cross-agent format, the payload is identical everywhere, and only the destination directory changes. Inside a project, that directory is converging on a shared `.agents/skills/`, which Cursor, Codex, GitHub Copilot, and Gemini CLI all read, with Claude Code the holdout at `.claude/skills/`. Installed globally, each agent still keeps its own home: `~/.claude/skills`, `~/.cursor/skills`, `~/.codex/skills`, `~/.copilot/skills`, `~/.gemini/skills`. The `npx skills` installer maps 70+ agents onto those paths for you, which is the argument for publishing one discovery index rather than a per-agent bundle five times over. #### How do I make my agent skills discoverable to LLMs? Publish a `.well-known/agent-skills/index.json` at your domain. This is a list of your skills, each with a `name`, `description`, `type` (`skill-md` or `archive`), a `url` to the payload, and a `sha256:` digest of that payload. The [`npx skills` installer](https://github.com/vercel-labs/skills) reads it when someone runs `npx skills add https://yourdomain.com/agent-skills --skill `—the full `https://` URL is what points the installer at your index instead of trying to clone a repo. It's the skills equivalent of `llms.txt`: a well-known, machine-readable list at a path agents already check. #### How do I share my agent skills with other developers? Publish them so any agent's installer can resolve them, instead of emailing a file around. Put each skill's `SKILL.md` in a GitHub repo, then serve a `.well-known/agent-skills/index.json` discovery index at your own domain listing each skill's `name`, `description`, `type`, `url`, and `sha256:` digest. Once that's live, anyone can install a skill with `npx skills add https://yourdomain.com/agent-skills --skill `, `gh skill install`, or a Claude plugin. Re-host the payloads on your domain rather than linking raw GitHub URLs, so the integrity digest matches the exact bytes you serve. That's how our own catalog at [/agent-skills](/agent-skills) is shared. #### Do I have to host skills on GitHub? For the repo-based install paths, yes: `gh skill install` and Claude's plugin marketplace both resolve straight from a GitHub repo, and the plugin `@name` comes from the repo's `.claude-plugin/marketplace.json`. But don't point your discovery index at raw GitHub URLs. Re-host the payloads on your own domain and compute the digest over the bytes you serve, so an upstream change can't break the integrity check behind a hash you already published. #### What's the difference between a single-file skill and a bundle? A single-file skill is one `SKILL.md`, served directly and marked `type: skill-md`. A multi-file skill ships extra files and is packed into a `type: archive` tarball. A bundle is one repo publishing several skills at once: each skill gets its own discovery entry and installs individually, and the bundle also offers a combined archive so `curl … | tar` can install the whole set in one command. As an example, our `storybook-workbench` bundle is eleven skills in one repo. #### Why offer more than one install command? Because your readers live in different tools. A Claude Code user wants the plugin marketplace. Someone scripting a container wants a dependency-free `curl` line. Someone who'd rather install from the repo they can read than from our domain wants `gh skill`. A less technical reader wants one copy-paste command for the tool they already have open. The `SKILL.md` payload is identical across agents, so the cost of offering four paths is a few command strings, and the payoff is not losing the reader who didn't use your default. #### Is a discovery index the same as llms.txt? Same *pattern*, pointed at a new payload. `llms.txt` (from [part one](/chronicles/how-to-make-your-website-visible-to-llms)) points agents at your content in Markdown. `.well-known/agent-skills/index.json` points them at installable skills, with integrity digests so an installer can verify what it downloaded. Both are machine-readable lists at a well-known path. If you already shipped one, the other is the same reflex applied to a new artifact. *Download Evil Martians' agent skill*: https://github.com/evilmartians/agent-skills Agent skills are young, and the conventions around them will shift, the way `llms.txt` and content negotiation are still shifting. But the shape of the answer is already familiar to us, because it's what package registries have always done: a well-known index, a canonical URL per artifact, and a checksum the client verifies before it installs anything. [Evil Martians](/) has been shipping into that model for years across npm, RubyGems, and Docker Hub: 128 open source projects, 186K+ GitHub stars, 25B+ downloads. Publishing skills didn't ask a new discipline of us, just an old one pointed at a new kind of consumer. The agents are already looking for the file. The only question is whether yours is there when they check! ### Fair by design: orchestrating background jobs in Ruby Date: 2026-08-11T00:00:00.000Z | Authors: Alexander Baygeldin, Travis Turner | Categories: Rails, Performance URL: https://evilmartians.com/chronicles/fair-by-design-orchestrating-background-jobs-in-ruby Are you treating your users fairly? They could be stuck in the queue while a greedy user monopolizes resources. And you might not even know it! In this post, you’ll see if it’s time for you to take background job prioritization seriously, and how to make it fair for all users. This article expands on my RubyConf 2026 talk, going deeper into the technical details and trade-offs behind each fairness strategy: *Meet Evil Martians at event*: https://evilmartians.com/events/rubyconf-fair-by-design Let's start with one of my favorite weird metrics: the [Pentagon Pizza Index](https://en.wikipedia.org/wiki/Pentagon_pizza_theory). As the theory would have it, when a political crisis is brewing, late-night pizza deliveries around the Pentagon spike. The staff can't go home, so they order food. Some have even tried to use this as a predictor of major geopolitical events. Fortunately, this post is not about geopolitics. It's about ~~pizza~~ background jobs. Still, imagine you own a pizza place. You've got a kitchen, chefs, and a single queue of orders. Your chefs take the next order from that queue. It's predictable and efficient. Then, "_Client X_" calls with a **massive** late-night order. You think: "Time for some sweet, sweet, revenue!" But a few minutes later, a "regular" customer walks in for one slice. The cashier has to turn them away: "Sorry, it'll be a couple of hours. We're backed up with a huge order." The customer leaves hungry and annoyed: "So unfair! I just wanted a single slice." And they would be right. It was unfair. Nothing dramatic happened and the kitchen did exactly what it was designed to do. But you just lost a customer. And next time, they may not come back, even on a quiet night. So, what happened? And could you have done something differently? --- *Need help with fair prioritization? Evil Martians can help you choose and build the right background job strategy.* [Contact Evil Martians](https://evilmartians.com/contact-us) --- #### Latency isn't the whole story In the world of background job processing, the first metric we look at is queue latency: the age of the oldest job still waiting. Latency is directly tied to quality of service (QoS). High latency makes users angry, just like in the pizza story. The natural reaction is to increase throughput by adding more workers. But workers cost money. Autoscaling helps you avoid paying for idle workers, but it also has a ceiling: a system has limited database connections, external API rate limits, and other shared resources. Even before that ceiling, throwing more money at the problem can stop paying off. Better QoS at higher operational cost does not always bring more customers or more revenue. In either case, you have a bottleneck. Every system has one. The only difference is whether it is wide enough for your peak traffic. If it is not, **latency becomes unavoidable**. And when latency is unavoidable, what do we do? In the pizza story, imagine we could pause the huge order, serve the single-slice customer, then resume. The large customer might not even notice. But the small customer definitely would. The total time to process all orders stays the same. So do overall throughput and queue latency. And yet, the QoS improves. Why? Because we treated our customers ***fairly***. #### What does "fair" mean? *Image: (source: [@urbandata on X](https://x.com/urbandata/status/695261718344290304))* There are actually formal ways to measure fairness. For instance, economists use the Gini index and network engineers use Jain's fairness index. But most teams do not need a fairness metric to know when a queue is unfair. If you have a first-in, first-out queue, occasional high latency, and tenants who can enqueue very different amounts of work, you probably have a fairness problem. From here on, I'll call the unit whose work we're scheduling fairly a _tenant_. Depending on the application, a tenant could be an individual user, an account, an organization, or a customer. Intuitively, fairness means that **everyone gets a piece of the pie**: a slice of the shared resource, up to their appetite, but never at the expense of others. In other words, everyone gets an equal share at first. If some tenants need less, the leftovers are split equally among those who still need more. This continues until the resource is fully allocated. If the shared resource is worker time, then the obvious implementation is to replace one global queue with one queue per tenant and poll those queues round-robin. As a good-enough approximation, you could also poll them at random. Either way, every tenant gets an equal chance to have one of their jobs processed. Everyone gets a turn. Tenants with more work can still consume more, but only after others have had a chance to make progress. Of course, this is the most basic case, where all tenants are treated equally. If you want to give some VIP tenants priority but still not at the expense of normal tenants, you need to change the algorithm slightly: assign different weights to the per-tenant queues and poll them according to those weights, rather than giving every queue the same probability. But either way, this means we would have a dynamic, potentially large set of queues, and most background job processors are not designed to handle this. #### A look inside background job processors Let's look under the hood of four popular Ruby background processors and see how they would handle a large, dynamic set of queues. #### Sidekiq In the [open source version of Sidekiq](https://github.com/sidekiq/sidekiq), every worker would make the following request to Redis every time it wants to pull a job for processing: ```redis BRPOP queue:tenant_1 queue:tenant_2 ... queue:tenant_N ``` [BRPOP](https://redis.io/docs/latest/commands/brpop/) checks `queue:tenant_1`, then `queue:tenant_2`, and so on until it finds a job. If all queues are empty, it waits instead of polling Redis continuously, so Redis doesn't get hammered. With [equal queue weights](https://github.com/sidekiq/sidekiq/wiki/Advanced-Options#queues), Sidekiq gives each queue the same chance of being checked first by shuffling the queue list before making a request. This is surprisingly close to the fair scheduler we sketched above. **The catch is dynamic queues.** Sidekiq workers subscribe to a fixed, static queue list. You can't add or remove tenant queues while the process is running. We'd also need to track "active" tenants, meaning tenants with jobs waiting to be processed, so we don't poll every tenant queue in the system every time. Imagine doing that with hundreds of thousands of tenants. Still, if Sidekiq supported a dynamic list of queues, implementing fair scheduling on top would be almost within reach. #### Sidekiq Pro Unfortunately, the open source version has one serious drawback: after `BRPOP`, the job is no longer in Redis. If a worker dies mid-job, that job can be lost. [Sidekiq Pro](https://sidekiq.org/products/pro/) fixes this [reliability problem](https://github.com/sidekiq/sidekiq/wiki/Reliability). It's one of the main reasons teams choose it over the open source version. But to make job processing more reliable, it had to change how workers pull jobs for processing: ```redis LMOVE queue:tenant_1 queue:sq||tenant_1 RIGHT LEFT LMOVE queue:tenant_2 queue:sq||tenant_2 RIGHT LEFT ... LMOVE queue:tenant_N queue:sq||tenant_N RIGHT LEFT ``` In Sidekiq Pro, every worker has its own set of "in-progress" queues. [LMOVE](https://redis.io/docs/latest/commands/lmove/) moves a job from the tenant queue into the matching worker queue. The worker removes the job only after processing it. If the worker dies, the job is still in Redis and can be recovered. The trade-off is losing the single blocking `BRPOP` across all queues. Because of Redis constraints, each worker now has to try all queues one by one with `LMOVE` until it finds work. Those calls are non-blocking, so a worker can end up polling Redis over and over just to fetch a single job. This does not scale well with the number of queues, so tracking the "active" queues would be absolutely essential: it would increase the chance of pulling a job in a few requests instead of going through the whole queue list. But Sidekiq Pro still doesn't support dynamic queues, so it was never an option anyway. #### Solid Queue [Solid Queue](https://github.com/rails/solid_queue) stores jobs in a relational database and uses `FOR UPDATE SKIP LOCKED` so multiple workers can poll efficiently without waiting on the same locked rows: ```sql SELECT job_id FROM solid_queue_ready_executions WHERE queue_name = 'tenant_N' ORDER BY priority ASC, job_id ASC LIMIT ? FOR UPDATE SKIP LOCKED; ``` Like Sidekiq Pro, it polls one queue at a time until it finds work, so it also doesn't scale well with the number of queues. However, unlike Sidekiq, Solid Queue actually *does* support dynamic queues: you can tell a worker to process all queues named like `tenant_*`. That can make polling less efficient than explicit queue names. The exact cost depends on whether you use MySQL, PostgreSQL, or SQLite, but in any case, Solid Queue has to make one extra request before each poll: ```sql SELECT DISTINCT(queue_name) FROM solid_queue_ready_executions WHERE queue_name LIKE 'tenant_%'; ``` Overall, that's not too bad. **The catch is that Solid Queue doesn't support weighted queues** like Sidekiq does. Queues are polled in exactly the order the database returns them. If it at least shuffled the list before polling, then it'd support the basic use case where all tenants are treated equally. For the VIP case, where some tenants are given priority, it would need to figure out where and how to store the weights. Still, Solid Queue seems much easier to adjust toward our idea of fair scheduling with dynamic per-tenant queues than Sidekiq. #### GoodJob Like Solid Queue, [GoodJob](https://github.com/bensheldon/good_job) is backed by a relational database—PostgreSQL, in this case. Instead of `FOR UPDATE SKIP LOCKED`, it defaults to using advisory locks to avoid lock contention when pulling jobs from the database: ```sql WITH rows AS MATERIALIZED ( SELECT id, active_job_id FROM good_jobs WHERE queue_name = 'tenant_N' AND () ORDER BY priority DESC NULLS LAST, created_at ASC LIMIT ? ) SELECT id FROM rows WHERE pg_try_advisory_lock() LIMIT 1 ``` Additionally, it relies on [PostgreSQL notifications](https://www.postgresql.org/docs/current/sql-notify.html) as a best-effort wake-up signal, so the workers don't have to poll the database as much. However, it supports neither dynamic queues nor weighted queues, making GoodJob an even worse fit for our use case than either Sidekiq or Solid Queue. --- Unfortunately, none of the background job processors we looked at are built with fairness and multi-tenancy in mind. That's not an oversight; they simply optimize for different things: performance and reliability. It's difficult enough to achieve those things while being general-purpose. Fairness is a specialized requirement, and it adds overhead. So, what are we supposed to do? Building a specialized in-house background job processor would be _insane_. These processors aren't FIFO queues with a few extra features. They handle retries, graceful shutdowns, and years of hard-learned production edge cases. So, instead of replacing the processor, let's try to work with what we already have. #### Fairness strategies We're going to look at four strategies that could bring some fairness into our background job processing while building on top of the existing infrastructure. #### Strategy 1: shuffle-sharding This strategy is primarily about workload isolation: if something bad happens to one part of the workload, it doesn't affect the rest. The term _shuffle sharding_ was coined by the AWS team while building Amazon Route 53. Check out the ["Workload isolation using shuffle-sharding" article](https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/) on the Amazon Builders' Library to see how they apply this technique to battle DDoS attacks. We could achieve isolation with traditional sharding: assign each tenant to one shard and dedicate a set of workers to that shard. Then, when one tenant blocks their shard, only tenants assigned to the same shard are affected. If we used Sidekiq with ten shards, we could route the workload between them like this: ```ruby SomeHeavyJob.set(queue: "shard_#{tenant_id % 10}").perform_async(tenant_id) ``` Then we could launch ten Sidekiq processes, one for each shard: ```sh bundle exec sidekiq -q shard_0 bundle exec sidekiq -q shard_1 # ... bundle exec sidekiq -q shard_9 ``` Assuming tenants are distributed uniformly, the chance that two tenants share a shard is one in ten, or 10%. The more shards we create, the fewer tenants one greedy tenant can affect. If we created as many shards as we have tenants and allocated the same resources to each shard, we would essentially achieve fairness. However, every shard needs at least one dedicated worker. With too many shards, we're bound to see workers sitting idle because their shards have no work. In other words, as the number of shards grows, the blast radius of one greedy tenant shrinks, but the risk of underutilizing our resources grows. Now imagine that, instead of routing every tenant's jobs to just one shard, we deterministically assign each tenant a random pair of shards. Then we choose one of those two shards at random for every job: ```ruby def shards_for(tenant_id) # Seed the random number generator to get the same shards every time. (0..9).to_a.sample(2, random: Random.new(tenant_id)) end SomeHeavyJob.set(queue: "shard_#{shards_for(tenant_id).sample}").perform_async(tenant_id) ``` Now each tenant's workload can be handled by two Sidekiq processes instead of one, improving resource utilization. However, if a greedy tenant previously monopolized one shard, they now monopolize two. That means each job from another tenant has a 20% chance of landing in one of the affected shards instead of 10%. On the surface, that sounds worse. But the chance that a greedy tenant _fully_ blocks another tenant's workload has dropped from 10% to about 2%: that's the chance that both tenants share the exact same pair of shards. Ten shards produce 45 unique two-shard combinations: `10 × 9 ÷ 2 = 45`. If each pair is equally likely, the chance that two tenants receive the same pair is `1 ÷ 45`, or about 2.2%. Even if a tenant shares one shard with a greedy tenant, the part of their workload routed to the other shard can still make progress. This follows the shuffle-sharding pattern described in the AWS article: instead of assigning each tenant to exactly one shard, we assign them to small, overlapping subsets of shards. *Image: With shuffle-sharding, tenants are routed to a few shards rather than just one.* The number of unique subsets grows combinatorially with the number of shards. With enough combinations, we could give every tenant a different subset. No two tenants would share every shard, so the chance of one fully blocking another would be 0%. That said, the main purpose of this approach is workload isolation. For example, if a poisonous HTTP request takes down all the servers handling one tenant, other tenants should still be able to use the service, as long as they don't share the same set of shards with that tenant. But when we apply it to fair scheduling for background jobs, it can get awkward. HTTP requests are synchronous and relatively short-lived; if one hits a bad server, it can be retried against another. A background job routed to a flooded queue simply stays there. This could leave a tenant importing 100 files from cloud storage with 99 files imported successfully while one remains stuck for a long time. That might be even more annoying than delaying the whole batch. If we want processing to remain consistent within a single tenant's workload, we can't route their jobs into multiple shards. Are we stuck with traditional sharding, then? Not necessarily! What allows shuffle-sharding to utilize more resources than regular sharding is that workers handle overlapping subsets of tenants. We can achieve a similar effect by assigning overlapping subsets of shards to Sidekiq processes. For example, each process could pull from three equally weighted shards: Mike Perham describes a slightly different approach in [Workload Isolation with Queue Sharding](https://www.mikeperham.com/2019/12/17/workload-isolation-with-queue-sharding/). Instead of equal queue weights, he uses Sidekiq's strict queue order: each process prioritizes a single shard and helps with another only when it has no work in its primary shard. ```sh bundle exec sidekiq -q shard_0,1 -q shard_1,1 -q shard_2,1 bundle exec sidekiq -q shard_1,1 -q shard_2,1 -q shard_3,1 # ... bundle exec sidekiq -q shard_9,1 -q shard_0,1 -q shard_1,1 ``` This significantly improves resource utilization, but the chance that one tenant blocks another remains 10%. And since processes are no longer tied to shards one-to-one, we could create more shards than processes, reducing that chance even further. This raises a question: why not launch a set of identical processes that pull from every shard and, thus, achieve full resource utilization? ```sh bundle exec sidekiq -q shard_0,1 -q shard_1,1 -q shard_2,1 ... -q shard_9,1 ``` With ten shards, we probably could. But this stops scaling as the number of shards grows, as we saw earlier when looking at how Sidekiq Pro pulls jobs from Redis. And that's shuffle-sharding in a nutshell. **It doesn't prevent unfairness**: when a greedy tenant blocks a shard, they block it for everyone else in that shard. But it can make the problem dramatically less noticeable, especially at scale. #### Strategy 2: interruptible iteration Another strategy that could help with fairness is interruptible iteration. Pioneered by Shopify with its [`job-iteration`](https://github.com/Shopify/job-iteration) gem, the idea has since made its way into both Sidekiq and Ruby on Rails. [Sidekiq Iteration](https://github.com/sidekiq/sidekiq/wiki/Iteration) arrived in Sidekiq 7.3. [Active Job Continuations](https://github.com/rails/rails/pull/55127) were introduced in Rails 8.1. The Judoscale article ["Sidekiq Iterable Jobs: With Great Power...."](https://judoscale.com/blog/sidekiq-iterable-jobs) gives a detailed comparison of iterable jobs and the alternative of fanning work out into many separate jobs. The main idea is straightforward: instead of doing all the work inside one long, indivisible job, **split it into small iterations and keep a cursor that records the last completed iteration**. If the job is interrupted, we can resume from that cursor next time instead of starting over. Here's how an iterable file import could look with Sidekiq: ```ruby class ImportFilesJob include Sidekiq::IterableJob def build_enumerator(import_id, cursor:) active_record_records_enumerator( Import.find(import_id).files.pending, cursor: cursor ) end def each_iteration(file, _import_id) file.import! end end ImportFilesJob.perform_async(import.id) ``` Sidekiq calls `each_iteration` once for every file and updates the cursor as it goes. On a graceful shutdown, it finishes the current iteration, saves the cursor, and re-enqueues the job. When a worker picks it up again, processing resumes with the next file. That behavior makes deployments and restarts safer, but iteration alone doesn't make scheduling fair. By default, an iterable job keeps running until it finishes or the worker is interrupted. It doesn't return to the back of the queue after every item. To turn interruptible iteration into a fairness strategy, the job needs to yield after a limited amount of time. Sidekiq's built-in `IterableJob` API doesn't currently support this, but if we used Shopify's `job-iteration` gem instead, we could configure a maximum runtime like this: ```ruby ImportFilesJob.job_iteration_max_job_runtime = 2.minutes ``` Once that time budget expires, the job finishes its current iteration, saves the cursor, and re-enqueues itself. This gives jobs from other tenants a chance to claim the worker. Overall, the interruptible iteration strategy is somewhat orthogonal to fairness. You'll probably want resumable jobs regardless of which fairness strategy you choose, because long, monolithic jobs are simply risky and cumbersome. But it could also solve our fairness problem because by processing each batch sequentially, we limit the execution to a single worker, and by time-bounding that execution, we prevent that batch from holding the worker indefinitely. However, if one tenant can submit many large batches at once, their jobs can still occupy every worker and crowd out everyone else. Limiting each tenant to one active batch prevents that, but if we have more workers than active batches, some workers will sit idle. So, this strategy works well when the workload naturally comes in batches _and_ there are enough active batches to keep the worker pool busy. #### Strategy 3: throttling Unlike the two strategies above, throttling targets fairness directly. It does so by punishing greedy tenants: we detect when a tenant is being greedy, then route their excess jobs into a slower queue. How do we detect greediness? My favorite mental model is the leaky bucket algorithm. For a deeper explanation of the algorithm and a Ruby implementation, read Julik Tarkhanov's ["The unreasonable effectiveness of leaky buckets"](https://blog.julik.nl/2022/08/the-unreasonable-effectiveness-of-leaky-buckets). Mike Perham's ["The Leaky Bucket rate limiter"](https://www.mikeperham.com/2020/11/09/the-leaky-bucket-rate-limiter/) shows how the idea is implemented in [Sidekiq Enterprise](https://sidekiq.org/products/enterprise.html). Picture a bucket with water flowing in and leaking out. Water leaks at a configured rate. While the bucket has capacity, we can pour water in bursts or at a consistent rate. Once it fills, any additional water spills over. Now, suppose tenants pour water by enqueueing jobs (say, one job equals one liter of water). Before enqueueing a job, we check whether there is enough capacity in that tenant's bucket. If there is, we send the job to the main queue. If the water would spill, we send it to the slow queue. *Image: Picture showing a bucket with water leaking at a consistent rate.* Then we can configure Sidekiq to poll the main and slow queues like this: ```sh bundle exec sidekiq -q main,4 -q slow,1 ``` With those weights, Sidekiq checks the main queue four times as often as the slow queue. In a way, we assign a priority to each job dynamically. [Solid Queue](https://github.com/rails/solid_queue#queue-order-and-priorities) and [GoodJob](https://github.com/bensheldon/good_job#job-priority) both support assigning priorities before enqueueing, but those priorities work differently: lower-priority jobs wait in the queue until there are no more higher-priority jobs. Unlike in shuffle-sharding, all workers process both queues. If the main queue is empty, those same workers can drain the slow queue at full capacity. As long as either queue contains jobs, we don't leave workers idle simply because they were assigned to the wrong queue. I won't go deeper into the implementation here because we already have an article explaining this approach and presenting a gem that implements it for Sidekiq. It uses Redis sorted sets and sliding time windows instead of leaky buckets, but the overall idea is the same. *Read also Evil Martians' article*: https://evilmartians.com/chronicles/fair-multi-tenant-prioritization-of-sidekiq-jobs-and-our-gem-for-it I'll only mention that throttling has two catches, both related to the fact that **it approximates fairness at enqueue time**. First, bursty workloads are hard to judge fairly. One tenant may place 100 orders in a day, then go silent for a month. Another might place 10 orders every day for that entire month. If your threshold treats more than 10 orders on any single day as greedy, you punish the first tenant even though they submit less work over the month. Second, once a job is routed to the slow queue, it is stuck there until processed. If the tenant stops being greedy later, the already-throttled work does not magically move back. However, if your tenants' workloads are fairly consistent over time—even if some tenants submit more work than others—throttling is a strong fit. It is simple, scales amazingly well (especially with leaky buckets), and avoids the underutilization problem. #### Strategy 4: per-tenant queues + custom scheduler The previous strategies have something in common: they scale well, but they are imperfect. They improve fairness for specific workloads, yet each has edge cases. If we want a universal solution—one that can enforce our definition of fairness across workload shapes—we need something more radical: a scheduler. In other words, **we need to hire a manager**. *Image: Let* Let's go back to the pizza story. Instead of sending every order straight to the kitchen queue, suppose we hire a manager with a ledger. Every incoming order goes into that ledger first. The manager watches the shared kitchen queue, sleeps while the chefs have enough work, and wakes up periodically to schedule another batch for processing when they are running low. For the chefs, nothing has changed. In fact, they may not even know they have a manager. They still pull orders from a single shared queue and process orders. Instead, the burden of fairness now falls on the manager. The manager can be a dictator, play favorites, or treat everyone equally. That's totally up to them. But we can envision that they implement the ideal fairness strategy we described at the beginning: the ledger holds a virtual queue for each customer, and the manager picks randomly among the active queues, giving each an equal chance. Now, if that sounds like a potential bottleneck, that's because it is. But _not all bottlenecks are created equal_! If the scheduler can push a new batch into the main queue faster than workers can finish the previous batch, then it isn't the thing that's limiting throughput. In my experience, this is often the case for the jobs where fairness matters most: AI workflows making slow HTTP requests, data imports streaming large files, report generation running heavy database queries and compiling PDFs, and media processing. These jobs occupy workers long enough for one tenant's batch to hold everyone else back. Some of this work is CPU-bound—for example, media processing with ImageMagick or FFmpeg—but much of it leaves the Ruby process waiting on input/output (I/O). Thus, it's tempting to think that adding more threads, or using a fiber-based processor such as [`async-job`](https://github.com/socketry/async-job), would reduce queue latency enough that we could forget about fairness. But look at these jobs from the perspective of the whole system: a self-hosted LLM model may be constrained by GPU capacity; an import may be constrained by cloud storage; report generation may be constrained by the database. Increasing concurrency would simply move the bottleneck down the stack. > As long as the system contains a scarce shared resource, a bottleneck is unavoidable. At a small to medium scale, throughput is more likely to be limited by that resource than by the scheduler, so the scheduler's potential bottleneck may not matter in practice. A typical implementation of this approach looks like this: I extracted this exact flow into [`sidekiq-fairplay`](https://github.com/baygeldin/sidekiq-fairplay), a reference implementation for Sidekiq. - Intercept jobs before they enter the main queue. - Store each intercepted job in a virtual queue for its tenant. - Keep track of tenants that currently have jobs waiting. - Run a periodic planner job, perhaps once a minute. - Have the planner check the main queue's latency. If it is already high, the planner goes back to sleep. If it is below a threshold, the planner selects jobs from active tenants according to your fairness policy and promotes them to the main queue. That policy can be as opinionated as you need: equal weight for everyone, more weight for paying tenants, or a cooldown for tenants that just consumed a large share. Some libraries, such as [`fairway`](https://github.com/customerio/fairway), take a different route and modify how the processor pulls jobs from its storage backend. While clever, this approach couples the implementation to the processor's internals, making the whole thing fragile and hard to maintain. The best part of this design is that **it leaves the background processor's core alone**. It needs only a way to intercept jobs before they reach the main queue—for example, with Sidekiq middleware—and access to the main queue's latency metric. Once the scheduler promotes a job, the processor remains responsible for execution, retries, and crash recovery. Another advantage is how straightforward it is to reason about. We have only two main knobs: how often the scheduler wakes up to push the next batch and the size of that batch. If we see resource underutilization, each batch is being processed before the scheduler promotes the next one. In that case, we can make the scheduler wake up more often. We only need to ensure that it can finish planning before the next wake-up. If it cannot finish within that shorter interval, we can keep the current interval and increase the batch size instead. If calculating tenant weights is the expensive part, promoting two jobs per tenant instead of one can roughly double the amount of scheduled work without doubling the planning time. There is a limit. As the number of active tenants grows, the planner takes longer to make its decisions. To keep workers busy, it must promote larger batches, and those larger batches compromise fairness: a newly active tenant has to wait behind more work already sitting in the main queue. At that point, it may be time to switch to one of the more scalable, workload-specific strategies above. Still, I suspect this design can stretch quite far. If the planner runs once a minute, how many active tenants does it take before the planner cannot decide what to pull within a minute? Quite a lot, I would think. This is the approach that worked best for me personally. I tried the alternatives; this one held up, and we did not run into scaling problems in practice. But your mileage may vary. #### Choosing a fairness strategy We have explored four strategies, each with its own trade-offs and suitable workloads. There are also many variations on these ideas. None is universally best. For an experimental comparison, check out Vladimir Dementyev's [`faqueue`](https://github.com/palkan/faqueue). It measures per-tenant latency while emulating shuffle-sharding, throttling, interruptible iteration, and several other strategies. If you ask me where to start, I would try per-tenant queues with a custom scheduler first. It is the most flexible option, and it was the only one that consistently worked for me in practice. I would choose something else only if I already had reason to believe the planner could not handle the number of active tenants. In that case, look at the shape of your workload: - If you have many active tenants and occasional _hogging_ is tolerable, **try shuffle-sharding**. It isolates most tenants while accepting that an unlucky few may still share a busy shard. - If work naturally arrives in large batches, **try interruptible iteration**. Limit how many batches a single tenant can run concurrently and time-bound each batch's execution to free worker capacity for other tenants. - If tenants enqueue jobs at a consistent rate rather than in large bursts, and you mainly need to slow down the greedy tenants, **try throttling**. It scales well and is straightforward to configure. > Whichever strategy you choose, don't ignore the problem. Once queue latency becomes visible to users, fairness stops being an abstract scheduling concern and becomes part of the product experience. A large customer should be able to place a large order. But a small customer should still be able to get that one slice. Design for this _before_ the queue backs up, and you have a much better chance of never turning that customer away. ### 5 best practices for preventing chaos in Tailwind CSS Date: 2026-08-05T00:00:00.000Z | Authors: Nina Torgunakova, Travis Turner | Categories: URL: https://evilmartians.com/chronicles/5-best-practices-for-preventing-chaos-in-tailwind-css Working with Tailwind CSS is pretty fast and easy (that's why it's received such wide recognition). You just paste a list of different classes in your HTML—and your interface immediately becomes attractive! But, as the application grows, the lists of classes grow. Then, one day you realize you can't understand your code, you're confused with the structure of the application and magic variables, and work becomes a struggle. This article is all about avoiding this scenario, sharing some best practices to ensure you stay aloft when using Tailwind CSS. **Want your agent to do this for you?** We packaged the guide as a skill you can download that includes every practice below, plus the anti-patterns to catch. *Download Evil Martians' agent skill*: https://github.com/evilmartians/agent-skills We can prevent headaches and resolve any problems (for the most part) by using Tailwind precisely and wisely. But, there are two requirements your project must meet, and if it doesn't, Tailwind can actually make your job very difficult. *Read also Evil Martians' article*: https://evilmartians.com/chronicles/better-dynamic-themes-in-tailwind-with-oklch-color-magic **First, you should have a design system in your project**. Tailwind’s philosophy couples with a design system where designers and developers use consistent design tokens. Design tokens are atomic values (like colors, spacing, or typography scales) that define a design’s properties and that are reused throughout the project. Let's imagine we have a standard button and some tabs that need to be the same color as that button: ```css .button { background-color: oklch(45% 0.2 270); } .tab { background-color: oklch(45% 0.2 270); } ``` If we decide to change the color scheme of the project a little, we’ll need to find every instance of this color (which looks like a magic variable) and update them everywhere. This can be inconsistent and hard to maintain. --- *We built open source projects PostCSS and Autoprefixer, used by millions of engineers. Hire us to optimize your frontend workflow and build scalable solutions.* [Contact Evil Martians](https://evilmartians.com/contact-us) --- > Design tokens help prevent these problems and ensure uniformity across UI elements. Luckily, to implement design tokens, we only need to define them once—directly in your CSS via the `@theme` directive (or in `tailwind.config.js` if you still use Tailwind v3): ```css /* Tailwind v4 — your CSS entry file */ @theme { --color-primary: oklch(45% 0.2 270); } ``` ```js // Tailwind v3 — tailwind.config.js module.exports = { theme: { colors: { primary: "oklch(45% 0.2 270)", }, }, }; ``` After adding a new color with the name `primary`, we can use `bg-primary` for our background color or `text-primary` for the text color throughout the application: ```html First tab ``` This way, when you want to change the color scheme in the project, you only need to replace the color in one place. It's better to **avoid using Tailwind if you haven't considered a design system** because you'll have to write magic values in the class lists (like `'p-[123px] mb-[11px] gap-[3px]'`) or add a lot of new tokens (`15px`, `16px`, `17px` in the spacing config), and this will eventually bring a lot of mess to your code. > Having a consistent design system is good because it can help the development and design teams understand each other better. For instance, within Figma, you can have a single shared source of truth for any values in your design system. But to make this system truly maintainable, you'll need to introduce some conventions regarding token grouping and naming—which we'll get into later in this article. *Read also Evil Martians' article*: https://evilmartians.com/chronicles/5-signs-your-project-needs-a-design-system This is the second requirement your project needs to meet: **you should already be using a component-based approach**. The utility-first approach can lead to quite cluttered and verbose HTML structures since Tailwind classes apply directly to elements. This can mean the markup is harder to read and maintain, especially noticeable as your project grows. > The solution: actively using a component-based approach that encapsulates frequently used patterns (in our case, HTML elements appearing more than once) as separate components. With this approach, we can keep things [DRY](https://en.m.wikipedia.org/wiki/Don't_repeat_yourself). Moreover, we'll still have a single source of truth for our Tailwind styles, and we can easily update it together in one place: ```jsx Custom Button ``` If your development tool doesn't allow you to split your code into components, it's likely that the utility-first approach of Tailwind will only make development harder, and you should probably look to other CSS frameworks-for example, [CSS Modules](https://github.com/css-modules/css-modules). And one last thing regarding a component-based approach: **avoid using the `@apply` directive**: ```css .block { @apply bg-red-500 text-white p-4 rounded-lg active:bg-blue-700 active:text-yellow-300 hover:bg-blue-500 hover:text-yellow-300; } ``` Yes, by using this directive, your code may look cleaner, but it throws away the key advantages of Tailwind: less mental overload when coming up with names for CSS classes, and the absence of regressions when changing styles (since with `@apply` they won't be isolated within the component). Further, using it increases CSS bundle size. In Tailwind v4, `@apply` also needs an explicit `@reference` import to access your theme when used in a separately bundled stylesheet (CSS Modules, or a `