https://www.shayon.dev/Recent content on Shayon MukherjeeHugo -- gohugo.ioen-us© 2026 Shayon MukherjeeSat, 10 Oct 2026 10:00:00 -0400https://www.shayon.dev/post/2026/283/versioned-filesystem-for-disposable-sandboxes/Sat, 10 Oct 2026 10:00:00 -0400https://www.shayon.dev/post/2026/283/versioned-filesystem-for-disposable-sandboxes/livedraftsandbox 1$ python sync_leads.pysandbox 2$ node enrich.jssandbox 3, days later$ python report.py41424344reads 43branch/mainfork, nothing copiedbranch/drafta sandbox on the draft$ python score_v2.pyEach sandbox commits a new snapshot to the volume, live keeps moving forward, and a draft forks off snapshot 43 without copying anything. At Tines we built 3B around a simple idea, which is that every step in a workflow is a computer. A step’s code, whether a person or an agent wrote it, runs inside a fresh Linux sandbox that gets its credentials through a proxy, does its work, and then gets thrown away so no leftover state can cause security issues later.https://www.shayon.dev/post/2026/220/every-fast-write-moves-work-somewhere-else/Sat, 08 Aug 2026 10:00:00 -0400https://www.shayon.dev/post/2026/220/every-fast-write-moves-work-somewhere-else/Every storage engine has to decide what must finish before it tells a client that a write succeeded. The quickest answer is to return after copying the bytes into memory. A local durable write waits for fdatasync() on an SSD in the database host. Keeping the write after that host disappears means waiting for a network volume, an object store, or several database servers to save their own copies. Those choices move latency and durability together because returning after memory is fast, but a machine crash can lose the write.https://www.shayon.dev/post/2026/161/building-a-tiny-fuse-filesystem/Wed, 10 Jun 2026 10:00:00 -0400https://www.shayon.dev/post/2026/161/building-a-tiny-fuse-filesystem/Lately I have been working around sandboxing, storage, and networking, and a lot of that work keeps coming back to files, which makes sense since Unix has organized itself around everything is a file for over fifty years. Your terminal and random number generator are device files you can open and read (/dev/tty, /dev/urandom), and even network sockets, which are created with their own system call rather than opened by path, are read and written through the same interface afterwards.https://www.shayon.dev/post/2026/118/i-am-not-leaving-github-any-time-soon/Tue, 28 Apr 2026 12:00:00 -0400https://www.shayon.dev/post/2026/118/i-am-not-leaving-github-any-time-soon/I created my GitHub account in October 2011. User 1,100,970. My first repository was a small project for an openSUSE summit, pushed in February 2012. I’ve opened GitHub almost every day since then for nearly fifteen years. It’s where most of my professional life has happened. So when Mitchell Hashimoto wrote last week that GitHub “is no longer a place for serious work” and that he’s moving Ghostty off the platform, I paid attention.https://www.shayon.dev/post/2026/65/linux-page-faults-mmap-and-userfaultfd/Fri, 06 Mar 2026 14:00:00 -0500https://www.shayon.dev/post/2026/65/linux-page-faults-mmap-and-userfaultfd/I recently went down a rabbit hole trying to understand how Linux handles page faults, what mmap actually does at the physical page level, and how userfaultfd lets userspace take over that fault handling. The motivation was a specific problem, which was making Virtual Machine (VM) snapshot restore fast by lazily populating guest memory. But the underlying mechanisms are general Linux concepts that I think are worth understanding on their own.https://www.shayon.dev/post/2026/52/lets-discuss-sandbox-isolation/Sat, 21 Feb 2026 14:00:00 -0500https://www.shayon.dev/post/2026/52/lets-discuss-sandbox-isolation/There is a lot of energy right now around sandboxing untrusted code. AI agents generating and executing code, multi-tenant platforms running customer scripts, RL training pipelines evaluating model outputs—basically, you have code you did not write, and you need to run it without letting it compromise the host, other tenants, or itself in unexpected ways. The word “isolation” gets used loosely. A Docker container is “isolated.” A microVM is “isolated.” A WebAssembly module is “isolated.https://www.shayon.dev/post/2026/38/understanding-how-gil-affects-checkpoint-performance-in-pytorch-training/Sat, 07 Feb 2026 14:00:00 -0500https://www.shayon.dev/post/2026/38/understanding-how-gil-affects-checkpoint-performance-in-pytorch-training/I have been spending time learning about model training infrastructure lately, and something that stood out to me was GIL contention when saving training checkpoints in PyTorch. Having spent years in the Ruby world dealing with the GVL (Global VM Lock, Ruby’s equivalent), I was naturally drawn to it. The symptoms are familiar, like - you spin up background threads expecting parallelism, and instead everything gets slower. So, as one does, I decided to go down some rabbit holes as an opportunity to learn more about what the GIL actually is, why it makes saving checkpoints in the background counterproductive when done with threads, what the fix looks like, and how I measured all of it on an H100 (rented for a few hours) with a real Llama model to get a better mental model.https://www.shayon.dev/post/2026/19/software-engineering-when-the-machine-writes-code/Mon, 19 Jan 2026 16:04:00 -0500https://www.shayon.dev/post/2026/19/software-engineering-when-the-machine-writes-code/In 1968, a group of computer scientists gathered at a NATO conference in Garmisch, Germany, and coined the term “software crisis.” The problem they identified wasn’t that computers were bad or unreliable. It was that computers had become too powerful for the existing methods of programming to handle. Edsger Dijkstra later put it memorably: “As long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem.https://www.shayon.dev/post/2025/314/a-hypothetical-search-engine-on-s3-with-tantivy-and-warm-cache-on-nvme/Mon, 10 Nov 2025 10:00:00 -0400https://www.shayon.dev/post/2025/314/a-hypothetical-search-engine-on-s3-with-tantivy-and-warm-cache-on-nvme/I’ve been curious about how far you can push object storage as a foundation for database-like systems. In previous posts, I explored moving JSON data from PostgreSQL to Parquet on S3 and building MVCC-style tables with constant-time deletes using S3’s conditional writes. These experiments showed that decoupling storage from compute unlocks interesting trade-offs while lowering costs and simpler operations in exchange for higher cold query latency. Search engines traditionally don’t fit this model.https://www.shayon.dev/post/2025/300/diwali/Mon, 27 Oct 2025 01:11:27 -0400https://www.shayon.dev/post/2025/300/diwali/Every year, I use Diwali as a moment to pause and reflect. Not in any formal way, not tied to ritual or ceremony but just a natural checkpoint (a database intended pun) where the lights come on and I look back at the year behind me, and forward at what’s ahead. It’s one of my favorite festivals, and I’d be lying if I said the sheer diversity and volume of sweets I consume during this period doesn’t have something to do with it.https://www.shayon.dev/post/2025/285/mutable-atomic-deletes-with-parquet-backed-columnar-tables-on-s3/Sun, 12 Oct 2025 09:00:00 -0400https://www.shayon.dev/post/2025/285/mutable-atomic-deletes-with-parquet-backed-columnar-tables-on-s3/In the previous post, I explored a Parquet on S3 design with tombstones for constant time deletes and a CAS updated manifest for snapshot isolation. This post extends that design. The focus is in file delete operations where we replace a Parquet row group and publish a new footer using S3 Multipart Upload (MPU) and UploadPartCopy without having to download and rebuild unchanged bytes. We preserve the same system model with immutable data files and a manifest pointer that serializes visibility.https://www.shayon.dev/post/2025/277/an-mvcc-like-columnar-table-on-s3-with-constant-time-deletes/Sat, 04 Oct 2025 08:00:00 -0400https://www.shayon.dev/post/2025/277/an-mvcc-like-columnar-table-on-s3-with-constant-time-deletes/Parquet is excellent for analytical workloads. Columnar layout, aggressive compression, predicate pushdown, but deletes require rewriting entire files. Systems like Apache Iceberg and Delta Lake solve this by adding metadata layers that track delete files separately from data files. But what if, for fun, we built something (arguably) simpler? S3 now has conditional writes (If-Match, If-None-Match) that enable atomic operations without external coordination. Let’s explore how we might build a columnar table format on S3 that gets most of Parquet’s benefits while supporting constant-time deletes.https://www.shayon.dev/post/2025/276/exploring-postgresql-to-parquet-archival-for-json-data-with-s3-range-reads/Fri, 03 Oct 2025 10:00:00 -0400https://www.shayon.dev/post/2025/276/exploring-postgresql-to-parquet-archival-for-json-data-with-s3-range-reads/PostgreSQL handles large JSON payloads reasonably well until you start updating or deleting them frequently. Once payloads cross the 8 KB TOAST threshold and churn becomes high, autovacuum can dominate your I/O budget and cause other issues. I have been exploring the idea of moving older JSON data (read: cold data) to Parquet on S3 while keeping recent data hot in PostgreSQL daily partitions, then simply dropping those partitions instead of running expensive DELETE operations and subsequent vacuum cycles.https://www.shayon.dev/post/2025/221/bypass-postgresql-catalog-overhead-with-direct-partition-hash-calculations/Sat, 09 Aug 2025 22:30:00 -0400https://www.shayon.dev/post/2025/221/bypass-postgresql-catalog-overhead-with-direct-partition-hash-calculations/PostgreSQL’s hash partitioning distributes rows across partitions using deterministic hash functions. When you query through the parent table, PostgreSQL must perform catalog lookups to route each query to the correct partition. This results in measurable overhead for high-throughput applications, especially if you decide to use multi-level partitioning schemes where PostgreSQL must traverse deeper catalog structures to identify the target partition. Let’s take a look at some findings on speeding up the part where you already know the partition key values.https://www.shayon.dev/post/2025/172/is-agi-paradoxical/Sat, 21 Jun 2025 22:30:00 -0400https://www.shayon.dev/post/2025/172/is-agi-paradoxical/A developer types implement user authentication and watches as Cursor generates 50 lines of secure, production-ready code in seconds. It’s remarkable—the AI understands context, follows best practices, even adds appropriate error handling. But here’s what’s fascinating: every pattern it used was learned from millions of human-written codebases. The AI didn’t invent authentication; it synthesized decades of human security knowledge at unprecedented speed. This same pattern plays out across today’s most impressive AI systems as we pursue Artificial General Intelligence.https://www.shayon.dev/post/2025/164/pitfalls-of-premature-closure-with-llm-assisted-coding/Fri, 13 Jun 2025 05:46:48 -0400https://www.shayon.dev/post/2025/164/pitfalls-of-premature-closure-with-llm-assisted-coding/A 51-year-old man walked into the emergency room with chest pain. The symptoms seemed clear enough: elevated blood pressure, chest discomfort, some cardiac irregularities. The emergency physician, attending doctor, and cardiologist all converged on the same diagnosis—acute coronary syndrome or accelerated hypertension. The classic signs of anything more serious simply weren’t there. But one hospitalist wasn’t satisfied. Despite multiple colleagues ruling out aortic dissection as unlikely, something felt incomplete. The pieces fit the common diagnosis, but not perfectly.https://www.shayon.dev/post/2025/124/another-look-into-postgresql-cte-materialization-and-non-idempotent-subqueries/Sun, 04 May 2025 08:00:38 -0400https://www.shayon.dev/post/2025/124/another-look-into-postgresql-cte-materialization-and-non-idempotent-subqueries/A few days ago, I wrote about a surprising planner behavior with CTEs, DELETE, and LIMIT in PostgreSQL, a piece I hastily put together on a bus ride. That post clearly only scratched the surface of a deeper issue that I’ve since spent way too many hours exploring. So here are some more formed thoughts and findings. The core issue revisited Let’s quickly recap: when using a query like the following, the planner might execute your query in ways you don’t expect.https://www.shayon.dev/post/2025/119/a-postgresql-planner-gotcha-with-ctes-delete-and-limit/Tue, 29 Apr 2025 08:00:38 -0400https://www.shayon.dev/post/2025/119/a-postgresql-planner-gotcha-with-ctes-delete-and-limit/I recently discovered an unexpected behavior in PostgreSQL involving a pattern of using a Common Table Expression (CTE) with DELETE ... RETURNING and LIMIT to process a batch of items from a queue-like table. What seemed straightforward turned out to have a surprising interaction with the query planner. The scenario Let’s say you have a task_queue table and want to pull exactly one task for a specific queue_group_id. A common approach is using a CTE:https://www.shayon.dev/post/2025/75/selective-asynchronous-commits-in-postgresql-balancing-durability-and-performance/Sun, 16 Mar 2025 11:24:26 -0400https://www.shayon.dev/post/2025/75/selective-asynchronous-commits-in-postgresql-balancing-durability-and-performance/I was recently looking into some workloads that generate a lot of I/O and CPU contention on some very high-write code paths and came across synchronous_commit (https://www.postgresql.org/docs/current/wal-async-commit.html). It can be very tempting to turn this off globally because the performance gains in terms of I/O, CPU, and TPS (transactions per second) are very hard to overlook. I noticed I/O completely gone, CPU down 20% (at peak), and a 30% increase in TPS.https://www.shayon.dev/post/2025/53/challenging-ai-generated-code-from-first-principles/Sat, 22 Feb 2025 16:50:20 -0500https://www.shayon.dev/post/2025/53/challenging-ai-generated-code-from-first-principles/Prototyping new features and fixing bugs has become so much faster now that we have coding copilots like Cursor and LLMs (Large Language Models). We can generate boilerplate code in no time, saving time that would have gone into repetitive tasks. AI suggestions often feel like magic—type a quick prompt, accept a snippet, and plug it in. However, this speed comes with a downside: it’s too easy to bypass deep thinking.https://www.shayon.dev/post/2025/40/scaling-with-postgresql-without-boiling-the-ocean/Sun, 09 Feb 2025 08:59:32 -0500https://www.shayon.dev/post/2025/40/scaling-with-postgresql-without-boiling-the-ocean/“Postgres was great when we started but now that our service is being used heavily we are running into a lot of ‘weird’ issues” This sentiment is frequently echoed by CTOs and senior engineers at high-growth startups when I speak with them. Scaling PostgreSQL successfully doesn’t always require a full team of DBAs and experts. The beauty of PostgreSQL is that solutions often lie within the database itself - by rethinking your data access patterns from first principles, you can solve many business problems at scale.https://www.shayon.dev/post/2024/365/database-mocks-are-just-not-worth-it/Mon, 30 Dec 2024 14:50:09 -0500https://www.shayon.dev/post/2024/365/database-mocks-are-just-not-worth-it/It’s tempting to rely on mocks for database calls. Mocking is faster and often feels more straightforward. However, testing against a real database uncovers hidden pitfalls that can appear as the application matures. Issues like unique constraint violations, default value handling, or even performance bottlenecks may only surface when the code is exercised against actual data. The importance of real database testing Consider a simple example where you create a user in your application.https://www.shayon.dev/post/2024/303/using-ctid-based-pagination-for-data-cleanups-in-postgresql/Tue, 29 Oct 2024 07:44:24 -0400https://www.shayon.dev/post/2024/303/using-ctid-based-pagination-for-data-cleanups-in-postgresql/When dealing with very large PostgreSQL tables (we’re talking 15TB+), sometimes routine maintenance like archiving very old data can become surprisingly challenging. Despite having good indexes. I recently faced this issue when trying to clean up very old data on a very large and legacy table. The Problem Initial approach used standard ID-based pagination. Imagine a query like this: DELETE FROM large_table WHERE id IN ( SELECT id FROM large_table WHERE created_at < '2023-04-01' ORDER BY id LIMIT 10000 ); While totally harmless on the surface, this kept timing out (10 min statement_timeout 😬).https://www.shayon.dev/post/2024/244/pg_easy_replicate-supports-schema-change-tracking-during-logical-replication/Sat, 31 Aug 2024 12:19:30 -0400https://www.shayon.dev/post/2024/244/pg_easy_replicate-supports-schema-change-tracking-during-logical-replication/I have been meaning to support common DDLs (Data Definition Language) for pg_easy_replicate for quite some time now and I am super stoked that it is now finally out. This new capability addresses one of the limitations of PostgreSQL’s native logical replication, bringing more flexibility to database migrations and replication through pg_easy_replicate. What is pg_easy_replicate? For those new to the project, pg_easy_replicate is a CLI orchestrator tool that simplifies the process of setting up and managing logical replication between PostgreSQL databases.https://www.shayon.dev/post/2024/225/stop-relying-on-if-not-exists-for-concurrent-index-creation-in-postgresql/Mon, 12 Aug 2024 13:43:48 -0400https://www.shayon.dev/post/2024/225/stop-relying-on-if-not-exists-for-concurrent-index-creation-in-postgresql/As a developer, you might have encountered situations where creating an index in PostgreSQL fails due to lock timeouts. In such scenarios, it’s tempting to use the IF NOT EXISTS as a quick fix and move on. However, this approach can lead to subtle and hard-to-debug issues in production environments. Let’s understand how PostgreSQL handles concurrent index creation When we initiate CREATE INDEX CONCURRENTLY, PostgreSQL first creates an entry for the index in the system catalogs (specifically in pg_index) and marks it as invalid.https://www.shayon.dev/post/2024/203/the-tech-industrys-moral-vacuum/Sun, 21 Jul 2024 15:15:41 -0400https://www.shayon.dev/post/2024/203/the-tech-industrys-moral-vacuum/The New York Times recently reported on how a group of tech elites helped J.D. Vance leap into power. This story isn’t just about one candidate; it’s a symptom of a broader shift in the tech industry’s ethos. These influential figures are pouring millions into supporting candidates and a party that, quite frankly, hold regressive values that clash violently with the image the tech world once projected. Let’s be clear: the Republican party these billionaires are backing has consistently opposed women’s reproductive rights.https://www.shayon.dev/post/2024/195/use-pg_easy_replicate-for-setting-up-logical-replication-and-switchover-in-postgresql/Sat, 13 Jul 2024 09:11:27 -0400https://www.shayon.dev/post/2024/195/use-pg_easy_replicate-for-setting-up-logical-replication-and-switchover-in-postgresql/Logical replication is a powerful feature in PostgreSQL that allows for real-time data replication between databases. It can be used for performing major version upgrades using a blue/green setup where you have two databases, allowing you to test and switch over to a new version with minimal downtime. Logical replication can also be use to facilitate database migrations between different environments, using the same technique and tooling. In this post, I will describe the process of setting up simple replication and switchover between two databases using pg_easy_replicate.https://www.shayon.dev/post/2024/148/fast-simple-and-metered-concurrency-in-ruby-with-concurrentsemaphore/Mon, 27 May 2024 08:00:38 -0400https://www.shayon.dev/post/2024/148/fast-simple-and-metered-concurrency-in-ruby-with-concurrentsemaphore/Let’s say you need to fetch a lot of data from an upstream API, then you want to manipulate that data, maybe even enrich it, and then send it downstream to a database or another API. You aim for high concurrency in fetching data (since it’s allowed), but you need to be cautious when sending the enriched data to the downstream API due to system limits. You can’t send events downstream at the same rate as you fetch them upstream.https://www.shayon.dev/post/2024/104/the-value-of-sitting-on-an-idea/Sat, 13 Apr 2024 09:37:51 -0400https://www.shayon.dev/post/2024/104/the-value-of-sitting-on-an-idea/Have you ever had a brilliant idea that you wanted to act on immediately? I think its safe to say we’ve all been there, and it’s tempting to jump right in. But what if I told you there’s immense value in simply sitting on an idea? Let’s explore this concept. Sometimes, the excitement of a new idea can cloud our judgment. It feels like the idea is burning a hole in our pocket, urging us to act.https://www.shayon.dev/post/2024/89/incidents-and-the-requirement-of-slowing-down/Fri, 29 Mar 2024 10:31:16 -0400https://www.shayon.dev/post/2024/89/incidents-and-the-requirement-of-slowing-down/Speed is key, except when its not. This adage, while seemingly prudent, overlooks a fundamental truth of complex systems: the paradox that slowing down can often be the fastest route to resolution. This contemplative approach to incident management is not merely a tactical choice but a philosophical stance on navigating the intricacies of system failures. It’s an acknowledgment that incidents have a tendency to breed more incidents. Imagine a scenario in which a production cluster experiences a sudden blip, leading to saturated connections and dropped requests for critical workflows.https://www.shayon.dev/post/2024/69/embracing-the-weeds/Sat, 09 Mar 2024 18:51:20 +0530https://www.shayon.dev/post/2024/69/embracing-the-weeds/We often hear the mantra “move fast and break things,” but what if I told you that “moving fast” also requires diving deep — getting into the weeds of every discussion, every idea, every potential innovation as much as possible. Yes, even when it seems counterintuitive. The Art of Specifics High-level conversations are comfortable. They’re the cruising altitude of discussions where the air is clear, and the view is broad. But it’s also where details get lost in the clouds.https://www.shayon.dev/post/2024/55/100x-faster-query-in-aurora-postgres-with-a-lower-random_page_cost/Sat, 24 Feb 2024 12:13:34 +0530https://www.shayon.dev/post/2024/55/100x-faster-query-in-aurora-postgres-with-a-lower-random_page_cost/Recently I have been working with some queries in Postgres where I noticed either it has decided not to use an index and perform a sequential scan, or it decided to use an alternative index over a composite partial index. This was quite puzzling, especially when you know there are indexes in the system that can perform these queries faster. So what gives? After some research, I stumbled upon random_page_cost (ref).https://www.shayon.dev/post/2024/7/shipping-fast-requires-a-high-degree-of-trust/Sun, 07 Jan 2024 10:56:48 -0500https://www.shayon.dev/post/2024/7/shipping-fast-requires-a-high-degree-of-trust/In the fast-moving world of startups, shipping code is a crucial competitive advantage. But rapid deployment goes beyond just having the latest tools or the most comprehensive test suites; fundamentally, it’s also about trust. Trust within your team is transformative. It means believing that each member will effectively handle their responsibilities, understanding that collective problem-solving is more powerful when issues arise, and recognizing that fast doesn’t equate to reckless. This trust is the foundation upon which successful, agile startups are built.https://www.shayon.dev/post/2023/363/introducing-pg_easy_replicate-2.0/Fri, 29 Dec 2023 11:53:10 -0500https://www.shayon.dev/post/2023/363/introducing-pg_easy_replicate-2.0/It’s exciting to share that pg_easy_replicate, a project I’ve been maintaining, has hit a new milestone with its 2.0 release! This tool is more than a practical solution for setting up logical replication between PostgreSQL databases—it’s been a profound source of learning about PostgreSQL concepts for me. Engaging with the community and discovering the ways they use pg_easy_replicate has been equally satisfying, making this journey an enjoyable and insightful experience.https://www.shayon.dev/post/2023/355/do-you-really-need-foreign-keys/Thu, 21 Dec 2023 09:25:15 -0500https://www.shayon.dev/post/2023/355/do-you-really-need-foreign-keys/Before we dive in, let’s set the stage with a few pointers: YMMV. There’s no one-size-fits-all. My insights are primarily drawn from my experience of handling large-scale, high-transaction Ruby on Rails / Go applications backed by PostgreSQL. I am not here to tell you Foreign Keys are bad. They are great. This post isn’t about that. If there’s one takeaway I’d love for you to have from this post, it’s this: Revisit your database schemas with a critical eye.https://www.shayon.dev/post/2022/47/pg-osc-zero-downtime-schema-changes-in-postgresql/Wed, 16 Feb 2022 11:50:49 -0500https://www.shayon.dev/post/2022/47/pg-osc-zero-downtime-schema-changes-in-postgresql/Schema changes are usually critical operations to perform on a high volume database. One thing off, and you are looking at an outage. PostgreSQL has a lot of nice alternatives to make these schema changes safe. However, depending on the kind of schema migration, you would need to know exactly what the alternatives are and perform it exactly in the prescribed way. While you can build some automation around it to make them easy, it still induces cognitive load on a Product Engineer.https://www.shayon.dev/post/2022/17/why-i-enjoy-postgresql-infrastructure-engineers-perspective/Mon, 17 Jan 2022 14:33:20 -0500https://www.shayon.dev/post/2022/17/why-i-enjoy-postgresql-infrastructure-engineers-perspective/While I enjoy working with MySQL (first database) both as a product and infrastructure engineer, lately, I have come to appreciate PostgreSQL as well. I recently read this post on why infrastructure engineers prefer MySQL, and I think it’s spot on. FWIW: This is not a MySQL vs PostgreSQL post. This is just a small summary of what I have come to appreciate about PostgreSQL as an Infrastructure Engineer. Schema Changes Throw a stone in the pond, and you shall have Infrastructure Engineers telling tales of how schema changes brought down production.https://www.shayon.dev/publications/Sun, 06 Dec 2020 00:00:00 +0000https://www.shayon.dev/publications/ Scaling Postgres Without Boiling the Ocean — Postgres Meetup for All (2026) PGConf NYC 2024 - Using Logical Replication for Major Version Upgrades and Tenant Migrations RailsConf 2023 (Video): Beyond CRUD: the PostgreSQL techniques your Rails app is missing 97 Things Every Cloud Engineer Should Know - O’Reilly Media Authored Chapter: Handling Network Failures in the Cloud Design Patterns for Cloud Native Applications - O’Reilly Media Technical Reviewerhttps://www.shayon.dev/post/2020/130/handling-network-failures-in-the-cloud/Sat, 09 May 2020 13:49:53 -0700https://www.shayon.dev/post/2020/130/handling-network-failures-in-the-cloud/Originally wrote this as a part of a submission (Dec 2019) for an upcoming O’Reilly Book in 2021. Stay tuned.https://www.shayon.dev/about/Sat, 25 Apr 2020 00:00:00 +0000https://www.shayon.dev/about/I am a Software Engineer with Product and Infrastructure engineering experience. I enjoy building and scaling software led infrastructure. Some areas that are of interest to me (in no specific order or category): Distributed Systems, Site Reliability, Operations, Incident Management, Databases, Neural Networks, Running, Gulab Jamun and Biryani. shayonj on twitter, github, linkedin, and gmailhttps://www.shayon.dev/incident/Sat, 25 Apr 2020 00:00:00 +0000https://www.shayon.dev/incident/rollback scale down / up turn off feature flags keep looking a deploy went out, but unlikely related rollback turned on a feature flag few hours ago, not likely it turn it off error rate isn’t high, not the last deploy rollback none of the code paths touched seem suspicious rollback scaled up with a config change, its harmless scale down, rollback hm, what if its something else roll back and keep lookinghttps://www.shayon.dev/reading/Sat, 25 Apr 2020 00:00:00 +0000https://www.shayon.dev/reading/books current Einstein in Bohemia backlog Designing Data-Intensive Applications (Refresher) Code: The Hidden Language of Computer Hardware and Software The Manager’s Path Napoleon: A Life Sapiens: A Brief History of Humankind The Systems Bible The Pragmatic Programmer (half-read) Moonwalking with Einstein: The Art and Science of Remembering Everything The Design of Everyday Things previously Born to Run The Practice of Cloud System Administration Rework Release It!: Design and Deploy Production-Ready Software UNIX and Linux System Administration Handbook, 4th Edition The DevOps Handbook (Refresher) Site Reliability Engineering TBD… papers / tech journals / posts current Raft: In Search of an Understandable Consensus Algorithm backlog S3(AWS Storage Services) Whitepaper-v9 Paxos Made Simple Meaningful Availability Autopilot: workload autoscaling at Google Building for reliability at HelloSign Firecracker: Lightweight Virtualization for Serverless Applications The 2019 Accelerate State of DevOps The QUIC Transport Protocol: Design and Internet-Scale Deployment QUIC, a multiplexed stream transport over UDP TBD… previously Dynamo: Amazon’s Highly Available Key-value Store Cloud Architectures White Paper Jul 16, 2008 Deterministic Aperture: A distributed, load balancing algorithm TBD…https://www.shayon.dev/post/2017/363/fetch-current-signal-handlers-without-overriding-in-ruby/Fri, 29 Dec 2017 14:36:51 -0800https://www.shayon.dev/post/2017/363/fetch-current-signal-handlers-without-overriding-in-ruby/Lately I have been very interested about Signal handling. So, this holiday season I embarked on a journey of writing a small project. One thing I noticed when inspecting signals is that there isn’t really an easy way of doing so. Or, especially, getting the “current” action handler for a given Signal. For instance, whether a signal (TERM, INT, etc) has an action of SIG_DFL or SIG_IGN or a pointer reference.