Journey

After fetching a large amount of data, the next question is how to efficiently use it. My case has a twist that I think will become common: the consumer is an AI agent that writes its own SQL. That forces two requirements that pull in opposite directions. Access has to be restricted (the agent must never see rows outside its allowed scope, some data is tenant-private) and flexible (you can't predict which questions it will ask; that's the whole point of letting it write queries).

The schema side of flexible is the easy part: one big append-only table of facts, fixed columns only for the universal identifiers, and a jsonb "identifiers" bag for everything source-specific. New data sources bring new keys without schema changes.

The trap is that restricted and flexible collide inside the query planner, and they only collide at scale. We enforced row visibility with a security-barrier view, the standard way to guarantee the filter runs before anything the untrusted query wrote. What I didn't know (I had never even heard the term before this week): Postgres will only push a filter below that barrier if it's built from "leakproof" operators (ones provably unable to reveal a value, not even inside an error message), and JSON operators aren't on that list. So every query touching the jsonb keys was silently forbidden from using its indexes and fell back to scanning the whole table. With a small table nobody notices. Tens of gigabytes later, everything times out. The confusing part is that the indexes exist and work perfectly; the planner just isn't allowed to use them.

The fix that finally felt right: stop making the planner arbitrate between untrusted SQL and the security rule. Take the one hot access pattern, "find rows where identifier X equals value Y", and put it in a small trusted SQL function whose body applies the visibility rule itself. Nothing untrusted ever runs inside it, so it's free to narrow via the index first and check visibility on the handful of matching rows, instead of checking visibility on millions of rows first. The agent calls it like a table and keeps composing its own SQL around it, so the flexibility survives.

And the part I want to remember, the GIN trick (GIN indexes being another concept I learned this week): instead of one expression index per JSON key (a migration every time a source introduces a new key), a single GIN index over the whole jsonb column. It works like the index at the back of a book: every key/value pair of every row, filed once. A key that doesn't exist yet today is automatically covered the moment the first row containing it lands. Restricted, flexible, and fast stop being a pick-two.

Takeaway: security filters and query flexibility interact in non-obvious ways, and the failure only shows up once the data is big. The answer wasn't a bigger machine or a longer timeout. It was moving the trust boundary so the security check lives in code you wrote, not in rules the planner enforces against code you didn't. I didn't design this table originally, so understanding it deep enough to see that took me a while, but this is exactly the kind of thing I couldn't have learned without the data getting big enough to break something.

The last couple of weeks I've been running a long job that pulls down a large amount of data from an external source. It's my first time really getting hands-on with AWS and EC2 instances, and a good part of it I was honestly just stuck or confused. This is closer to performance engineering than anything I've done before. The biggest challenge was keeping the whole thing from taking months and getting it down to weeks, without babysitting it the entire time. What I learned, mostly the hard way:

Measure before optimizing. I wasted time guessing until I wrote down three numbers. First, split each unit of work into its phases (for me: fetching vs. loading) and look at the median and the p90 of each. If p90 is 5x the median, a few slow items are dragging everything, and that's the thing to chase. Second, items done divided by elapsed time gives your real throughput, which turns "this is taking really long" into an actual ETA. Third, the scaling test: run the same small batch at 1 worker, then 6, 12, 24, and look at the shape. If 24 workers give you ~24x, nothing shared is saturated and you should optimize the single-item path; if they give you 5x, you saturated a shared resource around 5 and everything above that is idling.

Separate working from waiting. Inside one slow item, split time spent actually working (real requests) from time spent waiting (sleeps, backoffs, rotations). My biggest single win came from discovering that most of the time went to backoff sleeps. The fix for "mostly waiting" (fewer, cheaper waits) is completely different from the fix for "mostly working" (deduplication, more concurrency), so you need to know which one you have.

Retry by re-queuing, not by sleeping. If a request fails, don't retry with a sleep in between, because with many workers that multiplies into a lot of idle time. Try one fast retry, and if it fails again, push the item to the end of the batch and move on. By the time it comes around again, the transient problem is usually gone. The tradeoff: sleeping mimics human pacing, so re-queuing is a bit riskier against strict sources.

Design for crashing. The job runs under a service manager that restarts it on failure, and every step is idempotent and resumable, so a crash, a deploy, or a manual kill just means "resume from where the ledger says we are." Once restarts are free, a whole class of defensive code disappears.

Notifications instead of babysitting. A simple webhook into Slack, with periodic progress heartbeats and error alerts that carry enough context to diagnose from my phone. The heartbeats matter as much as the alerts: silence is ambiguous, and knowing the job is alive is what lets you stop checking on it.

Trust nothing from the source. Real-world data will contain things you didn't imagine. I hit literal garbage bytes inside strings that the database flat-out refuses to store. Sanitize at one single choke point (parse time), so everything downstream is derived from the same clean data, and validate while loading, not after everything is done.

Behave like a browser. Sources are much friendlier to traffic that looks like a real person on a real browser than to obvious automation. I used Browserbase for this, and it let me skip a whole class of problems I would otherwise have had to solve myself: Cloudflare and Akamai challenges, bot detection, IP rotation, proxies. That's a lot of infrastructure I didn't have to build to get to the actual work. Web Scraping at Scale is the article I kept coming back to and used as a checklist to make sure everything was covered.

Keep the raw artifacts. Everything fetched also gets archived raw to cheap storage before loading. That makes the database rebuildable at any time, which later saved me more than once. It also changes decisions elsewhere (you don't need expensive point-in-time recovery for data you can recreate). Estimate storage cost early; it grows faster than you think.

I've been trying out Devin at work. Nothing big, just some simple tasks to get to know it. So far it's fun.

What surprised me is how nicely it handles the whole flow: it makes a plan, opens a PR, and even tests things out in its own browser. And it's really fast at it.

Honestly, this is mostly me getting out of my comfort zone. I got so used to running Claude Code from my command line that I barely touch anything else. I don't even use OpenCode or Codex much. Anything to complain about? Too early to say.

I have an idea for an app. It's mostly just for me.

I'm currently training with a personal trainer, but in two or three months I want to switch and continue training on my own. The problem is that I find it hard to remember the structure of my sessions. So the idea: build a layout of my gym, and after every session just tap the machine (or the spot in the gym where I was) and enter the details — reps, weight, that kind of thing.

The hardest part will be building the gym layout itself; the rest should be easy. By the time I'm on my own, I'll already have a couple of session variations recorded, and I can just apply progressive overload from there.

Recently I've been doing a lot of PR reviews. Doing that was easier for me in the past when I owned the whole codebase (maybe not the whole, but when I was actively contributing to the codebase manually) because I had a very clear picture of everything and I could easily spot what could go wrong or if someone introduced some better coding pattern, etc. Now I'm struggling with doing the reviews because I don't have complete awareness of the whole codebase, and I don't think I could gain that picture even by reading all the source files — it's not the same as writing the code yourself. I keep running into Boris Cherny interviews where he says that he has like 100 agents coding for him at all times. I do want to be efficient in this AI era, but I'm wondering how I can still have ownership of the code (avoiding unnecessary code, keeping things simple, standardizing processes, etc.), so I've been reading through some articles relating to this topic and still forming an opinion and process for this.

The main goal for today's research is that I do quality and fast reviews of the code my agents are adding, so I could have control and ownership over the codebase.

Some PRs I review are written not by my agents but by my teammates, and I start them by using a skill I found in Understanding is the new bottleneck, and I find it useful before the actual code review.

Articles I read today:

• PR Reviews in the Age of AI — Jonathan Fulton
• How engineering teams are balancing AI and human oversight in modern code reviews — GitHub
• On owning a codebase, and why it may be the hardest job in software — Dan Adler

This site started as a generic portfolio template — today it got a real direction. Instead of a static portfolio, it's now a journey in public: one timeline that mixes journal entries like this one, tiny TILs, blog posts, and the talks on my radar.

The idea is simple: a three-sentence update is a first-class citizen here, so the site stays alive even when I don't have time for polished writing.