I was one of the 16 devs in this study. I wanted to speak on my opinions about the causes and mitigation strategies for dev slowdown. I'll say as a "why listen to you?" hook that I experienced a -38% AI-speedup on my assigned issues. I think transparency helps the community.

Firstly, I think AI speedup is very weakly correlated to anyone's ability as a dev. All the devs in this study are very good. I think it has more to do with falling into failure modes, both in the LLM's ability and the human's workflow. I work with a ton of amazing pretraining devs, and I think people face many of the same problems. We like to say that LLMs are tools, but treat them more like a magic bullet. Literally any dev can attest to the satisfaction from finally debugging a thorny issue. LLMs are a big dopamine shortcut button that may one-shot your problem. Do you keep pressing the button that has a 1% chance of fixing everything? It's a lot more enjoyable than the grueling alternative, at least to me.

I think cases of LLM-overuse can happen because it's easy to optimize for perceived enjoyment rather than time-to-solution while working. Me pressing tab in cursor for 5 hours instead of debugging for 1:

Third, it's super easy to get distracted in the downtime while LLMs are generating. The social media attention economy is brutal, and I think people spend 30 mins scrolling while "waiting" for their 30-second generation. All I can say on this one is that we should know our own pitfalls and try to fill this LLM-generation time productively: - If the task requires high-focus, spend this time either working on a subtask or thinking about followup questions. Even if the model one-shots your question, what else don't I understand? - If the task requires low-focus, do another small task in the meantime (respond to email/slack, read or edit another paragraph, etc). As always, small digital hygiene steps help with this (website blockers, phone on dnd, etc). Sorry to be a grampy, but it works for me :)

Some final statements: - METR is a wonderful organization to work with, and they are strong scientists. I've loved both participating in this study and reading their results. - I am not some LLM guru trying to preach. Think of this as me publishing a personal diary entry, and hoping that others can benefit from my introspection.