I think that Anthropic did something very clever when they gave their product a human name: Claude. It feels somehow natural to say "Claude wrote some code", more so than saying "I used Claude to generate some code". In this example, it isn't false to express this thought by making Claude the subject of the sentence, but doing so makes it very easy to feel that Claude really has agency, and that it is somehow equivalent to a real human worker.
Machines are not equivalent to people. There is however an attempt to act as if they are and, through the language used to speak about them, spread the notion that this equivalence is real. Beyond the name of the product, we speak of Claude and others as being programming "agents". It might feel like this is a fair name to give to the product, because the way it's used is the way that you might deal with an agent, and it's made in such a way as to feel equivalent to interacting with a real person. You send it a message in natural language, it then interprets, acts, and responds. (Surely using a chatbot only feels like interacting with a person because it has become possible in recent decades to do so in the dramatically simplified format of text on a computer screen.)
Agents and machines both have an inner logic which goes beyond the way that they are interacted with. The actions taken and the responses given in either case depend on this inner logic. Imagine you hire an agency to do some work on your house. They send someone out, one of their agents. It's amazing the power that money has, you pay a company, they tell one of their employees what they're doing that day, and the employee does it. Why do they do it? Because they have bills to pay. They do an adequate job, but not as good a job as they would have done if it was their own house. Why not? Because whether they do an adequate or exceptional job, they get paid the same. The agent's actions and responses had their origin in how well aligned they were to the person paying them (if the agent had been working for a close relative, perhaps they'd have done a better job). Agents take decisions on the basis of their priorities, which may change, and their inner logic is totally accounted for as such. To accomplish a goal via an agent is to entice the agent to change their priorities to align with your own, either by means of opportunity (help me and I'll pay you) or threat (do it or you're fired).
LLMs, like other machines, are an artefact of a sophisticated relationship between the maker and the controller of the machine. A controller may be a user, owner, or neither, but in any case, they directly benefit from the machine's being employed. The controller could be the owner and driver of their own car, or they may be someone who leases cars which are driven by their employees. In any case, the relationship between maker and controller is fulfilled by the machine being "employed". Rather than the simple relationship between principal and agent in which work is done for money, the maker provides a machine to the controller which will allow the work to be done by the controller at a lower resource cost than just paying an agent.
Looking at it this way it seems strange to focus attention on the machine itself, which is what most online posts do. Unlike an agent, machines don't care whether or not they are employed. They have no wants, and they cannot change their priorities, which are frozen from the moment they cease to be worked on by their makers. Their inner logic does not concern doing things for their own sake; it is about serving the controller of the machine in such a way that benefits the maker of the machine. The makers of the machine are the ones with the agenda, who commit resources to ensuring that the machine continues to be employed. Cars don't want to be driven, but automakers want people to drive them, etc.
LLMs, like other machines, are animated by the will of their makers and their controllers, when made properly. Anything else is just an accident, which will inevitably occur in new technologies, because the makers of those machines simply lack the experience to fully understand the machine and thus fully impose their will on it. The machine, the physical artefact, has the almost magical property of allowing the maker's will to reach into far away places, and beyond a single instant in time. Take smartphone navigation for example. The phone, running your maps app, directs you from point A to point B, and the whole function is accounted for by the will of the makers of the phone and app, who wanted people to use their product continually, and you, who wanted to reach your destination at that moment. In examples like this we understand the dynamic between makers and controllers of machines. So why is it that there are so many headlines saying things like "AI has surpassed the best mathematicians in the world"? The LLM is a machine, an extremely abstract, complex machine which in most cases consumed an incomprehensibly large amount of resources to develop, but it is still a machine, just doing the things it was made to do, when used. But notice that when "AI" achieves something, there is no mention of the people controlling the AI, probably because that would make the headline less engaging.
Why is it important to try to remain clear-headed and resist the language being used around AI? For me, it is simply a matter of not allowing myself to be deceived. I can only react to developments in this technology's capability and adoption. This technology is too new for there to be anyone who really knows how it will affect how we work and what we produce in the long term, despite the confidence with which it is spoken about. In the absence of anyone credible to defer to, I think it's best to try to think for oneself.
Agents will get the job done reliably without supervision, but have their own agenda, so cannot be fully controlled. Machines require supervision, but have no agenda, so can be fully controlled. Bosses want to see machines and people as equivalent. Perhaps they see LLMs as the perfect solution, an agent without an agenda, a machine which requires no supervision, all upside, no downside. Or perhaps they have simply revealed that they see their workers as machines, who consume more resources than necessary.