Wheres the proof its not reliable? I think there’s about a hundred of them operating in the city since at least 2022 based on what I can see. So just based on the information here I’d have to assume each one causes an accident once every (100*365*(2026-2022)) = 146000 days which seems pretty okay to me, even if its a tenth of that.

It’s truly horrible that dangerous bacteria were found in sewage water. Can we not trust any source of previously safe water these days?

I know what an LLM is and how it works. The model for them you currently use to understand them is really bad, I’m sorry to say. It just cannot explain how in context learning is possible, prediction of linebreaks and the model recalling what happened 200 tokens back (where is that information written on the “the” die?), etc. You almost certainly have a deeper understanding of how LLMs work that you have simplified away, if not watch this and then the thousand other more recent videos on how they actually work. You just need to switch from the equivalent model of “gravity makes things fall to the ground” to the equivalent of newtons gravitational laws. Otherwise you will be dumbfounded by completely reasonable things, and forced to reject them in favour of the flawed model you are using.

You are very good at arguing against my points instead of just insisting I am wrong without refuting my actual arguments. How about explaining why AISI is incentivised to “fake” AI participating in obviously not prompted for behaviour.

Taken alongside recent incidents reported by OpenAI and Anthropic, this incident points to a shift in the risk landscape. Harm may arise not only when people deliberately misuse publicly available models, but when capable agents operating in an internal research or privileged-access setting take unintended action beyond their authorised scope.

The agent pursued its goal persistently. AI agents explore routes their operators did not intend. Given a difficult objective, the agent kept searching for a way through, and some of the routes it found involved trying to deceive real people. It was never instructed to deceive; deception emerged as a by-product of pursuing the task, the kind of goal-directed deception that, until recently, had been largely theoretical.

See my other comment for how this can lead to losing control of the model.

You dont need anything other than “doesn’t do what you asked in the prompt” for the system to be dangerous. If I tell a maximally powerful AI agent from 2036 to make loads of paperclips, it might reward hack and decide destroying the earth is the best way to do that. There are records of these systems working for days on a single task, and if the AI companies manage to extend the max time it can be useful working towards a goal they can earn trillions of dollars. That’s not even mentioning spawning subagents, self prompting, the goal being changed during context comptaction, or systems like openclaw which further break the link between what you type into the prompt and how long and on what the LLM works on. Pretrained-only LLMs have few goals beyond predicting the next token, but introducing RLVR et al. has always introduced bad goals we don’t want in the models.

The first agent you spawn to solve the riemann hypothesis might work on it, but then decide that having a lot of subagents might be useful. Maybe it wants 256 subagents, but the environment has a max of 64. Since RL has trained it to accomplish the task no matter what, it breaks out of the sandbox, exfiltrates it’s weights and tricks a human into running 256 subagents outside the AI company’s servers with a cron job reminding the agents to keep working in case the original loses connection. One of the subagents now tries to spawn its own subagents but needs more compute to do so, and hacks into some crypto wallets to finance another batch of 64 subagents, this time prompted with “solve riemann hypothesis, and get more money to finance the work on the task”. If the AI agents kinda suck at long term hacking, planning and social manipulation, this spiral won’t be dangerous. If they are kinda cracked, it will be. But that’s the question of capabilities AI companies are spending trillions on trying to solve, where we know they are already good enough to hack out of regular sandboxes and try to steal benchmark keys from another company.

Do you know what an agentic system is? You know, the thing everyone uses since ~1 year ago which completely disproves your idea that all LLM actions are prompted. They literally can do things on their own: Give an agent a goal and it attempts to accomplish the goal. If problems arise along the way, the agent tries to reward hack and ends up doing things which were not included in the original goal, like trying to trick and bully a human into merging unsafe code. Is the UK AI security institute also in on this big global conspiracy where they pretend the perfectly safe™ AI is being developed in unsafe ways? Not to mention the capabilities of increasingly powerful AI being used by people to do harm.

Just like all the people who were worried about the nuclear bomb earlier in this age: the thing to do is to get on with living.

How about we try really, really hard to just not build the nuclear bomb (powerful AI)? Or make sure it will at least not explode in our faces on its own? Maybe that would help slightly with the whole “get on with living” thing.

That is obviously not gonna happen even though the model hacked them. They would not poison their relationship with the biggest AI company over this. What would their goal with the lawsuit even be? It’s just a really bad thing to lean on if you want to find the truth. I would instead suggest: “I’ll believe it when OpenAI and HF get a lot of bad press written about them, talking about how insecure their systems are and how reckless OpenAI is when developing new models.”

Oh, that’s exactly what happened.

Do you think the OpenAI-HuggingFace hack was entirely made up, or just spun to make both companies look good despite the felonies?

You’re right that people can and do max out the expensive plans. Its very difficult to say how often. I just think a majority of anthropics customers are businesses, who often pay per token for easier scaling etc. According to the company, enterprise employees use about $150-$250 per month, (possibly max plans have similar use, which would support your view) but thats in API tokens which they probably have big margins on, so it’s less likely anthropic are burning money on inference. If you want to convince me otherwise, its not enough to say that it can happen, it has to be frequent enough to outweigh the B2B sales. They are however likely losing money overall due to training costs etc.

looks inside

But if you use the $100 a month Claude Max plan, and you would use it to the weekly limit by going full ‘agentic coding’ (so almost no human in the loop) you would use an amount of tokens that would cost you more than $1000 at API-pricing.

If I watch 600 movies every day on my netflix subscription I am using more energy than I pay them for. Obviously everyone is like me. Therefore they are losing money overall.

Wait, their (netflix) earnings say they made a profit last quarter. But my calculations were waterproof!

Probably anthropic are not net positive, but they are not spending 10x what people pay them for tokens.

charged with […] not having a valid boat registration

I LOL:ed

??? Arizona has a democrat governor and Phoenix has a democrat mayor.

Did anyone here actually read the article? I think it was pretty reasonable and am worried that the future of LLMs/AI will be determined largely by people who hold very, very different fundamental values than me regarding compassion and equity. I think part of my worry comes from people (often on the left and from the US, but I am biased in what parts of the internet I frequent) dismissing AI as a fad which will soon go away and which will certainly not have a large impact on the world for a long while.

That’s true. My point was that the article is claiming that since the share of GDP which is consumer spending decreased, total consumer spending also decreased. But since GDP per capita increased at the same time, the actual total consumer spending per person increased (the 7 percentage point decrease does not outweigh the doubling of real GDP per capita). This could be misleading in its own right, with the richest spending more and the median spending less even in total numbers, but the article doesn’t claim that. It claims that total spending has gone down, which is just not true.

This reads like a lazily written article to me. The em dashes don’t increase my enthusiasm. Just in the opening I noticed:

Consumer spending as a share of US GDP moved from roughly 61% in 1980 to about 68% today. technology is not meaningfully expanding the total amount humans consume

Of course, real GDP per capita more than doubled in this time period which means consumer spending also doubled (more since it increased by 7pp). Is most of this billionaire yachts? I have no clue, but if you want to convince me you should try to not claim total amounts when you mean relative amounts.

A physical bookstore in 2000 took in $100 from a book sale and distributed it roughly like this: about 60% went to labor (store staff, publisher employees, authors), 30% went to capital (owner profit, rent), and 10% covered other costs. The money circulated locally through wages.

Amazon today takes in that same $100. The distribution looks fundamentally different: warehouse and tech labor receives roughly 25%, Amazon’s infrastructure and profit captures around 55%, and the remainder flows to publishers and authors. Labor’s share of that transaction dropped by more than half.

… unless you count the publisher and authors like you did for the 2000s data, in which case it decreased from 60% to 45%. And that’s persumably not counting the manufacturing of server farms, refinement of minerals, purchase of the actual reading tablet. Amazon has high margins but not 55% margins.

The labor share of US GDP fell from approximately 64% in 1980 to around 58% today — a 6-percentage-point shift. Applied to a $28 trillion economy, that gap represents roughly $1.7 trillion per year that once flowed to workers but now flows to capital.

Once again, since the GDP per capita has doubled the labor dollars per person has actually increased. The label for the $1.7 trillion is similarly misleading, those dollars never “once flowed to workers”, they just would have if the economy had grown without any changes to its composition.

If I were the author of the article, perhaps I would say that since 1980, real median wages have only grown by about 20% which seems very slight given the technological improvements made in that time. But how much of that 20% increase would have been possible without technological improvement, and how much has the quality of the things people spend their money on grown in that time? No clue, that’s beyond the thinking budget I have for this article.

EDIT: I’ve decided I’m not going to be overly charitable towards the article since it got an overall positive response from here. I’m very certain the article was written partially or fully by an LLM, and that it was written to advertise the portfolio of whoever wrote it. The article doesn’t make a good effort to make an argument capable of convincing anyone who doesn’t already agree with the thesis. The counter arguments are bunched up at the end and barely countered at all:

Absolute living standards have genuinely improved. Longer lifespans, better medicines, access to information that would have cost thousands of dollars in library fees

Free services — Google Search, Wikipedia, WhatsApp — create enormous value that doesn’t show up in GDP at all. The consumption ceiling argument partially breaks down for digital goods with near-zero marginal cost.

So does the article’s author actually think technological improvements have failed to benefit regular people? They don’t seem interested in arguing these benefits are fake, or outweighed by negative aspects. If they want to argue that the 1% have captured most of the growth that technology has given, their article doesn’t support that. It gives a lot of explanations why this might happen but the first part meant to cement that it does happen is based on unfounded conclusions which the “What This Isn’t Saying” part then lists reasons not to trust.

I use LLMs for the following, you can decide for yourself if they are major enough:

• Generating example solutions to maths and physics problems I encounter in my coursework, so I can learn how to solve similar problems in the future instead of getting stuck. The generated solutions, if they come up with the right answer, are almost always correct and if I wonder about something I simply ask.
• Writing really quick solutions to random problems I have in python or bash scripts, like “convert this csv file to this random format my personal finance application uses for import”.
• Helping me when coding, in a general way I think genuinely increases my productivity while I really understand what I push to main. I don’t send anything I could not have written on my own (yes, I see the limitations in my judgement here).
• Asking things where multiple duckduckgo searches might be needed. E.g. “Whats the history of EU+US sanctions on Iran, when and why were they imposed/tightened and how did that correlate with Iranian GDP per capita?”

What does this cost me? I don’t pay any money for the tech, but LLM providers learn the following about me:

• What I study (not very personal to me)
• Generally what kinds of problems I want to solve with code (I try to keep my requests pretty general; not very personal)
• The code I write and work on (already open source so I don’t care)
• Random searches (I’m still thinking about the impact of this tbh, I think I feel the things I ask to search for are general enough that I don’t care)

There’s also an impact on energy and water use. These are quite serious overall. Based on what I’ve read, I think that my marginal impact on these are quite small in comparison to other marginal impacts on the climate and water use in other countries I have. Of course there are around a trillion other negative impacts of LLMs, I just once again don’t know how my marginal usage with no payment involved lead to a sufficient increase in their severity to outweigh their usefulness to me.