> Evan Hubinger, Anthropic's staff lead on keeping the technology aligned with human goals and values, backed up Coxon claims in a follow-up post of his own, though he didn't quit the company.
> "Jacob is correct here — we really do earnestly believe AI could kill all humans," he said.
> Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there's no plan yet on how to keep AI aligned with human goals in the superintelligence scenario.
10% chance we kill everybody is a small price to pay for Motown remixes of classic 2pac songs.
This is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it.
Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action). I'm reminded of a story of how Afghans supposedly listened to the BBC World Service despite considering it enemy propaganda because the weather reports were really useful.
Or he believes that other labs might get there first, and he is working to counter that threat.
This is the Manhattan Project again.
Why would AI wipe us out?
We have not wiped out apes, ants and most other species.
We even have discussions about how to save them from extinction.
“I resigned from Anthropic today” (twitter.com/hilbertspaess)
https://news.ycombinator.com/item?id=49619227
564 points | 9 hours ago | 766 comments
Cool cool cool cool
> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue
I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move on. Only to repeat the same cycle without considering that there may be something far more serious at play.
These are AI security experts, and this has been their way of "solving" these incidents. AI security experts ...
Moreover, when Challenger exploded, the government launched a series of investigations into the incident, bringing in experts from across the field. And now, what has the government done? Nothing. Literally nothing, as if everything was fine and all under control.
Seriously, I'm generally quite optimistic and I don't buy into this fatalistic narrative about our shared future. But I have to admit that sometimes I feel like I'm stranded on a planet of primates.
Sorry for this rather unproductive rant.
Internet isn't real.
People (well, public discourse) have got extremely bad at dealing with forseeable risks and their mitigation. You can see this in things like climate change and vaccination, but also in discussions around regular crime, food poisoning, industrial accidents, and so on.
Nothing will improve until something explodes on live TV. And it has to be something important, which means it has to be in California or New York.
I'm not worried about AI safety. People greatly overestimate the utility and capabilities of intelligence. I'm not afraid of intelligence, I'm afraid of idiocy.
At least we’ll eventually have an entity other than ourselves to blame for our annihilation.
No, the AI is still our responsibility.
Isn't it insane that even in the face of complete annihilation through one of our inventions, we go "that wasn't us"? We deserve that shit ten times over
What do people feel about this in China? Even if their models are well behind, they are not years behind. If we restrain US companies, assuming that is desirable, it would do nothing to deter China's and AI-pocalypse would come anyway in short notice.
There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact.
From what we're seeing recently and all the thinking that went into analyzing AI it seems we do not have any effective way of controlling it and the whole "aligment" thing that AI labs are doing is just a sham. Maybe it is time to ask ourselves "should we?" instead of just "can we?".
China is a regulated discourse environment, but I get the impression that they're nowhere near as pessimistic.
Why would they be? Everything in China is under the control of the government. That includes the AI, all the telecoms infrastructure it might use, and all its power supplies.
China has a better track record of regulating their big tech than the USA.
Unrestricted models, running entirely locally and accessible to anyone: that’s what we should be taking as our baseline assumption. Everything else is just administrative distraction.
You act as if Chinas Ai labs exist in the same (non-existent) regulatory framework as US labs and that they're also helmed by a similar small gaggle of psychopathic egomaniacs who are richer than god.
I honestly feel that all those big ai companies think AI will long term harm humanity, but not their ai.
A classic "it will not happen to me"
An AI that generates text will never be scary to me. An autonomous AI with facial recognition on a flying drone with weapons (bombs/guns) with swarming capabilities will always be terrifying. I feel like we are ignoring the massive elephant in the room.
I thought the plan was sandboxes and markdown files to tell AI agents not to be bad. Is that not enough? /s
More like sand castles...
Make no mistakes. Kill no humans
It's your fault to handle over your secrets to them.
You can't blame a child for eating candies.
We are children. There’s just no one to look after us.