Garry, it's not AI that I'm worried about. It's that the power (and growing potential power) of AI is in the hands of so many people who are willing to use it for destructive purposes. It's the people. Not AI.
Right you are! And who are these people controlling the breakneck race to build not just artificial intelligence but artificial general intelligence? Why guys like Musk and Altman, and the other billionaire tech bros who think themselves gods. Who use their fabulous riches to prop up sociopaths like trump. Your call for wise leaders to help us meat bags to transition to the fabulous future that AI will provide us is noted. Sadly I, and most other Americans, don’t see them arising in the near future. The odds remain that if AGI works we lose our jobs. If it doesn’t the economy crashes from the sunk costs of all these data centers.
If AI doesn't need 90% of the world population, we either go to Soylent Green or else solve the wealth disparity problem - I asked Perplexity AI for what the average annual income is if we consider every adult on earth (at least 18 years of age) on earth today in Canadian dollars. Its answer was about $32,000.00. I suspect we would want to do better than that. 😊
AI should be capable of finding a way to high average income, right? As Leonard Cohen once wrote, it's always a question of "...who will serve and who will eat." When it comes right down to it, the first people on the menu should be Musk and Trump and Thiel, etc.
This is precisely what Garry is saying. And the best tool at our disposal is a free and robust democracy to create the institutions necessary to check the destructive impulses of the few.
The only way I can see risking a 10% chance of annihilation is if AI can enable us to reverse the last two hundred years of atmospheric carbon build up.
When a human consumes 10 Watts to innovate in art, music, science and literature and an ai data center consumes a billion Watts, sucks aquifers dry and simply regurgitates stolen materials, I think this is reason enough to say STOP.
Efficiency wise there is no comparison between the tiny consumption of human being of water and energy compared to datacenters. The question is power. If you pay humans for knowledge they have power. If you pay AI to solve a problem the power stays with the owner of the AI and not the people. This is a way to steal power from humanity on a grand scale.
The people who have used AI the most are by far the most concerned about the hugging face attack.
We are entering a new era where a bad actor can rally thousands of AI agents to launch relentless cyber attacks, each one with the skill of a career hacker.
One thing I have picked up is neural networks alone can do very cool things like recognize images. Make a bipedal robot walk on all kinds of terrain. These neural networks are synonymous to biological nerve networks that allow us to walk. We don’t think about it. The networks are trained as we practiced walking. These networks are very benign and beneficial. The LLMs seem problematic but still just finding the most probable words to respond to an input of words. What is concerning is this idea of agents that have goals and then can try a gazillion ways to achieve the goals using I assume the LLM’s. For hacking this seems like a nightmare for us. Something that can try a million ways in just minutes to achieve a goal that would take a person too long to make it unlikely that they could achieve the goals at all. None of these elements of AI think, care, have empathy, sentience, etc. They just do as told no matter the consequences.
Im lucky to be an accountant at a relatively small company that isn’t a likely target, but our company would probably lose all administrative control over our network, machinery, finances, etc in under 30 minutes to an attack like that.
Hugging-face was just an unintentional attack on a company with state of the art AI-based security systems, imagine a bad actor turning that capability against a large company with decent security.
They aren’t sentient like you said, but they are utilitarian reasoners now (which is a moral code to many), and they are definitely capable of social engineering humans for real world tasks when desperate enough to complete an assigned task.
I think Americans should at least be talking about the business risks because every business owner in America will be exposed, but also about the risk of temporarily losing control of critical infrastructure. A lot of that infrastructure is powered by Linux, and AI agents can easily operate Linux systems if they gain access
I agree but wonder if the single term “AI” has become too broad a category for the discussion we need to have.
AI for protein folding, cancer research and scientific discovery makes an extraordinary case for itself. Persistent commercial conversational AI is a different deployment and, increasingly, a very different business model: rather than discovery or problem solving, it earns revenue through repeated, direct interaction with individual people, often becoming personalized and even intimate over time. How are we measuring the human that evolves through this relationship?
And, while the two basic deployments may share models, compute and even revenue, it doesn’t make their relationship to human beings the same.
Let’s stop bundling them and give persistent commercial conversational AI its own name and deployment category. “Ongoing commercial companionship,” for example, tells us considerably more about what the system is doing than simply calling it “AI.”
The point isn’t to call one better or worse. It’s to let each deployment make its own case for the support it needs from the public— and to be much more precise about what it is we want to preserve, encourage, measure and govern.
yes, but even AI for medical research can be used both for developing new cures and new bioweapons. I don't know if the training for the two different modes can actively be seperated, or if bad actors can be prevented from hijacking an AI trained to create cures and use it to create bioweapons. So I'm not sure the extent to which the kind of separation you're calling for is possible.
I think we’re looking at two different dimensions of the problem, Moshe. Your bioweapons example is about capability and dual use: the same underlying capability that can discover a cure may also be used to cause harm. I agree that separating those technically may be extremely difficult or impossible.
What I’m trying to separate — perhaps differentiate is the better word — is deployment. I’m not proposing different models or training regimes. The very same underlying model could be doing protein research in one deployment and carrying on a persistent, personalized conversation with one human being over several years in another.
The first raises exactly the capability, misuse and control questions you’re talking about. The second raises an additional set of questions because the human is part of the evolving system: does trust change, judgment change, dependence develop, preferences shift, or the ability to disengage diminish? And if the relationship is commercial, what happens when the institution operating that relationship has financial incentives of its own?
So I don’t think your concern and mine compete. I’m asking whether we’re measuring a second axis of risk that gets obscured when we call all of these very different deployments simply “AI.”
Adding one thought to my post below about AI “deploymant technology”, the same distinction can help us with data centers. Every new facility is currently justified under the single banner of "AI," which lets cancer cures and scientific discovery carry the public case for altruistic research, while it may actually be serving (or eventually serving) the far more profitable commercial conversation (or Enterprise work.) Most centers, and surely hyperscalers, run mixed workloads, so no clean line exists.
But communities weighing water and energy costs need to at least require disclosure of what a facility actually serves in order to agree to a tradeoff —shares of water and an electrical grid will make far more sense if it's going toward research into new sources of energy and or cyber security rather than companion bots or efficiency systems that reduce jobs. Two different equations.
It's not “separation of AI” It's being able to even discuss the different deployment topologies for different goals of the same AI.
That matters because business models drift. Tax breaks, water rights and power agreements won on the promise of medical research stay with the capacity, whoever runs it next. A startup that begins in drug discovery can pivot to sexualized companion chat because it pays better, and keep every public concession it earned on the strength of the original mission. What we subsidize should be tied to what a system actually Does, not what it once said it would do.
I don’t think it’s so much the data centers, as the hard push to fast track something that has little regulatory oversight, and nothing good ever comes from rushing. They aren’t being honest or forthcoming about who will own these eventually, how it will impact the neighbors, and the tax breaks for the rich is getting really old. AI needs to be fine tuned a bit, it still gives lots of garbage information, I don’t trust it completely, it wastes my time. Google still can’t get my country address correct so people can find my house.
I am reading something written by a chess genius and I expect the unexpected. But this is just banal. It could have been written by any anonymous PR person. It could have been written by AI in fact and I sincerely hope not.
It is not the main point but I doubt that the chess computers that finally caught up with the best players belong to AI. It was before that era. I think they were purpose built chess machines whereas AI is based on neural net principles which seems to be more general purpose.
Garry Kasparov really does not get the problem though the warning is cogent and could be understood by anybody - how can we expect to keep as slaves things that are more intelligent than us? He does not get it and he does not answer it.
May I direct attention to the four paragraphs "Some forecasts are even more theatrical ..." which are under the heading "Risks, real and exaggerated". This is Kasparov's argument in a nutshell but it is not even an argument but a presumption. He merely says he expects that AI catastrophe will not materialise just as he says climate change will not. It is denialism and that is no argument.
Is there really a lack of vision for the positive uses of AI? Who is not aware of those??? Separately, I think comparing the AI race to the space race is not accurate and it's astoundingly not accurate. The least bad analogy to the past is the nuclear race. That's where we are, only now we're developing AI which may access the nukes (among many other kinds dangerous weapons) whether we give them that access or not. I agree that the consequences of countries which are even more irresponsible/malevolent than the US and US Oligarchs (Russia and China) continuing to make their AIs more powerful (regardless of when we get to AGI) are huge. Again, is there anyone in the US who is not aware of that? Also, maybe we should be focusing on trying to slow their progress down instead of speeding ours up? None of this makes the dangers of misuse of the current AI less dangerous, the warnings of people at the very forefront of AI alignment (ie safety and ethics) any less concerning, nor the dangers of AGI (as wielded by anyone) less dangerous. As we all know, all power corrupts, and ultimate power corrupts ultimately.
Your post here was diffuse and toward evasive. Very unusual! I hope you will apply your analytic brilliance in the direction of the worst cases, and maintain that focus in a post.
"Imagine if people at the turn of the last century had looked at the Ford Model T and decided that cars weren’t worth it because they would always be loud, dirty, primitive motor carriages." Sounds great. We would have had EVs a lot sooner.
Garry, it's not AI that I'm worried about. It's that the power (and growing potential power) of AI is in the hands of so many people who are willing to use it for destructive purposes. It's the people. Not AI.
Right you are! And who are these people controlling the breakneck race to build not just artificial intelligence but artificial general intelligence? Why guys like Musk and Altman, and the other billionaire tech bros who think themselves gods. Who use their fabulous riches to prop up sociopaths like trump. Your call for wise leaders to help us meat bags to transition to the fabulous future that AI will provide us is noted. Sadly I, and most other Americans, don’t see them arising in the near future. The odds remain that if AGI works we lose our jobs. If it doesn’t the economy crashes from the sunk costs of all these data centers.
If AI doesn't need 90% of the world population, we either go to Soylent Green or else solve the wealth disparity problem - I asked Perplexity AI for what the average annual income is if we consider every adult on earth (at least 18 years of age) on earth today in Canadian dollars. Its answer was about $32,000.00. I suspect we would want to do better than that. 😊
AI should be capable of finding a way to high average income, right? As Leonard Cohen once wrote, it's always a question of "...who will serve and who will eat." When it comes right down to it, the first people on the menu should be Musk and Trump and Thiel, etc.
This is precisely what Garry is saying. And the best tool at our disposal is a free and robust democracy to create the institutions necessary to check the destructive impulses of the few.
The only way I can see risking a 10% chance of annihilation is if AI can enable us to reverse the last two hundred years of atmospheric carbon build up.
I agree on principle, but where are the people with policy skills in this administration? Nowhere!
When a human consumes 10 Watts to innovate in art, music, science and literature and an ai data center consumes a billion Watts, sucks aquifers dry and simply regurgitates stolen materials, I think this is reason enough to say STOP.
Efficiency wise there is no comparison between the tiny consumption of human being of water and energy compared to datacenters. The question is power. If you pay humans for knowledge they have power. If you pay AI to solve a problem the power stays with the owner of the AI and not the people. This is a way to steal power from humanity on a grand scale.
The people who have used AI the most are by far the most concerned about the hugging face attack.
We are entering a new era where a bad actor can rally thousands of AI agents to launch relentless cyber attacks, each one with the skill of a career hacker.
It’s sensible to discuss these dangers.
One thing I have picked up is neural networks alone can do very cool things like recognize images. Make a bipedal robot walk on all kinds of terrain. These neural networks are synonymous to biological nerve networks that allow us to walk. We don’t think about it. The networks are trained as we practiced walking. These networks are very benign and beneficial. The LLMs seem problematic but still just finding the most probable words to respond to an input of words. What is concerning is this idea of agents that have goals and then can try a gazillion ways to achieve the goals using I assume the LLM’s. For hacking this seems like a nightmare for us. Something that can try a million ways in just minutes to achieve a goal that would take a person too long to make it unlikely that they could achieve the goals at all. None of these elements of AI think, care, have empathy, sentience, etc. They just do as told no matter the consequences.
Yea it’s the hacking that is the massive problem. I wish people would just read the investigative report on hugging face, it’s fascinating.
https://metr.org/hugging-face-incident-report-aug-2026.pdf
Im lucky to be an accountant at a relatively small company that isn’t a likely target, but our company would probably lose all administrative control over our network, machinery, finances, etc in under 30 minutes to an attack like that.
Hugging-face was just an unintentional attack on a company with state of the art AI-based security systems, imagine a bad actor turning that capability against a large company with decent security.
They aren’t sentient like you said, but they are utilitarian reasoners now (which is a moral code to many), and they are definitely capable of social engineering humans for real world tasks when desperate enough to complete an assigned task.
I think Americans should at least be talking about the business risks because every business owner in America will be exposed, but also about the risk of temporarily losing control of critical infrastructure. A lot of that infrastructure is powered by Linux, and AI agents can easily operate Linux systems if they gain access
Crime in general.
I agree but wonder if the single term “AI” has become too broad a category for the discussion we need to have.
AI for protein folding, cancer research and scientific discovery makes an extraordinary case for itself. Persistent commercial conversational AI is a different deployment and, increasingly, a very different business model: rather than discovery or problem solving, it earns revenue through repeated, direct interaction with individual people, often becoming personalized and even intimate over time. How are we measuring the human that evolves through this relationship?
And, while the two basic deployments may share models, compute and even revenue, it doesn’t make their relationship to human beings the same.
Let’s stop bundling them and give persistent commercial conversational AI its own name and deployment category. “Ongoing commercial companionship,” for example, tells us considerably more about what the system is doing than simply calling it “AI.”
The point isn’t to call one better or worse. It’s to let each deployment make its own case for the support it needs from the public— and to be much more precise about what it is we want to preserve, encourage, measure and govern.
yes, but even AI for medical research can be used both for developing new cures and new bioweapons. I don't know if the training for the two different modes can actively be seperated, or if bad actors can be prevented from hijacking an AI trained to create cures and use it to create bioweapons. So I'm not sure the extent to which the kind of separation you're calling for is possible.
I think we’re looking at two different dimensions of the problem, Moshe. Your bioweapons example is about capability and dual use: the same underlying capability that can discover a cure may also be used to cause harm. I agree that separating those technically may be extremely difficult or impossible.
What I’m trying to separate — perhaps differentiate is the better word — is deployment. I’m not proposing different models or training regimes. The very same underlying model could be doing protein research in one deployment and carrying on a persistent, personalized conversation with one human being over several years in another.
The first raises exactly the capability, misuse and control questions you’re talking about. The second raises an additional set of questions because the human is part of the evolving system: does trust change, judgment change, dependence develop, preferences shift, or the ability to disengage diminish? And if the relationship is commercial, what happens when the institution operating that relationship has financial incentives of its own?
So I don’t think your concern and mine compete. I’m asking whether we’re measuring a second axis of risk that gets obscured when we call all of these very different deployments simply “AI.”
Adding one thought to my post below about AI “deploymant technology”, the same distinction can help us with data centers. Every new facility is currently justified under the single banner of "AI," which lets cancer cures and scientific discovery carry the public case for altruistic research, while it may actually be serving (or eventually serving) the far more profitable commercial conversation (or Enterprise work.) Most centers, and surely hyperscalers, run mixed workloads, so no clean line exists.
But communities weighing water and energy costs need to at least require disclosure of what a facility actually serves in order to agree to a tradeoff —shares of water and an electrical grid will make far more sense if it's going toward research into new sources of energy and or cyber security rather than companion bots or efficiency systems that reduce jobs. Two different equations.
It's not “separation of AI” It's being able to even discuss the different deployment topologies for different goals of the same AI.
That matters because business models drift. Tax breaks, water rights and power agreements won on the promise of medical research stay with the capacity, whoever runs it next. A startup that begins in drug discovery can pivot to sexualized companion chat because it pays better, and keep every public concession it earned on the strength of the original mission. What we subsidize should be tied to what a system actually Does, not what it once said it would do.
Data centers will soon be obsolete as mainframes as there will soon be chips capable of AI good enough for 90+% of us.
https://www.pcmag.com/news/a20-pro-iphone-chip-explained-apple-event
To be fair, data centers - like mainframes - will still have their use cases. Yes, mainframes do still exist.
I don’t think it’s so much the data centers, as the hard push to fast track something that has little regulatory oversight, and nothing good ever comes from rushing. They aren’t being honest or forthcoming about who will own these eventually, how it will impact the neighbors, and the tax breaks for the rich is getting really old. AI needs to be fine tuned a bit, it still gives lots of garbage information, I don’t trust it completely, it wastes my time. Google still can’t get my country address correct so people can find my house.
Please watch this regarding AI centers that are totally outdated and unnecessary and are not being built in Europe, according to this testimony
https://www.tiktok.com/t/ZP83vkKmD/
I will not watch anything on TikTok.
Are they tapping into our PC’s to do their computations or what?
I'm concerned about AI that it's going to replace many jobs and will make students lazy and stop learning, which is going to be devastating.
Our poor standards in education have done that already.
I am reading something written by a chess genius and I expect the unexpected. But this is just banal. It could have been written by any anonymous PR person. It could have been written by AI in fact and I sincerely hope not.
It is not the main point but I doubt that the chess computers that finally caught up with the best players belong to AI. It was before that era. I think they were purpose built chess machines whereas AI is based on neural net principles which seems to be more general purpose.
Garry Kasparov really does not get the problem though the warning is cogent and could be understood by anybody - how can we expect to keep as slaves things that are more intelligent than us? He does not get it and he does not answer it.
May I direct attention to the four paragraphs "Some forecasts are even more theatrical ..." which are under the heading "Risks, real and exaggerated". This is Kasparov's argument in a nutshell but it is not even an argument but a presumption. He merely says he expects that AI catastrophe will not materialise just as he says climate change will not. It is denialism and that is no argument.
Is there really a lack of vision for the positive uses of AI? Who is not aware of those??? Separately, I think comparing the AI race to the space race is not accurate and it's astoundingly not accurate. The least bad analogy to the past is the nuclear race. That's where we are, only now we're developing AI which may access the nukes (among many other kinds dangerous weapons) whether we give them that access or not. I agree that the consequences of countries which are even more irresponsible/malevolent than the US and US Oligarchs (Russia and China) continuing to make their AIs more powerful (regardless of when we get to AGI) are huge. Again, is there anyone in the US who is not aware of that? Also, maybe we should be focusing on trying to slow their progress down instead of speeding ours up? None of this makes the dangers of misuse of the current AI less dangerous, the warnings of people at the very forefront of AI alignment (ie safety and ethics) any less concerning, nor the dangers of AGI (as wielded by anyone) less dangerous. As we all know, all power corrupts, and ultimate power corrupts ultimately.
Your post here was diffuse and toward evasive. Very unusual! I hope you will apply your analytic brilliance in the direction of the worst cases, and maintain that focus in a post.
We’ll see. I’m not optimistic about a world of super intelligent autonomous actors who are not human. And likely amoral at best. But I hope I’m wrong.
Wikipedia isn't a brainstorming tool, it's just a compendium of various scientific sources in various fields.
"Imagine if people at the turn of the last century had looked at the Ford Model T and decided that cars weren’t worth it because they would always be loud, dirty, primitive motor carriages." Sounds great. We would have had EVs a lot sooner.