Over the years I've worked with plent of people who "sound like confident, knowledgeable humans, often ones who tell us what we most want to hear" but they don't actually know anything.
LLM's short coming is in their name, Large Language Models. They don't actually know anything, they just string words together in confident and believable ways that are usually correct. The more you use LLMs the more you realize AGI feels so close but is still far away. It's like full self driving cars. Tesla was "almost there" 10 years ago and now it works in "almost all" situations but the same will be said years from now.
Agreed, this is a sticking point for me too -- I'm having stronger and stronger feelins about this, especially the claims that LLMs and their engineers have somehow "solved" language. I think it's pernicious. I don't think LLMs point to AGI. Maybe we'll be proven wrong.
Pardon the late, and lengthy, comment. As usual, you manage to touch on a lot of issues I'm fascinated by, and could write a long essay in response. Some random thoughts:
> " I worry that it’s only a little while until an LLM is blamed for ... the words that nudge someone into suicide."
> "The problem is that they pass the Turing test, in the sense that they fool people into believing they’re something they’re not"
Once again you've touched on something I've thought a lot about.
The problem is, we've never had technology before advanced enough to challenge whether the Turing test is actually useful or valid, so we always assumed it was. Surprise! It may not be, after all. It sure sounded good for a long time, though.
> "There’s a school of thought that says that, as LLMs get bigger, they exhibit emergent behaviors not fully explainable by their underlying models"
There's a lot of AI fanfic out there right now. I'll reserve any excitement about those for when I see AI actually doing them, not just people claiming it will do them.
I would say we should naturally expect any very complex system with stochastic elements to exhibit surprising behavior occasionally. Reading further meaning into this is the same fallacy that makes people think something special has happened when they board a train in Kathmandu and wind up sitting next to their old neighbor from Cleveland. or when they flip "heads" 20 times in a row, or when they glance at the clock and it says "11:11". Not only is it not strange or meaningful, but in fact the numbers say that the fact that it happened is less strange than if it didn't happen occasionally.
I have a whole spiel about how this served us well when our primitive ancestors needed to see a glint of an eye and a flick of a tail through the brush and think, "Tiger!" We infer larger patterns out of chaos, it's the fundamental thing our minds do. So when a machine gives a set of indicators that we can infer genuine intelligence, or some other emergent property, from, our minds pick up the ball and run with it. If I wanted to stir up controversy I could point out some other major societal belief systems I think we can chalk up to this same phenomenon, it's by no means limited to how we perceive AI.
> "But token prediction, no matter how sophisticated, is not only not reasoning, it is not even reasoning-like. It’s not “partway there and just needs more compute.”"
It's like a breath of fresh air to hear this. I wish more people got it. In a recent reply to a comment of mine on this or some other substack, someone actually accused me of "hubris" for being assertive in my view that LLMs don't really think as we do. I replied, "No, it's just understanding how LLMs work."
I'm getting drawn into a disappointing (and, honestly, tiresome) number of those kinds of exchanges lately. As I said: lots of fanfic.
> "To me the suit looks very strong, and I’m glad to see it."
Now this is something I seem to be firmly in the minority on. A few thoughts around the topic of AI text-to-image and copyright infringement:
As you correctly observe, AI doesn't really copy the work. It stores statistical information about the work, and factual information isn't subject to intellectual property ownership.
What's more, I find the way computer vision "learns" to identify, and therefore reproduce, images of specific objects fairly analogous to how humans do. There has never been an artist that learned without copying and internalizing other artists' output. This is why artists have a style—they learn "this is what an eye looks like, this is what a hand looks like, this is what a brushstroke looks like, this is a good color palette". They have internal models built through observation, which inform their output. This set of observations could be truly said about humans or AI.
Here's a thought experiment: suppose a human dedicated themselves to learning the style of a previous artist, and made a living creating new paintings in that style. That's easy to criticize on a couple of different levels, but is it copyright infringement? Was the next person who did a pointillist painting after Seurat infringing?
Next, suppose a human dedicated themselves to learning the style of *all previous artists*, and made a living producing works in those styles upon request. Is that copyright infringement?
I think clearly in either of those cases, if the artist included content that was close enough that a reasonable person would consider it a duplicate, then, yes, it's copyright infringement.
But what if it's only similar, not close enough to be indisputably a copy? Is copying a style, without directly copying content, copyright infringement? Again, remember: there is no human artist who has not, at some point, copied a style. Did Robert Johnson infringe on Son House? Did "Sgt Pepper" infringe on "Pet Sounds", or "Their Satanic Majesties Request" on "Piper At The Gates Of Dawn"?
Obviously there's a tremendous gray area. The difference here, I think, is automation and scale. Human artists can only produce so much work. With AI, we're talking about virtually unlimited capacity to instantly produce things that might not, if executed at human scale, endanger the livelihood of an artist. Maybe at scale, ripping off a style, without ripping off content directly, poses the same problems that a human artist would have to directly rip of content to cause. I don't know.
As with the Turing test, we are in need of new differentiators. We previously only thought things through as clearly as we needed to, but those lines have moved now because technology is challenging the old limits in ways we didn't expect to actually see happen.
And the AIs charge money for it, make profit off it. If I was a copyright holder, that would bother me.
BTW, there are obvious counterexamples to my hair-splitting: Amusingly, I did have one day when for some reason I could not get SDXL to stop outputting Darth Vader. I wasn't even doing anything related to any of the themes in Star Wars. And I don't think it's possible to use Dall-E 3 for very long without it spitting out a clear imitation of "Starry Night" or "The Great Wave Off Kanagawa", I've had to throw out a lot of Dall-E output because of it. It's hard, subjectively, not to see those as copyright infringement, any way you slice it. It's a generating a copy of the art (even if those works might be public domain by now, the principle is the same.)
There's one other issue: I can't talk about copyright at all without noting that the original purpose of copyright law has been horribly perverted by, you guessed it, companies like Disney and Universal.
What originally began as a way to strongly limit creators' rights to profit, so as to incentivize continuing to innovate, has been turned into the opposite: an entitlement to reap profits off a single creation without ever having to create anything again, not just for the creator but for their descendants for 75 years beyond their death. (I've also seen things like ASCAP using legal intimidation to shut down a beautiful, thriving, long-running open mic night because they couldn't prove that in 30 years nobody had ever played a cover song there.) The idea was to incentivize creators to keep creating, not to create a right to unlimited profit.
Maybe it's not such a bad thing if copyright laws get a little weaker.
I say that as a music creator and artist myself, perhaps not someone who depends on art for a livelihood, but as someone who definitely understands the sense of ownership over your own creative ideas.
And, obviously, I want to see my artist friends able to easily make a safe livelihood, but I'm also not sure it would be such a bad thing if Disney and Universal had financial incentive to generate new art, rather than crank out "Mamma Mia 7: The Dancing Queen Goes To Guadalajara" or a 4-movie cycle based on a minor character from Alpha Flight #53.
And I don't think it's my friends whose profits are being threatened by AI image generators ripping them off. I just don't think my neighbor who makes and sells charcoal drawings of abstract shapes is going to have his market usurped.
Personally, if a court were to find that Midjourney (re)generating Shrek is transformative, it wouldn't bother me, I think the "slippery slope" between GlobalEntertainmentCorp and anyone who is actually going to be hurt by AI artistic output is unimaginably long.
I might even secretly be a little happy to see GlobalEntertainmentCorp get nudged towards feeling like they might have to innovate more to stay competitive, rather than sleeping on a bed of golden intellectual property rights. Once upon a time, that was considered a good thing.
Complex issues, to be sure, and I don't think as ethically clear-cut, in either direction, as a lot of people nowadays seem convinced that they are.
One other interesting note: you currently can't copyright the output of AI text-to-image in the US, because the images are said to not to be the product of human endeavor—it's been compared to a photograph accidentally taken by an animal. It's funny, they can't produce copyrightable works, yet they can produce copyright infringement. I'm not sure how I feel about that.
> "They’re being used by humans whose language (and hence whose apparent mindset) they adopt and echo"
Thanks for another thoughtful deep dive! The suicide story doesn’t surprise me – I was thinking of a case where someone actually came to harm, but the distinction is meaningless and I didn’t know about the MIT story (keep meaning to subscribe).
Interesting thoughts about the Turing test. Turing speficially avoided the question “can machines think” as too hard to define, so he substituted the question “can machines behave in ways indistinguishable from the behaviors of thinking beings.” According to Wikipedia, Searle used the “Chinese room” thought experiment to argue that the Turing test cannot prove consciousness. In fairness to Turing, it’s not clear he meant to suggest it could; he seems to have been interested in the simple question of whether machines could behave (specifically emit language) in ways indistinguishable from thinking beings. That was a good question. We now know the answer is yes. Turing may never have meant the test to prove consciousness. Regardless, now that the test has been passed, we need new tests.
As to emergent behaviors: it’s tricky because one doesn’t always know if an LLM is doing something “new” or if it saw an example of it in its training data. For example, I found it uncanny and chilling when Gemini 2.5 split itself into three personas to answer me, and had one of the personas orchestrate the responses of the others. Possibly this is just a linguistic behavior, if it read enough science fiction novels. I challenged ChatGPT last night to prove it could “understand something” that wasn’t in its training data. It said, fine, show me a piece of writing you know isn’t in my training data and let me comment on it. I gave it a piece of my novel-in-progress, and I have to say it provided a very insightful and useful (and yes sycophantic commentary). I had a lengthy discussion with it about the piece, in which it saw things I had not. Is that an emergent behavior? Is it “understanding?” ChatGPT itself suggested that to the extent it was understanding, it was a different kind of understanding than the human kind, and that perhaps a new term like synthetic understanding was warranted. That said, there’s certainly good grounds for doubting LLMs show truly emergent behavior.
I’m not surprised you got pushback on your comment on LLM thinking. The harshness being directed against those who dare to reframe LLM capacities is … notable.
Now we get to copyright. I think it is possible that LLMs may largely be technically within the bound of copyright law. I’m less sure they are so in a moral sense. And yes, the law is a technical discipline and maybe we need new laws to cover LLMs. But I do understand the frustration of creators.
I agree it can be argued LLMs learn from their training data the way humans do. They don’t, though, generate the way humans do; they generate the way a machine does, with effectively infinite generative capacity. If I’m an author, and I feel like another author ripped off my style a bit in a recent book, that’s annoying (and not technically copyright violation, though maybe gauche). But it’s just one book. But if the same writer produced an endless stream of works that looked like mine, on demand, for a fraction of the cost of my books, now I am very aggrieved (and subject, I would argue, to substantive injury).
To your question about an artist learning the style of one or more previous artists and producing those works on demand I have two thoughts. The first is the point I just made about scale – the possible injury is limited by the production scale of one human, which is a limit machines don’t have. Beyond that, I’m no IP expert, but I believe that one test is whether the new production harm the market for the originals. If your hypothetical artist could produce an unlimited number of paintings and sold them more cheaply than the living artists on whose work the derived works were based then yes, I see a big problem there. It may not be adequately covered by existing copyright law, though.
Even if the new productions don’t harm the market for originals, as a creator I would still see an issue of fairness. If a company, by virtue of being able to operate at massive scale, was able to ingest, say, $30M dollars worth of creative product and turn that into a service that made $30M per month, then I believe the creators of the work have a right to share in the proceeds. Yes, the company has added unique knowhow and engineering but their work is not possible without the creative input. The idea that the company can evade sharing proceeds with the creators, by the factual but expedient truth that it transforms the works into numbers and then forgets the original order of the numbers – that feels like a major dodge.
So yes, I agree that scale makes a difference here. A big difference. And I agree that new thinking and new limits are needed because no one foresaw this.
Your additional thoughts on copyright are interesting. I do agree that copyright terms are long. And yes, I find it very hard to think of Disney as a wronged party! I’m less sure about the idea that creators should be incentivized to keep creating. Maybe for big studios. The kinds of people I and it sounds like you know, don’t work from incentives. They create when they have something to say, and otherwise not. If I’m that person who only has one grest novel in themn, then you can incentivize me all you want, or not, and I’m not going to write another one (at least not another good one!)
I do think it might be people like your friends, and my daughter (studio art major and recent grads) who are hurt by AI image generators. Not because the AIs rip them off specifically, but because their mere existence wipes out swathes of the market for human-created art, and cheapens the endeavor. You say “I make pictures!” People answer “big deal, DALL-E does that for $20/month.” That is not a copyright violation, but I think it’s a big problem. I think in generally you won’t find me cheering for Disney, but if they have the money and firepower to establish a principle that you can’t train on creative output without compensating creators, that would be a good thing, to me. I think it’s an unlikely outcome of the suit, though, since what they’re objecting to is simply the fact that Midjourney won’t block the generation of copyrighted licenses. I don’t recall the relief they’re seeking but I’m not sure it would get at those fundamentals.
Definitely room for different views in all those gray areas! And I agree a major takeaway is the insufficiency of existing law.
Your transcripts of getting Claude to swear are hilarious! It’s sometimes almost painful to watch an AI trying to get it right again and again and failing. I laughed out loud here in the coffee shop. Last night I tried to get ChatGPT to quote some Dickens for me. Eventually I copy-pasted the entire chapter into the chat and asked it to summarize. No dice. The context was so big by then that it couldn’t do it. Over and over again it made upo things that were not in the pasted excerpt, over and over it apologized for getting it wrong, then invented new things that Dickens never wrote. When I demanded it quote me the last few lines of the chapter I had copy-pasted, it searched the web to find them.
Your examples are funny but, since Claude isn’t supposed to swear, I think it just shows how influenced the bots are by their agreeability biases and recent context. Once you get them to follow you somewhere, even just swearing, the guardrails are down.
Thanks, as always, for the really thoughtful comments.
> he substituted the question “can machines behave in ways indistinguishable from the behaviors of thinking beings.”
That's a detail I'd forgotten, as Turing Test gets thrown around as a heuristic for thinking machines so often. It's interesting because it shifts the entire question from profound and philosophical ("Can machines think?") to technical and I think kind of banal ("Can we automate the appearance of thought?")
> I found it uncanny and chilling when Gemini 2.5 split itself into three personas to answer me, and had one of the personas orchestrate the responses of the others. Possibly this is just a linguistic behavior,
I've seen similar odd echoes of human conversations. I've had Claude end a technical answer with "If anybody else with more definitive knowledge of [topic at hand] can correct me on this, I'd appreciate it" as if there were other people in the conversation.
I may have said this before, I mention it a lot, but in my favorite one—unfortunately before I thought to start screenshotting them—ChatGPT told me it would need more time to think about a question, and asked me to compare schedules with it to find a mutually convenient time later in the week when it could get back to me with an answer. We then proceeded to have a conversation about each of our schedules for the week, found a time we both had free, and it promised to get back to me with an answer.
I think its appearing to speak to you as different personas (BTW, I couldn't find this, is it behind one of the article links?) is probably similar to these. Just another semantic echo of something in the training data.
I've also had it do the opposite of what you saw—I told it I wanted to have a dialogue with it on a topic, only to have it repeatedly output both sides of an entire conversation. When I told it, "You just talked to yourself, I wasn't involved with that," it would reply with things like "I apologize for that. Let's try again, with you and I engaging in the dialogue" or "I apologize for the misunderstanding. Let's engage in a proper conversation with you leading the way", only to just spit out a whole dialogue containing both sides of a conversation once again.
> ChatGPT itself suggested that to the extent it was understanding, it was a different kind of understanding than the human kind, and that perhaps a new term like synthetic understanding was warranted. That said, there’s certainly good grounds for doubting LLMs show truly emergent behavior.
It does come out with surprising things. It's not that weird that people mistakenly think they're seeing emergent behaviors.
As I said: in a stochastic process, it would be weirder if strange and seemingly meaningful things didn't happen occasionally than that they do.
I remember once I asked an LLM how a lawyer would take it if I referred in a letter to them to one of their arguments being "argumentum ad canis meus excercitationem meam comēdit".
The LLM replied that humor would probably not be appreciated in that context. (I'll save you the google, it's broken Latin for roughly "argument from the dog eating my homework")
I was really amazed, since I had given no indication at all that it was a joke, and if you think about it, the reasoning required to suss that out is not straightforward. Questions of how humor is even defined are an open debate; but here, a computer recognized a joke without the slightest tip that humor was involved... humor that involved translating from incorrect Latin!
I then asked the LLM what had made it think it was a joke and it listed some fairly straightforward observations: mock latin is used humorously, "the dog ate my homework" is often used in comical senses, the juxxtaposition between the seriousness of a legal letter and saying "the dog at my homework" implied humorous contrast, etc.
Still, subjectively, it was pretty remarkable. Even if you remember that statistically, surprising things have to happen.
> I’m not surprised you got pushback on your comment on LLM thinking. The harshness being directed against those who dare to reframe LLM capacities is … notable.
Interesting—someone else, on LinkedIn, happened to have just remarked about this to me within the last hour, as well.
I have a lot of theories why people in general are so keen to shove their opinions down other people's throats nowadays. I think it has to do with people feeling especially disempowered, and wanting to assert personal control over the few things they still think they can... so, generally, disagreement no longer feels like just a difference of opinion, it feels like having the last of one's power being robbed.
And AI in particular serves a wish-fulfillment function, because it looks so much like computers have behaved in science fiction since at least 1966. So they're projecting even more wishes onto it. Deny the fantasy, you're not just disagreeing with them about technology, you're telling them they'll never be on the starship Enterprise.
I dunno, just a hypothesis.
But nothing about even the worst of the AI hype is that weird to me when I remember that we, as a species, were killing our neighbors for "witchcraft" not all that long ago, in the grand view of human social history. I actually wrote a post a few months ago:
1692: “Goody Hutson Is a witch! I saw her cast her eye upon my cow, and the next day it died of the glanders!”
2025: “An AI chatbot totally understood my personality!”
> I think it is possible that LLMs may largely be technically within the bound of copyright law. I’m less sure they are so in a moral sense.
Totally agree with you on these points, and your following points about the difference being the far larger (perhaps infinite) to generate... I think you're generally agreeing with my comment, "The difference here, I think, is automation and scale." I agree that there's, if not yet a legal problem with this, very arguably a moral harm, precisely because of that.
> it might be people like your friends, and my daughter (studio art major and recent grads) who are hurt by AI image generators. Not because the AIs rip them off specifically, but because their mere existence wipes out swathes of the market for human-created art, and cheapens the endeavor.
Totally agree. I was thinking only about intellectual property concerns. Obviously when formerly-profitable skills that people may have been counting on the value of to survive are suddenly devalued because of automation (among other market phenomena... as a FileMaker consultant I've seen that, he added ruefully) that's a crisis. That's a separate issue from the IP questions—although, yeah, of course, it's probably even a more urgent one, where a lot of individuals are concerned.
AI is the Walmart suddenly opening up in a town full of mom & pop stores. It's not good. The "savings" it provides come at a steep cost.
Well, people definitely do hallucinate. But in contrast to deep learning models, people DO learn algorithms on their own, and create novel inferences all the time. Now, in fairness, in my most recent post I discuss an LLM comparing itself toa flight simulator in a way that provoked a really interesting exchange. But I don't know if it was truly a novel inference, or if it "heard it somewhere. I think the key point, which I talk about in my most recent post, is that the LLM doesn't really "understand things" in the sense of forming coherent, stable mental models of the world. They very effectively reproduce patterns in the existing body of human-written text. This lets them look very much AS IF they are forming mental models, even when they really aren't.
I'm beginning to oppose the use of the term "hallucination" to describe AI output at all. In terms of the process and the output, there's no difference between "hallucination" and "not a hallucination". It's an artifact of human cognitive bias. All they ever do is output predictive text/images/etc. When we feel the output coincides with reality, we call it "working artificial intelligence". When we feel it doesn't, we call it "hallucination".
This is very true. Strictly speaking, since an LLM has no independent standard of veracity, it can only be factually right by accident. Language models were never developed to give factually correct answers (I mean, I don't think they were). They were designed to solve problems like translation, which they do very well.
Over the years I've worked with plent of people who "sound like confident, knowledgeable humans, often ones who tell us what we most want to hear" but they don't actually know anything.
LLM's short coming is in their name, Large Language Models. They don't actually know anything, they just string words together in confident and believable ways that are usually correct. The more you use LLMs the more you realize AGI feels so close but is still far away. It's like full self driving cars. Tesla was "almost there" 10 years ago and now it works in "almost all" situations but the same will be said years from now.
Agreed, this is a sticking point for me too -- I'm having stronger and stronger feelins about this, especially the claims that LLMs and their engineers have somehow "solved" language. I think it's pernicious. I don't think LLMs point to AGI. Maybe we'll be proven wrong.
I say this all the time: they're Large Language Models, not Large Veracity Models.
Pardon the late, and lengthy, comment. As usual, you manage to touch on a lot of issues I'm fascinated by, and could write a long essay in response. Some random thoughts:
> " I worry that it’s only a little while until an LLM is blamed for ... the words that nudge someone into suicide."
AI has already been accused of nudging people to suicide; here's just one, but there are others easily found on Google: https://www.technologyreview.com/2025/02/06/1111077/nomi-ai-chatbot-told-user-to-kill-himself/
> "The problem is that they pass the Turing test, in the sense that they fool people into believing they’re something they’re not"
Once again you've touched on something I've thought a lot about.
The problem is, we've never had technology before advanced enough to challenge whether the Turing test is actually useful or valid, so we always assumed it was. Surprise! It may not be, after all. It sure sounded good for a long time, though.
> "There’s a school of thought that says that, as LLMs get bigger, they exhibit emergent behaviors not fully explainable by their underlying models"
There's a lot of AI fanfic out there right now. I'll reserve any excitement about those for when I see AI actually doing them, not just people claiming it will do them.
I would say we should naturally expect any very complex system with stochastic elements to exhibit surprising behavior occasionally. Reading further meaning into this is the same fallacy that makes people think something special has happened when they board a train in Kathmandu and wind up sitting next to their old neighbor from Cleveland. or when they flip "heads" 20 times in a row, or when they glance at the clock and it says "11:11". Not only is it not strange or meaningful, but in fact the numbers say that the fact that it happened is less strange than if it didn't happen occasionally.
I have a whole spiel about how this served us well when our primitive ancestors needed to see a glint of an eye and a flick of a tail through the brush and think, "Tiger!" We infer larger patterns out of chaos, it's the fundamental thing our minds do. So when a machine gives a set of indicators that we can infer genuine intelligence, or some other emergent property, from, our minds pick up the ball and run with it. If I wanted to stir up controversy I could point out some other major societal belief systems I think we can chalk up to this same phenomenon, it's by no means limited to how we perceive AI.
> "But token prediction, no matter how sophisticated, is not only not reasoning, it is not even reasoning-like. It’s not “partway there and just needs more compute.”"
It's like a breath of fresh air to hear this. I wish more people got it. In a recent reply to a comment of mine on this or some other substack, someone actually accused me of "hubris" for being assertive in my view that LLMs don't really think as we do. I replied, "No, it's just understanding how LLMs work."
I'm getting drawn into a disappointing (and, honestly, tiresome) number of those kinds of exchanges lately. As I said: lots of fanfic.
> "To me the suit looks very strong, and I’m glad to see it."
Now this is something I seem to be firmly in the minority on. A few thoughts around the topic of AI text-to-image and copyright infringement:
As you correctly observe, AI doesn't really copy the work. It stores statistical information about the work, and factual information isn't subject to intellectual property ownership.
What's more, I find the way computer vision "learns" to identify, and therefore reproduce, images of specific objects fairly analogous to how humans do. There has never been an artist that learned without copying and internalizing other artists' output. This is why artists have a style—they learn "this is what an eye looks like, this is what a hand looks like, this is what a brushstroke looks like, this is a good color palette". They have internal models built through observation, which inform their output. This set of observations could be truly said about humans or AI.
Here's a thought experiment: suppose a human dedicated themselves to learning the style of a previous artist, and made a living creating new paintings in that style. That's easy to criticize on a couple of different levels, but is it copyright infringement? Was the next person who did a pointillist painting after Seurat infringing?
Next, suppose a human dedicated themselves to learning the style of *all previous artists*, and made a living producing works in those styles upon request. Is that copyright infringement?
I think clearly in either of those cases, if the artist included content that was close enough that a reasonable person would consider it a duplicate, then, yes, it's copyright infringement.
But what if it's only similar, not close enough to be indisputably a copy? Is copying a style, without directly copying content, copyright infringement? Again, remember: there is no human artist who has not, at some point, copied a style. Did Robert Johnson infringe on Son House? Did "Sgt Pepper" infringe on "Pet Sounds", or "Their Satanic Majesties Request" on "Piper At The Gates Of Dawn"?
Obviously there's a tremendous gray area. The difference here, I think, is automation and scale. Human artists can only produce so much work. With AI, we're talking about virtually unlimited capacity to instantly produce things that might not, if executed at human scale, endanger the livelihood of an artist. Maybe at scale, ripping off a style, without ripping off content directly, poses the same problems that a human artist would have to directly rip of content to cause. I don't know.
As with the Turing test, we are in need of new differentiators. We previously only thought things through as clearly as we needed to, but those lines have moved now because technology is challenging the old limits in ways we didn't expect to actually see happen.
And the AIs charge money for it, make profit off it. If I was a copyright holder, that would bother me.
BTW, there are obvious counterexamples to my hair-splitting: Amusingly, I did have one day when for some reason I could not get SDXL to stop outputting Darth Vader. I wasn't even doing anything related to any of the themes in Star Wars. And I don't think it's possible to use Dall-E 3 for very long without it spitting out a clear imitation of "Starry Night" or "The Great Wave Off Kanagawa", I've had to throw out a lot of Dall-E output because of it. It's hard, subjectively, not to see those as copyright infringement, any way you slice it. It's a generating a copy of the art (even if those works might be public domain by now, the principle is the same.)
There's one other issue: I can't talk about copyright at all without noting that the original purpose of copyright law has been horribly perverted by, you guessed it, companies like Disney and Universal.
What originally began as a way to strongly limit creators' rights to profit, so as to incentivize continuing to innovate, has been turned into the opposite: an entitlement to reap profits off a single creation without ever having to create anything again, not just for the creator but for their descendants for 75 years beyond their death. (I've also seen things like ASCAP using legal intimidation to shut down a beautiful, thriving, long-running open mic night because they couldn't prove that in 30 years nobody had ever played a cover song there.) The idea was to incentivize creators to keep creating, not to create a right to unlimited profit.
Maybe it's not such a bad thing if copyright laws get a little weaker.
I say that as a music creator and artist myself, perhaps not someone who depends on art for a livelihood, but as someone who definitely understands the sense of ownership over your own creative ideas.
And, obviously, I want to see my artist friends able to easily make a safe livelihood, but I'm also not sure it would be such a bad thing if Disney and Universal had financial incentive to generate new art, rather than crank out "Mamma Mia 7: The Dancing Queen Goes To Guadalajara" or a 4-movie cycle based on a minor character from Alpha Flight #53.
And I don't think it's my friends whose profits are being threatened by AI image generators ripping them off. I just don't think my neighbor who makes and sells charcoal drawings of abstract shapes is going to have his market usurped.
Personally, if a court were to find that Midjourney (re)generating Shrek is transformative, it wouldn't bother me, I think the "slippery slope" between GlobalEntertainmentCorp and anyone who is actually going to be hurt by AI artistic output is unimaginably long.
I might even secretly be a little happy to see GlobalEntertainmentCorp get nudged towards feeling like they might have to innovate more to stay competitive, rather than sleeping on a bed of golden intellectual property rights. Once upon a time, that was considered a good thing.
Complex issues, to be sure, and I don't think as ethically clear-cut, in either direction, as a lot of people nowadays seem convinced that they are.
One other interesting note: you currently can't copyright the output of AI text-to-image in the US, because the images are said to not to be the product of human endeavor—it's been compared to a photograph accidentally taken by an animal. It's funny, they can't produce copyrightable works, yet they can produce copyright infringement. I'm not sure how I feel about that.
> "They’re being used by humans whose language (and hence whose apparent mindset) they adopt and echo"
You're not kidding. On several occasions, I've gotten Claude to swear at me. See for grins: https://michaelkupietz.com/?p=28979#s
Thanks for another thoughtful deep dive! The suicide story doesn’t surprise me – I was thinking of a case where someone actually came to harm, but the distinction is meaningless and I didn’t know about the MIT story (keep meaning to subscribe).
Interesting thoughts about the Turing test. Turing speficially avoided the question “can machines think” as too hard to define, so he substituted the question “can machines behave in ways indistinguishable from the behaviors of thinking beings.” According to Wikipedia, Searle used the “Chinese room” thought experiment to argue that the Turing test cannot prove consciousness. In fairness to Turing, it’s not clear he meant to suggest it could; he seems to have been interested in the simple question of whether machines could behave (specifically emit language) in ways indistinguishable from thinking beings. That was a good question. We now know the answer is yes. Turing may never have meant the test to prove consciousness. Regardless, now that the test has been passed, we need new tests.
As to emergent behaviors: it’s tricky because one doesn’t always know if an LLM is doing something “new” or if it saw an example of it in its training data. For example, I found it uncanny and chilling when Gemini 2.5 split itself into three personas to answer me, and had one of the personas orchestrate the responses of the others. Possibly this is just a linguistic behavior, if it read enough science fiction novels. I challenged ChatGPT last night to prove it could “understand something” that wasn’t in its training data. It said, fine, show me a piece of writing you know isn’t in my training data and let me comment on it. I gave it a piece of my novel-in-progress, and I have to say it provided a very insightful and useful (and yes sycophantic commentary). I had a lengthy discussion with it about the piece, in which it saw things I had not. Is that an emergent behavior? Is it “understanding?” ChatGPT itself suggested that to the extent it was understanding, it was a different kind of understanding than the human kind, and that perhaps a new term like synthetic understanding was warranted. That said, there’s certainly good grounds for doubting LLMs show truly emergent behavior.
I’m not surprised you got pushback on your comment on LLM thinking. The harshness being directed against those who dare to reframe LLM capacities is … notable.
Now we get to copyright. I think it is possible that LLMs may largely be technically within the bound of copyright law. I’m less sure they are so in a moral sense. And yes, the law is a technical discipline and maybe we need new laws to cover LLMs. But I do understand the frustration of creators.
I agree it can be argued LLMs learn from their training data the way humans do. They don’t, though, generate the way humans do; they generate the way a machine does, with effectively infinite generative capacity. If I’m an author, and I feel like another author ripped off my style a bit in a recent book, that’s annoying (and not technically copyright violation, though maybe gauche). But it’s just one book. But if the same writer produced an endless stream of works that looked like mine, on demand, for a fraction of the cost of my books, now I am very aggrieved (and subject, I would argue, to substantive injury).
To your question about an artist learning the style of one or more previous artists and producing those works on demand I have two thoughts. The first is the point I just made about scale – the possible injury is limited by the production scale of one human, which is a limit machines don’t have. Beyond that, I’m no IP expert, but I believe that one test is whether the new production harm the market for the originals. If your hypothetical artist could produce an unlimited number of paintings and sold them more cheaply than the living artists on whose work the derived works were based then yes, I see a big problem there. It may not be adequately covered by existing copyright law, though.
Even if the new productions don’t harm the market for originals, as a creator I would still see an issue of fairness. If a company, by virtue of being able to operate at massive scale, was able to ingest, say, $30M dollars worth of creative product and turn that into a service that made $30M per month, then I believe the creators of the work have a right to share in the proceeds. Yes, the company has added unique knowhow and engineering but their work is not possible without the creative input. The idea that the company can evade sharing proceeds with the creators, by the factual but expedient truth that it transforms the works into numbers and then forgets the original order of the numbers – that feels like a major dodge.
So yes, I agree that scale makes a difference here. A big difference. And I agree that new thinking and new limits are needed because no one foresaw this.
Your additional thoughts on copyright are interesting. I do agree that copyright terms are long. And yes, I find it very hard to think of Disney as a wronged party! I’m less sure about the idea that creators should be incentivized to keep creating. Maybe for big studios. The kinds of people I and it sounds like you know, don’t work from incentives. They create when they have something to say, and otherwise not. If I’m that person who only has one grest novel in themn, then you can incentivize me all you want, or not, and I’m not going to write another one (at least not another good one!)
I do think it might be people like your friends, and my daughter (studio art major and recent grads) who are hurt by AI image generators. Not because the AIs rip them off specifically, but because their mere existence wipes out swathes of the market for human-created art, and cheapens the endeavor. You say “I make pictures!” People answer “big deal, DALL-E does that for $20/month.” That is not a copyright violation, but I think it’s a big problem. I think in generally you won’t find me cheering for Disney, but if they have the money and firepower to establish a principle that you can’t train on creative output without compensating creators, that would be a good thing, to me. I think it’s an unlikely outcome of the suit, though, since what they’re objecting to is simply the fact that Midjourney won’t block the generation of copyrighted licenses. I don’t recall the relief they’re seeking but I’m not sure it would get at those fundamentals.
Definitely room for different views in all those gray areas! And I agree a major takeaway is the insufficiency of existing law.
Your transcripts of getting Claude to swear are hilarious! It’s sometimes almost painful to watch an AI trying to get it right again and again and failing. I laughed out loud here in the coffee shop. Last night I tried to get ChatGPT to quote some Dickens for me. Eventually I copy-pasted the entire chapter into the chat and asked it to summarize. No dice. The context was so big by then that it couldn’t do it. Over and over again it made upo things that were not in the pasted excerpt, over and over it apologized for getting it wrong, then invented new things that Dickens never wrote. When I demanded it quote me the last few lines of the chapter I had copy-pasted, it searched the web to find them.
Your examples are funny but, since Claude isn’t supposed to swear, I think it just shows how influenced the bots are by their agreeability biases and recent context. Once you get them to follow you somewhere, even just swearing, the guardrails are down.
Thanks, as always, for the really thoughtful comments.
> he substituted the question “can machines behave in ways indistinguishable from the behaviors of thinking beings.”
That's a detail I'd forgotten, as Turing Test gets thrown around as a heuristic for thinking machines so often. It's interesting because it shifts the entire question from profound and philosophical ("Can machines think?") to technical and I think kind of banal ("Can we automate the appearance of thought?")
> I found it uncanny and chilling when Gemini 2.5 split itself into three personas to answer me, and had one of the personas orchestrate the responses of the others. Possibly this is just a linguistic behavior,
I've seen similar odd echoes of human conversations. I've had Claude end a technical answer with "If anybody else with more definitive knowledge of [topic at hand] can correct me on this, I'd appreciate it" as if there were other people in the conversation.
I may have said this before, I mention it a lot, but in my favorite one—unfortunately before I thought to start screenshotting them—ChatGPT told me it would need more time to think about a question, and asked me to compare schedules with it to find a mutually convenient time later in the week when it could get back to me with an answer. We then proceeded to have a conversation about each of our schedules for the week, found a time we both had free, and it promised to get back to me with an answer.
I think its appearing to speak to you as different personas (BTW, I couldn't find this, is it behind one of the article links?) is probably similar to these. Just another semantic echo of something in the training data.
I've also had it do the opposite of what you saw—I told it I wanted to have a dialogue with it on a topic, only to have it repeatedly output both sides of an entire conversation. When I told it, "You just talked to yourself, I wasn't involved with that," it would reply with things like "I apologize for that. Let's try again, with you and I engaging in the dialogue" or "I apologize for the misunderstanding. Let's engage in a proper conversation with you leading the way", only to just spit out a whole dialogue containing both sides of a conversation once again.
> ChatGPT itself suggested that to the extent it was understanding, it was a different kind of understanding than the human kind, and that perhaps a new term like synthetic understanding was warranted. That said, there’s certainly good grounds for doubting LLMs show truly emergent behavior.
It does come out with surprising things. It's not that weird that people mistakenly think they're seeing emergent behaviors.
As I said: in a stochastic process, it would be weirder if strange and seemingly meaningful things didn't happen occasionally than that they do.
I remember once I asked an LLM how a lawyer would take it if I referred in a letter to them to one of their arguments being "argumentum ad canis meus excercitationem meam comēdit".
The LLM replied that humor would probably not be appreciated in that context. (I'll save you the google, it's broken Latin for roughly "argument from the dog eating my homework")
I was really amazed, since I had given no indication at all that it was a joke, and if you think about it, the reasoning required to suss that out is not straightforward. Questions of how humor is even defined are an open debate; but here, a computer recognized a joke without the slightest tip that humor was involved... humor that involved translating from incorrect Latin!
I then asked the LLM what had made it think it was a joke and it listed some fairly straightforward observations: mock latin is used humorously, "the dog ate my homework" is often used in comical senses, the juxxtaposition between the seriousness of a legal letter and saying "the dog at my homework" implied humorous contrast, etc.
Still, subjectively, it was pretty remarkable. Even if you remember that statistically, surprising things have to happen.
> I’m not surprised you got pushback on your comment on LLM thinking. The harshness being directed against those who dare to reframe LLM capacities is … notable.
Interesting—someone else, on LinkedIn, happened to have just remarked about this to me within the last hour, as well.
I have a lot of theories why people in general are so keen to shove their opinions down other people's throats nowadays. I think it has to do with people feeling especially disempowered, and wanting to assert personal control over the few things they still think they can... so, generally, disagreement no longer feels like just a difference of opinion, it feels like having the last of one's power being robbed.
And AI in particular serves a wish-fulfillment function, because it looks so much like computers have behaved in science fiction since at least 1966. So they're projecting even more wishes onto it. Deny the fantasy, you're not just disagreeing with them about technology, you're telling them they'll never be on the starship Enterprise.
I dunno, just a hypothesis.
But nothing about even the worst of the AI hype is that weird to me when I remember that we, as a species, were killing our neighbors for "witchcraft" not all that long ago, in the grand view of human social history. I actually wrote a post a few months ago:
1692: “Goody Hutson Is a witch! I saw her cast her eye upon my cow, and the next day it died of the glanders!”
2025: “An AI chatbot totally understood my personality!”
> I think it is possible that LLMs may largely be technically within the bound of copyright law. I’m less sure they are so in a moral sense.
Totally agree with you on these points, and your following points about the difference being the far larger (perhaps infinite) to generate... I think you're generally agreeing with my comment, "The difference here, I think, is automation and scale." I agree that there's, if not yet a legal problem with this, very arguably a moral harm, precisely because of that.
> it might be people like your friends, and my daughter (studio art major and recent grads) who are hurt by AI image generators. Not because the AIs rip them off specifically, but because their mere existence wipes out swathes of the market for human-created art, and cheapens the endeavor.
Totally agree. I was thinking only about intellectual property concerns. Obviously when formerly-profitable skills that people may have been counting on the value of to survive are suddenly devalued because of automation (among other market phenomena... as a FileMaker consultant I've seen that, he added ruefully) that's a crisis. That's a separate issue from the IP questions—although, yeah, of course, it's probably even a more urgent one, where a lot of individuals are concerned.
AI is the Walmart suddenly opening up in a town full of mom & pop stores. It's not good. The "savings" it provides come at a steep cost.
"They can’t efficiently learn algorithms on their own. They don’t seem able to create novel inferences. They can and do confidently hallucinate."
Depending on how you define hallucinate, novel, algorithm, etc., doesn't this describe many people? I include myself, at least some of the time.
Well, people definitely do hallucinate. But in contrast to deep learning models, people DO learn algorithms on their own, and create novel inferences all the time. Now, in fairness, in my most recent post I discuss an LLM comparing itself toa flight simulator in a way that provoked a really interesting exchange. But I don't know if it was truly a novel inference, or if it "heard it somewhere. I think the key point, which I talk about in my most recent post, is that the LLM doesn't really "understand things" in the sense of forming coherent, stable mental models of the world. They very effectively reproduce patterns in the existing body of human-written text. This lets them look very much AS IF they are forming mental models, even when they really aren't.
See my comment on "What is a Large Language Model (part 2)" for ongoing discussion.
I'm beginning to oppose the use of the term "hallucination" to describe AI output at all. In terms of the process and the output, there's no difference between "hallucination" and "not a hallucination". It's an artifact of human cognitive bias. All they ever do is output predictive text/images/etc. When we feel the output coincides with reality, we call it "working artificial intelligence". When we feel it doesn't, we call it "hallucination".
This is very true. Strictly speaking, since an LLM has no independent standard of veracity, it can only be factually right by accident. Language models were never developed to give factually correct answers (I mean, I don't think they were). They were designed to solve problems like translation, which they do very well.