Donald Knuth challenged ChatGPT-4 with 20 questions and then submitted the results and his comments to Stephen Wolfram. The whole thing is fascinating in many ways and worth reading.
Some remarkable quotes:
Of course these are extremely impressive responses, sometimes astonishingly so; thus I totally understand why you and others have been paying attention to it.
The most immediate impression is the quality of the wordsmithing. It’s way better than 99% of copy that people actually write. It’s definitely not like a Markov model that uses the most predictable way to continue what’s already been said. On the other hand there are surprising lapses there too, as typical of any large system.
Answer #10 reads as though it’s the best answer yet. But it’s almost totally wrong!
It’s amazing how the confident tone lends credibility to all of that made-up nonsense. Almost impossible for anybody without knowledge of the book to believe that those “facts” aren’t authorititative and well researched.
I find it fascinating that novelists galore have written for decades about scenarios that might occur after a “singularity” in which superintelligent machines exist. But as far as I know, not a single novelist has realized that such a singularity would almost surely be preceded by a world in which machines are 0.01% intelligent (say), and in which millions of real people would be able to interact with them freely at essentially no cost.
I myself shall certainly continue to leave such research to others, and to devote my time to developing concepts that are authentic and trustworthy. And I hope you do the same.