Originally Posted by the_software_dev

ChatGPT scanned the entire internet. Now it knows what words come after every part of a sentence. It also has a simple system to "somehow understand" the topic at hand. So if you ask it "is water wet?" it's going to say "yes", because it found 5 million web pages where it says that water is wet. If it would've found 5 million pages saying that water is not wet, then it would've answered "no". Simply put, the system is not intelligent in any way, it just generates words based on an input, just because it found those words in order on the internet. (it's a bit more complicated than this, but that's the general idea of the article).

So their point is that software like this cannot be trusted to provide any accurate information about anything, simply because it was trained to spew out text it found on the internet. And we all know the internet is a very trustworthy source. It's a cool tool, but nowhere near "factual information", which is why Google (who had all this data since 25 years ago) didn't come forward with it.