In his latest blogpost, Scott Aaronson summarizes AI related concepts. Among many thoughts, he points out that there will be limitations on LLMs (large language models) in the future because of the limits on the computational power and training data. I want to comment on training data. The total amount of training data will be limited but more importantly the characteristics of training data will change in the future.
LLMs can write poems, stories, create images and videos. LLMs will be much more capable in the future, of course. Differentiating the LLM generated code, text, image or video from human generated ones will be very important. As a result of government regulations, we may have signatures on the LLM generated output in the future. But, I think, there will always be unregulated LLMs.
LLMs like GPT are trained on public data found on the internet. Knowledge represented by the reachable portion of the internet is only a tiny fraction of the knowledge accumulated by humanity. Most of the scientific papers are behind pay walls. Similarly, private datasets accumulated by commercial firms contain much more information than public data. There will be LLMs trained on private data. This is already happening. Many firms are creating their own LLMs. There is no guarantee that these private LLMs will implement signatures.
Saturation
If the problem of differentiating human vs AI generated output exists in the future, then it will be inevitable that the future LLMs will be trained on a mixture of human generated and AI generated data. As the amount of AI generated data dwarfs human generated data, the future LLMs will be biased towards AI generated data. It is very hard to predict the outcome of such a bias. This will eventually lead to the saturation of the LLM capabilities including creativity, in my opinion.
Knowledgebase
I dream of a world-wide-knowledgebase. The information capacity of the internet is growing at an incredible rate but internet as a knowledgebase is still in its infancy. What we have so far is information curated by human beings. In the next step, information of the world will be curated by machines. That phase will be followed by another phase where human beings will organize the machine-curated information into an early form of world-wide-knowledgebase. Then there will be another phase dominated by machine processing. These human-touch and machine-touch dominated phases will follow each other.
The current development of LLMs corresponds to the phase “information of the world will be curated by machines” mentioned above. In the next phase “human beings will organize the machine-curated information into an early form of world-wide-knowledgebase“.
Epistemic uncertainty
Epistemic uncertainty increases as we come up with more and more models of Reality. I gave examples here. LLM (Large Language Model) refers to a model. I would argue that as the number of LLMs increases the epistemic uncertainty will increase.
Comments by Marc Kroeks
I wonder if in the future the AI will be trained on information it gathers itself with all possible kinds of sensory data. The human mind is in a way reflected in the language that is produced by human minds. These models reverse engineer the mind that created the data. When the AI becomes smarter than the human mind, like in the case of chess and go, it will begin to behave in a way that is new to us. This can be scary, because out sense of others is then no longer dominated by human beings as agents. The new agents will be “the new eyes of God” in this way, that these complex systems will be able to understand our universe and life on a deeper level than our tiny minds can. Thus they may become like the telescope of Coupernicus, that they show us things we could not perceive with our own eyes. In our mind we create a simulation of th world. We make a model with which we work when we interact with the world. When the AI behaves in a new way, we must learn to understand why and how it did that. What does AI understand that we do not?
AI could begin with converting all kind of sensory input, like audio and video and all kind of sensors into descriptive text. That text could then tell the story of what is in the data. It may be so however, that our human language so far lacks certain words and grammar in order to describe the deeper mysterious nature of existence. How is AI going to solve that problem? It has to be creative, intuitive and spiritual. Maybe internally it may use new language, but when it communicates with humans there has to be some sort of translation and explanation. Why did you make move 37? Can you invent a solution to problems we face now?
In short, maybe language is not the essence of intelligence, understanding is and the actions that follow how we give meaning to what we perceive. I think future AI will do just that, understand the game of life better than we do. How is it possible then to know what is right and what is wrong?
Suresh Emre
Renaissance Universal 