Ilya Sutskever: pre-training is over
Ilya’s speech primarily revolved around the development of AI models, self-awareness, and superintelligence. He emphasized that in the future we will face extremely unpredictable AI systems that can understand things from limited data without getting confused. At the same time, he believes that self-awareness is useful as it forms part of ourselves and also a part of our world model.
Ilya predicted that when all these features are combined with self-awareness, they will bring properties and characteristics that are completely different from existing systems, which will have incredible abilities. Although he cannot determine when superintelligence will be achieved, he believes it will happen eventually.
In the Q&A session, Ilya answered questions from the audience, discussing topics such as biological inspiration, automatic error correction in models, and whether large language models can generalize beyond distributed multi-hop reasoning. His speech sparked controversy and discussion among industry professionals, including Google’s Logan Kilpatrick and former Meta chief of staff Drew Brough.
They disagreed with Ilya’s conclusion that pretraining is terminated, stating that human data has not been exhausted yet, and that we are primarily dealing with written text issues but still have great potential in the visual domain and other multimodal data. They also mentioned the possibility of future robots collecting vast amounts of multimodal data, further enriching our data resources.
Therefore, most industry professionals believe that pretraining will not end, but rather continue to evolve and improve to adapt to growing data demands and increasingly complex tasks.
Translation
Ilya 的演讲主要围绕了 AI 模型的发展、自我意识和超级智能等主题。他强调将来我们将面对一些极其不可预测的 AI 系统，它们能够从有限的数据中理解事物，而不会感到困惑。同时，他认为自我意识也是有用的，它构成了我们自身的一部分，也是我们世界模型中的一个部分。
Ilya 预言了当所有这些特性与自我意识结合在一起时，会带来与现有系统完全不同的性质和特性，这些系统将拥有令人难以置信的惊人能力。尽管他不能确定超级智能何时实现，但相信它终将发生。
在问答环节中,Ilya回答了观众的问题，讨论了生物学上的灵感、模型的自动纠错，以及大语言模型能否泛化超出分布的多跳推理等问题。他的演讲结束后引起了一些业界人士的反驳和讨论，其中包括谷歌大佬洛根·基尔帕特里克和前Meta具身智能团队高级总监德鲁夫·巴特拉。
他们表示不同意Ilya关于预训练终结的结论，认为人类的数据还没有用完，目前我们主要是处理了书写文本的问题，但在视觉领域和其他多模态数据方面，我们仍然有很大潜力。同时，他们也提到了未来机器人能够采集海量多模态数据的能力，这会进一步丰富我们的数据资源。
因此，大部分业界人士认为，预训练并不会终结，而是会持续发展和改善，以适应不断增长的数据需求和越来越复杂的任务。