Variant Investment Partner: The Dilemma and Breakthrough of Open Source AI, Why is encryption technology the final piece of the puzzle?

Author: Daniel Barabander Translation: Deep Tide TechFlow Summary The current development of basic AI is dominated by a few technology companies, showing characteristics of closedness and lack of competition. Although Open Source software development is a potential solution, basic AI cannot operate like traditional Open Source projects (such as Linux) due to a "resource problem": Open Source contributors not only need to invest time but also bear computing and data costs beyond individual capabilities. Through encryption technology, by incentivizing resource providers to participate in basic Open Source AI projects, this resource problem can be addressed. Combining Open Source AI with encryption technology can support the development of larger-scale models and drive more innovation, thus creating more advanced AI systems. Introduction According to a survey conducted by the Pew Research Center in 2024, 64% of Americans believe that the influence of social media on the country is more harmful than beneficial; 78% believe that social media companies have too much power and influence in politics; 83% believe that these platforms are likely to deliberately censor political views they disagree with. Dissatisfaction with social media has become almost a consensus in American society. Looking back over the development of social media in the past 20 years, this situation seems to have been predetermined. The story is not complicated: a few large technology companies have captured users' attention, more importantly, they have control over user data. Although people initially hoped for open data, these companies quickly changed their strategies, using data to establish unbreakable network effects and cutting off outside access. This ultimately led to the current situation: less than 10 large tech companies dominate the social media industry, forming an "Oligopoly monopoly" pattern. Due to the favorable nature of the current situation for them, these companies have little incentive to change. This model is closed and lacks competition. Today, the development trajectory of AI technology seems to be replaying this scene, but the impact this time is more profound. A few technology companies have built basic AI models by controlling GPU and data resources and have closed access to these models to the outside world. For newcomers without billions of dollars, developing a competitive model is almost impossible. Because the computing cost of just training a basic model requires billions of dollars, and those social media companies benefiting from the previous technological wave are using their control over proprietary user data to develop models that competitors find difficult to reach. We are repeating the mistakes of social media and heading towards a closed and uncompetitive AI world. If this trend continues, a few technology companies will have unrestricted control over information and opportunities. Open Source AI and the "resource problem" If we do not want to see a closed AI world, what are our choices? The obvious answer is to develop basic models as Open Source software projects. Historically, countless Open Source projects have successfully built the foundational software we rely on daily. For example, the success of Linux has proven that even core software like operating systems can be developed through Open Source. So why can't LLMs (large language models) be the same? However, the special restrictions faced by basic AI models make them different from traditional software, greatly reducing the feasibility of them as traditional Open Source projects. Specifically, basic AI models require huge computing and data resources, far beyond the scope of individuals. Unlike traditional Open Source projects that rely solely on people's donated time, Open Source AI also requires people to donate computing power and data resources, which is the so-called "resource problem." Taking Meta's LLaMa model as an example, we can better understand this resource problem. Unlike competitors such as OpenAI and Google, Meta has not hidden the model behind a paid API, but has publicly provided the weights of LLaMa for anyone to use for free (with certain limitations). These weights contain the knowledge learned by the model during Meta's training process and are essential for running the model. With these weights, users can fine-tune the model or use the model's output as input for a new model. While Meta's release of LLaMa weights is commendable, it cannot be considered a true Open Source software project. Meta controls the training process of the model behind the scenes, relying on its own computing resources, data, and decisions, and unilaterally decides when to open the model to the public. Meta has not invited independent researchers or developers to participate in community collaboration because the resources required for training or retraining the model far exceed the capabilities of ordinary individuals. These resources include tens of thousands of high-performance GPUs, data centers storing these GPUs, complex cooling facilities, and tens of billions of tokens (text data units required for model training). As pointed out in the 2024 Stanford University Artificial Intelligence Index report, "the sharp rise in training costs has effectively excluded top universities, traditionally the stronghold of AI research, from developing top basic models." For example, Sam Altman mentioned that the cost of training GPT-4 can be as high as $100 million, and this does not even include capital expenditures for hardware facilities. In addition, in the second quarter of 2024, Meta's capital expenditure increased by $2.1 billion compared to the same period in 2023, mainly for servers, data centers, and network infrastructure related to AI model training. Therefore, although the community contributors to LLaMa may have the technical ability to improve the model architecture, they lack sufficient resources to implement these improvements. In summary, unlike traditional Open Source software projects, Open Source AI projects not only require contributors to invest time but also require them to bear high computing and data costs. It is unrealistic to motivate enough resource providers solely through goodwill and volunteer spirit. They need further incentive mechanisms. Taking the Open Source large language model BLOOM as an example, this model with 176 billion parameters has gathered the efforts of 1,000 volunteer researchers from over 70 countries and more than 250 institutions. Although the success of BLOOM is admirable (I fully support this), it took a year to coordinate a training session and relied on a €3 million grant from a French research institution (excluding the capital expenditure for the Supercomputer used to train the model). Relying on a new round of funding to coordinate and iterate BLOOM is too cumbersome and cannot keep up with the development speed of large tech labs. It has been over two years since BLOOM was released, and there have been no reports of the team developing any follow-up models. To make Open Source AI possible, we need to find a way to incentivize resource providers to contribute their computing power and data resources, rather than having Open Source contributors bear these costs themselves. Why encryption technology can solve the "resource problem" of basic Open Source AI The core breakthrough of encryption technology lies in enabling high-resource-cost Open Source software projects through an "ownership" mechanism. It addresses the resource problem of Open Source AI by incentivizing potential resource providers to participate in the network, rather than requiring Open Source contributors to bear these resource costs upfront. BTC is a good example. As the earliest encryption project, BTC is a completely Open Source software project, and its code has been open from the beginning. However, the code itself is not the key to BTC. Simply downloading and running BTC Node software and creating a blockchain locally have no practical meaning. The true value of the software is only realized when the computing power required for Mining Blocks exceeds the capability of any single contributor: maintaining a Decentralized, uncontrollable ledger. Similar to basic Open Source AI, BTC is an Open Source project that requires resources beyond individual capabilities. Although the reasons for the computing resource requirements of the two are different—BTC requires computing resources to ensure the network is tamper-proof, while basic AI requires computing resources to optimize and iterate models—the commonality is that both require resources beyond individual capabilities. BTC, as well as any other encryption network, can incentivize participants to provide resources for Open Source software projects through the secret of ownership provided by Tokens. As Jesse stated in Variant's founding concept in 2020, ownership provides powerful motivation for resource providers to contribute resources in exchange for potential returns in the network. This mechanism is similar to startup companies solving early funding shortages through "Sweat Equity"—by primarily paying early employees (such as founders) in the form of company ownership, startups can attract labor that they could not otherwise afford. Encryption technology extends the concept of "Sweat Equity" from time contributors to resource providers. Therefore, Variant focuses on investing in projects that use ownership mechanisms to build network effects, such as Uniswap, Morpho, and World. If we want Open Source AI to become a reality, the ownership mechanism achieved through encryption technology is the key solution to the resource problem. This mechanism allows researchers to freely contribute their model design ideas to Open Source projects, as the computing and data resources required to implement these ideas will be borne by resource providers, and resource providers will receive fractional ownership of the project as a reward, rather than requiring researchers to bear the high upfront costs. In Open Source AI, ownership can take on various forms, but one of the most anticipated is ownership of the model itself, as proposed by Pluralis. This approach proposed by Pluralis is called Protocol Models. In this model, computing resource providers can train specific Open Source models by contributing computing power and, in return, receive fractional ownership of the model's future inference income. Because this ownership is tied to specific models and its value is based on the model's inference income, computing resource providers will be incentivized to train the best model and not falsify training data (as providing useless training will directly drop the expected value of future inference income). However, a key question is: How does Pluralis ensure the security of ownership if the training process requires sending the model's weights to the computing provider? The answer lies in using "Model Parallelism" technology to shard the model among different workers. An important feature of neural networks is that even if only a tiny part of the model weights is known, the worker can still participate in training, ensuring that the complete weight set cannot be extracted. In addition, because many different models are being trained simultaneously on the Pluralis platform, trainers will face a large number of different weight sets, making it extremely difficult to reconstruct the complete model. The core idea of Protocol Models is that these models can be trained and used but cannot be fully extracted from the protocol (unless the computing power used exceeds the resources needed to train the model from scratch). This mechanism addresses a common criticism from Open Source AI critics, that closed AI competitors may steal the labor results of Open Source projects. Why encryption technology + Open Source = better AI At the beginning of the article, by analyzing the control of AI by large tech companies, I highlighted the ethical and normative issues of closed AI. However, in an age of powerless networking, I am concerned that such arguments may be difficult to resonate with the majority of readers. Therefore, I want to present two reasons from the perspective of practical effects to explain why Open Source AI supported by encryption technology can truly bring about better AI. Firstly, the combination of encryption technology with Open Source AI can coordinate more resources, thus driving the development of the next generation of Foundation Models. Research has shown that both increasing computing power and data resources contribute to improving model performance, which is why the scale of foundation models has been continuously expanding. BTC has shown us the potential of Open Source software combined with encryption technology in computing power. It has become the world's largest and most powerful computing network, far exceeding the cloud computing resources owned by large tech companies. The uniqueness of encryption technology lies in its ability to transform isolated competition into collaborative competition. By incentivizing resource providers to contribute resources to solve common problems instead of working separately and duplicating efforts, encryption networks efficiently utilize resources. Open Source AI powered by encryption technology will be able to leverage global computing and data resources to build models on a scale far beyond closed AI. For example, Hyperbolic has demonstrated the potential of this model through an open market where anyone can rent GPUs at a lower cost, fully utilizing distributed computing resources. Secondly, the combination of encryption technology with Open Source AI will accelerate innovation. This is because once the resource problem is solved, machine learning research can return to its highly iterative and innovative Open Source essence. Before the emergence of Foundation Models such as LLM, researchers in the machine learning field typically openly released their models and their replicable design blueprints. These models usually use Open Source datasets, and the computing requirements are relatively low, allowing researchers to continuously optimize and innovate based on them. It is this open iterative process that has led to many breakthroughs in the field of sequence modeling, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and attention mechanisms, ultimately making the Transformer model architecture possible. However, this open research approach changed after the launch of GPT-3. OpenAI's success with GPT-3 and ChatGPT demonstrated that with sufficient computing resources and data inputs, language-capable large language models can be trained. This trend has led to a sharp rise in the resource threshold, excluding academia gradually, and large tech companies no longer publicly share their model architectures to maintain a competitive edge. This situation limits our ability to push the boundaries of AI technology. Open Source AI achieved through encryption technology can change this situation. It allows researchers to iterate on cutting-edge models again, discovering the "next Transformer." This combination not only solves the resource problem but also reactivates the innovation vitality of the machine learning field, opening up broader paths for the future development of AI.