The development of current basic AI is dominated by a few technology companies, presenting characteristics of closedness and lack of competition.
Although open source software development is a potential solution, basic AI cannot operate like traditional open source projects (such as Linux) because it faces a 'resource problem': open source contributors not only need to invest time, but also bear computing and data costs beyond their personal capabilities.
Encryption technology incentivizes resource providers to participate in basic open-source AI projects, offering a potential solution to this resource issue.
Combining open-source AI with cryptographic technology can support larger-scale model development and drive more innovation, thus creating more advanced AI systems.
Introduction
According to a 2024 survey by the Pew Research Center, 64% of Americans believe that the impact of social media on the country is more harmful than beneficial; 78% of people believe that social media companies have too much power and influence in politics; 83% of people believe that these platforms are likely to deliberately censor political views that they disagree with. Dissatisfaction with social media has become one of the few consensuses in American society.
Looking back at the development of social media over the past 20 years, this situation seems to have been predetermined. The story is not complicated: a few large technology companies have seized the attention of users, and more importantly, have control over user data. Although people initially had hope for open data, these companies quickly changed their strategies, using data to establish unbreakable network effects and closing off external access. This ultimately led to the current situation: less than 10 large technology companies dominate the social media industry, forming an oligopoly. As the current situation is extremely advantageous to them, these companies have almost no incentive to change. This model is closed and lacks competition.
Today, the development trajectory of AI technology seems to be replaying this scene, but the impact this time is more profound. A few technology companies have built basic AI models by controlling GPU and data resources, and have closed access to these models. For newcomers without billions of dollars in funding, developing a competitive model is almost impossible. Because the cost of training a basic model alone requires billions of dollars, social media companies that have benefited from the last wave of technology are using their control over proprietary user data to develop models that competitors cannot reach. We are repeating the mistakes of social media and moving towards a closed and uncompetitive AI world. If this trend continues, a few technology companies will have unrestricted control over access to information and opportunities.
Open Source AI and the "Resource Issue"
If we don't want to see a closed AI world, what are our choices? The obvious answer is to develop the base model as an open source software project. Historically, we have had countless open source projects successfully build the foundational software we rely on every day. For example, the success of Linux has proven that even core software like operating systems can be developed in an open source manner. So, why not LLMs (large language models)?
However, the special constraints faced by basic AI models make them different from traditional software, which greatly undermines their feasibility as traditional open source projects. Specifically, basic AI models require huge computing and data resources, far beyond the capabilities of individuals. Unlike traditional open source projects that rely solely on people donating time, open source AI also requires people to donate computing power and data resources, which is the so-called "resource problem".
Using Meta's LLaMa model as an example, we can better understand this resource issue. Unlike competitors such as OpenAI and Google, Meta does not hide the model behind a paid API; instead, it openly provides the weights of LLaMa for anyone to use for free (with some limitations). These weights contain the knowledge the model learned during Meta's training process and are essential for running the model. With these weights, users can fine-tune the model or use the model's output as input for a new model.
Although Meta has released the weight of LLaMa, it cannot be considered a truly open-source software project. Meta controls the training process of the model behind the scenes, relying on its own computing resources, data, and decisions, and unilaterally deciding when to open the model to the public. Meta did not invite independent researchers or developers to participate in community cooperation, because the resources needed for training or retraining the model far exceed the capabilities of ordinary individuals. These resources include tens of thousands of high-performance GPUs, data centers to store these GPUs, complex cooling facilities, and tens of trillions of tokens for training (text data units required for model training). As pointed out in the 2024 Stanford University Artificial Intelligence Index report, 'the sharp rise in training costs has effectively excluded traditional AI research powerhouses from top basic model development.' For example, Sam Altman mentioned that the cost of training GPT-4 was as high as 100 million U.S. dollars, and this does not even include capital expenditures for hardware facilities. In addition, Meta's capital expenditures in the second quarter of 2024 increased by $2.1 billion compared to the same period in 2023, mainly for servers, data centers, and network infrastructure related to AI model training. Therefore, although LLaMa's community contributors may have the technical capability to improve the model architecture, they lack sufficient resources to implement these improvements.
In summary, unlike traditional open-source software projects, open-source AI projects not only require contributors to invest time, but also require them to bear high computing and data costs. It is unrealistic to rely solely on goodwill and volunteer spirit to motivate enough resource providers. They need further incentive mechanisms. Taking the open-source large language model BLOOM as an example, this model with 1760 billion parameters has brought together the efforts of 1000 volunteer researchers from more than 70 countries and over 250 institutions. Although BLOOM's success is admirable (I fully support this), it took a year to coordinate one training session and relied on 3 million euros in funding from a French research institution (not including the capital expenditure for the supercomputer used to train the model). Relying on a new round of funding to coordinate and iterate BLOOM is too cumbersome and cannot keep pace with the development speed of large tech labs. It has been over two years since BLOOM was released, and there is currently no news of the team developing any follow-up models.
In order to make open source AI possible, we need to find a way to incentivize resource providers to contribute their computing power and data resources, instead of letting open source contributors bear these costs themselves.
Why can encryption technology solve the "resource problem" of basic open-source AI
The core breakthrough of encryption technology lies in the 'ownership' mechanism, which makes it possible for high-resource-cost open source software projects. It solves the resource problem of open source AI by incentivizing potential resource providers to participate in the network, instead of having open source contributors bear these resource costs in advance.
Bitcoin is a great example. As the earliest cryptocurrency project, Bitcoin is a completely open-source software project, and its code has been open from the beginning. However, the code itself is not the key to Bitcoin. Simply downloading and running Bitcoin node software to create a blockchain locally has no practical meaning. The true value of this software can only be reflected when the computational power of mining blocks exceeds the ability of any single contributor: maintaining a decentralized and uncontrolled ledger. Similar to open-source AI, Bitcoin is also an open-source project that requires resources beyond individual capabilities. Although the reasons for the computational resource requirements of the two are different - Bitcoin requires computational resources to ensure the network is tamper-proof, while basic AI requires computational resources to optimize and iterate models - they both rely on resources that surpass individual capabilities.
Bitcoin, as well as any other cryptocurrency network, has the secret of incentivizing participants to provide resources for open-source software projects through tokens that provide network ownership. As Jesse described in the founding idea he wrote for Variant in 2020, ownership provides powerful motivation for resource providers to contribute resources in exchange for potential returns in the network. This mechanism is similar to how startups solve early funding problems through 'sweat equity' - by compensating early employees (such as founders) primarily in the form of company ownership, startups can attract labor that they would not have been able to afford otherwise. Cryptocurrency technology extends the concept of 'sweat equity' from time contributors to resource providers. Therefore, Variant focuses on investing in projects that utilize ownership mechanisms to build network effects, such as Uniswap, Morpho, and World.
If we want open-source AI to become a reality, the ownership mechanism implemented through encryption technology is the key solution to address the resource issue. This mechanism allows researchers to freely contribute their model design ideas to open-source projects, as the computation and data resources required to realize these ideas will be provided by resource providers who will receive partial ownership of the project as compensation, instead of requiring researchers to bear the high upfront costs themselves. In open-source AI, ownership can take various forms, but one of the most anticipated forms is ownership of the model itself, which is also the solution proposed by Pluralis.
The method proposed by Pluralis is called Protocol Models. In this mode, computational resource providers can contribute computing power to train specific open-source models and therefore obtain partial ownership of the future inference income of the model. Because this ownership is tied to specific models and its value is based on the inference income of the model, computational resource providers will be incentivized to choose the best model for training, and not falsify training data (as providing useless training will directly reduce the expected value of future inference income). However, a key issue is: how does Pluralis ensure the security of ownership if the training process requires sending the model's weights to the computation provider? The answer lies in using "Model Parallelism" technology to distribute model slices to different workers. An important feature of neural networks is that even with only a small part of the model weights, the trainer can still participate in the training, ensuring that the complete weight set cannot be extracted. In addition, because many different models are trained simultaneously on the Pluralis platform, trainers will face a large number of different weight sets, making it extremely difficult to reconstruct the complete model.
The core idea of Protocol Models is that these models can be trained and used, but cannot be fully extracted from the protocol (unless the computing power used exceeds the resources required to train the model from scratch). This mechanism addresses a common criticism from open-source AI critics, that closed AI competitors may misappropriate the labor of open-source projects.
Why does cryptography + open source = better AI
At the beginning of the article, I analyzed the control of AI by major technology companies and explained the ethical and normative issues of closed AI. However, in an era of powerlessness, I am concerned that this argument may be difficult to resonate with most readers. Therefore, I want to start with the actual effects and present two reasons why open-source AI supported by encryption technology can truly bring better AI.
First, the combination of encryption technology and open-source AI can coordinate more resources to promote the development of the next generation of Foundation Models. Research shows that increasing both computing power and data resources contributes to improving model performance, which is why the scale of Foundation Models has been continuously expanding. Bitcoin has demonstrated the potential of open-source software combined with encryption technology in computing power. It has become the world's largest and most powerful computing network, far surpassing the cloud computing resources owned by major tech companies. The unique aspect of encryption technology is its ability to transform isolated competition into collaborative competition. By incentivizing resource providers to contribute resources to solve common problems, instead of working separately and duplicating efforts, the encryption network achieves efficient resource utilization. Leveraging open-source AI with encryption technology will enable the utilization of global computing and data resources to build models that far exceed closed AI systems in scale. For example, Hyperbolic has already demonstrated the potential of this model. They have enabled anyone to rent GPUs at a lower cost through an open market, fully utilizing distributed computing resources.
Secondly, the combination of encryption technology and open-source AI will drive the acceleration of innovation. This is because once the resource problem is solved, machine learning research can return to its highly iterative and innovative open-source nature. Before the emergence of large language models (LLM), researchers in the field of machine learning would usually publicly release their models and their replicable design blueprints. These models typically use open-source datasets and have relatively low computational requirements, allowing researchers to continuously optimize and innovate on them. It is this open iterative process that has led to many breakthroughs in the field of sequence modeling, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), and attention mechanisms, ultimately making the Transformer model architecture possible. However, this open research approach has changed after the launch of GPT-3. OpenAI has demonstrated through the success of GPT-3 and ChatGPT that with sufficient computational resources and data, it is possible to train large language models with language understanding capabilities. This trend has led to a sharp increase in the resource threshold, gradually excluding the academic community, and large tech companies no longer openly share their model architectures in order to maintain a competitive advantage. This situation limits our ability to advance AI's cutting-edge technology.
Open-source AI enabled by encryption technology can change this situation. It allows researchers to iterate on cutting-edge models again, thereby discovering the "next Transformer." This combination not only solves resource issues but also reactivates innovation in the field of machine learning, paving the way for a broader future for AI.
The content is for reference only, not a solicitation or offer. No investment, tax, or legal advice provided. See Disclaimer for more risks disclosure.
Variant Investment Partners: The dilemma and breakthrough of Open Source AI, why encryption technology is the last piece of the puzzle?
Author: Daniel Barabander
Compile: DeepTech TechFlow
Brief summary
The development of current basic AI is dominated by a few technology companies, presenting characteristics of closedness and lack of competition.
Although open source software development is a potential solution, basic AI cannot operate like traditional open source projects (such as Linux) because it faces a 'resource problem': open source contributors not only need to invest time, but also bear computing and data costs beyond their personal capabilities.
Encryption technology incentivizes resource providers to participate in basic open-source AI projects, offering a potential solution to this resource issue.
Combining open-source AI with cryptographic technology can support larger-scale model development and drive more innovation, thus creating more advanced AI systems.
Introduction
According to a 2024 survey by the Pew Research Center, 64% of Americans believe that the impact of social media on the country is more harmful than beneficial; 78% of people believe that social media companies have too much power and influence in politics; 83% of people believe that these platforms are likely to deliberately censor political views that they disagree with. Dissatisfaction with social media has become one of the few consensuses in American society.
Looking back at the development of social media over the past 20 years, this situation seems to have been predetermined. The story is not complicated: a few large technology companies have seized the attention of users, and more importantly, have control over user data. Although people initially had hope for open data, these companies quickly changed their strategies, using data to establish unbreakable network effects and closing off external access. This ultimately led to the current situation: less than 10 large technology companies dominate the social media industry, forming an oligopoly. As the current situation is extremely advantageous to them, these companies have almost no incentive to change. This model is closed and lacks competition.
Today, the development trajectory of AI technology seems to be replaying this scene, but the impact this time is more profound. A few technology companies have built basic AI models by controlling GPU and data resources, and have closed access to these models. For newcomers without billions of dollars in funding, developing a competitive model is almost impossible. Because the cost of training a basic model alone requires billions of dollars, social media companies that have benefited from the last wave of technology are using their control over proprietary user data to develop models that competitors cannot reach. We are repeating the mistakes of social media and moving towards a closed and uncompetitive AI world. If this trend continues, a few technology companies will have unrestricted control over access to information and opportunities.
Open Source AI and the "Resource Issue"
If we don't want to see a closed AI world, what are our choices? The obvious answer is to develop the base model as an open source software project. Historically, we have had countless open source projects successfully build the foundational software we rely on every day. For example, the success of Linux has proven that even core software like operating systems can be developed in an open source manner. So, why not LLMs (large language models)?
However, the special constraints faced by basic AI models make them different from traditional software, which greatly undermines their feasibility as traditional open source projects. Specifically, basic AI models require huge computing and data resources, far beyond the capabilities of individuals. Unlike traditional open source projects that rely solely on people donating time, open source AI also requires people to donate computing power and data resources, which is the so-called "resource problem".
Using Meta's LLaMa model as an example, we can better understand this resource issue. Unlike competitors such as OpenAI and Google, Meta does not hide the model behind a paid API; instead, it openly provides the weights of LLaMa for anyone to use for free (with some limitations). These weights contain the knowledge the model learned during Meta's training process and are essential for running the model. With these weights, users can fine-tune the model or use the model's output as input for a new model.
Although Meta has released the weight of LLaMa, it cannot be considered a truly open-source software project. Meta controls the training process of the model behind the scenes, relying on its own computing resources, data, and decisions, and unilaterally deciding when to open the model to the public. Meta did not invite independent researchers or developers to participate in community cooperation, because the resources needed for training or retraining the model far exceed the capabilities of ordinary individuals. These resources include tens of thousands of high-performance GPUs, data centers to store these GPUs, complex cooling facilities, and tens of trillions of tokens for training (text data units required for model training). As pointed out in the 2024 Stanford University Artificial Intelligence Index report, 'the sharp rise in training costs has effectively excluded traditional AI research powerhouses from top basic model development.' For example, Sam Altman mentioned that the cost of training GPT-4 was as high as 100 million U.S. dollars, and this does not even include capital expenditures for hardware facilities. In addition, Meta's capital expenditures in the second quarter of 2024 increased by $2.1 billion compared to the same period in 2023, mainly for servers, data centers, and network infrastructure related to AI model training. Therefore, although LLaMa's community contributors may have the technical capability to improve the model architecture, they lack sufficient resources to implement these improvements.
In summary, unlike traditional open-source software projects, open-source AI projects not only require contributors to invest time, but also require them to bear high computing and data costs. It is unrealistic to rely solely on goodwill and volunteer spirit to motivate enough resource providers. They need further incentive mechanisms. Taking the open-source large language model BLOOM as an example, this model with 1760 billion parameters has brought together the efforts of 1000 volunteer researchers from more than 70 countries and over 250 institutions. Although BLOOM's success is admirable (I fully support this), it took a year to coordinate one training session and relied on 3 million euros in funding from a French research institution (not including the capital expenditure for the supercomputer used to train the model). Relying on a new round of funding to coordinate and iterate BLOOM is too cumbersome and cannot keep pace with the development speed of large tech labs. It has been over two years since BLOOM was released, and there is currently no news of the team developing any follow-up models.
In order to make open source AI possible, we need to find a way to incentivize resource providers to contribute their computing power and data resources, instead of letting open source contributors bear these costs themselves.
Why can encryption technology solve the "resource problem" of basic open-source AI
The core breakthrough of encryption technology lies in the 'ownership' mechanism, which makes it possible for high-resource-cost open source software projects. It solves the resource problem of open source AI by incentivizing potential resource providers to participate in the network, instead of having open source contributors bear these resource costs in advance.
Bitcoin is a great example. As the earliest cryptocurrency project, Bitcoin is a completely open-source software project, and its code has been open from the beginning. However, the code itself is not the key to Bitcoin. Simply downloading and running Bitcoin node software to create a blockchain locally has no practical meaning. The true value of this software can only be reflected when the computational power of mining blocks exceeds the ability of any single contributor: maintaining a decentralized and uncontrolled ledger. Similar to open-source AI, Bitcoin is also an open-source project that requires resources beyond individual capabilities. Although the reasons for the computational resource requirements of the two are different - Bitcoin requires computational resources to ensure the network is tamper-proof, while basic AI requires computational resources to optimize and iterate models - they both rely on resources that surpass individual capabilities.
Bitcoin, as well as any other cryptocurrency network, has the secret of incentivizing participants to provide resources for open-source software projects through tokens that provide network ownership. As Jesse described in the founding idea he wrote for Variant in 2020, ownership provides powerful motivation for resource providers to contribute resources in exchange for potential returns in the network. This mechanism is similar to how startups solve early funding problems through 'sweat equity' - by compensating early employees (such as founders) primarily in the form of company ownership, startups can attract labor that they would not have been able to afford otherwise. Cryptocurrency technology extends the concept of 'sweat equity' from time contributors to resource providers. Therefore, Variant focuses on investing in projects that utilize ownership mechanisms to build network effects, such as Uniswap, Morpho, and World.
If we want open-source AI to become a reality, the ownership mechanism implemented through encryption technology is the key solution to address the resource issue. This mechanism allows researchers to freely contribute their model design ideas to open-source projects, as the computation and data resources required to realize these ideas will be provided by resource providers who will receive partial ownership of the project as compensation, instead of requiring researchers to bear the high upfront costs themselves. In open-source AI, ownership can take various forms, but one of the most anticipated forms is ownership of the model itself, which is also the solution proposed by Pluralis.
The method proposed by Pluralis is called Protocol Models. In this mode, computational resource providers can contribute computing power to train specific open-source models and therefore obtain partial ownership of the future inference income of the model. Because this ownership is tied to specific models and its value is based on the inference income of the model, computational resource providers will be incentivized to choose the best model for training, and not falsify training data (as providing useless training will directly reduce the expected value of future inference income). However, a key issue is: how does Pluralis ensure the security of ownership if the training process requires sending the model's weights to the computation provider? The answer lies in using "Model Parallelism" technology to distribute model slices to different workers. An important feature of neural networks is that even with only a small part of the model weights, the trainer can still participate in the training, ensuring that the complete weight set cannot be extracted. In addition, because many different models are trained simultaneously on the Pluralis platform, trainers will face a large number of different weight sets, making it extremely difficult to reconstruct the complete model.
The core idea of Protocol Models is that these models can be trained and used, but cannot be fully extracted from the protocol (unless the computing power used exceeds the resources required to train the model from scratch). This mechanism addresses a common criticism from open-source AI critics, that closed AI competitors may misappropriate the labor of open-source projects.
Why does cryptography + open source = better AI
At the beginning of the article, I analyzed the control of AI by major technology companies and explained the ethical and normative issues of closed AI. However, in an era of powerlessness, I am concerned that this argument may be difficult to resonate with most readers. Therefore, I want to start with the actual effects and present two reasons why open-source AI supported by encryption technology can truly bring better AI.
First, the combination of encryption technology and open-source AI can coordinate more resources to promote the development of the next generation of Foundation Models. Research shows that increasing both computing power and data resources contributes to improving model performance, which is why the scale of Foundation Models has been continuously expanding. Bitcoin has demonstrated the potential of open-source software combined with encryption technology in computing power. It has become the world's largest and most powerful computing network, far surpassing the cloud computing resources owned by major tech companies. The unique aspect of encryption technology is its ability to transform isolated competition into collaborative competition. By incentivizing resource providers to contribute resources to solve common problems, instead of working separately and duplicating efforts, the encryption network achieves efficient resource utilization. Leveraging open-source AI with encryption technology will enable the utilization of global computing and data resources to build models that far exceed closed AI systems in scale. For example, Hyperbolic has already demonstrated the potential of this model. They have enabled anyone to rent GPUs at a lower cost through an open market, fully utilizing distributed computing resources.
Secondly, the combination of encryption technology and open-source AI will drive the acceleration of innovation. This is because once the resource problem is solved, machine learning research can return to its highly iterative and innovative open-source nature. Before the emergence of large language models (LLM), researchers in the field of machine learning would usually publicly release their models and their replicable design blueprints. These models typically use open-source datasets and have relatively low computational requirements, allowing researchers to continuously optimize and innovate on them. It is this open iterative process that has led to many breakthroughs in the field of sequence modeling, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), and attention mechanisms, ultimately making the Transformer model architecture possible. However, this open research approach has changed after the launch of GPT-3. OpenAI has demonstrated through the success of GPT-3 and ChatGPT that with sufficient computational resources and data, it is possible to train large language models with language understanding capabilities. This trend has led to a sharp increase in the resource threshold, gradually excluding the academic community, and large tech companies no longer openly share their model architectures in order to maintain a competitive advantage. This situation limits our ability to advance AI's cutting-edge technology.
Open-source AI enabled by encryption technology can change this situation. It allows researchers to iterate on cutting-edge models again, thereby discovering the "next Transformer." This combination not only solves resource issues but also reactivates innovation in the field of machine learning, paving the way for a broader future for AI.