Exploring Open-Source LLMs

The concept of AI democratization represents a shift in the landscape of technological innovation by making advanced AI technologies available to a wider base of users beyond the large tech corporations and specialized research institutions. This movement is essential in the realm of large language models because of the complexity and resource requirements of developing and training such large models. AI democratization seeks to empower individuals, startups, and academic researchers with the tools they need to explore, innovate, and contribute to the field of AI.

Open-source LLMs are at the heart of this movement serving as a catalyst for innovation and collaboration. By providing access to pretrained models and the source code, open-source initiatives encourage an environment where knowledge and resources are shared freely. This approach not only reduces the financial obstacles associated with AI development and training but also promotes transparency in AI research and applications. As a result, open-source LLMs facilitate AI advancements in many sectors, such as healthcare, education, and environmental science.

Open source refers to the practice of sharing the original source code of software with the public, allowing anyone to inspect the code, modify it, or enhance it. This is essential for collaboration purposes, whereby collaborative development can lead to more reliable, secure, and efficient products. What makes an open-source open is the license under which it is released, which dictates the usage, permissions, and distribution rights. Building open-source software or models means that resources such as the source code, model architecture, and documentation are available freely to the public. The open-source initiative emerged as a countermeasure to the constraints of proprietary software, encouraging freedom in the development and use of software and technology. This has led to the creation of communities and organizations that support open-source initiatives, such as the Apache Software Foundation.

On the other hand, closed-source LLMs are propriety models that can be accessed through online platforms or APIs. Neither the source code is shared with the wider audience, nor are the algorithms used revealed, nor is the training data that was utilized detailed. Closed-source LLMs are paid models either per token or as subscription-based models.

It is not a secret today that commercial LLM models, or in other words, closed-source models, are dominating the market. Only a handful of open-source models reach somewhere close to the quality of the commercial models, such as GPT models by OpenAI. Just look at the number of closed-source models and their performance in the market. However, the open-source community is trying hard to bridge this gap between open-source and closed-source models through projects that aggregate resources, such as datasets and online computing power, to train models that can compete with their commercial counterparts in terms of quality.

At first glance, these closed-source models seem cheap, costing not more than 0.001 dollars per token. However, as soon as we scale our applications in production, and as soon as we start having thousands of users, we realize that the cost of these models is huge.

Let’s find out the cost of utilizing closed-source models by calculating the number of users, the cost per token, and the pattern of usage per day. For example, we can take the cost per token for one of the premium models. On average, the input costs around 0.005 dollars per 1,000 tokens, and the output costs ...