DeepSeek-R1 is a powerful reasoning model developed by the Chinese AI research lab, DeepSeek. It has taken the world by surprise with its impressive capabilities which are comparable to those of OpenAI's ChatGPT-4, Anthropic’s Claude, and Google’s Gemini. This is particularly impressive because DeepSeek is believed to have been developed without the most advanced AI chips available to its American competitors¹.

Unlike most other commercial AI research labs, DeepSeek has open-sourced its models, which makes the source code freely available for anyone to use, modify, and share - including for commercial purposes. The open-source nature of this project begs the question: Can DeepSeek be used as a teaching model to train other student models? If so, what are the implications of this readily available and cost-effective technology?

Let’s start by examining some of the significant challenges and misconceptions facing the widespread adoption of DeepSeek-R1. Then we’ll delve into the more promising aspects of open-source AI - striving for a balanced approach to assess the current state of this powerful technology.

Weak Safety Guardrails

Reporting has emerged from credible sources such as Cisco Systems and the University of Pennsylvania² contending DeepSeek-R1 exhibits weak safety guardrails as compared to leading closed-source LLMs (Large Language Models), raising serious concerns about its security and potential for misuse.

If you are considering deploying DeepSeek-R1 or a distilled model derived from it (as discussed later in this article), please be aware²:

• DeepSeek-R1 exhibited a 100% attack success rate when tested against harmful prompts from the HarmBench dataset.
• It failed to block a single harmful prompt across categories including cybercrime, misinformation, illegal activities, and general harm.
• This performance contrasts sharply with other leading models that demonstrated at least partial resistance to such attacks.
• DeepSeek-R1's claimed cost-efficient training methods, including reinforcement learning and chain-of-thought self-evaluation, may have compromised its safety mechanisms.

Industry Response to Weak Guardrails

In light of these security concerns, major cloud providers are implementing additional safeguards:

• Amazon Web Services (AWS) is offering Amazon Bedrock Guardrails to provide configurable safeguards for DeepSeek-R1 deployments.
• Microsoft is implementing security measures for DeepSeek-R1 on Azure AI Foundry, including rigorous red teaming, safety evaluations, and built-in content filtering.

These findings highlight the critical importance of robust guardrails and security measures in LLM development and deployment, especially as these models become more powerful and widely used.

DeepSeek Training Cost Controversy

DeepSeek initially claimed that training R1 cost a mere $6 million. To put this in context, the leading AI models from American competitors cost hundreds of millions and sometimes even billions of dollars to train.

Understandably, DeepSeek’s initial claim of around $6 million, while attention-grabbing, has been met with skepticism from industry analysts⁹. The $6 million likely represents only a portion of the total cost, specifically the GPU time for pre-training. It fails to account for many other essential expenses such as:

• Research and Development
• Data Acquisition and Preparation
• Personnel Costs
• Infrastructure Costs

A more realistic estimate of DeepSeek's total investment in AI development is around $1.6 billion. This figure encompasses the cost of hardware, software, data, personnel, and research. While significantly higher than the initial claim, this figure is still lower than the investments made by some American competitors^{9, 10}.

Efficient Open-Source Engineering

While DeepSeek’s initial claim of ultra-low-cost training was likely exaggerated for marketing purposes, it is evident that the AI firm has legitimately made significant strides in optimizing both architecture and training methods to reduce costs. These innovations have the potential to disrupt the AI industry, putting pressure on American companies to find new ways to improve efficiency and reduce the expenses associated with training large language models.

DeepSeek-R1’s efficiency and performance stems from several important engineering decisions:

• It utilizes a decoder-only transformer architecture with multi-head latent attention³.
• DeepSeek-R1 combines chain-of-thought reasoning with reinforcement learning, where an autonomous agent learns to perform a task through trial and error without human instruction¹.
• R1 uses a mixture of experts (MoE) architecture, which is less resource-intensive to train. The MoE architecture divides an AI model into separate entities or subnetworks, each specializing in a subset of the input data⁴. Then the model only activates the specific experts needed for a given task, making it more efficient¹.

AI Model Distillation: A Primer

Instead of training a smaller model from scratch, model distillation offers a far more efficient approach by transferring knowledge from a larger, more complex model (the "teacher") to a smaller model (the "student"). The goal is to achieve comparable performance with the smaller model while reducing computational costs and latency⁵. If done correctly, this knowledge transfer does not lead to a loss of validity in the student model⁶.

The process involves generating a dataset where the teacher model provides outputs for a wide range of inputs. This dataset captures the teacher's behavior and decision-making patterns. The student model is then fine-tuned using this dataset, learning to mimic the teacher's responses. Techniques like temperature scaling are often employed to soften the output probabilities of the teacher, making it easier for the student to learn nuanced patterns⁵.

There are different types of model distillation, each with its own approach to knowledge transfer:

• Response-Based Distillation: The student model focuses on mimicking the teacher's predictions⁴.
• Feature-Based Distillation: The student model learns the internal features or representations learned by the teacher⁴.
• Relation-Based Distillation: The student model learns to understand the relationships between inputs and outputs⁴.

The choice of distillation process depends on the specific task and the desired outcome. Additionally, there are different training methods in model distillation, including offline distillation, where the student model learns from a static dataset generated by the teacher, and online distillation, where the student learns interactively from the teacher during training⁷.

DeepSeek as a Teaching Model

Given its open-source nature and impressive capabilities, DeepSeek is a strong contender to serve as a teaching model. Its comprehensive architecture and ability to perform complex reasoning tasks make it ideal for transferring knowledge to smaller, more specialized models.

Researchers and developers can leverage DeepSeek's open-source code and pre-trained weights to create datasets for distilling knowledge into student models. This can be achieved through various techniques, including response-based distillation, where the student model learns to mimic DeepSeek's outputs, or feature-based distillation, where the student model learns the internal representations of DeepSeek.

The availability of DeepSeek's architecture and training details allow for a deeper understanding of its inner workings, enabling developers to fine-tune student models more effectively. This can lead to the development of specialized models that excel in specific domains while maintaining efficiency and accuracy⁸.

Furthermore, using DeepSeek as a teaching model aligns with the broader movement towards transparency and wider participation in AI development⁹. By making its models open and accessible, DeepSeek encourages a collaborative approach to AI innovation, allowing developers and researchers to learn from and build upon its advancements.

Business Implications of Less Expensive Model Building

The open-sourcing of DeepSeek and the subsequent potential for less expensive model building have significant business implications:

• Reduced Development Costs: Distilling knowledge from DeepSeek can significantly reduce the cost of developing new AI models. This is particularly beneficial for startups and smaller companies that may not have the resources to train large models from scratch.
• Faster Time-to-Market: With reduced development costs and time, businesses can bring AI-powered products and services to market faster, gaining a competitive edge.
• Increased Accessibility: Less expensive model building makes AI technology more accessible to a wider range of businesses and organizations, democratizing access to advanced AI capabilities¹¹.
• Enhanced Customization: Open-source models like DeepSeek allow for greater customization, enabling businesses to tailor AI solutions to their specific needs and industry requirements. This is a key advantage over closed models, which often offer limited flexibility⁸.
• Innovation and Growth: The availability of cost-effective AI models can foster innovation and drive the development of new applications across various industries¹¹.

However, there are also potential challenges:

• Competition: The proliferation of AI models could lead to increased competition, potentially impacting the profitability of existing AI providers.
• Security Risks: Open-source models could be vulnerable to exploitation by malicious actors, requiring robust security measures to mitigate potential risks¹².
• Ethical Concerns: The widespread use of AI models raises ethical concerns, such as bias and fairness, that need to be addressed through responsible development and deployment practices¹⁵.

Exponential Proliferation of Specialty Models

With the availability of DeepSeek and other open-source models, we will likely see an exponential proliferation of new specialty models. The reduced cost and increased accessibility of model-building technology will empower developers to create AI solutions tailored to specific domains and use cases.

This proliferation will likely lead to a surge in AI applications across various industries, including healthcare, finance, manufacturing, and more. We can expect to see specialized models for tasks such as medical diagnosis, fraud detection, customer service, and personalized education.

The open-source nature of these models will also foster collaboration and knowledge sharing, accelerating the pace of innovation in the AI field. This collaborative environment will drive the development of more sophisticated and effective AI solutions, addressing a wider range of challenges and opportunities.

This proliferation of models is not just about quantity; it's about a fundamental shift in how we approach technological discovery. Openness in this process is key to surviving threats and ensuring that power dispersion is necessary for technological progress¹⁶. This democratization of AI development has the potential to unlock new levels of innovation and problem-solving, leading to solutions that benefit a wider range of individuals and communities.

Conclusion

The release of DeepSeek-R1 as an open-source model marks a significant milestone in the evolution of artificial intelligence. Its potential to serve as a teaching model for distillation, coupled with the reduced cost of model building, will undoubtedly lead to an exponential proliferation of new specialty models. This will have profound implications for businesses, industries, and society as a whole, driving innovation, growth, and the democratization of AI technology.

This shift towards open-source AI has the potential to reshape the AI landscape, fostering greater collaboration, transparency, and accessibility. It could lead to a more diverse and inclusive AI ecosystem, where innovation is driven by a global community of developers and researchers. However, it is crucial to address the potential challenges and ethical concerns associated with this proliferation to ensure responsible and beneficial AI development and deployment.

Thank you for reading and I would love to hear your thoughts about DeepSeek and open-source AI on Bluesky: @benjaminpatch.com. Until next time, take care!

Works Cited

1. What is DeepSeek? AI Model Basics Explained - YouTube, accessed February 13, 2025, https://www.youtube.com/watch?v=KTonvXhsxpc
2. Evaluating Security Risks in DeepSeek and Other Frontier Reasoning Models - Cisco Systems, accessed February 13, 2025, https://blogs.cisco.com/security/evaluating-security-risk-in-deepseek-and-other-frontier-reasoning-models
3. DeepSeek - Wikipedia, accessed February 13, 2025, https://en.wikipedia.org/wiki/DeepSeek
4. A pragmatic introduction to model distillation for AI developers - Labelbox, accessed February 13, 2025, https://labelbox.com/blog/a-pragmatic-introduction-to-model-distillation-for-ai-developers/
5. Model Distillation - Humanloop, accessed February 13, 2025, https://humanloop.com/blog/model-distillation
6. Knowledge distillation - Wikipedia, accessed February 13, 2025, https://en.wikipedia.org/wiki/Knowledge_distillation
7. What is Model Distillation? - Labelbox, accessed February 13, 2025, https://labelbox.com/guides/model-distillation/
8. How Open-Source Generative AI Models Affect Applications In Vertical Markets - Forbes, accessed February 13, 2025, https://www.forbes.com/councils/forbestechcouncil/2024/10/08/how-open-source-generative-ai-models-affect-applications-in-vertical-markets/
9. DeepSeek’s $6 Million AI Claim Debunked: True Costs Revealed - PC Outlet, accessed February 13, 2025, https://pcoutlet.com/software/ai/deepseeks-6-million-ai-claim-exposed-as-myth-true-costs-revealed
10. DeepSeek might not be as disruptive as claimed, firm reportedly has 50,000 Nvidia GPUs and spent $1.6 billion on buildouts - Tom’s Hardware, accessed February 13, 2025, https://www.tomshardware.com/tech-industry/artificial-intelligence/deepseek-might-not-be-as-disruptive-as-claimed-firm-reportedly-has-50-000-nvidia-gpus-and-spent-usd1-6-billion-on-buildouts
11. Open Source AI Models: Coding Outside the Proprietary Box - Neil Sahota, accessed February 13, 2025, https://www.neilsahota.com/open-source-ai-models-coding-outside-the-proprietary-box/
12. Open-Source AI — Challenges, Opportunities & Ecosystem | by Abel Samot - Medium, accessed February 13, 2025, https://medium.com/red-river-west/open-source-ai-mapping-advantages-debate-dd6be433eff6
13. Risks and Opportunities of Open-Source Generative AI - arXiv, accessed February 13, 2025, https://arxiv.org/html/2405.08597v1
14. With Open Source Artificial Intelligence, Don't Forget the Lessons of Open Source Software, accessed February 13, 2025, https://www.cisa.gov/news-events/news/open-source-artificial-intelligence-dont-forget-lessons-open-source-software
15. Why open-source is crucial for responsible AI development - The World Economic Forum, accessed February 13, 2025, https://www.weforum.org/stories/2023/12/ai-regulation-open-source/
16. Surviving a technological future: Technological proliferation and modes of discovery - PMC, accessed February 13, 2025, https://pmc.ncbi.nlm.nih.gov/articles/PMC7094529/