Downloading an AI model and running it on your own servers sounds like the safest option. Your data stays in-house, and no outside provider controls the model. But according to new guidance from Australia, the moment you take that model in, you also take on every risk that comes with it. ASD’s ACSC has released two publications on open-weight AI models, warning that the control they offer comes with hidden behaviours, tampered files and safeguards that can quietly disappear.
The first, Open-weight AI models: Opportunities and cyber security risks, explains what these models are and the risks that come from publicly releasing model weights. The second, Selecting, deploying and operating open-weight AI models securely, gives practical guidance on governance, model selection, deployment, monitoring, maintenance and incident response.
What Makes Open-Weight AI Models Different
ASD’s ACSC sorts AI models into three groups:
- Closed models, such as Mythos and Opus, are accessed as a service. The weights never leave the provider, which runs the safeguards and can update or withdraw the model.
- Open-weight models, such as Kimi K3, GLM5.3 and the Nemotron 3 family, publish their trained parameters for download, usually without training data or code. Anyone can run, fine-tune and redistribute them, subject to licence terms.
- Open-source models, such as the Olmo 3 family, also publish training data, code and recipes. The guidance notes that genuinely open-source releases remain rare.
The agency says recent open-weight releases have performed close to leading closed models on selected benchmarks, though not all vendor-reported results may have been independently verified. It warns that current and emerging models may be capable of supporting vulnerability discovery, autonomous multi-step operations, software supply chain compromise and sophisticated social engineering. That makes them of growing interest to defenders and malicious actors alike.
Why Organisations Want Them
The benefits are real, according to the guidance. Self-hosting can keep sensitive data local, reduce dependence on offshore providers, support sovereign research and help defensive cyber security work. Organisations can run models in disconnected environments, choose when to update and avoid relying on one provider’s policies or commercial decisions.
But self-hosting also shifts responsibility for security, monitoring and incident response onto the organisation. The agency advises weighing the total cost of ownership, including infrastructure, energy, security controls and skilled staff.
The Hidden Risks of Open-Weight AI Models
The risks exist because released weights can be copied, modified and redistributed by anyone, without the original developer’s oversight.
Unclear origins. Models may come from third parties that have converted, quantised or fine-tuned them, and documentation may be incomplete or wrong. ASD’s ACSC says organisations should not adopt a model whose model provenance cannot be established to a level appropriate for its use.
Tampered files. Like any software, these models carry supply chain risks. Some model formats can execute code, and tampered weights or manipulated datasets may introduce backdoors that routine testing is unlikely to catch. Some hidden behaviours may only appear when the model responds to particular identities, organisations, jurisdictions or topics.
Leaking fine-tuned data. Models fine-tuned on an organisation’s own data may memorise and reveal it. If the files are stolen, attackers could extract that information.
Disappearing safeguards. Built-in safeguards can be weakened or removed through fine-tuning or modification. That raises the chance of successful prompt injection and misuse of connected tools. The risk grows in agentic systems, where a model can act on its own, because a failure can lead to unauthorised actions rather than just unsafe text.
What ASD’s ACSC Recommends
The guidance urges organisations to:
- Get models only from approved, assessed sources, and verify hashes and signatures.
- Load new models in a sandbox first, and prefer file formats that cannot contain executable code.
- Test for hidden behaviours, prompt injection, and data leaks before deployment.
- Encrypt and restrict access to fine-tuned model files.
- Place extern
- al controls between the model and sensitive systems, rather than relying only on built-in safeguards.
- Require human approval for high-risk or irreversible actions.
The agency’s bottom line is that organisations should first decide whether they can actually manage the ongoing security responsibilities before adopting an open-weight model at all.
