Nvidia staffs AI safety team as open models power 25% of tokens

Nvidia is quietly staffing a new AI safety and security engineering team to evaluate AI agents and patch software vulnerabilities, as it pushes open-weight models that already generate about one in four AI tokens.

"Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty," Nvidia Chief Executive Jensen Huang said in his first post on X last month, sharing a letter urging US policymakers to support open-weight AI.

The team, described in job listings as "rooted in the firm belief that open-weight models, transparency, and broad scientific scrutiny are foundational to American AI leadership and cybersecurity defense," will evaluate AI agents before deployment and build AI-powered tools to patch software vulnerabilities. Nvidia is hiring a distinguished engineer as "founding technical leader," plus a security research engineer, an evaluation engineer, and a senior manager. The hiring follows Nvidia's June 4 release of Nemotron 3 Ultra, a 550-billion-parameter open-weight model that scores 48.2 on the Artificial Analysis Intelligence Index, trailing Claude Opus 4.8's 61.4 and GPT-5.5's 60.2.

The safety push and open-model strategy serve a commercial goal: open weights put AI in the hands of far more customers, creating more demand for the Nvidia chips needed to power them. As AI shifts from chatbots to agents that can access sensitive company data and take real-world actions, trust could become the linchpin for widespread adoption — and for Nvidia's data center revenue.

Open Weights as a Demand Engine

Roughly one in four AI tokens generated today already come from an open model, Huang has said, and he wants that share to climb. Every open model that gets fine-tuned, deployed, and scaled by a startup or enterprise is inference workload that has to run somewhere — and Nvidia wants that somewhere to run on its chips.

The company released Nemotron 3 Ultra on June 4 under a permissive OpenMDW-1.1 license, making the 550-billion-parameter model available on Hugging Face. The hybrid Mamba-Attention mixture-of-experts architecture activates only about 55 billion parameters per token, supporting a 1-million-token context window. Nvidia reports a 94.7 percent score on the RULER long-context retrieval benchmark at full context length.

The model trails frontier closed systems on general reasoning — 26.7 percent on Humanity's Last Exam — but is competitive on coding and agentic tasks, scoring 71.9 percent on SWE-bench Verified. Moonshot AI's Kimi K2.6, an open-weight rival, scores 53.9 on the same index, about six points ahead of Nvidia's flagship.

The Policy Fight Over Open Weights

The safety team's formation coincides with a coordinated lobbying push. On July 24, a coalition organized with Microsoft published the "Open Weights and American AI Leadership" letter, urging policymakers to expand compute access for startups and researchers, invest in shared training infrastructure, and avoid "premature restrictions" on open-weight models. The letter grew from roughly 25 signatories to about 50 within two days, with OpenAI and Google added after launch; Amazon and Anthropic did not sign.

Nvidia also became a founding member of the Open Secure AI Alliance, a group of 120 companies including Microsoft, Palantir, SpaceX, and Hugging Face building open-source security tools for AI. The job listings for Nvidia's safety team appeared before that announcement and describe many of the same responsibilities.

The policy stakes are direct: if the White House moved to restrict open-weight distribution, whether aimed at Chinese labs or releases generally, it would threaten the open-model market Nvidia is counting on to drive chip demand beyond the small circle of companies that can afford frontier closed-model training runs.

For investors, the strategy is a bet that making AI universal grows the pie for Nvidia faster than it feeds rivals. Open models also let competitors, including China, build competitive AI cheaply, and the gap between the best American and Chinese open models is narrowing. Nvidia did not respond to a request for comment on the safety team's formation.

This article is for informational purposes only and does not constitute investment advice.