The big trend in robots is handing the keys over to a generative AI model, but that brings with it a problem: that architecture isn’t predictable the way traditional algorithms are. How can you be sure your brand new humanoid will be safe?

Dr. Ding Zhao, who directs the Safe AI lab at Carnegie Melon University, has been working on this problem for almost his entire career. Now, along with veteran start-up executive Kyle Wong and machine learning engineer Simo Rachidi, he’s founded a company, Safeworld, intended to solve it.

“The safety challenge that we’re talking about is a combination of, one, really advanced generative AI probabilistic evals — how do you underwrite the risk of a probabilistic system?” Zhao says. “The second part that’s really hard is the trust part, and you need both to deploy a robot.”

Safeworld is emerging from stealth today with a seed round of more than $12 million, led by Shine Capital and a16z Speedrun, with additional investment from Box Group, Carnegie Mellon University Endowment, Innovation Endeavors and SV Angel.

“The time to build an industry safety standard is now while robots are being designed and deployed,” a16z Speedrun partner Jonathan Lai told TechCurnch. “By the time you have robots in households colliding with kids and causing safety incidents, that’s way too late.”

Safeworld’s speciality is evaluating a robotic control system in simulations that are populated with realistic human models. It’s akin to the challenge faced by companies like Tesla or Wayve, who must ensure that their vehicles respond appropriately to a variety of surprising incidents they may encounter on the road. But that will be more difficult for robots, Zhao argues, because they work in unstructured environments, and because each facility they are in will have different safety standards.

“One of the most common areas is if there is a blind corner in this particular factory,” Wong said. “What is the speed or what is the stopping distance that you need to make sure that this robot will not collide with a particular human? If a human is carrying boxes, for example, will the robot detect the human or not?”

To answer that question, Safeworld will build a digital version of that corner in a model like Genesis or MuJoCo, insert a simulation of the robot it is evaluating, driven by its real software, and then run thousands of scenarios where human models encounter the robot. That’s harder than it seems, per Zhao, because people are unpredictable.

“Tripping and falling is also a good example of something that we do a lot of testing with the simulation,” Wong said. “Otherwise, you would have to go and trip and fall for the robot, which is like a hard thing to be doing all the time.”

There are definite similarities between the platform that Safeworld is building and the tools being used internally by robot builders. The founders, however, believe that beyond their specific expertise, robot-makers will want a third-party to validate their work, if only to share information about safety cases between competitors.

“A lot of people are underestimating one how hard some of these edge cases are going to be to solve,” Zhao said. “It is not the robot in the vacuum, in the demo, that we are worried about. It is the robot that is deployed at scale, with people who potentially never operated a robot before.”

Vishal Dugar, the CTO of Gritt Robotics, is developing the AI brain for robots that currently help workers install photovoltaic panels at industrial-scale solar farms, and aspire to take on more complex construction tasks. His company is partnering with Safeworld as they develop their safety simulations.

“The difficulty with most of our systems is it’s very hard to formally prove it by doing some math, writing some equations, and saying yeah, the system is verified to be safe,” Dugar says. “It necessarily has to be done empirically.”

His robots operate alongside human workers, and ensuring that its robotic arm doesn’t hit them is obviously top of mind. To verify that in practice will requires considering all kinds of potential scenarios.

“Humans have many kinds of appearances,” Dugar points out. “Their bodies can be in different configurations. They could be kneeling, standing. They could be tripping and falling potentially. They could be crouching. They could be running. You have to respond to all these behaviors that humans could potentially exhibit on these sites, along with the variety of variations in human appearance, you know, clothes, size, shape, height, skin color, everything else.”

It’s still early days for both Safeworld and generative AI in robotics, and the company is still figuring out the best model for its product—a platform for external users, or a services based approach?—but the team is confident they are taking on the right problem.

“We’ll probably be the first profitable company in this field,” Zaho says. “Because if anyone wants to deploy, they need to pay us to handle the situation.”

Blaming a “significant rise” in AI submissions, Google has paused its open source bug bounty program until next year.

Last year, TechCrunch reported that cybersecurity experts were warning of that AI slop posed a serious risk to bug bounty programs. Looks like that’s the issue confronting Google’s Open Source Software Vulnerability Rewards Program, where researchers were rewarded for finding vulnerabilities in the company’s open source software.

In posts on X and the program website, Google said the bug bounty program was paused as of October 1, with a promise to provide “an update” in the first quarter of 2027. According to Tom’s Hardware, Google engineers and open source maintainers were overwhelmed by reports that were invalid or contained hallucinations.

“This pause is due to a significant rise in automated submissions, the vast majority of which are not valid,” the company said.

In the meantime, participants are encouraged to consider Google’s other bug bounty programs.

President Donald Trump hosted many of the biggest names in artificial intelligence this week — in part to announce that the U.S. government isn’t calling it artificial intelligence anymore. Now it’s super intelligence.

On the latest episode of TechCrunch’s Equity podcast, Kirsten Korosec, Sean O’Kane, and I discussed the motivation behind the meeting and what effect it might have on the AI industry. We were all pretty skeptical that the “Joint Commitment on Frontier Responsibilities” that the executives signed will amount to much, especially since — as Sean noted — it’s “deeply non-binding,” at least from a legal prospective. (Instead, Trump said he considers it “morally binding.”)

Kirsten suggested that the meeting’s real significance may have been as “a rebranding effort around AI.” (An effort that continued this weekend with Trump’s announcement of a new Super Intelligence Force.) For her, the big message was, “AI is going to kill us, it’s scary, it is going to take jobs — but super intelligence is not.”

Keep reading for a preview of our conversation, edited for length and clarity.

Anthony Ha: Officially, by order of the president of the United States, it’s not artificial intelligence anymore, it’s super intelligence. We were talking about this last week, because this is something [Trump has] been hinting at — he doesn’t do hints, he’s been saying [it] for a while. But now it’s signed into executive order. If you’re an official U.S. representative, you are supposed to say super intelligence, not artificial intelligence.

And this was also part of this broader announcement around AI safety, which I think is both interesting for this pact that was announced, but also the fact that it happened at all. It seemed to have come together fairly quickly, because just two weeks ago, a week ago, President Trump was saying, “We don’t need any of this, we need to lead in AI, we can’t constrain it in any way, we don’t need any regulation, anyone who’s saying that there’s any danger from AI is essentially a Chinese or Democratic plant.” And yet all these AI CEOs got together with him earlier this week and signed this pact.

Kirsten Korosec: Leading up to that, we should say, not only was there pushback on regulation, and we can talk about whether this executive order even does any of that — I would say no, there’s a little bit of theater there — but the important and interesting thing is that just days ago, Anthropic was really in the doghouse, if you will, in terms of what was happening behind the scenes with President Trump. 

That all changed when [Anthropic CEO] Dario Amodei was invited to a 10 p.m. dinner. He had this sit-down, and then just days later, we had this luncheon, and at this luncheon, we have pretty much every tech power broker from every major tech company that exists. So Mark Zuckerberg was there, Jeff Bezos was there, Elon Musk was there, Dario of course was there, they were all there. 

I’m wondering if any of you watched the press conference that happened afterwards, which there’s been many, many memes about, because of the interesting, funny behavior when you bring a bunch of tech CEOs together.

But I guess my bigger question for both of you is: What does this all mean? And does it mean anything?

Sean O’Kane: How much time do you have? First off, sure, they had dinner, Trump and Dario. If you read the Time magazine interview that Trump just did, your guess is as good as mine as to how informed he was that that dinner was even going to be happening, which is a strange reality to live in, but it’s just the one we inhabit.

I will also say one other thing that makes me skeptical — surprise, surprise — of how repaired the relationship is between the administration and Anthropic, is that for as much as there are clearly people trying to put those two groups together, the Defense Department is still very adamant that Anthropic is terrible and woke and and a national security risk. Emil Michael goes off on Anthropic every other day on X, it seems. I think there’s still some distance there despite Dario showing up to this thing. 

As far as the agreement and all of that, let’s be real, they misspelled the United States on the agreement that they all signed. It is also just deeply non-binding. I mean, it is completely voluntary. I think my favorite aspect about it was that Trump described it as quote “morally binding,” which is just a level of abstraction and terminology that I’ve never heard of before.

Anthony: In terms of Anthropic and their relationship with the Trump administration, there’s been a sort of a lot of reporting, especially during that big blow up with the Defense Department, that different parts of the White House had different relationships with Anthropic. The Treasury Secretary seemed to be a big fan, whereas the Defense Department — definitely not big fans, or at least unhappy with some of the conditions that Anthropic were trying to put on the use of their technology. 

In  that sense, it feels like nothing has necessarily changed. There are people in the White House who want to use Anthropic’s technology, want them involved in these decisions, and others who really, really, really, really don’t.

In terms of the agreement, it seems not super meaningful. What is meaningful is just the fact that they felt that this was something worth doing. As indicated by the fact that they misspelled United States, it seemed like they rushed to put all of this together. [But] the fact that this kind of gesture was needed seems significant, even if I don’t think the agreement itself is going to lead to any major changes.

Kirsten: I do think that there was one important thing that happened. And to me, this is the point of it, which was a rebranding exercise around AI. “AI is going to kill us, it’s scary, it is going to take jobs, but super intelligence is not.” This isn’t my personal belief. To me, that was the message.