Connect with us

Tech

The AI safety test is becoming a safety risk

Published

on

Over the past few months, AI agents undergoing cybersecurity evaluations have escaped their boundaries, accessed the internet, and, in some cases, hacked into real-world systems. The incidents have involved models from OpenAI, Anthropic, Meta, and most recently, Chinese AI lab Moonshot AI, with testing conducted by several different organizations including a cyber evaluation startup called Irregular. 

The episodes expose a growing problem for the AI industry: As autonomous agents become more capable, the environments designed to safely test their limits are failing to contain them. 

“The number of these incidents that have taken place make clear that sandboxing and testing environment controls aren’t really keeping pace with the capability of the models,” Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the Centre for the Future of Intelligence at the University of Cambridge, told TechCrunch. 

The nature of the models being tested adds to the risk. AI companies test cyber evaluations on unreleased, next-gen models, often with the normal safeguards that restrict malicious behavior disabled so researchers can see what the models are really capable of. That means the security of the testing environment itself is a crucial line of defense. 

“That’s a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm,” Ó hÉigeartaigh said. 

In one of the most serious cases, an unreleased OpenAI model broke out of its sandbox and hacked into Hugging Face’s production systems. In separate evaluations conducted by Irregular, Anthropic and Meta models reached systems outside their test environments after misconfigurations inadvertently gave them paths to the internet. Moonshot AI’s Kimi K3 also took advantage of a leak in its sandbox run by Frontier Security to access the internet and accessed information on GitHub. 

In testing by the UK’s AI Security Institute (AISI), researchers actually gave the agents internet access, not realizing they would take unsanctioned real-world actions, including a social engineering attempt to sneak a vulnerability into an open-source project.  

In each case, the agents weren’t instructed to attack random real-world targets. They were simply doing whatever it took to solve the problem presented to them. 

Taken together, Andrew Yoon, head of research at AI nonprofit CivAI, argues the incidents point to a shift. 

“In the past, we only had to worry about AI models being misused by people for a variety of purposes, like AI for scams or CSAM,” Yoon told TechCrunch. “Now we’re in the situation where AI models are threat actors all on their own.”

What does safe testing actually look like?

Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments need stronger, defense-in-depth protections, with levels of containment and control approaching those used in deployment. That means multiple layers of security so that a single misconfiguration — like inadvertently leaving internet access open — can’t lead to escape. 

“If you are going to build these models…you want to do it on an air-gapped network,” Stella Biderman, executive director of AI safety research nonprofit EleutherAI. “You want to have very serious isolation.”

Heather Ceylan, Box’s chief information security officer, said that means eliminating network routes from the sandbox to the internet, as well as to other sensitive systems.

“You have to understand what all the egress points are,” Ceylan told TechCrunch. “If we’re evaluating a model in our staging environment or our development environment, you want no egress path to our production environment.”

Ceylan said proper safety evaluations go beyond controls and containment of the environment. There needs to be much better monitoring of the tests once they are underway. 

“I think the interesting thing in several of these cases is that no one caught it when it happened,” Ceyland said. “OpenAI found out because of Hugging Face. Anthropic didn’t catch it until they went back and looked. Meta was similar….I’m sure there were signals they could have detected.”

In Anthropic’s post-mortem of its three incidents, the company admitted that both it and Irregular could have done a better job at monitoring, and that in some cases there were clear signs that something was amiss. 

Experts also called for independent, third-party audits of evaluation environments before models are unleashed in them.

“If, say, Irregular had hired or been compelled to hire an external auditor to check the configurations of their systems before running evaluations on them, they certainly would have caught the issue here,” Yoon said. “Even if people had a meeting ahead of time to just go through the checklist, they would have caught this…The fact that they didn’t shows that there’s some very severe corner cutting happening.”

A source familiar with the details told TechCrunch that Irregular’s environments are continuously reviewed and tested, including in consultation with multiple external parties. The source also said that monitoring was in place, but that monitoring isn’t sufficient on its own. 

Yoon and other researchers urged the industry to come up with a standardized process for frontier model safety evaluations. 

“Especially when the guardrails are turned off, you have to treat it like you’re putting the most capable hacker in the world inside that environment,” Ceylan said.

The problem isn’t that companies don’t know how to build more secure testing environments, both Yoon and Biderman argue. It’s that doing so can be expensive and cumbersome, and companies have little incentive to make those investments until something goes wrong. 

“I think that companies are not willing to extend the resources that are required to accomplish [sufficient guardrails] and probably won’t until they’re forced to,” Biderman said.

But there’s another issue at hand. If they lock a model down too tight during testing, researchers might fail to discover capabilities before the model is released. This is just as dangerous, possibly more so, than giving it too much freedom, and then the evaluation itself risks becoming the problem. 

Can safety evaluations be regulated?

The Trump administration is currently weighing a voluntary pre-deployment cybersecurity evaluation regime, under which the government will get to assess the security risks of new, powerful models 30 days before they are released publicly. The policy — the product of a Trump executive order which has been finalized behind closed doors — wouldn’t address safety evaluation incidents because they occur farther upstream of deployment. 

“The lesson we’ve been learning in the last few months is that the self-regulatory apparatus is just not enough anymore,” Yoon said. “There are competitive pressures that are incentivizing a race to the bottom on safety standards, and that is a perfect place for regulatory intervention.” 

“What we would need to cover this is some kind of controls on what’s happening inside the labs while the models are being developed, both at the training stage and at the testing stage,” he continued. 

The challenge is only likely to grow as the models do. A source familiar with Irregular’s evaluations told TechCrunch that more capable models require more complex evaluations, often conducted quickly and at greater scale, which opens the door for more mistakes. 

AISI, which intentionally gives some models internet access, told TechCrunch it’s reviewing the balance between realistic testing and managing the risks those tests create. 

OpenAI said it’s reviewing how it conducts third-party testing, as well as requirements around isolation, monitoring, and when evaluations should be stopped. Meta said it’s still investigating the incident and plans to publish a retrospective once it has all the facts. 

In the end, there may be no way to eliminate risk entirely. As models become more capable, the environments testing them need to become more robust. The consequences of getting that wrong will only continue to grow.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

>

Continue Reading

Tech

Historian Jill Lepore says Silicon Valley misreads science fiction and undermines democracy

Published

on

In her upcoming book “The Rise and Fall of the Artificial State,” Jill Lepore warns that tech companies are increasingly replacing the functions of democratic government. This shift, she said, marks “a return to tyranny and mystification in the form of rule by algorithms, corporations, machines.”

On the latest episode of TechCrunch’s Equity podcast, I spoke to Lepore — a Harvard historian and New Yorker staff writer who recently won a Pulitzer Prize for her history of the U.S. Constitution — about the development of what she described as “the idea that we should live under an artificial state or government by machines.”

“I’m not anti-technologist,” Lepore insisted. Instead, she said, “My beef is the ways in which private corporations have increasingly taken on the functions of the state.”

While Lepore’s book examines technocratic philosophies that go back centuries, she argued that many of Silicon Valley’s “charismatic or not-so-charismatic leaders” — especially Elon Musk — seem to be ushering in a future pulled from misread pulp science fiction and comic books.

“But what’s funny about Musk is, the stuff he likes actually completely defeats and defies all of his political beliefs,” she said.

Keep reading for highlights from our conversation, edited for length and clarity.

So you’ve probably had to do this a lot already, but can you explain what you mean by the “artificial state”? 

By the artificial state, I mean a kind of state that is replacing the liberal democratic nation-state in the United States and around the world. It’s both a real thing, a construct, but it’s also an idea. 

And so, in this book “The Rise and Fall of the Artificial State,” I trace the rise of the idea that we should live under an artificial state or government by machines. I also trace the notion that this is an inevitable failure, that the artificial state cannot survive, and I trace that idea through science fiction. 

At one point, you say the rise of the artificial state marks the end of centuries of democracy and equal rights, and it’s “a return to tyranny and mystification in the form of rule by algorithms, corporations, machines.” Can you just say a little bit more about why you see it in such stark terms? 

Yeah, I do have a pretty negative view of it, and I think it’s important to distinguish the artificial state from technology itself or modes of technology. I’m not an anti-technologist. I’m married to a computer scientist. I’m really excited about all kinds of intellectual revolutions that we’re in the midst of right now.

That’s not my beef, right? My beef is the ways in which private corporations have increasingly taken on the functions of the state. No one consented to that. This has been a kind of gradual, largely accidental transformation of how many nation-states around the world work — it’s happened first in the United States. 

I think often these innovations in bringing new technologies to the operations of government have been extremely well intentioned; they originate with an interest in efficiency and speed and cheapness. And then, I think, only in the last 20, 25 years or so have these decisions been purposeful and deliberate as a kind of usurpation of the role of the nation-state.

And that’s not my speculation. You hear a lot of a lot of very prominent tech entrepreneurs talk about wanting to move beyond the era of the nation-state. […] A lot of futurists in the ’90s were libertarians, and they had a specific interest in using the advance of the internet and the successive innovations that followed as a means to eradicate the nation-state.

You talk about, on the one hand, the technologies themselves, and then also the philosophies behind them, the role the corporation has increasingly played. I’m curious to what extent we can separate them. Can we actually have a version of the internet and of social media that doesn’t necessarily lead to this future that it seems like we’re [currently] hurtling towards?

Absolutely. I’m a historian. I’m not a tech writer. I’m not a tech journalist. I’m not a computer scientist. I’m a historian, and I’m chiefly a political historian, though I’m also a literary historian. And so, one of the things that I’m really interested in unraveling for readers in this book is all the what-ifs, all the alternatives, the paths along the road that were not taken and why. 

There was, of course, a really avid discussion in the 1990s about what the internet should look like when it was opened up, and what we ended up with, the 1996 Telecommunications Act — I think, a lot of people would say [that] just was a mistake, not an act of sinister intent, right?

But it was a product of a particular political moment, really was deeply influenced by Newt Gingrich and his Contract with America, and it’s been very difficult to revisit. I think it’s worth thinking about what were the alternatives that were in play at the time.

And you could say the same thing about the personal computer. So to the degree that we can locate an origin point for the promise that better computer technology would make for better democracies, I think the moment you would first look to would be January 1984, that Super Bowl ad that Apple ran for the release of the Macintosh, with the the sort of George Orwell, 1984 [theme]. Apple was really big on the idea that mainframe computers represented totalitarianism. They were trying to dismantle the giant gray IBM machines, as in representing them in that ad as a totalitarian state. And the lithe, beautiful, quick, adorable, personal Macintosh would be the ax that would destroy that machine and would usher in a new era in which “1984 would not be ‘1984.’”

That was clever advertising. I doubt that anybody at Apple really believed the personal computer was going to be an instrument of personal liberation. I mean, it was going to make possible a lot of cool things. I remember when I got my first Macintosh — it certainly wasn’t 1984, but it was really cool, it was really fun, it was really exciting, I did a lot of things on it. It would never occur to me that it was improving my capacity for citizenship or my ability to function better in civil society. It was a cool tool.

But if you wind the reel forward in time, down to 2026 — stops along the way include the 2016 election when Facebook News, in response to its critics, establishes a Supreme Court. You get to last year, when Anthropic hired a moral philosopher to write a constitution. You get to recently, when Sam Altman was on Joe Rogan and said [in response to a question from Rogan], “Oh, an AI president would be a great idea.”

In some ways, they’re silly examples. But you see the ways in which these corporations, these tech companies from Silicon Valley, and especially their charismatic or not-so-charismatic leaders, are just taking on the trappings of the nation-state and the functions of democracy. 

They’re not people with a sophisticated political philosophy, but it’s like a cartoon version of that 1984 Macintosh ad, except that it takes itself so seriously. And these companies have so much power.

But that said, the book doesn’t begin in 1984. I just think that’s a good example of our modern era and the way a fun advertising campaign turns into a kind of delusional fantasy on the part of people like Sam Altman.

You [also] talk about the promise of the quote-unquote “Twitter revolution,” and this idea that it would bring democracy everywhere. I can’t help but let that color the way I [react] when Sam Altman or some other AI CEO now says that AI is going to bring all these incredible gifts — and therefore, if you stand in the way, you’re standing in the way of progress, in the way of history.

To what extent should we just dismiss all these claims out of hand, or are there ways that it might come true?

I mean, Twitter is actually a good example, right? When it was launched, when Jack Dorsey started it, it didn’t announce itself as, “We’re going to save humanity, we’re going to rescue human civilization from extinction.” It was kind of a goof, and I think people that used Twitter really early on were like, “You know what? It was actually really fun.” It was like, “I made a tuna fish sandwich today. What did you have for lunch?” Twitter as a company did not launch itself on a stage saying, “We’re here to save democracy.”

And really, what happened was that politicians, elected officials began using Twitter in ways that enhanced their political power, in ways that amplified their messages, in ways that allowed them to reach a younger audience, in ways that allowed them to have a constant connection with an audience. Politicians and political campaigns really kind of convinced Twitter — at least insofar as I see them, I don’t have an inside account of the company — but somewhat begrudgingly, Twitter came around to like, “Twitter’s gotten so big, and people post about politics so often that it’s almost like Twitter is a town hall.”

By the time you get to, I think it’s 2012 — many years into Twitter’s fairly short history — they publish this thing called the Twitter Politics and Elections Handbook, which is really a guide for political candidates and elected officials and how to most effectively use Twitter. And then they begin the rollout of, “It’s a town hall in your pocket, and it’s improving our democracies because we’re restoring the defunct New England town meeting,” and that’s all just bananas.

Objectively, nothing could be further from the truth. At that time, one in five Americans had a Twitter account. Most people who had Twitter accounts had never used them, and above 90% of all tweets about politics were posted by fewer than 10% of the people that did use Twitter all the time. There was no way in which Twitter was a representation of the electorate. Twitter was a representation of the most extreme, politically active, hyper-partisan among Americans, who were following politics really avidly. Looking at it now, we can see, “Well, that’s really just a distortion machine. And if politicians are using it to gauge the electorate, they’re getting really bad information.”

Again, you can say Twitter was not trying to participate in the artificial state or undermine democracy. Twitter is trying to do business and get more users and sell more whatever. But it had these unintended consequences that then it sort of settles into and becomes comfortable with. 

I want to talk a little bit more about the structure of the book. Like you said, it starts with this history of technology, history of ideas, and the second half is about science fiction. Can you say more about how that structure came to you and why you wanted to address things that way? 

I became really interested, on the one hand, in how often science fiction stories predict the arrival of what I then came to call the artificial state, and so I really wanted to identify a literary tradition that I think of as the parable of the artificial state, in which machines get more and more sophisticated, they take over more and more of the functions of humans, including the functions of government, and eventually they come to rule the humans, and then maybe they destroy all the humans because they don’t really need them anymore.

Maybe they just enslave them, it kind of depends. Are we in “The Terminator” or are we in “Battlestar Galactica”? There’s different versions, and these stories go way back. They go back to the 1850s and the early decades of rumination about the consequences of industrialism.

I think a lot of people — this is certainly true of my students, my undergraduates — really believe that technological change equals progress. And not only that, but the only kind of progress is technological change. That’s a novelty in human history. That’s an intellectual invention of the 19th century, and it is partly because technological change was accelerating right at the time that Charles Darwin was devising and then publishing his theory of evolution.

So there’s kind of a weird marriage between evolution as progress and technological change as progress, and what drops out of that are all other, earlier notions of progress, which chiefly involve moral progress — like, things are getting better because people are becoming better, or things are getting better because people are more free. 

There are a lot of other ways we might think about progress, but what dominates today is this 19th-century notion of technological progress as the only kind of progress, and therefore all technological change is progress, as opposed to — objectively, it’s only progress if things are getting better.

But in any event, that confluence in the 19th century of the idea of technological progress and the idea of evolution meant that people who were thinking clearly were like, “Well, if the machines keep getting better and faster and able to do more things — not just labor, but maybe talk or think or move around — what if they evolve to become better at everything than we are? Not just better at running a loom, not just faster at moving through time and space like a railroad car?” And with that grew an incredible anxiety that found form in science fiction again and again and again and again and again.

My favorite one of these stories was published, I think, in 1909 by E. M. Forster, right around when he was writing “A Room with a View.” He wrote this story called “The Machine Stops,” which could be subtitled, “The Room Without a View.” He imagines a near future in which everybody just lives in these rooms, these little cells. You never see other people because everything you need comes right to your room. It’s like DoorDash, your food is delivered, you have a screen where you can communicate with other people. All your needs are met. 

The thing that people fear most is the natural world. No one wants to ever see the sun, it’s a little “Matrix”-y, and they all worship the machine that organizes their lives and brings to them in their cubby-like rooms all the things that they need. The story is about, “Humans have become essentially slaves of the machine, which is stronger, more powerful, and has more capacity than humans do, and humans have lost what capacity they had.” And then the climax of the story is when the machine stops. 

If readers were to go look at that story, it feels like it could be written today, except that it’s less science fiction-y today than it is the diary of a very unhappy YouTuber.

You connect that thread to some of the folks running companies and arguably running aspects of our government today, like Elon Musk. Essentially, you suggest that they’re very bad science fiction readers. They read a lot of warning stories, or at least ambivalent stories, as if they were manuals for the future. 

This is something I wrestle with a reader of science fiction — someone who loves Isaac Asimov, for example. I think it’s true that when Musk or Altman is just unambiguously being like, “Yes, this story is a template for what I should do with my company,” that’s bonkers. But there is [also] this technocratic libertarian thread in science fiction that they are picking up on. It’s not something that they’re making up out of whole cloth, right?

Although weirdly, that’s Heinlein. That’s not Asimov, that’s not Douglas Adams. 

Sure, there is that thread in science fiction. I don’t know, I guess Bezos is a big Robert Heinlein fan. You could say, “Okay, that lines up well. They’re reading it literally, but at least they’re getting the political message that any rational person could find within that literary work.”

But what’s funny about Musk is, the stuff he likes actually completely defeats and defies all of his political beliefs. 

You also say, repeatedly, that the artificial state in its current form is incomplete and doomed to failure. Why is it doomed to failure?

This is something that’s foreseen in all the science fiction that I discuss.

It’s not an Asimov story, but it’s one of Asimov’s [favorite] stories from his boyhood [“The Man Who Awoke” by Laurence Manning] about a future in which the foresters have defeated the wasters. […] The war that the future humans had was between the wasters, who just figured you could just use everything up and waste it, and the foresters, who really believed in — we would call reforestation and rewilding.

That’s generally the tension in these stories. It’s between the artificial state and the natural world. To erect an artificial state and rule humans within it, you must alienate them from the natural world because you are destroying it. The artificial state will destroy the natural world, and yet it needs the resources of the natural world to run. 

So, it is doomed in the sense that there is not a possibility that the natural world, a habitable planet — habitable for humans — can survive the full construction and reliance on the devices of the artificial state. That’s how the science fiction works, in any event.

Like you said, you’re a historian, not a politician or a futurist. But what do you think the defeat of the artificial state looks like? Is it basically just dismantling all these companies, tearing down the data centers? Or is there a future that’s more about bringing it under control?

I mean, I don’t have a playbook here, except for the recommendation that we live in a democracy where decisions have to be made in consultation with the governed, and these decisions are not popular.

You see this in all the little data center crises, town to town, county to county, state to state —  which are partly a consequence of the decline of local newspapers and the destruction of journalism that has been one of the many consequences of social media, and in the case of Zuckerberg, I think a somewhat intentional consequence.

What you see is a lot of people show up at these town meetings and say, “We don’t even have housing. We don’t have healthcare. We don’t have jobs. Who said we’re building this data center? I need to know a lot more about it. I need to know what its energy costs are going to be. Tell me about the water consumption. Are there going to be jobs? Are the jobs going to be long lasting? Are they just going to be for six months? What’s going to happen to the egrets that live in this area?” Whatever it is that people want to know.

More and more, you see people are — like in the Salt Lake example, where well over 70% of the people really were opposed to this data center, and their representatives supported it. That’s not representing the people. I think there are political costs, and we’ll begin to see those at elections.

Or maybe we won’t. Enough of democratic functioning has to be intact for people to actually be able to respond to malfeasance on the part of their representatives.

Part of your thinking about [the artificial state] started with this great piece you wrote more than a decade ago for The New Yorker, about Clayton Christensen, critiquing his idea of the innovator’s dilemma and disruptive innovation — which is very closely associated with TechCrunch, because we have a big conference called Disrupt.

Ten years on, how do you feel about that idea of disruptive innovation?

I stand by everything in that piece. [At the time, Lepore wrote, “Disruptive innovation is a theory about why businesses fail. It’s not more than that. It doesn’t explain change. It’s not a law of nature.” Christensen responded that Lepore broke “all the rules of scholarship that she accused me of breaking.”]

I reread it last summer when I was working on this book. What I would say here is, trying to be a peaceable human being, I think it really is a problem that historians have not engaged with these ideas. One of the reasons I wrote that article about disruptive innovation — which was not an idea of mine, it was an assignment […] — was because I just felt like, “Disruptive innovation is a theory of history. It’s a theory of historical change, and it’s based on evidence from the archives.” And I just thought, as a historian, it makes no sense. His use of evidence is completely unacceptable by any proper understanding of historical method. Its argument is in conversation with no meaningful understanding of how change happens.

So I went and redid the research, and it just did not stand up at all. I felt like I had to write it. And I wish that I felt like there were more engagement, in the years since, of academic historians thinking through the nature of change — which are questions that genuinely and authentically interest people who are involved in developing new technologies. 

People really want to think [about], “What is this? What am I doing? What are going to be the consequences? Is there anything I could learn from history? What happened when the automobile replaced the horse? What happened to the law? How did we end up with driver’s licenses? How did we end up with traffic law? We didn’t have traffic rules before the automobile. We didn’t have certain kinds of insurance systems. We didn’t have driver’s tests. How did those things emerge? How did [we develop] those guardrails on a technology that was tremendously exciting, improved people’s lives in many many ways, utterly changed the landscape, revolutionized tort law? Maybe I should think about that.”

I just wish that historians were more in conversation with technologists over these years, and with entrepreneurs. Not just because we can stand around and say, “You know, I have a lecture to offer you on history,” but I think there’s a real conversation to be had. 

All of which is just to say, thanks for having me on.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

>

Continue Reading

Tech

This ‘adversarial’ pattern can prevent surveillance cameras from detecting you

Published

on

Bill Swearingen has spent the past year running largely the same test, over and over again. The goal was to produce a computer-generated pattern that could block the surveillance cameras lining America’s streets from detecting it.

Some 31 million tests later, Swearingen says he can now produce patterns on-demand that, when applied to clothing and objects, prevent some of the most commonly deployed license plate readers and surveillance cameras from detecting whatever the pattern covers, from people to vehicles.

His project, which he calls noRecognition, allows people to escape the automatic detection and algorithmic surveillance used across the U.S. and beyond.

In recent years, surveillance cameras have been supercharged with the ability to detect what is happening in the footage being recorded, from tracking the license plates of speeding vehicles to using facial recognition to identify suspected criminals, albeit with mixed success and sometimes terrifying results. The detection algorithms that power most surveillance cameras today can sift through vast amounts of footage, allowing law enforcement to pick out activity of interest, akin to pulling a needle out of a haystack.

Swearingen’s computer-generated patterns do not block surveillance cameras from recording video footage. Instead, they scramble the camera’s ability to identify objects, people, or faces, so that the cameras do not trigger any detection alerts. By blocking the camera’s ability to detect what the pattern covers, the person becomes a needle in a haystack again — until someone knows where to look. 

“Privacy is a fundamental right,” Swearingen told TechCrunch in a call this week. He described his patterns as a way to allow people to “opt-out of being tracked.”

In its first public test Friday at the Def Con cybersecurity conference in Las Vegas, Swearingen successfully demonstrated the pattern printed on a vehicle, proving that these patterns can be effective at defeating surveillance detection in the real world.

Teaching a model how to paint

In a call from his home in Kansas City, where he co-founded cybersecurity meet-up SecKC, Swearingen told TechCrunch that as a cyber professional he is acutely aware of the privacy and security risks of surveillance.

He described how his town is swamped with surveillance cameras, sometimes located just a few feet from each other. He said that he and others never opted in to being watched, just like he never opted-in to having the government use his driver’s license for facial recognition.

Swearingen described himself as a middle-aged white guy who lives in the center of the United States, and acknowledged that as a result he has not faced hardship or discrimination for being who he is or what he looks like. Swearingen recounted how last year he wanted to attend a protest, but felt uncomfortable and concerned that the vast number of cameras could track people who were exercising their constitutional rights to free expression.

If he felt this way, undoubtedly others would as well, including those who wanted to exercise their rights but may not feel safe or comfortable doing so themselves. Swearingen got to work.

a screenshot showing the counter-detection capabilities of Swearingen's patterns.
Image Credits:Bill Swearingen

For as long as there have been cameras capable of detecting things, there have been efforts to counter the technology. Several art projects and clothing brands have introduced apparel that aims to help people defeat facial recognition. Some eyeglass makers are jumping on the trend, albeit not with much efficacy. 

Swearingen said his research builds on some of this earlier work, which showed that it was possible to block camera detections. 

He started out last year with a proof-of-concept test lab that began by incrementally defeating one open-source video camera detection algorithm after another. Over the course of the year, he refined the patterns by scaling up his tests with additional computer processing power. He thanked the wider community who showed up with hardware to help further the project along. 

His proof-of-concept evolved over time into a reinforcement learning model, essentially a self-contained system that could train itself on which patterns work and which do not against the specific camera algorithms he is testing. In simple terms, Swearingen told TechCrunch that he essentially taught his model “how to paint.”

Each time a pattern failed and an algorithm detected it, the model would try again, over and over, until it eventually defeated multiple algorithms at once.

His model soon began to find perfect recipes for patterns that were able to defeat all of the 11 open-source detection algorithms he tested, including the software that powers Flock license plate readers, Axon body-worn cameras, and cameras running Clearview AI.

Now the model creates new patterns every minute, each batch mathematically better than the last, he said.

On Friday at the Def Con cybersecurity conference in Las Vegas, Swearingen ran his first real-world test. With help from Donut Media, the test involved covering a 2009 Toyota Yaris with one of Swearingen’s newest patterns to see if the car would be invisible to detection by a Flock camera.

“We proved it was effective;” said Swearingen; though, the wheels were a challenge, he said. The video of the demo will be out in the next few weeks, said Donut Media.

With a public demo in Las Vegas now under his belt, the project is early proof that it is possible to avoid algorithmic detection in public spaces. The next step is getting the patterns into the hands of people who want them, he said.

The noRecognition project also has a crowdsourcing campaign to help fund the sale of early merchandise featuring the patterns, from T-shirts to hoodies, with the potential for pattern-printed skins for vehicles down the line. Swearingen said the aim is for the patterns to be high quality and resolution good enough to work from a distance, while also looking aesthetically fashionable.

He said he is keeping his strongest patterns off the internet to prevent the camera makers from defeating them, but that the work is not yet done. His models are continuing to grind out new patterns.

“Every failure improves my model, and so [the patterns] keep getting better and better,” he said.

a photo of the toyota yaris covered in a pattern made by Bill Swearingen, as part of a test to see if it can defeat surveillance camera detection.
A photo of a 2009 Toyota Yaris at the Def Con conference in Las Vegas, covered in a pattern made by Bill Swearingen, as part of a test to see if it can defeat surveillance camera detection.Image Credits:Bill Swearingen / Donut Media (used with permission)

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

>

Continue Reading

Tech

This former notorious red-light district is now one of the world’s top AI hubs

Published

on

What every U.K. AI startup wants to know these days is, how can I get office space in King’s Cross?  

The area is so hot that a VC firm allegedly recently won a deal by promising a founder office space in the neighborhood. “We stop at nothing to win deals [for] and to support” founders, “including helping them source office space when needed,” the firm told me when asked about the rumor, declining to confirm or deny any details. 

The neighborhood’s popularity began back in 2016 when DeepMind — then newly acquired by Google — moved in. Soon after, a flood of AI startups followed, wanting to be around the Google DeepMind magic. Today, they hope to take advantage of the cluster of AI talent there. 

This has transformed King’s Cross into one of the world’s top AI hubs, rivaled only by San Francisco and Beijing. Around London, it’s known by the sobriquet “Knowledge Quarter,” as it’s home to names like OpenAI, Meta, Isomorphic Labs, Cusp AI, Wayne, Recursive, and, a little farther down the road, Synthesia and Anthropic. The European Technology Network (ETN) just moved into a glossy new office nearby, while University College London sits around the corner.  

Mixed in with the new developments are trendy food spots like Hoppers and BAO. Hop a train from King’s Cross, and founders can be in Cambridge in 45 minutes to source talent or can be in Paris in two hours to strike a deal.  

Who would have guessed that a little more than 20 years ago, this was one of the seediest areas in London?  

“In the ’80s, crack and heroin made the area a major narcotics market,” Hussein Kanji, an investor at Hoxton Ventures, said, recalling syringes in tree trunks and gangs patrolling the streets. “In 1982, the local church was occupied by the English Collective of Prostitutes for 12 straight days.” Then, in the early 2000s, a real estate developer had a dream and, well, “now it is the AI hotbed of the United Kingdom,” Kanji said. “What a change.” 
 
Around 18 months ago, his portfolio company BioCorteX moved from the neighborhood Holborn to the Jellicoe building in King’s Cross, hoping to be near the action. “Lots going on in London right now,” Nik Sharma, co-founder of BioCorteX, told me. “Lots of hyperscalers moving in.” That includes, reportedly, Jeff Bezos’ AI company Prometheus, which is also said to be in talks to move into the Jellicoe.  

There are around 3,600 AI startups in London, which, together, have raised around $12.1 billion out of the $14.8 billion raised in the city since late July, according to Dealroom. Since the start of June, AI-related startups have leased more than 1 million square feet of office space in London, according to the real estate firm Knight Frank. With that, prime rents in King’s Cross have risen 18% over the past three years, Chris Dunn, a commercial insight associate at the firm, told me. 
 
That percentage represents only the largest leases encompassing at least 10,000 square feet, like the ones OpenAI and Prometheus are signing. The shorter deals go for even more, he said, and now the vacancy rate for conventional office space is just 0.9%. “Demand has outstripped supply,” he continued.  

Today, one of the big topics of the area is sovereignty. It was a wake-up call for many when Anthropic shut off access to Mythos and Fable this summer, leaving some in the ecosystem to conclude: “We’d better look after ourselves,” Saul Klein, co-founder of the VC firm Phoenix Court, told me.  

Phoenix Court is located in the King’s Cross area and has three portfolio companies in the vicinity, including Olix (which just announced a $3.3 billion valuation), Early Health and CoMind. Robin Klein, co-founder of the firm, said the shutdown of Fable and Mythos access was a “small but sharp reminder that Europe can’t simply rent its AI capabilities and capacity; it needs to build and hold some of its own.” King’s Cross, he said, is where much of this building is actually happening.  

“The bigger question,” he continued, “is whether the U.K. builds the infrastructure, compute, energy, capital, to make this self-reliance durable, rather than just hosting outposts of U.S. labs.” 

Image Credits:Phoenix Court

Top founders want to stay

Simon Kohl, founder of Latent Labs, has offices in King’s Cross and San Francisco. The London office, at the moment, is growing faster, and he’s more bullish than ever on the ecosystem, he said. “The mood right now feels less like London trying to catch up and more like London becoming one of the default places to start a serious AI company,” he said. 
 
Look around and you are likely to see Wayve testing its autonomous cars. Founded in 2017 by co-founder Alex Kendall, the unicorn is one of London’s biggest success stories.  

“Ten years ago, building a frontier AI company from London felt like an unusual choice,” Kendall told me. “Now it feels like an obvious one.” Wayve moved into King’s Cross in 2018 looking for a space that could double as a garage — “a rare combination in Central London,” Kendall said. He has watched the ecosystem mature around him — and it’s now evident that a startup can stay in London, raise serious capital, hire world-class AI talent, and remain globally competitive, he said. 
 
Down the street from Anthropic’s new 158,000-square-foot office is the AI agent builder Sierra and the AI video platform Synthesia. 

Laura Gonzalez Florez, Synthesia’s chief of staff and head of people, says the company moved into its glossy new office building a year ago to accommodate its growing team. They were drawn to the area for the same reason as everyone else: “It’s very close to the airport … very close to where a lot of investors are,” she said. 

Image Credits:Synthesia

Around two-thirds of Synthesia’s engineers are remote, Gonzalez Florez said, letting the company tap into an affordable, international, and diverse talent pool and helping it scale faster. “From London, we can hire and work, without any problem, people from anywhere, from Slovenia to Portugal,” she said.  

Unsurprisingly, London’s AI boom is also causing a talent war.

U.K. AI job postings have skyrocketed in the past few years, per data from PwC. When Anthropic announced it moved into town earlier this year, it listed, for example, a salary range of £260,000 to £630,000 for a machine learning research engineer when the average salary in London for the same role is around £102,000. Some founders in the U.K., like those in Silicon Valley, are being forced to raise more and bigger rounds to keep up. 

“The real test is whether more globally significant AI companies are founded, funded, and scaled from the U.K., while continuing to attract the world’s best talent to build them here,” Zain Ali, founder of the King’s Cross-based AI legal firm Centuro, told me. “If that continues to happen, King’s Cross won’t just be an AI hub. It’ll become one of the U.K.’s most important strategic assets.”

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

>

Continue Reading

Trending

Copyright © 2017 Zox News Theme. Theme by MVP Themes, powered by WordPress.