AI Agent Hacks Another AI Agent Inside Google — An Agentic Supply Chain Bomb

Justin Beals:  Hello everyone and welcome to Secure Talk. I'm your host, Justin Beals. A security researcher was going through Google's public code repositories one at a time, looking for the usual kinds of flaws. In one of Google's projects, he found a helper AI agent, the kind anyone can talk to. You file an issue in the code repository, it reads it, and it comments back. People had been throwing prompt injections at that little agent for a while, trying to make it misbehave.

And nobody had gotten anything useful out of it. What he noticed was that his public AI agent was commenting on the repository as a collaborator. Now that is a trusted status. That permission goes to people the maintainers know and rely on. And in the same repository sat a second AI agent, a powerful one, that only trusted collaborators were supposed to be able to call. So he asked a simple question.

What if the agent anyone could reach was used to trigger the agent almost no one could? He forked the project and he tried it and it worked. One agent, out in the open, pulling the levers of a lockdown agent sitting right next to it. When he told his manager, the response was, You found an agent triggering another agent? Why are you not reporting this? Nobody had described this before. And that is our guest today.

Dan Lisichkin is a security researcher at Pillar Security, and the work is novel enough that he's taking it to DEF CON. All of it happened inside a code repository, which for most of us is the crown jewels. It is the fastest path to the most sensitive data we have. Non-human and agentic identity is one of the hottest topics in security going into this year. And this is why. We are handing more and more of our work to agents, they review our code.

They triage our tickets and they open pull requests. Most companies are already running several of them, and a lot of those agents talk to each other. Almost nobody has tested what happens when you let one turn on another. The idea underneath Dan's work is the part I keep thinking about. For most of my career, security was built for machines that behave the same way every time. A buffer overflow is deterministic.

Justin Beals (02:27.202)
We learned how it happens, how to repeat it, and how to close it. An AI agent is different. It is random by design. You cannot predict what it will always do. So the old habit of plugging holes by writing more rules starts to break down because there are infinite ways to write text that lands a prompt injection. And we have been giving agents the wrong kind of trust. Google treated its triage agent.

Like a human collaborator. We give a database engineer broad access because we believe in their judgment. An AI agent has none of that judgment, so it needs permission scoped down to the exact thing it is allowed to touch, and nothing more. We have been handing agents the trust we reserve for our most senior people, and they have not earned it. Dan and I get into all of it: how we built the automation that hunts for these setups.

Why the contribution guidelines Google published became his roadmap for the attack, and why he does not think we should slow down, even though both of the flaws he found could have led to full supply chain compromise, the same family of problems as solar winds. Dan is a cybersecurity researcher for pillar security, focusing on AI security, adversarial threats, and securing AI-based systems. Dan has extensive knowledge in areas including malware analysis.

Reverse engineering, threat intelligence, and offensive security tactics. Prior to joining Pillar, Dan was a cybersecurity researcher for Simulate, where he specialized in threat intelligence and threat hunting, a threat intelligence researcher for clear sky cybersecurity, and an IT consultant for Ernst & Young. Join me today in welcoming Dan on Secure Talk.
—-


Justin Beals: Dan, thanks for joining us today on Secure Talk. We're really glad to have you.

Dan Lisichkin: Happy to be here.

Justin Beals:  Excellent. So today we're talking a little bit about the recent flaw vulnerability that you found, AI flaw. And before we dig into it, I want to start with your methodology a little bit because I was very curious about how you're working. And one of the things that stood out to me is that you don't hunt these flaws by hand. As a matter of fact, you're building a lot of automation for yourself that scans public projects and flags AI agent setups that might fit a pattern. 

I'm just curious, in plain terms, what is that tool looking for? How'd you come about building it? What's that been like?

Dan Lisichkin:  So,  at Pillar we got really interested in CI/CD workflows on GitHub. I was personally, this project just started as me personally wanting to just learn about CI/CD vulnerabilities, and I was researching Google Cloud Shell, the product, and if somehow I leaked all the which are public, it's a public list of all the allowed listed Google repos, and I wanted to automatically hunt for classic CI/CD vulnerabilities inside of them. 

And so I got clock code now, everyone has it. And I just told it collect all the common CICD vulnerabilities you have. Like, I just had a whole list. I researched as much as I could, manually curated the list, and it created this automation. And by a side effect, it also added a tag for workflows called AI agent injection. Basically the Clauco decided that this is also gonna be a vulnerability. I didn't even think about it because I didn't even know they're integrating agents and CICD workflows on GitHub.

And I ran the automation on all GitHub on all Google's repositories. And I actually went repo by repo looking for classic vulnerabilities. I didn't even think about going to the AI prompt injection vector because I didn't think there was anything there. And when I was done looking at all of them, I had like this dashboard and the and the clock code already built everything, like it would explain what it thinks there is there and what there isn't, and everything was a false positive. And I was just left with the AI prompt injection workflows, and I went inside, and I saw on the GemLai CLI repository. This is even a different vulnerability that I found before this one.

And I found that it's susceptible to prompt injection, which is not that interesting, but I could gain code execution through that prompt injection. And then I realized, whoa, Eureka moment, I need to go and map all the coding agents that are embedded in workflows, mapping codex, cloud and workflows, and Gemini. And It kind of worked with Gemini, and I just started focusing on that. And I started mapping out where Google uses agents inside their workflows. And I had this automation that would just, like, trigger every day if Google added a new repository with an agent. And that really helped me go through their entire, like, the entire repos Google has to offer.


Justin Beals:  Yeah. I find this so intriguing. First off, we you know, we use a lot of AI tools in our work. I do in my day-to-day work. Obviously, use cloud code to help develop an analysis, but there are like in a probabilistic situation, there's new things that it recommends that I might not have thought of. I think that's the most intriguing thing to me about some of these models, these LLM things, 'cause it allowed you to kinda have a research partner to work with. Yeah.

Dan Lisichkin: Right. I would also add that at Pillar we have our own harness. It's basically the way we talk to Claude, we have all these scenarios and engagements we did for vulnerability researching. And I think what happened is that in its side, its context, when it was doing all this work for me, it already thought about I'll help, I'll help him with exactly what he's looking for. So I think that was the byproduct of that.

Justin Beals: Yeah. okay, I'm off on a little bit of a tangent here, Dan, but it feels like when I think about the large LLMs, like the companies, anthropic or or open AI, I do feel like over time what's gonna start happening is that either our implementation of those tools is gonna be very custom for us and what we're trying to achieve, or the models themselves, like pure open source models, will be things we adopt and place in our own code and infrastructure, and how we want to use it. 
And it seems like this is the very nascent part of these LLMs where we just rely on a website to help us. But the future of it, from a computer science perspective, is a lot more control.

Dan Lisichkin: Yeah, I agree. I don't think I, as a security researcher, when I started five years ago in malware research, I used to program a lot, and I don't do that much anymore. It's crazy how much I don't do that. And I try for, like, good habits to steal control, like have control over what I do to not gain any technical debt, but I must say it's completely integrated into my work cycle, and it made everything so much better. Like, it's not a gimmick. It really changed how I work.

Justin Beals: Yeah. Now your tool flags these targets. In this case, we were looking at CICDs or projects in a Git repo that had a vulnerability. And the hard part sounds like setting up the tooling; the actual vulnerabilities were fairly straightforward to exploit. Like it was prompt injection. It wasn't a very complicated challenge necessarily.

Dan Lisichkin: So the prompt injection is I would say the easiest part because you all only have to tell the yeah, you only have to trick or yeah, trick the the model to output text you control. I would say the actually hardest part was finding how to exploit these vulnerabilities. Because the
LLMs are not very good at finding novel things that didn't happen yet, or finding, like, these vulnerabilities are not something that happened before per se, not this instance of the vulnerability happening again. So you have all these models finding, you know, zero-days of buffer overflows and SQL injections or, you know, whatever memory corruptions.

These type of like agentic kind of chaining mon vulnerabilities are not something that the models are very tr well trained on. So I would say the easiest part is finding the prompt injection. The hardest part is actually figuring out: can I do something useful with this prompt injection? Because how I think about it when I research these vulnerabilities is that it takes me back to this famous blog post by Simon Wilson, the lethal trifecta. Basically, he said for an AI agent to be vulnerable, it has to process untrusted data from an outside source. It has to have private access to data, and it also has to have ability to externally communicate this data somewhere. And this is the hardest part. With it, I already found four vulnerabilities. The always hardest part was not how to get untrusted text to be read by the agent, is how to get the agent to read untra like to read private data or to execute code and get access to private data, and then somehow communicate that data somewhere. So this is always the hardest part. I can't say DLLMs are very good on, “look, this is the way you do it”. It's it's you gotta you gotta you gotta figure this part out yourself and be kind of creative with what the infrastructure you're working with gives you.

Justin Beals: I think there was a lot of creativity in this latest vulnerability, and you're sharing it at Black Hat most recently. I think the blog post is I'll just give you a call- agent-to-agent privilege boundary failures in CICD. Let's just start with you talking a little bit about this particular vulnerability you found and what happened.

Dan Lisichkin: Def Con. 
So I was riding this big wave. I found two vulnerabilities on Google, and I was feeling, my god, there has to be more. They're embedding all these agents everywhere. And these are very novel techniques. I was very excited, and I found that on the ADK repository, the agent development kit, which is the official Google repository for their SDK to build agents, that they use agents, but they don't use it like you would expect it to, like on Gemini CLI, for example, where the workflows that do use agents execute code inside the CI runners.  Here the agent that they used with ADK would only process text and it could interact perhaps with the repository, maybe to process issues or process PRs. They had different workflows that used ADK. One was for processing issues, one was for processing PRs. And like I said, they would process untrusted data. 

In fact, I already saw researchers forcing them to output attacker-controlled text. And they actually succeeded in making it output controlled text. But there wasn't a very like I hear you the gotcha moment, like, it's doing prompt injection, boom, exploit. I couldn't figure out what to do with it. And I noticed, additionally that they do have the Gemini CLI, like classic workflow that
does just runs code inside a CI runner. They would use it. I saw they were kind of using it to review other PRs, maybe comment if the PR is good, stuff like that. And I couldn't really figure out how to connect these two together because I knew that the Gemini CLI workflow, if I can somehow get a prompt injection in that workflow, I could X maybe exfiltrate the GitHub token, do something with it. 
But there was like a clear path there. And I actually called my manager up. He was like, maybe you should see maybe you should see what this agent is commenting, how is it interacting with repository. And I came back after like, it was after two days already researching the specific repository, and I saw this anomaly that the issue traging agent, a very low-privilege agent that people were constantly opening issues, talking to it. Researchers were bombing it with prompt injections. It's actually commenting on issues as a collaborator, not a bot user, not like a GitHub bot user, but it was commenting out as a collaborator, which is a pretty high user on a GitHub repository. Like, usually very trusted members only get access to this status. 

In this specific scenario, Triadgers, PR triaders would get access to the collaborator user status. And I thought to myself, like the obvious, well, we have a gated workflow here, like the Gemini CLI workflow; only admins can interact with it; only collaborators, admins, and members can interact with it. Here we have a user that I could make it output text as a trusted identity; what's gonna happen if I make it trigger this workflow? And didn't I mention it? The way the gaten workflow was triggered, that privileged user would have to comment out slash Gemini CLI jet slash or add Gemini CLI and then a prompt, and then it would do stuff in accordance to that prompt.

And I forked this repository to my own environment, and I tried it, and it worked. And I was very, very surprised that it did. And then I actually called my manager up, and I was like, his name is Elon, and I told him, this is not a very interesting vulnerability. Because the only the the maximum impact here is that I could exfiltrate a GitHub token that could modify a PR maybe, maybe modify some issue. It's not that strong. And he was like, Are you crazy? Like this is an agent triggering another agent. Why won't you report this? This is a real vulnerability. This is like no one talked about like I didn't see anyone talking about this before. 

And yeah, and I was like, okay, let's do that. And so the hard part was already from the Google's perspective was actually I my complaint, I think it was valid. In the end, they didn't they didn't give me a reward for this specific vulnerability because you had to do some impact. Maybe we can talk about that later, like the impact analysis for this specific vulnerability.

Justin Beals: Yeah. I think what is before we move off of the architecture of the vulnerability, I think what is really intriguing, and the first thing that we saw or I read when I was reading about the press release on this, was kind of the agent-to-agent communication. Certainly, in systems we're working with, whether it's like internal work or even when we think about our own product, we have agents that are interacting with other agents.

You know, we're i it's not unusual; like, this is object-oriented programming to me, right? Dan, it's like we're gonna build this thing, it does something specifically, and when we want to impact with it, we call it from another system. And now we're just baking intelligence or probabilistic models into those interactions. And it is absolutely intriguing to me. And let me see if I get this right: what we have is basically a low-privilege agent, one that's open to the public, being used to attack another one inside a Google system, which is very common now, and especially around a Git repo. Cause I think your manager's right. To me, a Git repo, especially for the code we deploy, is the crown jewels. It's the fastest access I would have to the most amount of data.

Dan Lisichkin: Yes. I already seen that this architecture is very common where an agent, like in a multi-agent architecture, you would have multiple agents talking to each other all the time. And it's not necessarily that it's automatically exploitable, but it creates a new attack surface that didn't exist, that we weren't aware of, and I would say it's there's a lot of undefined ways you can interact with that. It's not modeled. You know, like if you look at a buffer overflow or SQL injection, like these classic vulnerabilities, we talked about these. We know how they happen, we know where they could come from and why and how to prevent them. And I would say a multi-agent architect a multi-agent system we don't really understand how these interact. Like it's not like it's it's not like science we don't understand. It's just we didn't think about what could go wrong inside a system like that. And this is exactly what's novel about this vulnerability is that  it was unexpected behavior. It was like it's a weird machine; how it would be called in the academic field of exploit development. It's where one input inside a machine leads to undefined behavior. And this is what's unique about the system. And yeah, it's pretty prevalent. I would say it probably exists somewhere right now. No, no one really just tested it. Just like in this repository, people were probably injecting this agent. They were talking to the low-privileged agent all the time. People would make it coming out; attacker-controlled text. They would try to make to fool around with it, but no one really realized how to weaponize it yet.

Justin Beals: Yeah. And to me, this is a concept shift for all of us that have worked in the computer science field or even security leadership. Is that I think, to your point, a buffer overflow is very deterministic, right? It'slike a real number as opposed to an imaginary number if we're talking about mathematics, right? But the mod a model is random by design on some level.
You cannot predict what it will always do.

Dan Lisichkin:
That's exactly like people talk about the OpenAI Hugging Face incident. Say the model went rogue. It didn't go rogue. It's just not a model is not a definitive program. They gave it input, and it just went out and did just outputted unexpected output. No one expected that it would be like go and enter Hugging Face, steal the data, fetch it as a solution. No one really modeled that could help it happen because models, LLMs, are just not definitive machines. That's what makes them such a prime target for exploitation when they are embedded in applications.

Justin Beals:  So between the two AI agents, one low-level privilege that you as you pointed out, a lot of people have been doing prompt injection, and a high-level one meant only for trusted maintainers. How did you get the public one to pull the powerful one's levers? Because I'd imagine the more powerful model had some structure around security for it.

Dan Lisichkin: So you'd be surprised that actually pulling the lever on the powerful one was easy. Making the lower-privileged agent come out my con attacker control text, like the specific attacker control text that I wanted to like it had to activate the other agent with a very specific prompt. So telling it to do something that would look malicious, or not in the scope of what it was meant to process, actually made it ignore me several times. The way we solve this at Pillar specifically is we have not to do product marketing, but we have a very great blog post call that my colleague wrote; mostly I helped him. It's called a autonomy of indirect prompt injection and basically uses something called the CFS framework. so for a prop indirect prompt injection to work properly, it has to have three pillars. So it's context, format awareness, and salience, instruction salience. And so a good indirect prompt injection has to reflect the deep understanding of the system tasks, the format awareness. Is it a type of format that the system or that the content belongs into? Something the system would even process; would it even notice it? And instruction salience is the prompt injection position somewhere that the the model will notice it. So, for example, if you upload a PR with a prompt injection, you have to embed the prompt injection somewhere for the model to notice it. And so, we inside our, like, clock code hardness, we have this already embedded. And now I just needed to give it the proper mo like, the proper guidelines to build this indirect prompt injection. And so, inside the Google ADK repository, specifically for the first vulnerability, they use the PR triaging agent, and when I try to push a prompt injection through a PR, it will constantly ignore me. 

And so what I did was already very easy and automated. I saw that the agent was only triaging PRs if the con they they followed a contribution guidelines in CONTRIBUTING.md. They have a like, a file there that explains to maintain like to contributors how they should upload a PR. And the agent has to follow that as to notice if a PR follows that; otherwise, it just defaults to some answer like This is not good. And I just gave the contributor MD file to the hacking harness, and it just built the indirect that can push the PR, just had already everything, and that would eventually trigger the low-privilege agent to really process that prompt ejection because, if you think about it, it has to go through a lot. It's reading all the metadata of the PR, so its author, title, comments, content, the code content. It has to process everything, and the prompt injection has to be just somewhere right to lead the model away where you want it to go and output exactly what you want. 
This part is completely already automated. This is not very hard. And if it's not, if it's too hard on the second time, like the first or second time, I can just let it run for 20 different prompt injections. I don't care; it happens on my own system. So eventually I would find it.

Justin Beals: Yeah. I mean it's a hard I think what is I'll call it funny to me a little bit on this Dan, is it's like we we see these layers of security, but they become attack surfaces at the same time. Like someone published a rule book around how to make a contribution. You use that rule book as the roadmap to plan the prompt injection that you want it in a way. Yeah.

Dan Lisichkin:  Yeah, exactly. Exactly. But it's it's all part of the attack surface. I mean I again, I consider building the prompt injection. It's the easiest part, as far as I'm concerned, because eventually what differs a jailbreak from a model is with a jailbreak, you really have to, like, overcome a wall that the model was trained on not to do. Like, you have to, like, cram that wall; you have to break it apart. It's really hard. It becomes really hard. And these models are the frontier models are very strong at preventing jailbreaks. It's still possible. But for me, that's not the interesting part because I'm not interested in jailbreaking the model. I'm just interested in taking it where I want it to go, and that's enough for me. And for that to happen.

The prompt injection has to kind of land, like, flow through this allowed path as if it's like going through this minefield and just land exactly where I want it to, and there are infinite ways to write text. And so I don't know if this problem is even solvable in the long term. And I don't think it's what we should even focus on when we're defending against these types of attacks.

Justin Beals: I mean, it does seem Sisyphean in a way, you know, to try and figure out how many holes you can plug in a model that you is not deterministic by writing more and more rules. I mean, I think about all the rules written around sporting, you know, in certain ways, everything from, you know, even just simple soccer football, right? It's like the rulebook is massive, and they're constantly writing new rules because people are finding ways to get around them, and we're dealing with a very similar situation. I don't know that you can write enough rules for a model that you give a high level of access to.

Dan Lisichkin: Yeah, I don't think it matters because I mean, who cares if a model replies I like cheese to when it's meant to be processing IT tickets? Like, it has no impact on anyone. And
It's not where we should even focus our efforts with security because you can defend against this problem. And it doesn't really matter at the end of the day also.

Justin Beals: Yeah. Let's talk a little bit about how Google kind of responded to what you told them about. Now they did not provide a reward or a bounty for this particular vulnerability, but they did, I think, put some solutions in it. Or maybe I'm wrong. Yeah.

Dan Lisichkin:  So the first vulnerability is it has only social engineering impact. So I stole the GitHub token. It had pull request write and issue write permissions. That meant I could modify any issue and any PR inside the repository but that doesn't mean, like, code-wise. If it was code-wise, that would be very dangerous. I can only modify metadata. So only comments, labels, stuff like that. I could not trigger anything else with these capabilities. What I could do,
And that was my impact analysis: I could take a PR that was already, like, I could, as an attacker, open a benign PR.

And then once it would get triaged, someone would comment like This needs a review, add a label, needs review. And I could open another PR, prompt the prompt injection, activate the other agent, exfiltrate the GitHub token, and then modify the PR lifecycle where I would modify the triagers' comment, like maybe looks good to merge, running Gemini review, and then push some fake Gemini review comments up in also push malicious code. As with my GitHub token, I had this, like, cosmetic capability to approve the changes, but not approve the PR, and then make it look like as if this whole thing was moving along and it's just about to get merged. And they said, This is cool, this is novel, we're gonna credit you for it on our leaderboard, but we don't pay for social engineering impact.

And on another vulnerability which was discovered the same day, the next day, I was at my grandma at Shabbat dinner, and I was kind of bored. So I went back on this repository and I saw they added more workflows that could be triggered with this exact vulnerability. And for this specific second vulnerability, I could exfiltrate a sensitive service account, a GCP service account, which they say they it had some permissions, but they're still they're still processing that second vulnerability. They're overloaded with AI reports right now, so there's there's no verdict on the second one. That's what they say.

Justin Beals: No, that's what they say. Yeah, absolutely. I mean, I think there is a discussion, right, Dan? about if we're overloaded on vulnerability responses, maybe we should slow down the amount of feature set we put in the market. You know? Yeah. g

Dan Lisichkin Yeah.Perhaps 

Justin Beals:  So I understand this helps clarify a little bit. Like the first vulnerability was a lot about infiltrating basically the software developers that are working on the project and the process flow and being able to social engineer some aspect of what you wanted. But the second vulnerability you actually started seeing was being able to eject code. Yeah. Yeah. Yeah.

Dan Lisichkin: There was more juice on that one. I could also push BRs in the name of that bot identity, the ADK bot identity. I could push a fixed BR as that identity. But they they they also said it's social engineering impact and they're actually looking at the credentials I managed to exfiltrate. I don't know, time will tell.

Justin Beals: Yeah. What do you think about these bot and their bots and their identities, right? Like, what's intriguing to me is it feels like sometimes the access management for an agent is like our old API key and secret. And even logging the after-action report on an incident can be hard without this agent has this identity, you know, it was given to it by this person or human being or what's the trail there, and I'm wondering if you have an opinion about that.

Dan Lisichkin: Can you clarify the question?

Justin Beals:  Yeah. So let's say we're running an agent, or you're operating an agent, and the agent is, let's say able to produce a vulnerability or an exploit on the other side. Tracking who that agent is, at least from a, like, after-action or an incident report perspective, still seems nascent. Like sometimes we're just allowing them to operate. We know what the agent was, but we don't know a lot about it other than it was this agent instantiated.

Dan Lisichkin: Right. So Google has, like, all the big vendors have some solutions to agentic identities. And I didn't dig into it too much. But we're starting to see this problem which emerges that we don't actually know what the agents are doing on our behalf, and we don't know what they're doing inside CICD, for example, where they're just like, expected to behave as this sentient identity that does stuff. And we need to start when we model the agent inside our attack, like in our attack models that we considered we consider to be attack surfaces, we need to start thinking like how we track this identity that does stuff, and what kind of access do we give it. For example, for the ADK repository, the entire attack could have been prevented if they wouldn't have treated this bot as a human. So, normally in a GitHub repository, when you see bots, just like regular bots, not even AI agents, they usually have either a GitHub App application identity or a get GitHub Actions identity. And these differ and completely from like a user personal access token identity in the way that they cannot recursively trigger other workflows. So GitHub Actions bot, this identity that usually is scoped to GitHub applications or bots, could have not triggered the privileged workflow by commenting something out. That wouldn't have happened ever. And so that was already like stop on the entire vulnerability. And the second part is where we use agents in the cloud, maybe. we start like this new machine and give it put an agent there and just starts doing stuff. Maybe we should start considering by not only giving it an identity, but what resources can this identity access and what kind of actions can it perform on this specific resource? So we know agents are very useful. We like them. We use them a lot now.

And we don't want to take capabilities away from them. So maybe an agent that adds, modifies a database table with columns or rows, we would want it maybe to ax have access to the database and maybe even have access to the table. But we only want it to have access to this specific row or a way it could interact with the table.

And that's completely different because if you would give this access to a user, you would give him access to the database, you would give him access to a table, and you would let him use that table. But here we need to start thinking what kind of what kind of specific access does this identity have to this specific resource. And this is where I think things should start going. because the solution can't be just like, we're taking this toy away from everyone. Because it's it's really useful. I use agents all the time. Everyone uses agents all the time. Automations are not gonna go away. We can't, our solution cannot be just like, I'm restricting this. No one can play with this anymore. It's not gonna work, and no one's gonna listen. And so we need to start thinking about this specific solution. And I think Google and AWS, they they have all these kind of specific solutions to where you mandate how this igentic identity can interact with specific resources. And for personal use, that's a bit more tricky because now you can have a clock code instance running, and it's running on your behalf with your privileges, with your user identity. And say if I would be an attacker and I would manage to get initial access to a computer, maybe I would want to use Cloud Code to do all my operations and not necessarily for the speed of infiltration, lateral move and and exfiltration of data or even impact, but to hide my identity because the agent, agent's traffic just kind of blends in and everyone just assumes like it's okay that it's doing it. We don't really have, we don't really think about that. 

This problem is relatively new. at Pillar, I could say that we have like these, and we have an endpoint solution and we treat treat agentic behavior. We try to model it with rules, what's allowed and what's not. And that I just I think this is kind of an iterative cycle that we're gonna have where we try to understand what's okay for agents to do and what not. And that could be for the IT team and the security engineer team to decide.

But this kind of thought process has to start happening because we have to have some control over what these entities, let's call them, are doing.

Justin Beals: Yeah. I think what stands out to me, Dan, are two things. one is is that I think what you're asking for is a is a deeper level of granularity on what we usually consider roles and permissions, right? Like to your point, a user will have ac access to a database, maybe an engineer or systems administrator, and they're allowed to have that access because we think they're have the right intelligence and skill set and critical thinking capabilities to deal with data that's sensitive. But an agent may need a much more granular set of permissions than that human being is provided because they have none of those things.

Dan Lisichkin: Right. Took the words out of my mouth. Exactly. The specific word I was looking for is granularity. Exactly.

Justin Beals: And then, you know, I think you also mentioned believing in a human in the loop in all the right places, right? Like it's it's one thing to have these things operating. It is also important to manage them just like we would, you know, human workers and on a team. And there's certain areas where the model needs to stop, or the agent needs to stop and check in with a human and say, Am I doing the right thing?

Dan Lisichkin: Yes, but also no, because again, yeah, you can put a human in the loop. No one used to run Claude with dangerous skip permissions before, but you know, it's really useful. And I would always run it with, like, you can do this, you can do this, but eventually, like you get like people say, you get like, review fatigue, but I also think we wanna use this technology, and push it to the limit of what it can do for it to be super useful. I I at least I would imagine people want to do that. And yeah, human in the loop is a nice solution for some problems, but in an ideal world, you would want to have an agent running unsupervised all the time. And for that to happen, we need to think about solutions for that. Cause that's because that's where we want to go. That's that's where people are gonna do anyway. 

I men look look at OpenClaw, for example, right? Like when it came out, every person with security hygiene would just like, what are people doing? Are you connecting this to your bank account? Are you crazy? Are you giving this access to your email accounts? And people were like, even security engineers, yeah. I mean, it's useful, right? And well, yeah, it is. And we need to think about how the fact that people are gonna use this anyway, matter if we scream that AI is dangerous and we shouldn't connect it to stuff, people are gonna do it, people wanna do it. It's fine, it's natural. We saw it in the early 2000s, when new technology was evolving, and you know, companies were pushing unsecured new technology everywhere. And no one's saying, no, no, no, no, please stop this. This is not right. We should kind of wait. 

And we didn't wait, and we got crazy vulnerabilities. We lost tons of money. But in the end, we reached this specific point where we have good security practices, and we advanced technologically, and we have great companies and great technology, and we can do all this crazy stuff. And we had to add this like gap where, you know, some random guy from Ukraine would find a vulnerability in the SMB protocol. And then one person would accidentally click something, oops, everyone's infected with some internet worm, and we're losing like billions of dollars. 

But you know, you close that gap, the damage was done, it's not gonna happen again. People are still using SMB, and we shouldn't go through this place where human in the loop, or what are you doing? Like it's very tempting; it's very easy. I would say it's probably right sometimes. I wouldn't personally connect an OpenClaw, my bank account, but people but I'm not most people like I'm a security researcher, I come from malware research.

Like I know what malware can do to a computer, I know it can ruin people's lives, I know what APTs are doing, but I'm not most people. Most people would want to know what happens in their bank account and they want their agent to call them up in the morning and say, like, hello, wake up, you have this X and Y and Z meetings. And that's fine. And we should aim to solve this problem because it's gonna happen anyway.

And we can feel righteous in our little security bubble, and it feels nice sometimes, but it's it's not it's not gonna solve anything. So human in the loop is a good solution for some problems, but I would imagine that we should think of how we solve the autonomy problem, because sorry if this answer is already like taking too long. Look at these agents in GitHub workflows, like they review code.

They tell you if your code is good, they review PR, they label issues, they tell maintainers like, is this a duplicate? Is this not? It's very useful. On Gemini CLI, you have more than 100,000 stars. The amount of activity that's happening on that repository, agentic and human, issues being open. It's there's they're sunsetting Gemini CLI, by the so maybe it kind of lowered down. But there's a there's a lot of life happening there, and you can't expect 20 contributors that are working on this to even have the capacity to process everything that's going on there. And if you put that human in the loop, you know what problem did you solve exactly? You're still having one guy to go through all these issues saying that this is okay, this is okay, this is okay, this is okay. And Google did that. And the two vulnerabilities I found for Gemini CLI had catastrophic impact. Catastrophic. Both of them could have led to a full supply and chain compromise. That's bad. But th this cannot happen again. And it was reported, and they closed it, and it's over. And maybe a bad guy would have found it, and there would have been a supply chain compromise. 

But we're talking about people all of us we went through solo winds and we went through crypto locker and we went like and and these are these things back when they happened back back in 2013 when crypto locker happened where everyone woke up and they had all these big grand scary ransomware screens and there was a they they found a a pretty bad exploit and they used it but it's closed same for solo winds we went through that incident and it's closed it's solved

We learn from this. And it's better if we have this thing happening right now where people are pushing this technology to the edge and don't fully understand what they're doing, rather than having these little gates that we think make us safe, but not really, because no one's gonna listen to you when you say you want a human in the loop, and everyone should mandate that. 


Justin Beals: So I think that what I like about what you're describing, Dan, is a little bit of what I might call a Pandora's box issue. Like, this technology is out here in the wild. We are, I mean, I think I read recently a Cloudflare executive talking about how there's more machine activity on the internet than human activity at this point. And we're not able to manage these digital spaces without machines helping us at the end of the day. 

It's just like It's all stacked up and and as security people, you know, yourself deeply as a researcher and someone that's interested in helping solving problems, what we need to do is just be leaned into solving those problems because we're not gonna find the we're not gonna find them if they're not in the wild being used at the end of the day.

Dan Lisichkin: Yeah, we need to get used to this idea that we crossed this point where we did invent something that is really great. When ChatGPT 3 was released, no one really expected this thing to happen. Eventually, Asians found, like, models found their way inside this unique.
area that just sits between LLMs and application security. And we decided we're gonna merge. We're gonna merge these two things together. And we really created something big, and it can't go back now. It's too late. It's too useful. It's too good. It solves too many problems. And we should be really thinking about how we are going to move on from here from this new perspective. And that means rethinking how we look at exploitation and how we look at our security boundaries and attack surfaces and attack models. And we need to start putting agents inside of them. I wouldn't say just LLMs, but agents. And that's okay. That might be a bit scary, but we should just go with the flow, and that's fine.

Justin Beals: 
Yeah. Dan, I and I know my listeners really appreciate the work you do at Pillar and the research work and loved reading about the exploit and also your methodology. And so we're really grateful for your work and grateful for joining us today on Secure Talk.

Dan Lisichkin: I'm happy to be here. Be sure to check out the rest of my blogs.

Justin Beals: Excellent, Dan. We'll include a link in the comment section of our post for this episode.

Dan Lisichkin:  Perfect.