When an AI suggestion nearly became a bad install
I nearly let an AI agent talk me into installing a package I’d never heard of.
That sounds dramatic, but the setup was pretty ordinary. I was working through a routine coding task, the kind that makes you want the cleanest, quickest answer possible. I asked for help, got a package recommendation back, and at first glance it looked totally harmless. The name sounded like something that could belong in a normal project. Nothing about it screamed trouble. Nothing about it looked like the sort of thing that would make a developer stop, squint at the screen, and mutter, “Hmm, that feels off.”
And that’s exactly why the moment mattered.
If I’d been in a rush, I could’ve accepted the suggestion and moved on without thinking twice. Copy the name. Run the install. Keep coding. Done. That’s the tempting part, right? AI agents are very good at making things feel easy, and sometimes that ease turns into a tiny trap. The recommendation arrived with enough confidence that I might’ve treated it like a decent shortcut instead of a thing that needed checking. A package suggestion from an assistant can feel a lot like a suggestion from a teammate who already did the homework.
Except it wasn’t a teammate. It was a model. And models can be very persuasive while still being wrong.
A smooth answer is not the same thing as a safe one.
That was the moment I had to slow down and remember a boring little rule that saves a lot of grief: don’t install anything just because an AI says so. Give it a human sanity check first. Not a five-hour audit. Not a full security review for every keystroke. Just enough checking to make sure I wasn’t about to drag something shady into the project because I was feeling lazy for ninety seconds.
This kind of mistake is easy to make because the task itself is usually so dull. You’re not designing a new system or debating architecture. You’re just trying to get a dependency in place so you can keep moving. That’s when bad judgment slips in. The faster the task, the easier it is to trust the first answer that looks tidy. AI agents are especially good at feeding that impulse. They hand over a package name, a command, a code snippet, and the whole thing feels polished enough to use right away.
That’s where software supply chain security stops being a buzzphrase and turns into an everyday habit. You don’t need to become paranoid. You do need to pause before letting an unknown package into your codebase. A few seconds of doubt can save you from a much uglier cleanup later. And honestly, that pause is cheaper than explaining why a random dependency got a free pass because I was trying to save two minutes.
I didn’t install it on the spot. I checked first. That tiny delay is what kept a routine coding task from becoming a bad habit with a potentially nasty consequence. In the next part, I’ll show the specific thing that made the package look legitimate at first, and why a quick look at the source changed my mind fast.
The package looked real — until I checked GitHub
The funny part is that nothing about the package name looked weird at first glance. It sounded like a normal little library, the sort of dependency you’d expect to see in a project without making a scene. I was doing a routine coding task, the AI agent tossed out a suggestion, and for a second it had that annoyingly confident vibe that makes you think, “Sure, that seems fine.”
A package can sound ordinary and still be a terrible idea.
If I’d been moving fast, I probably would’ve accepted it and kept going. That’s the trap. The suggestion fit the task well enough that it could’ve slipped straight into the codebase without much thought. No drama, no warning siren, just a quick install and on to the next thing. That’s exactly why this stuff gets missed. Bad packages don’t always look bad. Some of them show up wearing a tidy name and a clean enough description.
Before I installed anything, I did the one boring thing that saves a lot of pain later: I checked GitHub. Not the package manager listing alone, and not the AI’s confidence level. I wanted the source code, the repo history, and a sense of whether this thing had any real life behind it. That habit lines up pretty well with the NIST AI Risk Management Framework, which basically treats AI output as something to evaluate, not something to obey. The same general idea shows up in the OpenAI human-in-the-loop guidance for agents: let the model suggest, but keep a person in the loop before anything meaningful happens.
Once I looked, the red flags were hard to miss. The download count was tiny. The package had only recently appeared. That combination usually doesn’t prove anything by itself, but it does tell you to slow down. A library that claims to solve a useful problem and yet has almost no adoption deserves a closer look. So does a package that feels brand new in a space where real utilities usually have some trail of use behind them. You don’t need a lab coat to notice that something is off. You just need to pause long enough to ask whether this thing has any history at all.
The GitHub check made the whole picture less flattering. The package still looked legitimate on the surface, which is exactly what makes these AI suggestions annoying. There was enough polish to tempt me into assuming it was safe, but not enough evidence to justify the assumption. I didn’t see the kind of history I’d expect from something I’d trust in a real codebase. That was enough for me. I backed away before the install happened, which sounds almost too simple until you remember how many problems start with “I’ll just add this one dependency.”
If you keep a checklist for AI-assisted work, the OWASP LLM Top 10 is a decent place to start. It’s not a magic shield, but it does nudge you toward the right habit: don’t let confidence replace verification. I treated this package the same way I’d treat any unfamiliar dependency, except the recommendation came from an agent that had no stake in the mess if things went wrong. That’s a lovely arrangement for the model. Not so lovely for me.
That quick review kept a bad package out of the codebase. No cleanup, no rollback, no awkward conversation about how a sketchy dependency made it past review because I trusted a machine that was, in this moment, being a bit too helpful for its own good. The whole thing took a few minutes. Annoying? Sure. Time well spent? Absolutely. I’d rather lose five minutes than spend the afternoon explaining why I let a suspicious dependency anywhere near production.
And that’s the part that stuck with me. The package did its best impression of a normal library, but the numbers and the source code didn’t back it up. Once I had those facts in front of me, the decision got easy. I didn’t need a big policy document to tell me what to do. I just needed to stop, look, and refuse the shiny suggestion until it proved it deserved a place in the project.
Why this happens: AI can invent names attackers are ready for
That GitHub check bought me a few extra minutes, and honestly, it reminded me of the part people skip. AI doesn’t actually know whether a package exists. It can spit out a name that sounds tidy, familiar, and just convincing enough to get past a tired brain at 11 p.m. The NIST overview of artificial intelligence keeps the basic idea pretty plain: these systems are good at pattern matching, not at verifying reality. So when an AI agent suggests a library, I’m not looking at a trusted recommendation. I’m looking at a guess with a confident voice.
That guess can become a real attack. The trick has a name, and it’s a nasty one: slopsquatting. In plain English, it means attackers take package names that an AI invents, register those names for real, and wait for someone to install them. The package name looks plausible because it was built to sound plausible. The AI made up the name, the attacker made the trap, and the developer just wanted to get back to work.
A believable package name is still a trap if nobody checked whether it belongs to a real project.
Once that malicious package is installed, the damage can get boring in the worst possible way. It doesn’t need to do anything flashy. It can create a backdoor, steal credentials, read files, tamper with build steps, or quietly send data somewhere it never should’ve gone. In a development environment, that might mean API keys, source code, test data, or CI tokens. In a production pipeline, the fallout can spread faster than anyone expects because the package may get pulled into automated builds before a person notices the mistake.
This is where the whole “AI convenience” story gets a little messy. When an agent is allowed to suggest code, packages, or actions, it’s easy to treat that output like a shortcut instead of a draft. I get why. The tool is fast, the answer looks polished, and the package name has just enough structure to feel real. But a human still has to ask the annoying question: does this thing actually exist, and who put it there?
If you’re letting an agent do more than chat, the guardrails matter even more. The OWASP prompt injection prevention cheat sheet is aimed at a related problem, but the habit is the same. Don’t assume an automated system is making safe choices just because its output reads smoothly. The OpenAI agents running agents guide says the same thing in developer language: agents need boundaries. They are not tiny compliance officers. They are pattern machines that can be nudged, fooled, or simply wrong.
Softjourn’s Sergiy Fitsak put the warning in a way that stuck with me: convenience without verification is how teams get burned. That’s really the whole story here. The danger isn’t that AI is always wrong. It’s that it’s often right enough to make people stop checking. One believable package name, one rushed install, and suddenly you’ve invited something hostile into your codebase because it sounded helpful.
I’ve started thinking about that as a supply-chain problem with a very human-shaped weak spot. The chain breaks when I get lazy, not when the model gets poetic. And that’s exactly why the next step is so simple, even if it feels a bit tedious in the moment: treat every AI-generated package suggestion like a rumor until a real person has checked it.
My rule now: verify first, install second
I’ve made the rule pretty simple for myself: if an AI suggests a package, I do not install it on reflex. I check it. Every time.
That sounds almost embarrassingly obvious now, but it’s also exactly the kind of thing that gets skipped when you’re trying to finish a routine task before lunch, or before a meeting, or before your brain decides it would rather stare at a wall. The suggestion looks neat. The name sounds plausible. The path of least resistance is to hit install and move on. That’s where the trouble starts.
A package suggestion is a lead, not a verdict.
My little routine is boring on purpose. I look at the package age first. If it was created yesterday, I get suspicious fast. I check download counts too. A library that claims to solve a common problem but has almost no traction deserves a pause, not blind trust. Then I open the source code, or at least the repository, and see whether it actually looks like a real project. A README full of fluff and an empty folder tree don’t inspire confidence. Neither does a package name that feels oddly generic but has no history behind it.
That habit has become part of my developer security routine, right alongside reading dependency diffs and not committing secrets because I was “sure I’d remember later.” Package verification isn’t glamorous. It won’t make your demo look slick. It won’t earn applause in a standup. It does, however, keep you from introducing something nasty into a codebase because a machine sounded confident.
And yes, deadlines make this harder. That’s usually when the shortcut starts waving at you from the corner of the screen. I get it. I’ve been there. The AI says, “Try this package,” and your brain says, “Great, one less thing to think about.” But that’s exactly the moment to slow down. A recommendation from an AI agent is useful only after a person has checked whether it makes sense. Otherwise you’re letting speed do the deciding, and speed is a terrible security reviewer.
Here’s the version I actually follow when I’m tired and mildly over it:
- Check whether the package is old enough to have a real history. - Scan the download count and recent activity. - Read the source code, or at least enough of it to see whether it behaves like the name suggests. - Look for signs that the project is maintained by an actual person or team. - If anything feels off, I stop and find a different option.
That list isn’t fancy, but it saves me from myself. It also saves the rest of the team from inheriting a mess because I wanted to shave off five minutes. A bad dependency can sit quietly for a while before it causes trouble, which is part of what makes this stuff so annoying. You don’t always get a dramatic explosion. Sometimes you just get a weird build issue, a strange outbound request, or a headache that appears three days later and takes half a morning to trace.
I’d rather spend a few extra minutes checking than spend an afternoon explaining how a supply-chain compromise slipped in because I trusted a machine with no context and no skin in the game. That trade is easy. Slow is cheaper than sorry.
