The Traffic Wasn't Real: How I Found and Blocked Malicious Bots Hitting My Application

This one is really personal for me.

Traffic bots have gotten on my nerves before, but this time they actually got in the way of something I was building.

I was working on an application where traffic and the data coming from that traffic actually mattered. Then I started seeing some strange activity.

The numbers were moving, but not making sense.

So I got forced out of what I was doing just to fix this problem. Started looking at what was actually hitting the application.

Checked the server logs. The logs were much more interesting than the analytics dashboard.

I was seeing requests for pages that didn’t exist, attempts to reach common administrative paths, and strange payloads being sent to endpoints that had no reason to receive them.

It wasn’t people using the application.

It was automated probing.

I ended up having to learn how these bots were getting around, what they were looking for, and how to stop them.

Once I had that figured out, I added the same kind of security drill to the applications I build for clients.

Here’s what I found.

What were these bots actually trying to do?

Not every bot is malicious.

Search engines have bots. Monitoring services have bots. Other legitimate services send automated requests all the time.

The bots I was dealing with were different.

They were looking for something they could abuse.

That could be an exposed admin area, an outdated piece of software, a weak authentication endpoint, a vulnerable plugin or framework, or simply something that responds in a way that gives them useful information.

They don’t necessarily know what your application is before they start.

They send requests.

They see what responds.

Then they try something else.

And if they find something interesting, they can keep going.

That’s why the first thing I wanted to understand wasn’t “How do I block bots?”

It was:

What exactly are these requests trying to do?

The first sign: the traffic doesn’t behave like traffic

One of the first things that caught my attention was the behaviour of the traffic itself.

There were spikes, but the sessions weren’t behaving like normal users.

Very short sessions. Little or no meaningful interaction. Requests happening far too quickly to look like someone actually navigating the application.

A traffic spike by itself doesn’t mean you’re under attack. You can get legitimate spikes from a campaign, a crawler, a popular post, or something else.

But when the numbers don’t make sense alongside what people are actually doing on the site, that’s when I start investigating.

This is not normal.

The server logs told me what was actually happening

Once I looked at the logs, things became much clearer.

I found requests for paths that didn’t exist on the application.

I found attempts at common administrative paths.

I found requests containing payloads that made no sense for the endpoint they were hitting.

This is where I think a lot of people make the mistake of looking only at their analytics.

Analytics can tell you that 500 visitors arrived.

Your server logs can tell you that those “visitors” were actually machines making hundreds of requests looking for something that shouldn’t be there.

That’s a very different problem.

And in my case, the requests weren’t random enough to ignore.

They looked like someone — or something — was asking the application a series of questions.

Why were they requesting things that don’t exist?

Because they aren’t necessarily targeting you.

A lot of this activity is automated.

A scanner can work through thousands of websites looking for the same common weaknesses.

Try this path.

Didn’t work.

Try another.

Didn’t work.

Send this payload.

See what comes back.

Try another.

Most of the requests will fail.

They don’t care.

They only need one application somewhere to respond differently.

That’s what makes this kind of traffic annoying. Your application doesn’t have to be famous or specifically targeted for someone to start poking at it.

If it’s reachable, something will eventually find it.

How do they know what to try?

This is where things get interesting.

Different applications leave different clues about what they’re running.

The technology might be visible through responses, headers, URL structures, known files, error messages, or other behaviour.

Once a scanner gets an idea of what an application is using, it can start trying things that make sense for that technology.

That’s why you’ll sometimes see requests for paths or files you’ve never created.

The bot isn’t confused.

It’s checking whether somebody else’s version of the application has that path or vulnerability.

And if your application happens to expose something useful, the scanner has found a door worth trying.

This can mess up more than your security

This was another thing I had to pay attention to.

Bad traffic doesn’t just create a security problem.

It can also mess with the data you’re using to understand your business.

If you’re looking at traffic numbers and a large percentage of that traffic is automated probing, what exactly are you measuring?

You might think a page suddenly became popular.

You might think a campaign is performing better.

You might think users are behaving strangely.

When in reality, machines are just hammering your application.

That was one of the reasons I wanted to get the traffic under control instead of simply accepting it as “internet noise.”

How I decided to block it

For the applications I work on, one of the easiest places to put this protection is in front of the application.

I use Cloudflare for this. Once the domain is proxied, all traffic to my website will pass through Cloudflare before reaching my actual web server.

The basic idea is:

Internet → Cloudflare → Your Server/Application

Any request reaches Cloudflare first.

Cloudflare’s WAF can inspect incoming requests and apply rules based on things such as the URL, headers, IP information and other request characteristics. Depending on the rule, you can block the request, challenge it, or take other actions.

That means I don’t have to wait for every suspicious request to reach my actual application before doing something about it.

The rule I actually use

This is where I don’t want to give you a generic “block /admin” rule and pretend that’s security.

That would be a bad idea.

There are legitimate applications with admin areas. There are legitimate users accessing them. A rule that simply blocks a word in a URL can easily block the wrong people.

What I look for is a combination of signals. For example, for blocking requests with payloads to my JS files it would make sense to block POST requests to URLs that contain ”/_next/static/”. So the rule would look kind of like this:

(http.request.method eq "POST"
and http.request.uri.path
contains "/_next/static/")

And then I can bundle related patterns together where it makes sense.

(http.request.method eq "POST"
and http.request.uri.path
contains "/_next/static/") or
(http.request.method eq "POST"
and http.request.uri.path contains ".js") or
(http.request.method eq "POST"
and http.request.uri.path contains ".css")

The important part isn’t having a magical expression.

It’s understanding why you’re blocking the request and making sure the rule isn’t going to cause more problems than the traffic you’re trying to stop. You might need to have a little bit of knowledge about the tech stack you use to host/manage your business for you to have that full understanding.

If you don’t completely know you can use AI to help. Don’t blindly trust AI for stuff like this. Use it to help you not replace you because this is the kind of stuff that can ruin your business/application if you mess up.

You can even use Cloudflare’s AI to help create the rule

Cloudflare can also help generate rules from a description of what you’re trying to detect.

I’ve found this useful when I already understand the traffic I want to catch but don’t want to spend time remembering the exact expression syntax.

For example, you can describe the behaviour you’re seeing and ask it to create a rule around those signals.

You're a cybersecurity expert. Below
are the logs that include requests
from malicious bot traffic on my website:

[Paste your logs here]

I am using [tech stack you're
using here: Wordpress/Firebase/
.NET/whichever stack you use]
for my webiste. Create a WAF rule to
protect my website from malicious requests
without blocking legit user
traffic requests. Malicious
requests should be blocked.

But inspect throughly the result it gives you. This is security. Being a little paranoid is a good thing. Do this for all the malicious requests you see.

Cloudflare has a maximum of 5 rules on the free account. When you have exhausted all of them just bundle multiple rules into one rule by combining the rule expressions so that you don’t have to pay for this.

I don’t just block it and forget about it

This is probably the part that’s easiest to skip.

You create the rule.

You see the malicious traffic disappear.

You feel like you’re done.

You’re not.

I keep an eye on what the rules are catching and whether anything legitimate is getting caught along with it.

If a rule starts catching something I didn’t expect, I investigate it.

If the attackers change what they’re sending, I adjust or append more to the rules I have.

If I see a new pattern repeatedly, I decide whether it deserves another rule.

It’s more of a drill than a one-time fix.

What I do now

I don’t wait until an application is obviously under attack before thinking about this stuff.

The basic process I follow now is:

1. Put a security layer in front of the application.
2. Watch the traffic.
3. Look at the actual requests.
4. Identify the patterns.
5. Build rules around those patterns.
6. Monitor what gets blocked.
7. Adjust when necessary.

That’s it.

The important thing is that I’m not trying to block every automated request on the internet.

I’m trying to stop the requests that have no legitimate reason to be hitting the application in the first place, without accidentally blocking the people who actually need to use it.

One last thing

If your traffic suddenly looks strange, don’t immediately assume your website has become popular.

Look at it.

Look at what the visitors are doing.

And if it still doesn’t make sense, look at the server logs.

That’s where you may find the real story.

Because sometimes the “traffic” you’re looking at isn’t people visiting your website at all.

It’s machines asking your application questions.

And you probably don’t want to answer all of them.