I self-host a (non-critical) mail server and a few other things and occasionally look at live firewall logs, seeing the constant flow of illegitimate traffic hitting random ports all over the place, some hitting legitimate service ports but others just probing basically anything and everything. I decided to setup a series of scripts that detect activity on ports that aren't open (and therefore there's no legitimate reason for the traffic to exist) and block those IP addresses from the service ports since the traffic source isn't to be trusted.

Something that came out of analysis of the blocked IP addresses was that I discovered a few untrustworthy /24 networks belonging to a bunch of "internet security companies" whose core business seems to depend on flooding the entire IPv4 space with daily scans. Blocking these Internet scanner networks significantly reduced the uninvited activity on my open service ports. And by significantly I mean easily over 50% of unwanted traffic is blocked.

Network lists and various scripts to achieve my setup can be found here: https://github.com/UninvitedActivity/UninvitedActivity

Internet Scanner lists are here: https://github.com/UninvitedActivity/UninvitedActivity/tree/...

Large networks that seem responsible for more than their fair share of uninvited activity are listed here: https://github.com/UninvitedActivity/UninvitedActivity/tree/...

I'm semi-aware of the futility of blocking IP addresses and networks. I do believe, however, that it can significantly reduce the load on the next layers of security that require computation for pattern matching etc.

Be aware: there are footguns to be found here.

Good grief. A couple days ago I re-enabled password logins on a server that normally only accepts private keys, just to check something from a third location, and then forgot to turn it off. Two days later the server's logs were full of thousands of failed login attempts that started a few hours after I enabled passwords and then ramped up to dozens per minute.

Just because it didn't instantly say "Goodbye".

I checked ip locations on the biggest offensing addresses; all were in China.

I don't know what to call the idiocy and amorality that leads people to scan port 22 for a living (or the stupidity that leads them to guess random passwords for random usernames that don't exist), but I suppose that for every gardener there are a billion ants.

Coincidently, I recently visualized the scanners for fun by plotting them on a globe[1]. It gives a more comprehensive view of the locations and ASNs of the scanners. The demo data is generated from 1 day of logs.

[1]: https://github.com/simonmysun/where-are-the-scanners

Amazingly there's no request from same ASN. I believe this is because the VPS provider has a quite strict validation process, e.g. you have to upload a photo of yourself with your ID and your handwritten username, etc. I would suggest we consider the reputation or credibility of the data centers so that the data centers have the motivation of banning such users. In my case, a lot of the requests were sent from Tencent or Alibaba data centers.

If you have only public key authentication enabled with SSH I honestly don't understand why people bother with things like fail2ban. It just adds more moving parts with very little security gain.

The real risk is a zero-day in OpenSSH and fail2ban probably isn't going to protect you from that. In that case you are better served by putting another layer of defense in front of SSH like a VPN.

I am not sure why this should keep anyone from hosting their own servers and services.

I find it positive to know that whatever and whomever expose anything on the Internet someone will try to exploit it.

For 443 and 80, why the concern ? Outsiders can try all they want bit if you are certain the software you use is secure, there will be no cigar.

I'd much rather have these things out in the open than hiding things away with some obscure thought about that should help anything.

If something is difficult do more of it. The same goes for understanding security.

in the early 2000s I kept an anonymous ftp server open and would routinely get the latest cracked software delivered right to my hard drive. It was very convenient.

Over 90% of the ssh logins come from just a few China Telecom addresses. They just keep trying random ssh accounts over and over all day. I just geoblock China now. Maybe occasionally unblock it for a few minutes if the kids want to buy something from Shien. Then I honeypot the rest with the continuous ssh banner script.

I have a utility that parses ssh failed attempts and creates iptables blocklists:

https://gitlab.com/mtekman/iptables-autobanner

For those just wanting the blocklist, here is a table of malicious IP addresses, with columns of: address, number of ports tried, number of usernames tried.

https://upaste.de/bgC

Some time ago I set up a server for a website and I was appalled, like many others, by the number of SSH connection attempts. I decided to open SSH only in a randomly chosen port number above 1024 and now I have essentially zero probing attempt. It is trivial but for me is a satisfying configuration.

  > 1016 cd ~; chattr -ia .ssh; lockr -ia .ssh

Does anyone know what the "lockr" command is? I can't find any references to it besides people saying they observed malware trying to run it, usually (as is the case here) right after a chattr command with the same arguments.

> 8181 root

In 30 days? That's tad unrealistic.

Just checked and there are dozens root login attempts per minute on my colo'ed server in the EU. Virtually all from the Chinese and post-Soviet IP space. But mostly Chinese.

> In conclusion, these commands represent a clear strategy to infiltrate, assess, and establish control over targeted systems.

Oh hello, ChatGPT. You seem to be everywhere these days.

This story reminds me of the time I accidentally/naively set up a ssh honeypot when I configured the router I was using at the time (this was a long time ago) to forward incoming SSH connections to a ReadyNAS (which was using a Sun SPARC processor) in my house. I did that so I could log into it while I was away from my house. One day, I ssh'ed in and notice that the ReadyNAS was running very slow which surprised me because I thought it was idle. I checked the CPU usage and the sshd was using 100%. First, I thought it was a bug but it occurred to me I should check my incoming ssh connection attempts in my router log. Turns out there were a ton of ssh connection attempts coming from an IP address assigned to China. In response I changed the router port forwarding for incoming ssh connections to use a non-standard port number, like 55,243, and after that my ReadyNAS was no longer bombarded with ssh connection attempts. Lesson: try to avoid forwarding standard port numbers.

Amazing article!

It is actually amazing how fast and thorough the connection attempts happen as soon as you put anything online.

I've been playing around Hetzner and Coolify recently, and notice that, as soon as port 22 is opened, it is bombarded by those attempts. Several per second. It might be due to Hetzner IPs being reused, but happened to me every single time. Same with Postgres default port (those were the ones I've seen).

I have defaulted to use Terraform and bash to only open those ports in the Hetzner firewall (and more common ones like 3000 or 8000) to my own current ip. It does mean I'll get drift and need to reapply the Terraform code if I change ips, but seems to be at least one way to defend.

I fear that a lot of devs jumping into the "you only need a VPS" crowd on Twitter will end up with a huge attack surface on their apps and machines and most won't even know they are being targeted like that most of the time.

To this day I still find it hard to find a comprehensive security guide for those newer Linux fresh boxes (and the ones you find are all so very different with different suggestions). If anyone knows of a good one, please share with me!

Not sure if op will see this, but with regard to his comments on MikroTik routers and frequently seeing in his honeypot logs, the command: /ip cloud print

he is correct, This is a MikroTik command- although mikrotik has this feature, disabled/ off by default, a lot of users make use of it, and running that command will (if cloud dns enabled), will show the dynamic DNS entry of the device he is connected to. Ie if the cloud DNS is enabled, the output from that command will be something like: Detected public ip: 34.2.82.3 DynDns: djwisyehd.clouddns.mikrotik.com (which will always be updated to the detected public IP address of the router)

So I assume the attackers run this command so that they can still reach the router in case it’s public IP address changes at some point. (And assuming that the device will still be accessible after any public IP address changes).

(or perhaps they run that command to see if the cloud DNS service is disabled, which is the default, in which case they will then enable it so that they will have a dynamic DNS entry for the device).

What does `echo -e "\x6F\x6B"` do?

A bit tangential but is there a service or self hosted solution that would take a list of IPs and then keep scanning them periodically and alert me if any new ports have suddenly open?

I simply block traffic from countries where I do not do business in.

I used to see constant attempts to mess with Wordpress URLs, which I know is not legitimate because I don't run Wordpress.

Cutting out Russia & China basically removed this problem. I really hate locking up my tiny corner of the internet but I don't see another way.

If you want to try this yourself, although he carefully doesn’t mention his software, it appears he is running the Cowrie honeypot (https://github.com/cowrie/cowrie)

(Note, I’m the maintainer)

There's a project for running Honeypot as a Service: https://haas.nic.cz The data is public and you can register your router too

What is a `lockr` command? Is it file system specific or something? Never seen anything like this. It probably should lock permissions on .ssh, but how?

Somewhat related due to a weak password a mail server from a community I'm involved in send out lot's of spam mail, after analysing the log files I've had over 1500 different IP addresses that logged in to send spam, about 10 mails for each address. ASN and subnets where spread across over the whole world. It seems like these attacks are coordinated using vast botnets and the use of single ssh public key here seems to confirm this. I had similar experiences going after attacks on WordPress instances and there I've also found attacks spread out across lots of hosts.

I'm wondering if it's possible to pin down those behind these attacks, there must be mistakes.

Interesting article, sadly due to my exposure to LLMs I couldn't help but notice that the parts about "oinasf" and sakura.sh are AI-edited at least. Kind of a weird choice considering that a lot of the article was clearly human-written.

3 simple sshd config lines remove a gigantic amount of worries if you run open ssh servers:

   Port NN
   PasswordAuthentication no
   AllowUsers user1 user2 user3

change those, sleep at night.

Check out https://viz.greynoise.io/ especially the trends > anomalies tab is very interesting

Had a own server in university during mY PhD. Most request were trying to download scientific papers from large journals using absolute and not relative URLs after request.

I once tried hosting a web server at home by exposing ports 80 and 443 to the Internet. Hours later I reviewed the logs, thousands of attempts to hack into my lil Linux server. It spooked me to say the least, so I switched to using cloudflare tunnels instead.

Exposing ports on the Internet is dangerous, especially SSH. You're much safer using a proxy or gateway of some sort, or better yet a VPN if it doesn't need to be publicly accessible.

We run internal sites that are on the public facing web - the logs from Akamai are a daily list of mostly the same requests to find unsecured Wordpress and MySQL installs, .cgi and php files and paths like "..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../etc/profile"

In 24 hours theres anywhere from 7000-9000 log events just from the CDN

How do people feel about using docker as a way of avoiding 0 day vulnerability

It's all for personal use and maybe I'm just cosplaying as a sysadmin but I have apache proxy-pass ing to sets of docker containers. So as long as apache and ssh are kept up to date (on nixos), even if all my services are 0 day'd, they have to also escape the docker containment

If anyone is looking to run IP metadata based reports on their honeypot, I can suggest IPinfo's CLI (https://github.com/ipinfo/cli). Here is my summary report from Fail2Ban:

``` Summary - Total 1490 - Unique 153 - Anycast 0 - Bogon 0 - Mobile 52 - VPN 91 - Proxy 12 - Hosting 1003 - Tor 0 - Relay 0

Top ASNs - AS132203 Tencent Building, Kejizhongyi Avenue 409 (27.4%) - AS14061 DigitalOcean, LLC 148 (9.9%) - AS135377 UCLOUD INFORMATION TECHNOLOGY (HK) LIMITED 72 (4.8%) - AS16276 OVH SAS 72 (4.8%) - AS206264 Amarutu Technology Ltd 44 (3.0%)

Top Usage Types - Hosting 955 (64.1%) - ISP 418 (28.1%) - Business 49 (3.3%)

Top Routes - 43.134.64.0/18 (AS132203) 48 (3.2%) - 183.81.169.0/24 (AS206264) 44 (3.0%) - 43.156.192.0/18 (AS132203) 36 (2.4%) - 43.130.0.0/18 (AS132203) 36 (2.4%) - 43.134.0.0/18 (AS132203) 36 (2.4%)

Top Countries - United States 276 (18.5%) - Singapore 221 (14.8%) - China 141 (9.5%) - France 88 (5.9%) - Japan 86 (5.8%)

Top Cities - Singapore, Singapore, SG 221 (14.8%) - Santa Clara, California, US 100 (6.7%) - Hong Kong, Hong Kong, HK 80 (5.4%) - Tokyo, Tokyo, JP 61 (4.1%) - Amsterdam, North Holland, NL 50 (3.4%)

Top Regions - Singapore, SG 221 (14.8%) - California, US 134 (9.0%) - Tokyo, JP 86 (5.8%) - Hong Kong, HK 80 (5.4%) - New Jersey, US 70 (4.7%)

Top Carriers - Africell 12 (0.8%) - Claro 12 (0.8%) - Vivo 12 (0.8%) - WINDTRE 12 (0.8%) - Telekom 4 (0.3%)

Top Privacy Services - TunnelBear 36 (2.4%) - Best Proxy Switcher 12 (0.8%)

Top Domains - ovh.net 60 (4.0%) - googleusercontent.com 24 (1.6%) - prod-infinitum.com.mx 16 (1.1%) - poneytelecom.eu 12 (0.8%) ```

Here is the command I used:

``` cat /var/log/fail2ban.log | ipinfo grepip -o | ipinfo summarize ```

The CLI is free to use. You can also do `bulk` enrichment.

``` cat /var/log/fail2ban.log | ipinfo grepip -o | ipinfo bulk -c > fail2ban_ips.csv ```

Disclaimer: I work for IPinfo. However, the CLI is free to use, and the bulk feature will usually work within your free tier limits. Ping me if you have any questions

Why is 345gs5662d34 the 2nd most tried username after root? Bizarre.

I somehow found myself in charge of a computer lab two decades ago... and idiotically set up admin controls via SSH.

The entire lab was down for almost a week [immediately hacked], and then I suddenly moved a few states away.

I opened my personal server's 22 to the world because I screwed up my vpn config a couple weeks ago. I just had a look at the auth log and closed it again. It is non-stop.

A perfect example of why one should use SSH over a mesh network like Tailscale, and don't expose over the public internet. No attack surface means no attack.

I was always wondering, how do you guys geoblock entire countries/ip ranges? Do you just use free ip lists you find on the internet?

My new VPS got an SSH attempt in 5 minutes after I purchased it. I'm now in the progress of running a similar honeypot experiment.

I've been running self-hosted servers for the last 25+ years without an incident and its less complicated than it might seem if you learn a bit about securing unix-based systems (ok, I already had 10+ years of server admin knowhow for various systems, but anyway, it's not rocket science ;-). Yes, an hour or so after you connect any machine to the Internet, you'll see attempts to "talk" to your server. So don't wait to set up basic security. But it actually has never been so easy to "just give it a try" (see below), with all the virtual offerings today. So here's a short/raw sketch of basic things you'd need to do:

1. 25+ years ago I used http://easyfwgen.morizot.net/ to generate an iptables based local firewall. Still works fine (then and now tweaking some things) and allows only certain ports too be accessed at all. I just open email, ssh and a web server.

The generator is well documented and still works, although it would be nice to see an updated version to newer firewall software like pf.

2. server configs:

edit /etc/hosts.deny --> restrict all by default

  ALL: ALL

edit /etc/hosts.allow --> allow your service providers networks, e.g.

  sshd: .t-dialin.net
  sshd: .dip0.t-ipconnect.de

So you can connect to your machine for further setup, but not the whole world.

3. set up sshd:

edit /etc/ssh/sshd.config

  # allow key-based access only
  PasswordAuthentication no

Maybe change sshd's port (reduces log file entries) but don't forget to allow this port in your iptables setup and your /etc/hosts.allow

People have opinions an key-based access, I know. But my private and public key is stored in various secure locations, including my phone (password safe) and I can access my server even from my Android phone or tables via Termux.

4. set up email (I suggest postfix as an MTA):

configure restrictions in /etc/postfix/main.cf, e.g.

  # restrictions in the context of the RCPT TO command
  smtpd_recipient_restrictions =
        reject_invalid_hostname,
        reject_non_fqdn_hostname,
        reject_non_fqdn_sender,
        reject_non_fqdn_recipient,
        check_sender_access hash:/etc/postfix/sender_access,
        reject_unknown_sender_domain,
        reject_unknown_recipient_domain,
        permit_mynetworks,
        reject_unauth_destination,
        [...]

  # restrictions for clients connecting
  smtpd_client_restrictions =
        reject_unauth_destination,
        check_client_access hash:/etc/postfix/access_client,
        reject_unknown_client,
        reject_unauth_pipelining

This heavily reduces the amount of spam you'll see. I add greylisting too, as this even nowadays reduces even more unwanted traffic. Combine that with spamassassin if you like. This setup gives me maybe one spam per day reaching my inbox (actually the spam subfolder).

5. Learn by doing (not just reading stuff on the Internets ;-), that is, set up a machine, e.g.

If you'd like to experiment a bit, take a look at Hetzner's unexpensive cloud servers, these are easy to set up (incl. a virtual firewall in front of it) and take down after some experiments of a failure. You can do this in Hetzner's web interface, even if you misconfigure your server to be unaccessible. Cf.

https://docs.hetzner.com/cloud/servers/overview/

Tip: Hetzner's web interface allows you to pre-define an ssh key which they'll install automatically on your new machine (but they leave password login enabled, so change that asap).

Disclaimer: I'm just a happy customer, no other relation. And it might be as easy to do this with Digital Ocean, which have some nice tutorials too, for example on the set up of a web server:

https://www.digitalocean.com/community/tutorials/how-to-inst...

Last but not least No Starch Press overs some nice books like "How Linux Works" or "The Linux Command Line" (if you're not sure about that) or even "Linux Firewalls: Attack Detection and Response" ...

You learn most by trying.

I'm now heading for the beach to enjoy some offline adventures and will answer questions later if needed.

I guess may the compromised host was probably also use same weak password as it's Brute force other host.

My website backend APIs get repeated attempts at javascript prototype injection, all day, every day.

(Long shot) I really would like to USA a spare machine for web serving a Jupyter Notebook server, but I did not found a single resource that blocks everyone except a single IP or something like this. Supper annoying to pay some cloud providers to have a resource that I already have.

We use port knocking and haven’t had a single hack attempt in many years.

How did he expose his honeypots and make the bots aware of his existence?

I wanted to read more about the interesting part!

dont ever run publicly exposed production SSH. If there is vulnerability in your version of ssh, you risk getting pwned.

Most of this nonsense disappeared when I adopted wireguard and later Tailscale.

I am amazed we have not yet said "Hands off!" and coordinated physical interventions against the scum who attack our electronic brains.

Is it so hard to kick in the doors of those whose IP addresses are used to try to hack honeypots?

This lack of action is why I oppose all law enforcement. Until they do their jobs, they do not need to be paid.

[dead]