Shell Voodoo, Connected IPs, and Counting Total Connections

I’m posting this mostly as a note to myself, but if you, future visitor, stumble upon this post and have improvements or other things you’d like to share, be my guest. Posts that are overly critical of the methodologies provided by others, or those which otherwise add nothing to the discussion will be removed. This is especially true for those espousing beliefs that PowerShell is superior.

I won’t go into the exact details of why we needed to do this, but the general break down is thus:

  • Get a list of connected IP addresses
  • Sort them
  • Count how many connections were made from a single address

Fortunately, the solution turns out to be quite easy. For FreeBSD:

netstat -anfinet | grep -v 127.0.0.1 | awk '{ print $5 }' | \
grep -E '.*([0-9]{1,4}\.)+.*' | sed 's/\(.*\)\..*/\1/' | \
sort -g -k 1 | uniq -c | sort -n -k 1

And for most derivatives of Linux:

netstat -anW --tcp --udp | grep -v 127.0.0.1 | awk '{ print $5 }' | \
grep --color=never -E '.*[0-9]{1,4}(\.|\:).*' | sed 's/\(.*\)\:.*/\1/' | \
sort -g -k 1 | uniq -c | sort -n -k 1

You may need to modprobe sctp to get the --tcp and --udp netstat flags working. Also, both of these should work with IPv6 addresses, too, which is why I’ve tried to keep the sed regex as simple as possible.

What the Eff is This?!

Okay, I agree. I’ve probably made some kind of mistake somewhere; I don’t know awk or sed quite as well as I should (easily fixed, if I ever wanted to spend a weekend learning). That said, here’s my understanding of how this should work. First, we’ll deal with the FreeBSD derivative, line by line:

FreeBSD

Here is a breakdown for the FreeBSD-specific stuff:

netstat -anfinet | grep -v 127.0.0.1 | awk '{ print $5 }' | \

As with all platforms I’m aware, -an shows all connections by their numerical addresses. netstat prefers to perform a reverse lookup on every address, and this can take some time. However, the FreeBSD-specific option -f inet specifies to only show INET (IPv4/IPv6) addresses and eliminates much of the cruft associated with local Unix domain sockets. Likewise, we trim localhost from the list with grep -v, and we fetch the 5th output column using awk

grep -E '.*([0-9]{1,4}\.)+.*' | sed 's/\(.*\)\..*/\1/' | \

Moving on to the next line, we fetch only those lines that contain something that vaguely resembles an IP address with grep -E (I prefer to use -E here since it gives us the extended regex syntax), and we pass the results into sed to strip off the trailing remote host’s port number. Alternatively, you could use something like 's/^\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*/\1/' instead to filter out IPv4 addresses, but since we already know roughly what to expect from the input we can simplify our regex. Furthermore, we also know that the IP address of the remote host in FreeBSD will always have a dot followed by the port number appended, and we can naively remove this.

sort -g -k 1 | uniq -c | sort -n -k 1

Lastly, we sort (generically, with -gunique addresses in our list including their totals, and we sort numerically by the first column (now containing the count).

Linux

Here is a breakdown for the Linux-specific stuff:

netstat -anW --tcp --udp | grep -v 127.0.0.1 | awk '{ print $5 }' | \

Following in the footsteps of FreeBSD, we use -an to display all connected numeric addresses so we don’t waste time running reverse lookups. However, in most Linux distributions, lengthy columns–and especially IPv6 addresses–will be truncated by netstat’s output. To counter this, we use -W to show the wide listing, and we use --tcp and --udp to filter out only those protocols. You may need to modprobe sctp in order to get this to work; if you can’t, this string of commands might still work. Lastly, we filter connections to localhost with grep -v, and we fetch the 5th column using awk Easy enough, right?

grep --color=never -E '.*[0-9]{1,4}(\.|\:).*' | sed 's/\(.*\)\:.*/\1/' | \

In this next line, we use the extended regex feature of grep -E to filter out lines that look somewhat address-y, and we separate the remote host’s address from its port using sed. In this case, Linux appends port numbers using a colon (:), so we have to deviate slightly from the FreeBSD example. Also, since some distros might alias grep with grep --color=auto|always, we use --color=never to eliminate feeding ANSI control characters to sed.

sort -g -k 1 | uniq -c | sort -n -k 1

Lastly, we sort by the IP address using a generic sort (-g), filter out only those addresses that are unique, count them, and then sort by the count column which is now tacked onto the front.

Now we can get a fancy list of IP addresses, how many connections from them are being made to us, and sort them accordingly! Manipulating grep accordingly can re-introduce localhost or remove specific addresses that might not be of interest.

No comments.
***

Building Binary Packages in FreeBSD

It’s amazing how difficult it can be to find certain things on Google. It’s been a while since I’ve used FreeBSD, so I couldn’t precisely remember which way was the correct way of building binary packages. Some of the more “official” documentation seems to suggest creating a jail and building the packages from there. While the jailed approach is a good one, it isn’t exactly what I was looking for since the FreeBSD system I have running in a VM roughly mirrors my requirements for one that I’m intending to install on my desktop. As it turns out, I had forgotten about pkg_create; more importantly, I didn’t know that since FreeBSD 6.0, pkg_create allows for generating all dependencies for a given package (the last version of FreeBSD I used was 5.2).

Anyway, I’ll cut this short. I found this very handy link for creating binary packages in FreeBSD: http://www.math.colostate.edu/~reinholz/freebsd/pkg_create.html.

Edit May 25th, 2010: The link above now returns a 404 Not Found error, and I’ve long since forgotten what its contents were. However, archive.org still has a copy of this site from 2007. I’m not sure when it was taken down, and it’s possible that the version (above) doesn’t contain some newer, more interesting information. While I don’t remember precisely what excited me about the original link, I think it had something to do with the fact that pkg_create supports grabbing all of the dependencies in a single, fell swoop. While searching around to see if I could remember why I found this so exciting, I discovered that it’s possible to create a package of all installed ports on the system. You can find some tips here on how to do exactly that (scroll down to the second post).

No comments.
***

FreeBSD Ports: Making them Friendlier

The FreeBSD ports collection may be a tough nut to crack if you’re used to easier package distribution systems like those found on Ubuntu or Redhat. In many ways, I find it easier to manage. The ports collection grants you greater control over what is installed, how it is configured, what dependencies you’d like to build (or rebuilt), and what upgrade path you’d like to choose whenever upgrading packages. The ports collection is so good, in fact, that several Linux distributions now borrow from the principles set forth by the FreeBSD foundation what seems like eons ago (Gentoo and its derivatives, specifically). It can be daunting to manage ports at first, but you’ll find that it offers you freedom that simply couldn’t be had from binary packages.

After all, FreeBSD is about freedom. Read more…

No comments.
***