FAQ: How to block/allow packets sent by specific Operating System with iptables?

Question: How can I block traffic coming from specific operating system in Linux? In other words, how can I block traffic from Windows users on my firewall and allows other people?

Answer: There is an iptables module named OSF (passive OS Fingerprinting) that was written by Evgeniy Polyakov. This module allows passively detect OS packet was sent from and perform various netfilter actions based on this match. Packets with SYN bit set are analyzed.

In order to install OSF module, do the following:

1. Download latest release from here, for example as follows:
wget http://tservice.net.ru/~s0mbre/archive/osf/osf-2008_06_14.tar.gz

2. Edit Makefile from unpacked archive in order to set proper path to iptables headers (iptables.h and libiptc/ dir).

3. If your kernel sources can not be accessed via /lib/modules/$(shell uname -r)/build, you have to replace KDIR variable with the correct path to kernel sources.
4. Run make that should build ipt_osf.ko kernel module.
5. Run make lib that will build libipt_osf.so shared library (copy it to where all other iptables shared libs are placed in your distro e.g. /lib/iptables or /lib64/iptables in Fedora).

6. Run make bin that will build userspace applications which allows to load fingerprints and obtain information about matched packets (load, osfd, ucon_osf).

7. Download signatures list:
wget http://www.openbsd.org/cgi-bin/cvsweb/src/etc/pf.os

8. Install kernel module:
insmod ./ipt_osf.ko

9. Load signatures:
./load ./pf.os /proc/sys/net/ipv4/osf

10. Set up iptables rules allowing/disallowing packets generated by certain OS:
iptables -I INPUT -j ACCEPT -p tcp -m osf --genre Linux --log 0 --ttl 2

This example allows traffic from Linux systems and logs packets from other ones:
ipt_osf: Windows [2000:SP3:Windows XP Pro SP1, 2000 SP3]: 11.22.33.55:4024 -> 11.22.33.44:139

BTW, OSF has following options:

• –log
  If present, OSF will log determined genres even if they don’t match desired one.
  0 – log all matched and unknown entries.
  1 – only first one.
  2 – log all matched entries.
• –ttl
  0 – true ip and fingerprint TTL comparison. Works for LAN.
  1 – check if ip TTL is less than fingerprint one. Works for global addresses.
  2 – do not compare TTL at all. Allows to detect NMAP, but can produce false results.
• –connector
  If present, OSF will log all events also through netlink connector(1.0 id).
  More about connector can be found in Documentation/connector in kernel source tree.