Fyodor's Paper on Remote OS Detection: www.insecure.org/nmap/nmap-fingerprinting-article.html .
I found this paper to be extremely informative on the subject of OS fingerprinting.
Netfilter Kernel Modules: netfilter.samba.org.