lib/mdm/host/operating_system_normalization.rb
#
# Leverage the Recog gem as much as possible for sane fingerprint management
#
require 'recog'
#
# Rules for operating system fingerprinting in Metasploit
#
# The `os.product` key identifies the common-name of a specific operating system
# Examples include: Linux, Windows XP, Mac OS X, IOS, AIX, HP-UX, VxWorks
#
# The `os.version` key identifies the service pack or version of the operating system
# Sometimes this means a kernel or firmware version when the distribution or OS
# version is not available.
# Examples include: SP2, 10.04, 2.6.47, 10.6.1
#
# The `os.vendor` key identifies the manufacturer of the operating system
# Examples include: Microsoft, Ubuntu, Cisco, HP, IBM, Wind River
#
# The `os.family` key identifies the group of the operating system. This is often a
# duplicate of os.product, unless a more specific product name is available.
# Examples include: Windows, Linux, IOS, HP-UX, AIX
#
# The `os.edition` key identifies the specific variant of the operating system
# Examples include: Enterprise, Professional, Starter, Evaluation, Home, Datacenter
#
# An example breakdown of a common operating system is shown below
#
# * Microsoft Windows XP Professional Service Pack 3 English (x86)
# - os.product = 'Windows XP'
# - os.edition = 'Professional'
# - os.vendor = 'Microsoft'
# - os.version = 'SP3'
# - os.language = 'English'
# - os.arch = 'x86'
#
# These rules are then mapped to the {Mdm::Host} attributes below:
#
# * os_name - Maps to a normalized os.product key
# * os_flavor - Maps to a normalized os.edition key
# * os_sp - Maps to a normalized os.version key (soon os_version)
# * os_lang - Maps to a normalized os.language key
# * arch - Maps to a normalized os.arch key
#
# Additional rules include the following mappings:
#
# * name - Maps to the host.name key
# * mac - Maps to the host.mac key
#
# The following keys are not mapped to {Mdm::Host} at this time (but should be):
#
# * os.vendor
#
# In order to execute these rules, this module is responsible for mapping various
# fingerprint sources to {Mdm::Host} values. This requires some ugly glue code to
# account for differences between each supported input (external scanners), the
# Recog gem and associated databases, and how Metasploit itself likes to handle
# these values. Getting a mapping wrong is often harmless, but can impact the
# automatic targetting capabilities of certain exploit modules.
#
# In other words, this is a best-effort attempt to rationalize multiple competing
# sources of information about a host and come up with the values representing a
# normalized assessment of the system. The use of `Recog` and multiple scanner
# fingerprints can result in a comprehensive (and confident) identification of the
# remote operating system and associated services.
#
# Historically, there are direct conflicts between certain Metasploit modules,
# certain scanners, and external fingerprint databases in terms of how a
# particular OS and patch level is represented. This module attempts to fix what
# it can and serve as documentation and live workarounds for the rest.
#
# Examples of known conflicts that are still in progress:
#
# * Metasploit defines an OS constant of 'win'/'windows' as Microsoft Windows
#
# - Scanner modules report a mix of 'Microsoft Windows' and 'Windows'
# - Nearly all exploit modules reference 'Windows <Release> SP<Version>'
# - Nmap (and other scanners) also prefix the vendor before Windows
#
#
# * Windows service packs represented as 'Service Pack X' or 'SPX'
#
# - The preferred form is to set os.version to 'SPX'
# - Many external scanners & Recog prefer 'Service Pack X'
#
# * Apple Mac OS X, Cisco IOS, IBM AIX, Ubuntu Linux, all reported with vendor prefix
#
# - The preferred form is to remove the vendor from os.product
# - {Mdm::Host} currently has no vendor field, so this information is lost today
# - Many scanners report leading vendor strings and require normalization
#
# * The os_flavor field is used in contradictory ways across Metasploit
#
# - The preferred form is to be a 'display only' field
# - Some Recog fingerprints still append the edition to os.product
# - Many scanners report the edition as a trailing suffix to os.product
#
#
#
#
# Maintenance:
#
# 1. Ensure that the latest Recog gem is present and installed
# 2. For new operating system releases, update relevant sections
# a) Windows releases will require updates to a few methods
# 1) parse_windows_os_str()
# 2) normalize_nmap_fingerprint()
# 3) normalize_nexpose_fingerprint()
# 4) Other scanner normalizers
# b) Mobile operating systems are minimally recognized
#
#
# @todo Handle OS icon incompatiblities with new fingerprint names
# Note that VMWare ESX(i) was special cased before as well, make sure it still works
# 1) Cisco IOS -> IOS breaks the icon mapping in MSP/MSCE of /cisco/
# 2) Ubuntu Linux -> Linux breaks the distro selection
# The real solution is to add os_vendor and take this into account for icons
#
# @todo Implement rspec coverage for normalize_os()
# @todo Implement smb.generic fingerprint database (replace {#parse_windows_os_str}?)
# @todo Implement Samba version matching for specific distributions and OS versions
# @todo Implement DD-WRT and various embedded device signatures currently missing
# @todo Correct inconsistencies in os_name use by removing the vendor string (Microsoft Windows -> Windows)
# This applies to MSF core and a handful of modules, not to mention some Recog fingerprints.
# @todo Rename host.os_sp to host.os_version
# @todo Add host.os_vendor
# @todo Add host.os_confidence
# @todo Add host.domain
#
module Mdm::Host::OperatingSystemNormalization
# Cap nmap certainty at 0.84 until we update it more frequently
# XXX: Without this, Nmap will beat the default certainty of recog
# matches and its less-confident guesses will take precedence
# over service-based fingerprints.
MAX_NMAP_CERTAINTY = 0.84
#
# Normalize the operating system fingerprints provided by various scanners
# (nmap, nexpose, retina, nessus, metasploit modules, and more!)
#
# These are stored as {Mdm::Note notes} (instead of directly in the os_*
# fields) specifically for this purpose.
#
# The goal is to infer as much as we can about the OS of the device and the
# various {Mdm::Service services} offered using the Recog gem and some glue
# logic to determine the best weights. This method can result in changes to
# the recorded {#os_name}, {#os_flavor}, {#os_sp}, {#os_lang}, {#purpose},
# {#name}, {#arch}, and the {Mdm::Service service details}.
#
def normalize_os
host = self
matches = []
# Note that we're already restricting the query to this host by using
# host.notes instead of Note, so don't need a host_id in the
# conditions.
fingerprintable_notes = self.notes.where("ntype like '%%fingerprint'")
fingerprintable_notes.each do |fp_note|
matches += recog_matches_for_note(fp_note)
end
# XXX: This hack solves the memory leak generated by self.services.each {}
fingerprintable_services = self.services.where("name is not null and name != '' and info is not null and info != ''")
fingerprintable_services.each do |s|
matches += recog_matches_for_service(s)
end
#
# Look for generic fingerprint.match notes that generate a match hash from modules
# This handles ad-hoc os.language, host.name, etc identifications
#
generated_matches = self.notes.where(ntype: 'fingerprint.match')
generated_matches.each do |m|
next unless (m.data and m.data.kind_of?(::Hash))
matches << m.data.dup
end
# Normalize matches for consistency during the ranking phase
matches = matches.map{ |m| normalize_match(m) }
# Calculate the best OS match based on fingerprint hits
match = Recog::Nizer.best_os_match(matches)
# Merge and normalize the best match to the host object
apply_match_to_host(match) if match
# Set some sane defaults if needed
host.os_name ||= 'Unknown'
host.purpose ||= 'device'
host.save if host.changed?
end
# Recog matches for the `s` service.
#
# @param s [Mdm::Service]
# @return [Array<Hash>] Keys will be host, service, and os attributes
def recog_matches_for_service(s)
#
# We assume that the service.info field contains certain types of probe
# replies and associate these with one or more Recog databases. The mapping
# of service.name to a specific database only fits into so many places and
# Mdm currently serves that role.
#
service_match_keys = {
# TODO: Implement smb.generic fingerprint database
# 'smb' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match
# 'netbios' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match
'ssh' => [ 'ssh.banner' ], # Recog expects just the vendor string, not the protocol version
'http' => [ 'http_header.server', 'apache_os'], # The 'Apache' fingerprints try to infer OS/distribution from the extra information in the Server header
'https' => [ 'http_header.server', 'apache_os'], # XXX: verify vmware esx(i) case on https (TODO: normalize https to http, track SSL elsewhere, such as a new set of fields)
'snmp' => [ 'snmp.sys_description' ],
'telnet' => [ 'telnet.banner' ],
'smtp' => [ 'smtp.banner' ],
'imap' => [ 'imap4.banner' ], # Metasploit reports 143/993 as imap (TODO: normalize imap to imap4)
'pop3' => [ 'pop3.banner' ], # Metasploit reports 110/995 as pop3
'nntp' => [ 'nntp.banner' ],
'ftp' => [ 'ftp.banner' ],
'ssdp' => [ 'http_header.server' ]
}
matches = []
return matches unless service_match_keys.has_key?(s.name)
service_match_keys[s.name].each do |rdb|
banner = s.info
if self.respond_to?("service_banner_recog_filter_#{s.name}")
banner = self.send("service_banner_recog_filter_#{s.name}", banner)
end
res = Recog::Nizer.match(rdb, banner)
matches << res if res
end
matches
end
# Recog matches for the fingerprint in `note`.
#
# @return [Array<Hash>] Keys will be host, service, and os attributes
def recog_matches_for_note(note)
# Skip notes that are missing the correct structure or have been blacklisted
return [] if not validate_fingerprint_data(note)
#
# These rules define the relationship between fingerprint note keys
# and specific Recog databases for detailed matching. Notes that do
# not match a rule are passed to the generic matcher.
#
fingerprint_note_match_keys = {
'smb.fingerprint' => {
:native_os => [ 'smb.native_os' ],
:native_lm => [ 'smb.native_lm' ],
},
'http.fingerprint' => {
:header_server => [ 'http_header.server', 'apache_os' ],
:header_set_cookie => [ 'http_header.cookie' ],
:header_www_authenticate => [ 'http_header.wwwauth' ],
# TODO: Candidates for future Recog support
# :content => 'http_body'
# :code => 'http_response_code'
# :message => 'http_response_message'
}
}
matches = []
# Look for a specific Recog database for this type and data key
if fingerprint_note_match_keys.has_key?( note.ntype )
fingerprint_note_match_keys[ note.ntype ].each_pair do |k,rdbs|
if note.data.has_key?(k)
rdbs.each do |rdb|
res = Recog::Nizer.match(rdb, note.data[k])
matches << res if res
end
end
end
else
# Add all generic match results to the overall match array
normalize_scanner_fp(note).each do |m|
next unless m
matches << m
end
end
matches
end
# Determine if the fingerprint data is readable. If not, it nearly always
# means that there was a problem with the YAML or the Marshal'ed data,
# so let's log that for later investigation.
def validate_fingerprint_data(fp)
if fp.data.kind_of?(Hash) and !fp.data.empty?
return true
elsif fp.ntype == "postgresql.fingerprint"
# Special case postgresql.fingerprint; it's always a string,
# and should not be used for OS fingerprinting (yet), so
# don't bother logging it. TODO: fix os fingerprint finding, this
# name collision seems silly.
return false
else
return false
end
end
#
# Normalize matches in order to handle inconsistencies between fingerprint
# sources and our desired usage in Metasploit. This amounts to yet more
# duct tape, but the situation should improve as the fingerprint sources
# are updated and enhanced. In the future, this method will no longer
# be needed (or at least, doing less and less work)
#
def normalize_match(m)
# Normalize os.version strings containing 'Service Pack X' to just 'SPX'
if m['os.version'] and m['os.version'].index('Service Pack ') == 0
m['os.version'] = m['os.version'].gsub(/Service Pack /, 'SP')
end
if m['os.product']
# Normalize Apple Mac OS X to just Mac OS X
if m['os.product'] =~ /^Apple Mac/
m['os.product'] = m['os.product'].gsub(/Apple Mac/, 'Mac')
m['os.vendor'] ||= 'Apple'
end
# Normalize Sun Solaris/Sun SunOS to just Solaris/SunOS
if m['os.product'] =~ /^Sun (Solaris|SunOS)/
m['os.product'] = m['os.product'].gsub(/^Sun /, '')
m['os.vendor'] ||= 'Oracle'
end
# Normalize Microsoft Windows to just Windows to catch any stragglers
if m['os.product'] =~ /^Microsoft Windows/
m['os.product'] = m['os.product'].gsub(/Microsoft Windows/, 'Windows')
m['os.vendor'] ||= 'Microsoft'
end
# Normalize Windows Server to just Windows to match Metasploit target names
if m['os.product'] =~ /^Windows Server/
m['os.product'] = m['os.product'].gsub(/Windows Server/, 'Windows')
end
# Normalize OS Family
m = normalize_match_family(m)
end
m
end
# Normalize matches in order to ensure that an os.family entry exists
# if we have enough data to put one together.
def normalize_match_family(m)
# If the os.family already exists, we don't need to do anything
return m if m['os.family'].present?
case m['os.product']
when /Windows/
m['os.family'] = 'Windows'
when /Linux/
m['os.family'] = 'Linux'
when /Solaris/
m['os.family'] = 'Solaris'
when /SunOS/
m['os.family'] = 'SunOS'
when /AIX/
m['os.family'] = 'AIX'
when /HP-UX/
m['os.family'] = 'HP-UX'
when /OS X/
m['os.family'] = 'OS X'
end
m
end
#
# Recog assumes that the protocol version of the SSH banner has been removed
#
def service_banner_recog_filter_ssh(banner)
if banner =~ /^SSH-\d+\.\d+-(.*)/
$1
else
banner
end
end
#
# Examine the assertations of the merged best match and map these
# back to fields of {Mdm::Host}. Take particular care not to leave
# related fields (os_*) in a conflicting state, leverage existing
# values where possible, and use the most confident values we have.
#
def apply_match_to_host(match)
host = self
# These values in a match always override the current value unless
# the host attribute has been explicitly locked by the user
if match['host.mac'] && !host.attribute_locked?(:mac)
host.mac = sanitize(match['host.mac'])
end
if match['host.name'] && !host.attribute_locked?(:name)
host.name = sanitize(match['host.name'])
end
# Select the os architecture if available
if match['os.arch'] && !host.attribute_locked?(:arch)
host.arch = sanitize(match['os.arch'])
end
# Guess the purpose using some basic heuristics
if ! host.attribute_locked?(:purpose)
host.purpose = guess_purpose_from_match(match)
end
#
# Map match fields from Recog fingerprint style to Metasploit style
#
# os.build: Examples: 9001, 2600, 7602
# os.device: Examples: General, ADSL Modem, Broadband router, Cable Modem, Camera, Copier, CSU/DSU
# os.edition: Examples: Web, Storage, HPC, MultiPoint, Enterprise, Home, Starter, Professional
# os.family: Examples: Windows, Linux, Solaris, NetWare, ProCurve, Mac OS X, HP-UX, AIX
# os.product: Examples: Windows, Linux, Windows Server 2008 R2, Windows XP, Enterprise Linux, NEO Tape Library
# os.vendor: Examples: Microsoft, HP, IBM, Sun, 3Com, Ricoh, Novell, Ubuntu, Apple, Cisco, Xerox
# os.version: Examples: SP1, SP2, 6.5 SP3 CPR, 10.04, 8.04, 12.10, 4.0, 6.1, 8.5
# os.language: Examples: English, Arabic, German
# linux.kernel.version: Examples: 2.6.32
# Metasploit currently ignores os.build, os.device, and os.vendor as separate fields.
# Select the OS name from os.name, fall back to os.family
if ! host.attribute_locked?(:os_name)
# Try to fill this value from os.product first if it exists
if match.has_key?('os.product')
host.os_name = sanitize(match['os.product'])
else
# Fall back to os.family otherwise, if available
if match.has_key?('os.family')
host.os_name = sanitize(match['os.family'])
end
end
end
if match.has_key?('os.family')
host.os_family = sanitize(match['os.family'])
end
# Select the flavor from os.edition if available
if match.has_key?('os.edition') and ! host.attribute_locked?(:os_flavor)
host.os_flavor = sanitize(match['os.edition'])
end
# Select an OS version as os.version, fall back to linux.kernel.version
if ! host.attribute_locked?(:os_sp)
if match['os.version']
host.os_sp = sanitize(match['os.version'])
else
if match['linux.kernel.version']
host.os_sp = sanitize(match['linux.kernel.version'])
end
end
end
# Select the os language if available
if match.has_key?('os.language') and ! host.attribute_locked?(:os_lang)
host.os_lang = sanitize(match['os.language'])
end
# Normalize MAC addresses to lower-case colon-delimited format
if host.mac and ! host.attribute_locked?(:mac)
host.mac = host.mac.downcase
if host.mac =~ /^[a-f0-9]{12}$/
host.mac = host.mac.scan(/../).join(':')
end
end
end
#
# Loosely guess the purpose of a device based on available
# match values. In the future, also take into account the
# exposed services and rename to guess_purpose_with_match()
#
def guess_purpose_from_match(match)
# some data that is sent to this is numeric; we do not want that
pstr = ""
# Go through each character of each value and make sure it is all
# UTF-8
match.values.each do |i|
if i.respond_to?(:encoding)
i.each_char do |j|
begin
pstr << j.downcase.encode("UTF-8")
rescue Encoding::UndefinedConversionError
# rescue Encoding::UndefinedConversionError => e
# this works in Framework, but causes a Travis CI error
# elog("Found incompatible (non-ANSI) character in guess_purpose_from_match")
end
end
end
end
# Loosely map keywords to specific purposes
case pstr
when /windows server|windows (nt|20)/
'server'
when /windows (xp|vista|[78]|10)/
'client'
when /printer|print server/
'printer'
when /router/
'router'
when /firewall/
'firewall'
when /linux/
'server'
else
'device'
end
end
# Ensure that the host attribute is using ascii safe text
# and escapes any other byte value.
def sanitize(text)
Rex::Text.ascii_safe_hex(text)
end
#
# Normalize data from Meterpreter's client.sys.config.sysinfo()
#
def normalize_session_fingerprint(data)
ret = {}
case data[:os]
when /Windows/
ret.update(parse_windows_os_str(data[:os]))
# Switch to this code block once the multi-meterpreter code review is complete
=begin
when /^(Windows \w+)\s*\(Build (\d+)(.*)\)/
ret['os.product'] = $1
ret['os.build'] = $2
ret['os.vendor'] = 'Microsoft'
possible_sp = $3
if possible_sp =~ /Service Pack (\d+)/
ret['os.version'] = 'SP' + $1
end
=end
when /Linux (\d+\.\d+\.\d+\S*)\s* \((\w*)\)/
ret['os.product'] = "Linux"
ret['os.version'] = $1
ret['os.arch'] = get_arch_from_string($2)
else
ret['os.product'] = data[:os]
end
ret['os.arch'] = data[:arch] if data[:arch]
ret['host.name'] = data[:name] if data[:name]
[ ret ]
end
#
# Normalize data from Nmap fingerprints
#
def normalize_nmap_fingerprint(data)
ret = {}
# :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"2000" :os_accuracy=>"94"
ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy]
if (data[:os_vendor] == data[:os_family])
ret['os.product'] = data[:os_family]
else
ret['os.product'] = data[:os_family]
ret['os.vendor'] = data[:os_vendor]
end
# Nmap places the type of Windows (XP, 7, etc) into the version field
if ret['os.product'] == 'Windows' and data[:os_version]
ret['os.product'] = ret['os.product'] + ' ' + data[:os_version].to_s
else
ret['os.version'] = data[:os_version]
end
ret['host.name'] = data[:hostname] if data[:hostname]
if ret['os.certainty']
ret['os.certainty'] = [ ret['os.certainty'].to_f, MAX_NMAP_CERTAINTY ].min.to_s
end
[ ret ]
end
#
# Normalize data from MBSA fingerprints
#
def normalize_mbsa_fingerprint(data)
ret = {}
# :os_match=>"Microsoft Windows Vista SP0 or SP1, Server 2008, or Windows 7 Ultimate (build 7000)"
# :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"7" :os_accuracy=>"100"
ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy]
ret['os.family'] = data[:os_family] if data[:os_family]
ret['os.vendor'] = data[:os_vendor] if data[:os_vendor]
if data[:os_family] and data[:os_version]
ret['os.product'] = data[:os_family] + " " + data[:os_version]
end
ret['host.name'] = data[:hostname] if data[:hostname]
[ ret ]
end
#
# Normalize data from Nexpose fingerprints
#
def normalize_nexpose_fingerprint(data)
ret = {}
# :family=>"Windows" :certainty=>"0.85" :vendor=>"Microsoft" :product=>"Windows 7 Ultimate Edition"
# :family=>"Windows" :certainty=>"0.67" :vendor=>"Microsoft" :arch=>"x86" :product=>'Windows 7' :version=>'SP1'
# :family=>"Linux" :certainty=>"0.64" :vendor=>"Linux" :product=>"Linux"
# :family=>"Linux" :certainty=>"0.80" :vendor=>"Ubuntu" :product=>"Linux"
# :family=>"IOS" :certainty=>"0.80" :vendor=>"Cisco" :product=>"IOS"
# :family=>"embedded" :certainty=>"0.61" :vendor=>"Linksys" :product=>"embedded"
ret['os.certainty'] = data[:certainty] if data[:certainty]
ret['os.family'] = data[:family] if data[:family]
ret['os.vendor'] = data[:vendor] if data[:vendor]
case data[:product]
when /^Windows/
# TODO: Verify Windows CE and Windows 8 RT fingerprints
# Translate the version into the representation we want
case data[:version].to_s
# These variants are normalized to just 'Windows <Version>'
when "NT", "2000", "95", "ME", "XP", "Vista", "7", "8", "8.1"
ret['os.product'] = "Windows #{data[:version]}"
# Service pack in the version field should be recognized
when /^SP\d+/, /^Service Pack \d+/
ret['os.product'] = data[:product]
ret['os.version'] = data[:version]
# No version means the version is part of the product already
when nil, ''
# Trim any 'Server' suffix and use as it is
ret['os.product'] = data[:product].sub(/ Server$/, '')
# Otherwise, we assume a Server version of Windows
else
ret['os.product'] = "Windows Server #{data[:version]}"
end
# Extract the edition string if it is present
if data[:product] =~ /(XP|Vista|\d+(?:\.\d+)) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/
ret['os.edition'] = $2
end
when nil, 'embedded'
# Use the family or vendor name when the product is empty or 'embedded'
ret['os.product'] = data[:family] unless data[:family] == 'embedded'
ret['os.product'] ||= data[:vendor]
ret['os.version'] = data[:version] if data[:version]
else
# Default to using the product name reported by Nexpose
ret['os.product'] = data[:product] if data[:product]
end
ret['os.arch'] = get_arch_from_string(data[:arch]) if data[:arch]
ret['os.arch'] ||= get_arch_from_string(data[:desc]) if data[:desc]
[ ret ]
end
#
# Normalize data from Retina fingerprints
#
def normalize_retina_fingerprint(data)
ret = {}
# :os=>"Windows Server 2003 (X64), Service Pack 2"
case data[:os]
when /Windows/
ret.update(parse_windows_os_str(data[:os]))
else
# No idea what this looks like if it isn't windows. Just store
# the whole thing and hope for the best.
# TODO: Add examples of non-Windows results
ret['os.product'] = data[:os] if data[:os]
end
[ ret ]
end
#
# Normalize data from Nessus fingerprints
#
def normalize_nessus_fingerprint(data)
ret = {}
# :os=>"Microsoft Windows 2000 Advanced Server (English)"
# :os=>"Microsoft Windows 2000\nMicrosoft Windows XP"
# :os=>"Linux Kernel 2.6"
# :os=>"Sun Solaris 8"
# :os=>"IRIX 6.5"
# Nessus sometimes jams multiple OS names together with a newline.
oses = data[:os].split(/\n/)
if oses.length > 1
# Multiple fingerprints means Nessus wasn't really sure, reduce
# the certainty accordingly
ret['os.certainty'] = 0.5
else
ret['os.certainty'] = 0.8
end
# Since there is no confidence associated with them, the best we
# can do is just take the first one.
case oses.first
when /^(Microsoft |)Windows/
ret.update(parse_windows_os_str(data[:os]))
when /(2\.[46]\.\d+[-a-zA-Z0-9]+)/
# Look for older Linux kernel versions
ret['os.product'] = "Linux"
ret['os.version'] = $1
when /^Linux Kernel ([\d\.]+)(.*)/
# Look for strings like "Linux Kernel 2.6 on Ubuntu 9.10 (karmic)"
# Ex: Linux Kernel 2.2 on Red Hat Linux release 6.2 (Zoot)
# Ex: Linux Kernel 2.6 on Ubuntu Linux 8.04 (hardy)
ret['os.product'] = "Linux"
ret['os.version'] = $1
vendor = $2.to_s
# Try to snag the vendor name as well
if vendor =~ /on (\w+|\w+ \w+|\w+ \w+ \w+) (Linux|\d)/
ret['os.vendor'] = $1
end
when /(.*) ([0-9\.]+)$/
# Then we don't necessarily know what the os is, but this fingerprint has
# some version information at the end, pull it off, treat the first part
# as the OS, and the rest as the version.
ret['os.product'] = $1.gsub("Kernel", '').strip
ret['os.version'] = $2
else
# TODO: Return each OS guess as a separate match
ret['os.product'] = oses.first
end
ret['host.name'] = data[:hname] if data[:hname]
[ ret ]
end
#
# Normalize data from Qualys fingerprints
#
def normalize_qualys_fingerprint(data)
ret = {}
# :os=>"Microsoft Windows 2000"
# :os=>"Windows 2003"
# :os=>"Microsoft Windows XP Professional SP3"
# :os=>"Ubuntu Linux"
# :os=>"Cisco IOS 12.0(3)T3"
# :os=>"Red-Hat Linux 6.0"
case data[:os]
when /Windows/
ret.update(parse_windows_os_str(data[:os]))
when /^(Cisco) (IOS) (\d+[^\s]+)/
ret['os.product'] = $2
ret['os.vendor'] = $1
ret['os.version'] = $3
when /^([^\s]+) (Linux)(.*)/
ret['os.product'] = $2
ret['os.vendor'] = $1
ver = $3.to_s.strip.split(/\s+/).first
if ver =~ /^\d+\./
ret['os.version'] = ver
end
else
parts = data[:os].split(/\s+/, 3)
ret['os.product'] = "Unknown"
ret['os.product'] = parts[0] if parts[0]
ret['os.product'] << " " + parts[1] if parts[1]
ret['os.version'] = parts[2] if parts[2]
end
[ ret ]
end
#
# Normalize data from FusionVM fingerprints
#
def normalize_fusionvm_fingerprint(data)
ret = {}
case data[:os]
when /Windows/
ret.update(parse_windows_os_str(data[:os]))
when /Linux ([^[:space:]]*) ([^[:space:]]*) .* (\(.*\))/
ret['os.product'] = "Linux"
ret['host.name'] = $1
ret['os.version'] = $2
ret['os.arch'] = get_arch_from_string($3)
else
ret['os.product'] = data[:os]
end
ret['os.arch'] = data[:arch] if data[:arch]
ret['host.name'] = data[:name] if data[:name]
[ ret ]
end
#
# Normalize data from generic fingerprints
#
def normalize_generic_fingerprint(data)
ret = {}
ret['os.product'] = data[:os_name] || data[:os] || data[:os_fingerprint] || "Unknown"
ret['os.arch'] = data[:os_arch] if data[:os_arch]
ret['os.certainty'] = data[:os_certainty] || 0.5
[ ret ]
end
#
# Convert a host.os.*_fingerprint Note into a hash containing 'os.*' and 'host.*' fields
#
# Also includes a os.certainty which is a float from 0 - 1.00 indicating the
# scanner's confidence in its fingerprint. If the particular scanner does
# not provide such information, default to 0.80.
#
def normalize_scanner_fp(fp)
hits = []
return hits if not validate_fingerprint_data(fp)
case fp.ntype
when /^host\.os\.(.*_fingerprint)$/
pname = $1
pmeth = 'normalize_' + pname
if self.respond_to?(pmeth)
hits = self.send(pmeth, fp.data)
else
hits = normalize_generic_fingerprint(fp.data)
end
end
hits.each {|hit| hit['os.certainty'] ||= 0.80}
hits
end
#
# Take a windows version string and return a hash with fields suitable for
# Host this object's version fields. This is used as a fall-back to parse
# external fingerprints and should eventually be replaced by per-source
# mappings.
#
# A few example strings that this will have to parse:
# sessions
# Windows XP (Build 2600, Service Pack 3).
# Windows .NET Server (Build 3790).
# Windows 2008 (Build 6001, Service Pack 1).
# retina
# Windows Server 2003 (X64), Service Pack 2
# nessus
# Microsoft Windows 2000 Advanced Server (English)
# qualys
# Microsoft Windows XP Professional SP3
# Windows 2003
#
# Note that this list doesn't include nexpose or nmap, since they are
# both kind enough to give us the various strings in seperate pieces
# that we don't have to parse out manually.
#
def parse_windows_os_str(str)
ret = {}
# Set some reasonable defaults for Windows
ret['os.vendor'] = 'Microsoft'
ret['os.product'] = 'Windows'
# Determine the actual Windows product name
case str
when /\.NET Server/
ret['os.product'] << ' Server 2003'
when / (2000|2003|2008|2012)/
ret['os.product'] << ' Server ' + $1
when / (NT (?:3\.51|4\.0))/
ret['os.product'] << ' ' + $1
when /Windows (95|98|ME|XP|Vista|[\d\.]+)/
ret['os.product'] << ' ' + $1
else
# If we couldn't pull out anything specific for the flavor, just cut
# off the stuff we know for sure isn't it and hope for the best
ret['os.product'] = (ret['os.product'] + ' ' + str.gsub(/(Microsoft )|(Windows )|(Service Pack|SP) ?(\d+)/i, '').strip).strip
# Make sure the product name doesn't include any non-alphanumeric stuff
# This fixes cases where the above code leaves 'Windows XX (Build 3333,)...'
ret['os.product'] = ret['os.product'].split(/[^a-zA-Z0-9 ]/).first.strip
end
# Take a guess at the architecture
arch = get_arch_from_string(str)
ret['os.arch'] = arch if arch
# Extract any service pack value in the string
if str =~ /(Service Pack|SP) ?(\d+)/i
ret['os.version'] = "SP#{$2}"
end
# Extract any build ID found in the string
if str =~ /build (\d+)/i
ret['os.build'] = $1
end
# Extract the OS edition if available
if str =~ /(\d+|\d+\.\d+) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/
ret['os.edition'] = $2
else
if str =~ /(Professional|Enterprise|Pro|Home|Start|Datacenter|Web|Storage|MultiPoint)/
ret['os.edition'] = $1
end
end
ret
end
#
# Return a normalized architecture based on patterns in the input string.
# This will identify things like sparc, powerpc, x86_x64, and i686
#
def get_arch_from_string(str)
res = Recog::Nizer.match("architecture", str)
return unless (res and res['os.arch'])
res['os.arch']
end
end