Tag Archives: Splunk

Event logs with control characters


A WLS user contacted me and was having issues parsing a date from a data field in EventID 6008 (unexpected shutdown). Taking a look at my logs everything looked fine, even in a viewer like Notepad++ with Show View->Show Symbol->Show All Characters. Since I use Splunk, on the record in question I selected Event Actions->Show Source, and it looked fine there too. Next I did a right-click and Inspect on the web page and there it was: “‎” aka u200e, aka E2808E, aka “Left-To-Right Mark”.

lrm

Saving the event text to a file and opening it with a hex editor also shows the control character in question (e2 80 8e):

e2808e

Indeed these control characters are included in at least 8 other events and all appear to be in date fields.

In Splunk you can use rex/sed or replace to remove control characters before attempting a strptime or other function.

`wlslogs` EventID=6008 | rex field=Data1 mode=sed "s/\p{C}//g" | eval NewDate=strptime(Data1,"%m/%d/%Y")

or

`wlslogs` EventID=6008 | eval NewDate=strptime(replace(Data1,"\p{C}",""),"%m/%d/%Y")

Finding vulnerable OpenSSL DLLs (Heartbleed) with WLS and Splunk


If you are running WLS with ModuleMonitor enabled, you already have logs of all loaded DLLs.

Simply run the following search:

WLS_ModuleMonitor “openssl” FileVersion=”1.0.1*” NOT FileVersion=”1.0.1g”

Results will look similar to:

2014-04-24T11:28:38-05:00 [host] WLS_ModuleMonitor: LogType=”WindowsEventLog”, BaseFileName=”ssleay32.dll”, ChangeType=”Added”, CompanyName=”The OpenSSL Project, http://www.openssl.org/”, FileDescription=”OpenSSL Shared Library”, FileName=”C:\Program Files\MakerBot\MakerWare\ssleay32.dll”, FileVersion=”1.0.1e”, InternalName=”ssleay32″, Language=”English (United States)”, Length=”343040″, MD5=”A49B52FB216AD1524C902A41A433A1E1″, Process=”makerware”, ProductVersion=”1.0.1e”, SHA1=”F21CBCC5F291C1D841581FDBBBF854B315D71248″, WLSKey=”35505″, Zone=”0″


For more information on WLS, click “WLS Information” at the top, or here: WLS Information

If you’d like additional information about WLS, send me a note via the contact form. WLS is currently available to US entities, but does require a signed license agreement.

USB Device Tracking with WLS and Splunk


WLS provides the option to monitor plug and play devices. When enabled, a log will be generated for each state change containing the current state and detailed device information.

Enabling Device Monitoring

If WLS is already installed, device monitoring can be enabled by changing the registry value at HKLM\Software\KCP\WLS\Config\DeviceMonitor\Enabled from 0 to 1.

EnableDeviceMonitoring

If WLS hasn’t been installed, adding (or changing) the DeviceMonitor section and setting Enabled to 1 in the initial.xml will enable this feature when WLS is installed.

<WLS>
 <Config>
  <DeviceMonitor>
   <Enabled>1</Enabled>
  </DeviceMonitor>
 </Config>
</WLS>

Reading the logs

Once device monitoring is enabled, when a device state changes, entries like the following will appear in your logs. Each entry should include at least one key/value pair that contains a product id (PID), a vendor id (VID), and a serial number as well as another key/value pair that contains the Class, SubClass, and Protocol.

Device added

Sep 18 15:15:46 [host] WLS_DeviceMonitor: LogType=”WindowsEventLog”, Caption=”USB Mass Storage Device”, ChangeType=”Added”, ClassGuid=”{36fc9e60-c465-11cf-8056-444553540000}”, CompatibleID=”USB\Class_08&SubClass_06&Prot_50″, CompatibleID1=”USB\Class_08&SubClass_06″, CompatibleID2=”USB\Class_08″, ConfigManagerErrorCode=”0″, ConfigManagerUserConfig=”False”, CreationClassName=”Win32_PnPEntity”, Description=”USB Mass Storage Device”, DeviceID=”USB\VID_1043&PID_8012604261021070038″, HardwareID=”USB\VID_1043&PID_8012&REV_0100″, HardwareID1=”USB\VID_1043&PID_8012″, Manufacturer=”Compatible USB storage device”, Name=”USB Mass Storage Device”, PNPDeviceID=”USB\VID_1043&PID_8012604261021070038″, Service=”USBSTOR”, Status=”OK”, SystemCreationClassName=”Win32_ComputerSystem”, SystemName=”[host]”, TIME_CREATED=”130240089461584098″, WLSKey=”23559″

Device removed

Sep 18 15:16:01 [host] WLS_DeviceMonitor: LogType=”WindowsEventLog”, Caption=”USB Mass Storage Device”, ChangeType=”Removed”, ClassGuid=”{36fc9e60-c465-11cf-8056-444553540000}”, CompatibleID=”USB\Class_08&SubClass_06&Prot_50″, CompatibleID1=”USB\Class_08&SubClass_06″, CompatibleID2=”USB\Class_08″, ConfigManagerErrorCode=”0″, ConfigManagerUserConfig=”False”, CreationClassName=”Win32_PnPEntity”, Description=”USB Mass Storage Device”, DeviceID=”USB\VID_1043&PID_8012604261021070038″, HardwareID=”USB\VID_1043&PID_8012&REV_0100″, HardwareID1=”USB\VID_1043&PID_8012″, Manufacturer=”Compatible USB storage device”, Name=”USB Mass Storage Device”, PNPDeviceID=”USB\VID_1043&PID_8012604261021070038″, Service=”USBSTOR”, Status=”OK”, SystemCreationClassName=”Win32_ComputerSystem”, SystemName=”[host]”, TIME_CREATED=”130240089617443014″, WLSKey=”23569″

Device error

If the device fails to load properly, the Status field will be set to Error instead of OK.

Sep 18 15:15:45 [host] WLS_DeviceMonitor: LogType=”WindowsEventLog”, Caption=”USB Mass Storage Device”, ChangeType=”Added”, ClassGuid=”{36fc9e60-c465-11cf-8056-444553540000}”, CompatibleID=”USB\Class_08&SubClass_06&Prot_50″, CompatibleID1=”USB\Class_08&SubClass_06″, CompatibleID2=”USB\Class_08″, ConfigManagerErrorCode=”0″, ConfigManagerUserConfig=”False”, CreationClassName=”Win32_PnPEntity”, Description=”USB Mass Storage Device”, DeviceID=”USB\VID_1043&PID_8012604261021070038″, HardwareID=”USB\VID_1043&PID_8012&REV_0100″, HardwareID1=”USB\VID_1043&PID_8012″, Manufacturer=”Compatible USB storage device”, Name=”USB Mass Storage Device”, PNPDeviceID=”USB\VID_1043&PID_8012604261021070038″, Service=”USBSTOR”, Status=”Error”, SystemCreationClassName=”Win32_ComputerSystem”, SystemName=”[host]”, TIME_CREATED=”130240089461584098″, WLSKey=”23558″

Add more data!

Splunk is great for combining data from multiple sources, and in this case, data will be added to decode VID, PID, Class, SubClass, and Protocol.

The best place (I’m aware of) to get VID and PID information is http://www.linux-usb.org/usb-ids.html. Download the usb.ids file, work some spreadsheet magic, and you have a nice csv file for a Splunk lookup table.

Getting the Class, SubClass, and Protocol information isn’t quite as straightforward; I started here: http://www.usb.org/developers/defined_class and generated a csv file for another Splunk lookup table. Luckily these don’t change as often as VID and PID information.

Add these csv files to Splunk by going to “Lookups” in the Splunk Manager page, then choosing “Add new” on the “Lookup table files” line.

Both lookup tables are available to download at the end of this post.

Extracting lookup data

Now that lookup tables exist, the information to lookup needs to be extracted from the source logs. I created two macros, one to extract and lookup the hardware manufacturer and model, another to extract and lookup the class, subclass, and protocol. Both macros split a single field into parts, and lookup the individual parts. I assumed that a new PID may exist for an existing VID, and that the new PID may not be in the lookup table; so a double lookup is performed and the first non-null value is returned.

Extract and lookup VID and PID

eval DeviceModelData=split(HardwareID,”\\”)
| eval USBModelData=split(mvindex(DeviceModelData,1),”&”)
| eval USBMfr=mvindex(USBModelData,0) | eval USBModel=mvindex(USBModelData,1)
| eval USBInterfaces=mvindex(USBModelData,2)
| lookup USBIDS Mfr as USBMfr Model as USBModel OUTPUT MfrName as USBMfrName ModelName as USBModelName
| lookup USBIDS Mfr as USBMfr OUTPUT MfrName as USBMfrName1
| eval USBMfrName=mvindex(coalesce(USBMfrName,USBMfrName1),0)
| eval USBMfrName=coalesce(USBMfrName,USBMfr)
| eval USBModelName=coalesce(USBModelName,USBModel)

Extract and lookup Class, SubClass, and Protocol

eval DeviceData=split(CompatibleID,”\\”)
| eval USBData=split(mvindex(DeviceData,1),”&”)
| eval USBClass=mvindex(split(mvindex(USBData,0),”_”),1)
| strcat “Class_” USBClass USBClass
| eval USBSubClass=mvindex(USBData,1)
| eval USBProtocol=mvindex(USBData,2)
| lookup USBSpec Class as USBClass SubClass as USBSubClass Protocol as USBProtocol OUTPUT ProtocolDescription as ProtocolDescription2
| lookup USBSpec Class as USBClass SubClass as USBSubClass OUTPUT SubClassDescription as SubClassDescription1,ProtocolDescription as ProtocolDescription1
| lookup USBSpec Class as USBClass OUTPUT ClassDescription,SubClassDescription,ProtocolDescription
| eval ClassDescription=mvindex(ClassDescription,0)
| eval SubClassDescription=mvindex(coalesce(SubClassDescription1,SubClassDescription),0)
| eval ProtocolDescription=coalesce(ProtocolDescription2,ProtocolDescription1,ProtocolDescription)
| strcat ClassDescription ” ” SubClassDescription ” ” ProtocolDescription FullUSBDescription

Combined Result

With all the information combined, it’s time to make a dashboard. I created one that displays each class in it’s own titled section for readability. If the lookups are able to decode the VID, PID, SubClass, and Protocol, the decode is shown, otherwise the original undecoded value is shown.

Devices

When deployed enterprise-wide, logs now exist to uniquely track any plug and play hardware across all systems and users. The PID, VID, and serial number can be used to identify new and potentially unwanted devices. Combined with a process to issue hardware from a central location, the issuer can register the device and the end-user. This reduces the noise and false positive alerts, and provides user accountability to a specific device. A word of caution, I have seen serial numbers reused, in mass.

Here are the lookup tables I’m currently using, they may be out of date. They are renamed to .xls files so WordPress would let me upload them; rename to .csv after downloading.

USBIDS
USBSpec

Have other ideas for using the data WLS provides, or data you’d like to have logged? Let me know in the comments below or via the contact form.

For more information on WLS, click “WLS Information” at the top, or here: WLS Information

If you’d like additional information about WLS, send me a note via the contact form. WLS is currently available to US entities, but does require a signed license agreement.

Monitoring Windows security products


I came across the WMI namespaces ROOT\SecurityCenter (XP) and ROOT\SecurityCenter2 (Vista+) while doing some research. These namespaces provide the product and state for AntiVirus, AntiSpyware (SecurityCenter2 only), and Firewall as recognized by Windows.

Since WLS provides a generic interface for WMI logging, I created the entries and updated the configuration. I have both XP and Windows 7 systems and each has it’s own namespace, so I’ll need two AntiVirus and Firewall entries, but only one AntiSpyware, and I’d like the information reported every 24 hours. The update to the configuration looks like this:

<WLS>
  <Config>
    <WMI>
      <AntiSpyware2>
        <Enabled>1</Enabled>
        <Class>AntiSpywareProduct</Class>
        <Interval>86400</Interval>
        <Namespace>ROOT\SecurityCenter2</Namespace>
      </AntiSpyware2>
      <AntiVirus>
        <Enabled>1</Enabled>
        <Class>AntiVirusProduct</Class>
        <Interval>86400</Interval>
        <Namespace>ROOT\SecurityCenter</Namespace>
      </AntiVirus>
      <AntiVirus2>
        <Enabled>1</Enabled>
        <Class>AntiVirusProduct</Class>
        <Interval>86400</Interval>
        <Namespace>ROOT\SecurityCenter2</Namespace>
      </AntiVirus2>
      <Firewall>
        <Enabled>1</Enabled>
        <Class>FirewallProduct</Class>
        <Interval>86400</Interval>
        <Namespace>ROOT\SecurityCenter</Namespace>
      </Firewall>
      <Firewall2>
        <Enabled>1</Enabled>
        <Class>FirewallProduct</Class>
        <Interval>86400</Interval>
        <Namespace>ROOT\SecurityCenter2</Namespace>
      </Firewall2> 
    </WMI> 
  </Config> 
</WLS> 

The ROOT\SecurityCenter namespace may be invalid on Vista+, and the ROOT\SecurityCenter2 namespace is invalid on XP; WLS will report the error once at startup and disable the offending WMI entry.

The logs generated after applying the configuration look like this:

2013-09-04T16:15:50-05:00 [host] WLS_WMI: LogType=”WindowsEventLog”, GroupID=”8″, MonitorName=”AntiSpyware2″, WLSKey=”3635″, displayName=”Symantec Endpoint Protection”, instanceGuid=”{D8BEB080-B73A-17E3-1B37-B6B462689202}”, pathToSignedProductExe=”C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\WSCSavNotifier.exe”, pathToSignedReportingExe=”C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\Rtvscan.exe”, productState=”462848″

2013-09-04T16:08:31-05:00 [host] WLS_WMI: LogType=”WindowsEventLog”, GroupID=”2″, MonitorName=”AntiVirus”, WLSKey=”2″, companyName=”Symantec Corporation”, displayName=”Symantec Endpoint Protection”, instanceGuid=”{FB06448E-52B8-493A-90F3-E43226D3305C}”, onAccessScanningEnabled=”True”, productUptoDate=”True”, versionNumber=”11.0.7200.155″

2013-09-04T16:15:50-05:00 [host] WLS_WMI: LogType=”WindowsEventLog”, GroupID=”10″, MonitorName=”AntiVirus2″, WLSKey=”3636″, displayName=”Symantec Endpoint Protection”, instanceGuid=”{63DF5164-9100-186D-2187-8DC619EFD8BF}”, pathToSignedProductExe=”C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\WSCSavNotifier.exe”, pathToSignedReportingExe=”C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\Rtvscan.exe”, productState=”462848″

2013-09-04T16:08:32-05:00 [host] WLS_WMI: LogType=”WindowsEventLog”, GroupID=”5″, MonitorName=”Firewall”, WLSKey=”5″, companyName=”Symantec Corporation.”, displayName=”Symantec Endpoint Protection”, enabled=”True”, instanceGuid=”{BE898FE3-CD0B-4014-85A9-03DB9923DDB6}”, versionNumber=”10.0″

2013-09-04T16:15:50-05:00 [host] WLS_WMI: LogType=”WindowsEventLog”, GroupID=”13″, MonitorName=”Firewall2″, WLSKey=”3638″, displayName=”Symantec Endpoint Protection”, instanceGuid=”{5BE4D041-DB6F-1935-0AD8-24F3E73C9FC4}”, pathToSignedProductExe=”C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\Smc.exe”, pathToSignedReportingExe=”C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\Smc.exe”, productState=”266256″

The information from ROOT\Security center has fields defined such as “displayName”, “enabled”, “productUptoDate”, and “onAccessScanningEnabled”; whereas ROOT\SecurityCenter2 gives us “displayName” and “productState”. The productState is returned as a decimal representation of a hex value which contains the information we need, just encoded.

A bit more research turned up some helpful posts, notably http://neophob.com/2010/03/wmi-query-windows-securitycenter2/, which lead to the creation of a `decodeProductState` macro. The macro converts the productState to hex, trims the leading “0x”, and adds a leading 0 to pad the result to 6 digits. Each pair of digits represents a state, so I split them that way for ease of reuse. Then specific values are checked for enabled and productUptoDate and assigned to enabled2 and productUptoDate2. Since there will be mixed results from ROOT\SecurityCenter and ROOT\SecurityCenter2, coalesce will be used to keep the first non-null value of enabled or enabled2, and productUptoDate or productUptoDate2, assigning the result back to enabled and productUptoDate.

eval productStateHex=”0″.substr(tostring(productState,”hex”),3)
| eval productStateHex1=substr(productStateHex,0,2)
| eval productStateHex2=substr(productStateHex,3,2)
| eval productStateHex3=substr(productStateHex,5,2)
| eval enabled2=if(substr(productStateHex2,1,1)=”1″,”True”,”False”)
| eval productUptoDate2=if(productStateHex3=”00″,”True”,”False”)
| eval enabled=coalesce(enabled,enabled2)
| eval productUptoDate=coalesce(productUptoDate,productUptoDate2)

Finally I created a simple dashboard to display the results. This will get more refined as it’s utilized, but it’s a good starting point. This data can also be used to drive alerts if more than a certain percent or count of your hosts has outdated definitions, or to check for hosts that don’t have any products installed at all.

productexample

Have other ideas for using the data WLS provides? Let me know in the comments below or via the contact form.

What is WLS?

If you’d like more information on WLS, send me a note via the contact form. WLS is currently available to US entities, but does require a signed license agreement.

Tracking software versions with WLS and Splunk


While initially intended to aid in detecting malware, the data WLS provides has many other uses. One that we use is to track software updates and outdated software being used on our network. In theory you could schedule a report similar to the ones below and automatically email people who are using incorrect software versions.

The search I used for the examples is shown below, and could easily be turned into a Splunk form where you can specify any BaseFileName for a quick report.

(EventID=4688 OR EventID=592) BaseFileName=firefox.exe  | dedup host | stats count(MD5) by CompanyName, FileVersion, ProductVersion, MD5

Breakdown:

  • Find all process creation events
    • (EventID=4688 OR EventID=592)
  • Find processess named “firefox.exe”
    • BaseFileName=firefox.exe
  • Deduplicate hosts so we only get the most recent execution
    • dedup host
  • Calculate statistics (count all the MD5s) by the fields specified
    • stats count(MD5) by CompanyName, FileVersion, ProductVersion, MD5

Example results!

firefox.exe

FirefoxVersions

chrome.exe

ChromeVersions

iexplore.exe

IExploreVersions

acrord32.exe

AcroRd32Versions

java.exe

Too embarrassing to post 😦

Have other ideas for using the data WLS provides? Let me know in the comments below or via the contact form.

What is WLS?

If you’d like more information on WLS, send me a note via the contact form. WLS is currently available to US entities, but does require a signed license agreement.

Using Splunk to watch for new binaries


The method presented below can be used to track any log attribute in Splunk; this example demonstrates watching MD5 hashes of executed files and loaded modules.

I’ve enabled Process Auditing via the Group Policy Editor and configured WLS to provide MD5 hashes.

standardhashes

I also enabled the “ModuleMonitor” in WLS which tracks loaded modules by process

modulemonitoron

and configured it to provide MD5 hashes for these as well.

modulehashes

Now that we are receiving hashes for all executed files and loaded modules, let’s start tracking them in Splunk.

First we’ll need to create a lookup table, there are a few ways to do this, a quick way is simply:

| outputlookup md5tracker.csv

This will create an empty csv file named “md5tracker.csv”.

emptycsv

Next, we need to search for and add the desired data to the csv file. I like to preserve some of the metadata that WLS reports with each record for later use – avoid re-searching, etc.

index=windows MD5=* | dedup MD5 | lookup md5tracker.csv MD5 as MD5 OUTPUT FirstSeen as LookupFirstSeen | where NOT LookupFirstSeen LIKE “%” | eval FirstSeen=_time  | table FirstSeen, MD5, BaseFileName, CompanyName, FileDescription, FileVersion, InternalName, Language, Signed, Length | inputlookup md5tracker.csv append=t | dedup MD5 | outputlookup md5tracker.csv

OK, let’s break this down:

Find the desired records: index=windows MD5=*

Remove duplicates: dedup MD5

Lookup the MD5s in our lookup table, returning the date first seen: lookup md5tracker.csv MD5 as MD5 OUTPUT FirstSeen as LookupFirstSeen

Remove records that already exist (field will be non-null): where NOT LookupFirstSeen LIKE “%”

Preserve the time stamp as desired output field: eval FirstSeen=_time

Format the desired fields into a table: table FirstSeen, MD5, BaseFileName, CompanyName, FileDescription, FileVersion, InternalName, Language, Signed, Length

Bring all the old data in and append it: inputlookup md5tracker.csv append=t

Remove duplicates (just in case): dedup MD5

Write out the new + old data: outputlookup md5tracker.csv

After the first run, you should have the results from your chosen time period now stored in md5tracker.csv

firstresults

You’ll want to save this search

save1

and schedule it to run every x minutes for the last x minutes; I schedule mine for every 15 minutes.

save2

Once this is complete you’ll now have a search that keeps your lookup table up-to-date. Now what?

What you do next depends on how closely you feel this needs monitored. I run a second search every x minutes that alerts on all new entries in the last x minutes (based on the FirstSeen) field.

| inputlookup md5tracker.csv | where now()-FirstSeen < 2200 | table FirstSeen, MD5, BaseFileName, CompanyName, FileDescription, FileVersion, InternalName, Language, Signed, Length

This simply take the entire table and selects all entries in the last 2200 seconds (2200 / 60 = 36.6 minutes) and formats the results into a table. I scheduled it to run every 35 minutes with some overlap time (hence 2200 instead of 2100).

save3

I also like to take an export of the hashes every so often and check them against Team Cymru’s malware hash registry – https://hash.cymru.com/

| inputlookup md5tracker.csv | table MD5

Export the results from Splunk, open the file in a spreadsheet, and copy/paste them into Team Cymru’s lookup for a quick analysis. An enterprising person might also create a custom Splunk command that uses their DNS lookup service (https://www.team-cymru.org/Services/MHR/#dns) and puts the results into the lookup table itself…

I currently have 23,537 executable hashes and 131,885 module (dll, etc) hashes, and see a few new ones at most search intervals during normal business hours. After the initial gathering, the periodic alerts are easy to quickly review, and you’ll know everything that is running on your Windows hosts.