Setting Up Network Security Monitoring for Multiple Sensors in Security Onion

So the other day I was wondering about this, especially as I have used SecurityOnion in my MFIRE class. There are a few options when installing NSM on your VM... and I wondered how to set it up so you had a server (maybe with a sensor on it) and then a sensor off somewhere else feeding that server with information. That way, if you are ever on site and are collecting data, but really do not feel like going into each box separately... this would solve that issue. Well, no time like the present to see? Off we go!

1) Google is your friend (duh) I have never met the creator of SecurityOnion (Doug Burks) but I would definitely give him a big hug for creating this tool. Anywhoo... I found him talking about in 2011 doing this exact same thing. Score. 

2) Load up your VMs of Security Onion. Obviously we need two here. 

3) Set up your server FIRST

• Configure NSM
• Go to Applications > NSM > Setup and follow along
• Let's choose advanced... and for grins lets install BOTH a sensor and server

Configure what you want on this Instance

• Choose your IDS... (but choose wisely!) I will use Snort
• Next is the rules you want to use. If you do not have an oinkcode you must go for 'Emerging Threats GPL'. You can get an oinkcode for free over at the Snort website. I Don't worry if you set this up before your oinkcode, you can add it manually in the /etc/pulledpork/pulledpork.conf file.

Rulesets

• Enter your username. Write this down. Seriously- take my pen. Do it.
• Enter your email address. If you won't be sending yourself alerts don't stress this one too much. But write it down all the same.
• This is the big one-- password. Don't forget this!
• Ok now NSM tells you what its about to do. If you are OK with this hit 'Yes'
• Update rules/add any local rules. You can do this by running pulled pork (Applications > NSM > Rule Update)
• Before we leave, run 'ifconfig' in a terminal and note the IP

4) Set up sensor

• Configure NSM
• This is almost like above, except when you hit which components you would like to configure, hit 'Sensor'
• Remember how I said to note down that IP of the server? Well here is where you add it. You can add by host name if you prefer.

May want to ensure you can reach this first to save heartache...

• Now enter a username so NSM can ssh to the server. Note you need to add this user to the sudo list as well. For questions about adding a user or adding a user to a sudo list check out the respective links. (*note* be sure to add a home directory for the user '-m')

Here piggy piggy!

• Ok, choose whichever rules you chose for the server (for me Snort)
• Choose what interface you would like to listen on
• Again, we get NSM telling us what it will do and if you are happy click 'Yes'
• Don't go grab your coffee yet, you will need to enter the password for the SSH user! OK.. NOW go grab your coffee.
• Rules will be pulled from the server when you run 'Update Rules'

5)Fire up Sguil and let er rip!

Hopefully now everything is peachy. Lets fire up Sguil on the server, log in,  and see if we can see the interfaces we can monitor. Drumroll please!

Add What You Would Like to Monitor

Success! We see both the local sensor (melissa-desktop-eth0) and our sensor (MFIRE-eth4). If you are wondering what OSSEC is check it out here. Select the networks you want to monitor and then hit 'Start SGUIL'.

Oooh... data!

Look at all that fun stuff (I highlighted me deleting my user because I originally forgot its home directory. It labeled it as 'attack followed by the addition of a user'.. oops) I will not go into the inner workings of SGUIL here. I will say however if you want to look at this in prettier ways... you can log into Snorby or Squert. I hope you wrote down that information during setup!

Snorby Screenshot

Squert Screenshot

Oh and psh... did I mention Security Onion is free? :)

Memory

Whee! Memory forensics! It is scary how much information we can get from memory. If there is one thing you should grab from an image, its the memory (in my opinion!).

I am going to use Volatility- its free, open source, and pretty kickass. That is not to say there are no other memory forensics tools out there, HB Gary has one as does Mandiant. This is just the one I am most comfortable with.

ImageInfo
Before we delve into Volatility, its important for the program to know exactly which image it is trying to parse. Different OS flavors have different memory structures, even different service packs. Therefore it is important to figure out which OS we are dealing with. This is done with the plugin 'imageinfo' which search for KDBG (a certain set of characters found in the image which tells us which OS we are dealing with).

Now there are alot of things which this output tells us, but for us right now the most important thing is the Image Type and Suggested Profiles. We know it is for sure a WinXP image, and the Image Type tells us its Service Pack 3. So now for every command we run we want to add '--profile WinXPSP3x86' so Volatility knows the correct structure.

Processes
Now there are two different options we can do here- pslist (pstree follows the same concept as pslist) or psscan. What is the difference? Well pslist is following a double-linked list while psscan is simply looking to process headers in the pool. Still not make sense? Heh.. welcome to memory :D

Basically you know how some malware can hide their presence from users? Like you look at running process in Task Manager but don't see evil.exe running? The malware is using Direct Kernel Object Manipulation (DKOM) to unlink itself, therefore a normal LISTing of the process won't show it. However, if one simply SCANs (brute force searches) for the EPROCESS header... the evil.exe will show up. Malware you can run but you can't hide!

So let's run both to show the differences. This is a 'pslist'

And this is a 'psscan'

Psscan can find process that are not only hidden but also terminated as well (as long as the process header has not been overwritten! How handy is that.

Connections
Just like processes, one can walk the connections list or perform a brute force for connections.

We see our good ol friend PID 760. Again. Let's focus on this and see if we can pull anything out of memory about this.

Threads
Threads are the workhorse of processes. They are the ones doing the work. A process without threads is a process doing nothing. Within the thread is a wealth of information. It says the state of the thread, the state of the processor for that thread (think EIP, ESP, EBP), and any Tags associated with it. Now, I dont want to parse thru ALL the threads in my memory dump, I want to focus on the suspicious process. Luckily, with threads we can use the switch --p and limit our results a bit. 

So we see (again) a ton of information. What hits me is the Tag of 'HookedSSDT' as well as the ServiceTable (and based on the address running in kernel space) revealing the file pxhmnhtj.sys... our malicious kernel driver (again correlated with the address where its located).

SSDT
The System Service Description Table is basically a roadmap for the kernel on how to handle userland API calls. It is used by a lot of programs (including your AV) but malware also likes to lurk around here. So without getting into crazy detail,the files ntoskrnl.sys and win32k.sys are normal modules seen in the SSDT. If you could filter our those, it leaves a lot less for one to analyse. Basically if there is a hook for a userland API (EnumerateProcess for example) it can show up here.Based on our results from threads, our pxhmnhtj.sys file should be here..

Ha! CORRELATION! Hooray!

Now I know what you are thinking, wouldn't it be GREAT if we would pull out that driver? Well....
Volatility has a plugin called modules which list all the modules present in memory at the time of capture.It also has one called modscan (same concept again as pslist/psscan). Lets use that but pipe the results thru grep and see if we can figure out where the suspicious module is in memory. We see below we have found the module, sitting in virtual memory address 0xf899a000. Using that address we can extract the kernel level driver for analysis (as seen with moddump)






Now we have another file for analysis. Schaaawing!

Pulling out the Registry
So I remember how bummed everyone was when we didn't grab the NTUSER.DAT hive for our fateful user. Well never fear, volatility can also scan/list all the registry hives found in memory and them we can dump them out! >PHEW<

All we have to do now is use hivedump and provide the virtual address of the hive we would like to peruse, in our case 0xe1de0878. I would recommend saving it off to another file... its a deluge of data!

Oh and remember the MFT? WELL-- today is your lucky day, as there is a plugin for that too in Volatility :)

So one may say: then why should I bother grabbing anything else? Well for multiple reasons, but correlation is the key. If you need to put someone in prisons you want to be darn sure your evidence is backed up by even more evidence. It's all well and good to have the candlestick, but it would be better to have the candlestick, the partial print on the candlestick, a shoe print, and CCTV footage showing Miss Scarlet entering the Dining Room :)

Cheers from Denver! And Prefetch/Registry Goodness

I am getting ready to return back to Blighty, and I have thoroughly enjoyed myself in Denver. Not only because I was able to go to the FOR526 (Windows Memory Forensics In-Depth) Beta class, but also because Denver is a pretty kick ass city! Great beer, great food, friendly people, and epic mountains.... and some snow!

Today is our last posting before I get to the meat of our investigation... the memory dump. People will be shocked when you see all we can get from a memory dump :) 

Registry
Harlan Carvey is the godfather of the registry. He has contributed a lot to the community in terms of research, developments, and creation of tools. He talks about a lot of other pertinent topics in forensics too, highly recommend a visit to his blog. He created a tool called RegRipper, which parses thru a registry hive and looks for interesting artifacts which are generally useful in a forensic investigation. (I never recommend just trying to parse thru the registry blind... its just not a good idea).

So what is doing these searches? Well so glad you asked! When you download Harlan's tool you can also download all these awesome plugins, 235 to be exact. Here is a screenshot of me perusing the list. 

Well I know I am interested in autoruns... but if one checks, it is an NTUSER.DAT plugin, I did not grab NTUSER.DAT did I? Shame on me! It's now gone forever! Well, unless I can extract from memory! (Thats a hint). 

Well let's try another plugin, #60 'findexes' sounds promising.  Let's try it on the System hive:

First off we see exactly what this plugin is doing, searching the hive for any binary data starting with the characters MZ (the header for an executable). We have some hits, but none match out time frame... sheesh this could take forever. Wouldn't it be great if there was a way to run all the plugins associated with a hive all at the same time?! Well funny you should ask! You can! I will show you this using the GUI version of RegRipper (more than one way to skin a cat!). Its pretty self-explanatory:

The options in the 'plugin' dropbox can be modified to whatever plugins work for that hive.  You could open up the 'system-all' file and remove or add any compatible RegRipper plugin you wish. I for example added the findexes to mine.  You could create your own plugin-list (e.g. searching only for persistence) if you wanted too.

I am going to open up my output in Notepad++ and search for all instances of Oct 30 and see if we can see anything within that time frame.

 

devclass v.20100901 (System) 
Get USB device info from the DeviceClasses keys in the System hive

Tue Oct 30 09:16:59 2012 (UTC)
    Disk&Ven_SanDisk&Prod_U3_Cruzer_Micro&Rev_8.02,17387318C8507EF4&0
   
Tue Oct 30 09:16:59 2012 (UTC)
    ParentIdPrefix: 8&2fbb6782&0&RM

The parentIDPrefix is a key unique to each USB key inserted into the machine (its how these things get tracked). Check out the article from Hak5.

----------------------------------------
svc v.20080610 (System) 
Lists services/drivers in Services key by LastWrite times, short format

Tue Oct 30 09:17:53 2012Z,Micorsoft Windows Service,Micorsoft Windows Service,\??\C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\pxhmnhtj.sys,Kernel driver,Disabled,
Tue Oct 30 09:16:59 2012Z,USBSTOR,USB Mass Storage Driver,system32\DRIVERS\USBSTOR.SYS,Kernel driver,Manual,

  
Again we see the USB being invoked right before infection, and then we see a new malicious file pxhmnhtj.sys being started as a kernel level (thats bad mkay?) driver (however its disabled at the time the registry was grabbed). From where you ask-- the TEMP folder!

The software hive has potentially some other interesting tidbits:

----------------------------------------
userinit v.20080328
(Software) Gets UserInit value

Microsoft\Windows NT\CurrentVersion\Winlogon
LastWrite Time Tue Oct 30 09:17:47 2012 (UTC)

    Userinit -> C:\WINDOWS\system32\userinit.exe,,C:\Documents and Settings\Administrator\Local Settings\Application Data\agrsahog\ijjfkkxw.exe

Per references, content should be %SystemDrive%\system32\userinit.exe,  

 ----------------------------------------
schedagent v.20100817
(Software) Get SchedulingAgent key contents

Microsoft\SchedulingAgent
LastWrite Time Tue Oct 30 10:04:00 2012 (UTC)

OldName      = CONSULTA-142832
LogPath      = %SystemRoot%\SchedLgU.Txt
MaxLogSizeKB = 32
TasksFolder  = %SystemRoot%\Tasks
LastTaskRun  = 2012-10-30 10:04:01

Note: LastTaskRun time is written in local system time, not GMT

 So remember that userinit registry key we saw before? There it is! 
 
Prefetch
So remember we saw windows.pf in the 1st blog posting. Well let's see if we can glean any more juicy tidbits from prefetch. Enter PFDump, another tool from Mike Spohn.

Again, the usage is quite simple. Give it an input directory, in our case the prefetch folder I grabbed, and if wanted the computer name. How did I grab that? There is a regripper plugin for it called compname :) All thats left to do is throw the file into your favourite analyst tool. I tried to clean it up a tad so its a bit more legible. We really do not see anything new here, our two suspicious files showing last access time close to our timeframe. However, what I thought was interesting was the create time... it looks like the file was created 6 hours after the file was last accessed! Very strange... if anything its a flag that something is not ok. We can also see from PFDump the number of times an executable was run (both 1 on our cases) and the full path the executable ran from.

 At this point I would have uploaded all the files I recovered from the Temp folder  to VirusTotal (or heck ran them in my own sandbox). Heck even a hash search turns up badness I would also run strings on them as well as the two prefetch files and any other files I was able to extract. But what fun is that?! Where is that dead horse?! Memory forensics next!

Autoruns

Editors Note: I just published this and the formatting is FUBAR. I apologise for this... but jet lag is setting in so I will try and fix it up tomorrow after some sleep and c0ff33. Cheers!

Thanks to our previous MFT analysis, we saw the file 'ijjfkkxw.exe' being placed in the StartUp folder. We should probably check our autorunsc output to A) VERIFY this finding and B) potentially find additional autorun locations.

Again, Excel is my friend here and a separate the output into nice pretty columns. Reiterating the information I found last time, these are the potentially malicious files:

• windows.exe (Temp folder)
• ijjfkkxw.exe (StartMenu\Programs\StartUp)
• keqkhgks.log (Application Data)
• ijjfkkxw.exe (Application Data\agrsahog)
• areapmadsdctysii.exe (Temp folder)
  
  There are many ways to go about looking thru this output. Let's start by simply doing keyword searches. The only hit I get is for the file'ijjfkkxw.exe'. Here is a screenshot.

So it looks like Autorunsc has provided a few more locations where this piece of malware keeps itself in order to maintain persistence.

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Userinit: This registry key tells Microsoft which programs to run automatically when a user logs onto the machine. See the Microsoft page. C:\Documents and Settings\Administrator\Start Menu\Programs\Startup: This is simply a location within the OS where programs which should be started at StartUp (not just a clever name) can be placed. This is not a registry key, you can generally navigate to it via Explorer or the command line to see its contents (just check what you can and cannot view!) HKCU\Software\Microsoft\Windows\CurrentVersion\Run: For the CURRENT USER (in this case Administrator), this registry location has the programs which should automatically start at run time. More information is here. Based on the locations above, it seems like no matter who logs into the machine, the malware will be executed. This is important as it means that simply cleaning 'Administrator' may not clean all users who have logged onto this machine after infection. Another method analysts use when looking at this output is to focus on 'Description' or 'Publisher'. Malware writers can be very lazy and write little to no information in the Description section. If we filter for blanks under 'Description' we see a few more entries, but all three of the malicious exe file are still observed. Now for publisher, people can filter on 'Not Verified' or again blank.  In theory (of course we have seen examples contradicting this) if the file is 'Verified' and is a company you trust (for example Microsoft) you can be less concerned about this. Of course when you have malware using signed Microsoft drivers... well then you have big problems. However, lets assume we are NOT dealing with some APT here and further filter our blank description with removing all "Verified" Windows files. For the sake of trying to smoosh (thats the technical term) the relevant columns in on screen, I highlighted the 'known' bad files. Now whats left is verifying the remaining files. We see an autorun entry which is no longer valid as the file is gone (Google searches seem to indicate this is normal?). The rest are shell extensions for WinRAR. Basically what they allow a user to do is right click on a file (which displays what is known as the context menu) and then alter that file using common WinRAR action (compress/decompress/view archives/etc) or when a user drags and drops a file. You can also try to Google search the CLSID to see if it is associated with the program it says it is. Is the user supposed to have this on their machine? Again, this is where having a baseline comes in handy! Wow-- so what do we have left to analyze? Memory  Internet History Registry System Logs Prefetch We definitely have some good data already, but let's see if we can find more indications of badness!

Beginning Analysis

 So I ran all the commands I discussed previously on 30 October. Lets go thru some of them...

CPORTS: Shows Open TCP/UDP Ports
I open up the CSV in excel and do some fiddling so it looks pretty. Something immediately sticks out at me.

That looks like A LOT of svchost.exe (the same one I may add-- PID 760) going out to 192.168.0.5 (lets say its my proxy server). I decide to sort on the 'ProcessCreatedOn' column...

So they started at 0917 GMT (local time). How do I know it was local time? I ran unixdate when I started my analysis remember? This also give you an idea how long this has been running for. In our case, not long. I count 72 instances of some sort of connection attempt to 192.168.0.5 from PID 760, all at 0917GMT . Two of them are established, which means potentially there is still data going out.

P.S. A lot of times students say 'Hey! I see multiple svchosts.... that's bad right?' No, it doesn't not automatically mean badness. Check out this How-To Geek Article about it. Basically it is a container used to run services executed on the system. However, if we scroll ALL the way over in cport output to look at Process Service for our suspicious PID, we don't see any... but heck let's just check.

Services
I am hoping to see SOMETHING here. Luckily since my system administrator is amazing (*blush*) they have a list of  services installed on the standard image from this organization. This bring me to a big helpful piece of advice: KNOW YOUR NETWORK. If you know what is normal, then you will have a faster/easier time picking out things which do not belong.

Anyways, I check my services output with the list from my admin. I see the 'rpcapd' service which is a bit unnerving as it is used to capture packets off the wire for the machine. Something to bring up to the administrator (ie WHY do all the machines have this?! Why do they need it? It could be used for malicious purposes....) Nothing here though from the malware.. maintaining persistence elsewhere!

MFT
OK, so we have a timeframe (potentially). Let's see if we can use the good ol Master File Table and focus on that time frame (October 30, ~0917 GMT). Remember if we are dealing with NTFS then a file should have at least one entry in the MFT (even if it is deleted-- potentially). Time for MFTDump by Mike Spohn now that we have the $MFT.

So the command I entered was as follows:

mftdump.exe --hostname=VICTIM_PC F:\Output\MFT_victim.txt

 This created a file called 'mftdump_VICTIM_PC'. I throw this into Excel (every analyst's first love) and after making it pretty sort by created time. Lo and behold I hit some goodness...

Well THAT doesn't look normal! What are those weird looking executables? log files? Let me copy those to another worksheet and then lets also sort by accessed time as well...maybe something was created earlier but then was run again around that time? So now I have a new spreadsheet with created time around the suspicious time and accessed around the suspicious time. (I have linked here the excel spreadsheet if people want to follow along...)

I also want to point out the two different attributes for time here, the $Standard_Information(si) attribute and the $File_Name(fn) attribute. If you look you notice that they do not always match.  Why is that? Well its because of the way different attributes are updated based on actions. Lets quickly recap the definitions of each action (from whereismydata):

• File Created: This is the date the file was “created” on the volume. This does not change when working normally with a file, e.g. opening, closing, saving, or modifying the file.

• File Accessed: This is the date the file was last accessed. An access can be a move, an open, or any other simple access. It can also be tripped by Anti-virus scanners, or Windows system processes. Therefore caution has to be used when stating a “file was last accessed by user XXX” if there is only the “File Access” date in NTFS to work from.

• File Modified: This date as shown by Windows there has been a change to the file itself. E.g a notepad document is has more date added to it, would trip the date it was modified.

• MFT Entry Modified: A basic understanding of NTFS and the MFT is required for this section. This is date not shown by Windows Explorer or the average windows interface, but requires forensic tools , e.g EnCase, FTK, iLook, WinHex, etc.  This date shows when the MFT entry, which points to the file of concern, was changed. This means that if the record that points to the file  is changed, then this date would trip. As all the dates, file name, file sizes are stored in the MFT, if any of those are changed then the date will change. For example, if the file size changes then the MFT Entry modified date is changed. If the file name is changed, than the MFT entry modified is changed.

Dusting off the File System Forensic Analysis Book by Brian Carrier, Windows does not generally update the values in $FILE_NAME, therefore the $STANDARD_INFORMATION timestamps are more 'up-to-date' (these are the ones you normally see in a Windows GUI). However certain things like 'accessed times' need to be treated carefully, as 'accessed' do not always mean 'executed'.  Oh and don't forget timestomping.... Oh and now they can alter the $FILE_NAME attribute too... Sad panda :(

SIDE NOTE: You can also grab some interesting timestamp by parsing the $I30 (Index Attribute File). See the SANS blog post talking about it. You can download the INDXparser tool for free here. I have even used this before in a case, but completely forgot until now.

So right now my timeline is looking something like this (I have things I think are malicious bold in red, suspicious in yellow):

October 30 @ 0853 

• windows.exe is created in Temp directory for Administrator (I always find things created in Temp directory interesting, and it fits within the scope of the timeline)

October 30 @ 0904

•  Google Crashed (maybe related? not sure)

October 30 @ 0916

• RunDLL32 is accessed (seen by prefetch)
• Looks like a USB is connected (seen via the wav file and USBSTOR.SYS)

October 30 @ 0917

• rundll32 executed again (seen by rundll32.exe and prefetch)
• bin folder accesses from \MinGW directory
• Application Data folder accessed for Administrator
• windows.exe is accessed and run from Temp directory (see the prefetch?)
• ijjfkkxw.exe is created and accessed from  StartMenu\Programs\StartUp
• keqkhgks.log is created and accessed from Application Data
• ijjfkkxw.exe is created and accessed from Application Data\agrsahog
• desktop.ini is accessed
• areapmadsdctysii.exe is created, accessed, and run from Temp directory
• tourstart.exe accessed from system32
• migwiz.exe accessed from system32\usmt\
• xpsp1res.dll accessed from system32
• sti_ci.dll accessed from system32
• ...

October 30 @ 0922

• Folder agrsahog is accessed 

October 30 @ 0923

• IR First Response starts with UNIXDATE and FASTDUMP observed...

See how useful the MFT is?! Amazing! We have a good timeline and additional places to look for badness! Man how happy am I that I saved off the temp folder! Judging from size alone it looks like potentially areapmadsdctysii.exe, ijjfkkxw.exe, and windows.exe are the same files, just in various locations. Did you see that places itself in StartUp? Hooray we found one persistence mechanism!

In theory since we have the offending executable we could throw that bad boy into our lab for dynamic analysis... but whats the fun in that? Let's keep moving!! 
 
Also want to post these two additional links about $FILE_NAME and $STANDARD_INFORMATION attributes.  I hope I explained it properly... I was actually going back thru Brian's book in sheer amazement about the complexity of NTFS. It's a jungle out there!