Scripting - Making Life Easier

I was using INetSim (within REMNux) today to mimic internet services so I could fool my little malware to thinking it was connected to the real world. Anyways, the malware started looking for certain URLs, of course when INetSim dutifully returned with a page (but not the right one), it kept looking for other URLs. After 20 minutes I realized I would have a lot of IPs on my hands... scripting to the rescue!

So for those of you who use INetSim (or those who don't, we don't judge here), the output of the service log looks something like this:


OK, so first off, we need to figure out how we can narrow this file down to the date we want and the lines that only say 'Request URL'. This is where grep is your friend:

cat service.log | grep "\[2012-05-03" | grep "Request URL" > foo.txt

 Cool, that reduces some noise and leaves us with:

The google.com ones above were ones I did to test that InNetSim was up and running, so I can edit those out in vi. The juicy bits start where the censored blocks are (sorry!)

So now what? Well lets see if we can get any IP addresses from the domain names. In linux world we can use the 'host' command. Here is a quick example of host in action (if you want to see the syntax available just enter 'host':

Awesome, so if we want to get a list of IPs up we simply have to feed it all of the domains we have. Now just how many is that per chance? 'wc' with the 'l' switch is our friend!

O_o I don't know about you, but I don't want to manually feed those in! Plus we need to get rid of the extraneous info on each line, all we need is the domain. This is where the 'cut' command comes in:

cut -d/ -f4 foo.txt > bar.txt

So we are setting our delimeter to '/' because we do not need 'http://' for our lookups. the -f switch says what field (or column) we want to display. Looking at the 2nd screenshot and counting along, our domains lie in the 4th column. So now we should have each domain on its own line in the file 'bar.txt'. Ok onto the script!

file=bar.txt

for i in `cat $file`

do

      host $i

done

Yes this can be done from the command line, but with a bit of tweaking this little bad boy can be used for all sorts of things! All that's left is to chmod the script to be executable and run. If you want you can output to a file rather than the screen, or even use tee and have your cake and eat it too!
 
None of this is new, however its simply another way of tackling the problem of too much data :)