Log Agent, log file recollection tool By Floydman, Bachelor in Computer Sciences Floydian_99@yahoo.com Floydman@hacker.am September 25th, 2000 Updated November 13th, 2000 As Presented to Securidad en Computo 2000, November 28th, 2000, in Mexico City http://www.seguridad2000.unam.mx You can distribute this document freely, as long as no changes are made to the file, or as long as credit for it is not pretended by someone else. All comments and suggestions about the material presented here should be directed at floydian_99@yahoo.com. If future versions of this document include add-ons coming from other people than me, then proper credit to the various authors will be clearly identified. All version updates of this document are to be released by me. You can find this paper and more online at http://www.geocities.com/floydian_99/ Abstract The goal of this paper is to present a tool made in Perl for recollecting log files from various applications and various machines into a central location. Preface When comes the time to choose computer security tools, one most wanted feature is the ability to centralize the information contained in the log files. Also, this prevents the evidence from being tampered by a potential intruder. So because of this, somewhat good products are overlooked because they fail to provide this single feature, and sometimes this leads to purchasing a product that offers (and sells) many features not necessarily needed, or products that are not as flexible as desired when comes the time to make it work on your environment. In order to resolve this, I programmed LogAgent 1.0, which is an agent that you can run on all your Windows NT machines to monitor the log files of various unrelated applications and to redirect any new input made to these files to a central location. Targeted audience This document is presented to anyone who has interests in computer security, NT Administration, computer monitoring, intrusion detection, Perl programming and computing in general. Table of contents 1. About LogAgent 2. The source code 3. The experiment 4. Conclusion 1. About LogAgent First of all, I'd like to thank Amine Moulay Ramdane, who programmed the AdvNotify PPM package. AdvNotify provides the functions needed to monitor directories and report when changes are made to these directories. I practically used the example file she provides with it and made some modifications in order to capture data and redirect it. Here's how LogAgent works: @mondir is the list of directories containing the log files that you want to monitor. If one folder contains more than one log file, it is not necessary to specify each one of them, specifying the folder will catch them all. $mainlog is the central destination where you want to redirect all your log files, in UNC format. We also determine the machine host name, IP address and the username logged on the machine, and we store these values in variables. Then we start one thread for each element contained in the list @mondir. After the thread is started, we have to enable each one of them with EnableWatch(). Then, the program goes into an infinite loop in which it waits for changes to happen in the folders it monitors. When data is added to a log file, it is also captured by LogAgent, the host information is appended to the data, and then it sends it back to the central log file defined by $mainlog. If we get out of the loop (with [CRTL-C]), then we terminate all the threads we previously created. In the likely event that you don't want to install Perl on all your networked PCs, you can use perl2exe.exe to convert your Perl scripts to an executable .EXE file. With the full version, you can even add -gui option in order to prevent the console from showing, thus making LogAgent run in the background without being seen by the user. ActivePerl can be downloaded from www.activestate.com Perl2exe (demo) can be downloaded from www.indigostar.com 2. The source code #!C:\perl # Log Agent version 1.0 (c)2000 # By Floydman # floydian_99@yahoo.com # You can freely use and distribute this Perl program, and change the code to fit your purposes as well. # If you feel that somehow you can improve this program, please let me know so I can update the original. # I guess you can consider this is Open Source software. # Using Win32::AdvNotify # By Amine Moulay Ramdane # Website: http://www.generation.net/~aminer/Perl/ use Win32::AdvNotify qw(FILE_NAME SIZE INFINITE Yes No All %ActionName %ActionColor); use Socket; use Sys::Hostname; # Define username, IP address and hostname of the local machine my $host = hostname(); my $addr = inet_ntoa(scalar(gethostbyname($name)) || 'localhost'); my $login = getlogin || getpwuid($<) || "not logged"; # @mondir contains the list of directories to be monitored @mondir = ("C:/Program Files/Security/Grisoft/Avg6/Log/", "C:/Program Files/Utils/GetRight/Log/", "C:/Winnt/Internet Logs/", "C:/Program Files/Winetd/logs/"); # $mainlog contains the location of the master directory $mainlog = '//darkside/log$/'; my $obj = new Win32::AdvNotify()|| die "Can't create object\n"; # Note: you launch a thread for each directory you want to monitor, # meaning one thread per object in @mondir print "Log Agent 1.0 by Floydman\n"; my $Thr1 = $obj->StartThread(Directory => $mondir[0], Filter => All , WatchSubtree => No ) || die "Can't start thread\n"; my $Thr2 = $obj->StartThread(Directory => $mondir[1], Filter => All , WatchSubtree => No ) || die "Can't start thread\n"; my $Thr3 = $obj->StartThread(Directory => $mondir[2], Filter => All , WatchSubtree => No ) || die "Can't start thread\n"; my $Thr4 = $obj->StartThread(Directory => $mondir[3], Filter => All , WatchSubtree => No ) || die "Can't start thread\n"; print "\nThreads started successfuly ...\n"; #DO NOT FORGET TO ENABLE EACH ONE OF YOUR THREADS $Thr1->EnableWatch() || die "Problem starting EnableWatch()\n"; $Thr2->EnableWatch() || die "Problem starting EnableWatch()\n"; $Thr3->EnableWatch() || die "Problem starting EnableWatch()\n"; $Thr4->EnableWatch() || die "Problem starting EnableWatch()\n"; print "\nTo exit from the loop press [CTRL-C] ...\n\n"; while($Thr1->Wait(INFINITE))# exit with [Ctrl-C] signal { while($Thr1->Read(\@data))# exit when the list is empty {for($i=0;$i<=$#data;$i++) {$obj->TextColor($ActionColor{$data[$i]->{Action}}); open (LOGFILE, $data[$i]->{Directory}.$data[$i]->{FileName}) or die "Can't open log file"; flock (LOGFILE, 1) or die "Can't lock file"; @Lines = ; open (MAINLOG, ">>".$mainlog.$data[$i]->{FileName}) or die "Can't open master log file"; flock (MAINLOG, 2) or die "Can't lock file for writing"; print MAINLOG $host," ", $addr," ", $login," ", $Lines[-1] or die "Can't read from file"; close (MAINLOG) or die "Can't close master log file"; close (LOGFILE) or die "Can't close file"; }} } # never forget to terminate your threads. $Thr1->Terminate(); # terminate thread1 $Thr2->Terminate(); # terminate thread2 $Thr3->Terminate(); # terminate thread3 $Thr4->Terminate(); # terminate thread4 # never forget to call the destructor undef $obj; 3. The experiment What we just saw is the source code as I used it to perform an experiment to verify if LogAgent could deliver its promises (note: this test was performed before I added the variables containing the host information). So I decided to monitor the log files from 4 different, unrelated applications, and forward any input made to these files to a central log file directory located on UNC share name \\DARKSIDE\Log$. The applications monitored were Grisoft AVG AntiVirus, ZoneLabs ZoneAlarm, Headlight Software GetRight, and COTSE Winetd. I don't see how monitoring GetRight log files could be useful in a security context, but I needed a good test bed, and GetRight spans more log than the other applications, so this is why we see it here. The Log$ share was empty prior to the beginning of the test. I copied a shortcut pointing to LogAgent.pl in the startup folder, and rebooted in order to get a fresh user session. Then I proceeded to generate activity that would be logged by the mentioned applications. After I finished generating log activity from these applications, I proceeded to collect the results. Here's what was in the central Log$ directory: D:\Log>dir Volume in drive D is D Volume Serial Number is 0840-C01D Directory of D:\Log 09/25/00 08:31p . 09/25/00 08:31p .. 09/25/00 08:31p 373 bind.log 09/25/00 07:44p 2,483 getright.log 09/25/00 07:38p 8,768 IAMDB.RDB 09/25/00 08:31p 41 restart.log 09/25/00 08:31p 36 shutdown.log 09/25/00 08:31p 104 startup.log 09/25/00 07:38p 366 ZALog.txt 9 File(s) 12,171 bytes 902,415,872 bytes free bind.log, restart.log, shutdown.log and startup.log all belongs to Winetd, getright.log belongs of course to GetRight, and ZALog.txt belongs to ZoneAlarm. IAMDB.RDB also belongs to ZoneAlarm, but I don't know exactly what is its purpose. Since it is not relevant for our analysis (because it contains no useful data), I simply disregard it. Grisoft antivirus didn't detect any virus during the testing period, so it did not generate any log. Here's the content of the other files: bind.log bind() listening to port 3 - 18:53:18 - 09/25/2000 bind() listening to port 13 - 18:53:18 - 09/25/2000 bind() listening to port 23 - 18:53:18 - 09/25/2000 bind() listening to port 25 - 18:53:18 - 09/25/2000 bind() listening to port 137 - 18:53:18 - 09/25/2000 bind() listening to port 139 - 18:53:18 - 09/25/2000 bind() listening to port 139 - 18:53:18 - 09/25/2000 getright.log 2000/09/25-19:04:36: File: D:\downloads\Hack\telnet server\fd-update.zip.GetRight = http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:04:39: (Re)Started download: http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:04:40: Could not resume (restarting from 0): http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:04:58: (Re)Started download: http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:04:58: Could not resume (restarting from 0): http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:05:16: File: D:\downloads\Hack\telnet server\fd-update.zip.GetRight = http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:05:16: (Re)Started download: http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:05:17: Could not resume (restarting from 0): http://rapidus.tucows.com/files/fd-update.zip 2000/09/25-19:09:20: File: D:\downloads\Hack\proxy\CProxy.zip.GetRight = http://rapidus.tucows.com/files2/CProxy.zip 2000/09/25-19:09:21: (Re)Started download: http://rapidus.tucows.com/files2/CProxy.zip 2000/09/25-19:09:22: Resumed: http://rapidus.tucows.com/files2/CProxy.zip at: 0 2000/09/25-19:10:28: File: D:\downloads\Hack\proxy\as-setup.exe.GetRight = http://rapidus.tucows.com/files2/as-setup.exe 2000/09/25-19:10:28: (Re)Started download: http://rapidus.tucows.com/files2/as-setup.exe 2000/09/25-19:10:33: Resumed: http://rapidus.tucows.com/files2/as-setup.exe at: 0 2000/09/25-19:13:23: File: D:\downloads\Hack\telnet server\fd-update.zip.GetRight = http://cny.tucows.com/files/fd-update.zip 2000/09/25-19:13:27: (Re)Started download: http://cny.tucows.com/files/fd-update.zip 2000/09/25-19:13:27: Could not resume (restarting from 0): http://cny.tucows.com/files/fd-update.zip 2000/09/25-19:13:33: (Re)Started download: http://rapidus.tucows.com/files2/CProxy.zip 2000/09/25-19:13:35: Resumed: http://rapidus.tucows.com/files2/CProxy.zip at: 519767 2000/09/25-19:13:35: (Re)Started download: http://rapidus.tucows.com/files2/as-setup.exe 2000/09/25-19:13:40: Resumed: http://rapidus.tucows.com/files2/as-setup.exe at: 49638 2000/09/25-19:30:36: Finished downloading: http://rapidus.tucows.com/files2/CProxy.zip 2000/09/25-19:31:50: (Re)Started download: http://rapidus.tucows.com/files2/as-setup.exe 2000/09/25-19:31:51: Resumed: http://rapidus.tucows.com/files2/as-setup.exe at: 49638 2000/09/25-19:44:25: Finished downloading: http://rapidus.tucows.com/files2/as-setup.exe restart.log server restart- - 10:34:15 - 09/25/2000 shutdown.log shutdown - - 10:29:40 - 09/25/2000 startup.log Starting Server: darkside - 18:53:17 - 09/25/2000 Starting Server: darkside - 18:53:17 - 09/25/2000 ZALog.txt PE,2000/09/25,18:53:00 -5:00 GMT,ZoneAlarm Internet Security Utility,203.110.251.2:53,N/A FWIN,2000/09/25,18:57:00 -5:00 GMT,65.229.239.88:1901,65.227.240.78:139,TCP FWIN,2000/09/25,19:19:22 -5:00 GMT,65.228.69.89:2936,65.227.240.78:139,TCP PE,2000/09/25,19:37:49 -5:00 GMT,Xnews,203.110.251.2:53,N/A PE,2000/09/25,19:38:06 -5:00 GMT,Xnews,203.110.251.2:53,N/A For the sake of testing, here is what a log file looks like once I put in place the host information variables (using a new ZALog.txt): ZALog.txt test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:08 -5:00 GMT,10.1.4.68:1676,10.1.4.87:4,TCP test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:08 -5:00 GMT,10.1.4.68:1676,10.1.4.87:4,TCP test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:08 -5:00 GMT,10.1.4.68:1676,10.1.4.87:4,TCP test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:08 -5:00 GMT,10.1.4.68:1676,10.1.4.87:4,TCP test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:08 -5:00 GMT,10.1.4.68:1676,10.1.4.87:4,TCP test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:08 -5:00 GMT,10.1.4.68:1676,10.1.4.87:4,TCP test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:08 -5:00 GMT,10.1.4.68:1676,10.1.4.87:4,TCP test_host 10.1.4.87 Administrator FWIN,2000/11/14,15:50:54 -5:00 GMT,10.1.4.68:1683,10.1.4.87:65535,TCP We can see that the IP address (10.1.4.87) is written twice, but with an application that fails to give such information, it does get really helpful in identifying the origin of a log entry. 4. Conclusion LogAgent provides a very inexpensive way to track down all loggable activity on your Windows NT network. In fact, all you may have to pay is a license for Perl2exe full version, and this will provide you with the ability to make LogAgent (or any other of your Perl scripts) run as a background task without any visual indicator of their presence (except maybe for the Task Manager). With this simple but useful tool, you can now look at any security application that you may want to deploy, even if they don't provide the ability to log centrally. LogAgent will take care of this task for you. Unfortunately, AdvNotify will only work on NT, but I am working to see if I can make a LogAgent that could work also on Windows 9x machines as well. Your input on this project is welcome.