Virus protection in a Microsoft Windows network, or How to stand a chance by Floydman, Bachelor in Computer Sciences Floydian_99@yahoo.com May 30th, 2000 You can distribute this document freely, as long as no changes are made to the file, or as long as credit for it is not pretended by someone else. All comments and suggestions about the material presented here should be directed at floydian_99@yahoo.com. If future versions of this document includes add-ons coming from other people than me, then proper credit to the various authors will be clearly identified. All version updates of this document are to be released by me. You can find it online at http://www.geocities.com/floydian_99/ Preface Computer viruses have always been a weird part of the computer security game. It is the aspect of computer security that gets the most press coverage, while it is probably the less dangerous to deal with (compared to trojans or intrusion). To many security experts, viruses are not such a big threat because you don't get infected if you practice safe computing practices. While this may have been the truth for a while, but it isn't the case anymore. For the past five years, the Internet have grown up quite a bit, now having millions of people with poor computer litteracy online, from their houses or from businesses. While UNIX used to be a big part of the Internet (and still is), the fact remains that there are a lot of Microsoft networks connected to it at this time. A virus launched from the Internet can cripple down a business if appropriate measures are not taken. I think small and medium enterprises here, but also big corporations. The last breeds of Macro.Viruses are just a hint of what may soon happen. Most of these viruses only slowed down servers to a halt, but what will happen when they start to really get nasty? Abstract The goal of this paper is to present some strategies that can (and should) be implemented in corporate or non-corporate networked sites using Microsoft products as operating systems in order to maximize overall virus protection of said sites. I state here sites using Microsoft products only because it is the most widely virus-attacked platform, but the strategies described here could be applied in other platforms subject to virus infections. Also note that the strategies that I am about to describe have been applied on the most part with McAfee antivirus software, ranging versions 2.X to 4.X, which was the product used at my workplace at the time I was there. This is mostly a recollection of the experiences and results that I had made at the time and, and I present here the results I had achieved from such a setting. This document should be in *no way* a starting ground as to if McAfee or Norton (or any other virus scanning software) is the better virus scanner, nor is it to be ground to Microsoft bashing. Commercial products mentionned here are so only because these are the products that were used at the time, and in no way should be considered as my preferences over other products. Targeted audience This document is presented to anyone who has interests in computer security, network administration, virus prevention and computing in general. Table of contents 1. In the beginning 2. The obvious 3. The batch file strategy 4. Then came autoupdate 5. Batch, batch and more batch 6. McAfee Customer Support 7. My Web 8. Strategies to adopt 9. Real-life crisis case study 10. The brown stuff 11. The sad thruth 12. In conclusion Appendice A: Something extra 1. In the beginning After I finished University, I had found myself a job at a large corporation as a job for desktop support and server administration for a whole department, about 300 people, 6 remote sites in two major cities, about 8 servers. We were two guys in one city, and two others in the other one, and helped each other when massive work had to be done. We had just finished doing a cut-over from OS/2 to Windows 95 desktops and laptops along with NT 3.51 servers (that was in 96). Converting all the stations one-by-one, by hand, it didn't take long to learn quicker ways to install software, in order to save time. The task was huge, but when it was done, we were proud of ourselves, we had done a great job, and no major problems happened during the conversion. But the truth was bitter: as soon did we finish, as soon we had to do it again - at least partially. If we wanted our site to be up to date in virus protection, we had to go on each station - again and again every month - to update the virus scanning software. After three months of this treatment, I could suffer this no more - nothing is more boring than installing the same software again and again, repeatedly, especially on week-ends. I went to my boss and said "let me take care of this." This is the final result. 2. The obvious In this section, I will cover the *really obvious* things if you're serious about virus scanning. First, if you have to administer a site and want to stand a chance at protecting your site, read. Read about new viruses outbreaks as soon as they get out, learn the different methods they propagate, know the difference between a virus, a worm and a trojan, and learn why some malicious code can belong to more than one of these categories, ... Mailing lists are a good idea. Second, give yourself a chance, DON'T USE Outlook or Outlook Express if you can avoid it. No personal anti-Microsoft feeling here, but get real! For the past year or so, most new viruses worth mentioning (Melissa, I Love You and variants...) use flaws present in this software. They all use the same method of picking addresses in Outlook adress books, auto-launches other programs, and stuff like that. By using software other than Outlook, you already have a better protection against these kind of viruses, even if you do not use virus scanners at all! Of course, this would not be a good idea to only rely on this. But it works. Remember Worm.Explore.Zip? It became famous for being quite fast-spreading and destructive. It replicated by sending itself by using Outlook, Outlook Express and Exchange, installed itself in the windows directory, and destroyed files dear to both management and software development staff. I had to handle with Worm.Explore.Zip. I was saved *only* because we were using a combination of Lotus Notes and Netscape, and no Outlook at all. It infected only one machine, proving to be a dead-end (well, almost, more on that later). I wonder why companies using Microfoft mail clients bother to run a firewall at all; they have a big hole on ports 25 and 110 (SMTP and POP3 protocols in case you're wondering). Third, to be effective, virus scanners have to be updated. If you don't take care of it, nobody will. (And especially not your users.) Fourth, whenever you try new versions of software, or new ways do remotely do things, test it before deploying it. I know this sounds obvious (which is why I discuss about it here), but I have seen too many software deployment projects go ka-boom because of poor testing. It is easier to correct a mistake or a flaw on the original before deployment, than to correct it manually on each machine after (you can easily put yourself in deep shit remotely, but most of the time you have to dig out of it locally) :-) Fifth, if you install software or updates remotely, let your users know it. You don't want to be running updates during lunchtime and have zealous (but little computer-litterate) users noticing unknown activity on their machines and trying to prevent what appears for them to be a virus infection. Let them know about serious threats (only the really serious ones, because if you put notices up too often, people stops reading them, which defeats the purpose) Also, make clear once a while what are the various precautions taken at the site to prevent virus infections, what they should do if they think they are infected with a virus (-->you want them to call you!), and what are your policies about how virus-related information and virus alerts should be handled at your site (-->all hoaxes should be sent to you, and *only* to you for your evaluation/authentication of the so-called alert). There again, once in a while, but not too often. Too much information is the same as no information. Sixth, knit your network tightly. At least the part you have control over, the one that is relevant to virus infections. That means know what's installed on your users PC. Make restrictions about desktop modifications but don't be too harsh, because people will go against it anyway. Let people change their bitmaps and the colors if they want, but make it clear that no custom screen saver or non-approved software should be installed on their machines. You don't need to implement restrictive schemes with policy editor to achieve this, simply make a clear policy that absolutely NO support will be provided by the techs if such a software or screen saver is found on a machine, no matter if the problem is caused by such software/screen-saver or not. That should scare enough people that they won't try anything too fancy, and the power users would have found a way to do it anyway, so when they get stuck, you let them deal with the brown stuff. (I hate these super users, they are the worst. They actually know enough about technology to be not impressed by it, but they have the astounding ability to make just one "enhancement" too many to completely screw up a system.) Make sure that everyone who logins gets automatically a login script, and make sure that you follow some standards in your login scripts to easily manage them (if you want a batch to run, make a single batch file and call it from the login scripts, don't put the batch commands in the scripts themselves). Seventh, know your virus scanning software. Know what options are available, how they work, and compare it from time to time with competitors software to see if one implements better detection schemes than other. You have to take the edge if you want to stay ahead. Also analyze where your software installs itself, what files or registry entry controls what settings, ... Know your software inside out. And of course, scan everything. You can never be too careful (in college, there was only two time that I got a virus encounter on college computers, and these were the only two times that I thought I didn't have time to do scan the machine first, and I lost files on both occasions; Murphy's law in action), and with the processing power at hand these days, there's no reason to limit only to certain files. Next thing you know, a new type of virus will propagate in .XYZ files by some stealth process changing the file's extension to .exe prior to execution. And when I say scan everything, I also mean scan your files servers (daily, system AND data), scan all mail at the server level if you can, and get all your stations to get up to date and scanned entirely at regular intervals (a weekly basis seems reasonable). Use live scanning software also, these will prevents machines from being infected at all before detection. I could also add... Know your users, and their machines. I know this isn't something that can be easy to do in some corporate environments, just because of the way computer support is dealt with. Some guys on the servers, changing passwords and restoring files from the backup tapes. Unnexperienced tech support people who answer calls from a remote site, some guys knowing only about the e-amil system, some others knowing only about in-house apps, and on-site people always on the go fixing this and that. But if you can actually land in a place where you get to do support and server admin, don't disregard it. I know that to some (especially devellopers), desktop support isn't very glamorous in the computing jet-set, but you get to be in the first seats of your network. You never know what you might learn while your answering to a support call, or just by chatting with a usually-happy-from-your-services user. You get to see the new junk such as internet newsticker with push technology and the likes when there's still only a few people having it. That gives you the chance to get the thingy, try it, evaluate impact on network traffic, Windows stability (yes, Windows 95 CAN be stable, but you have to be careful with it) and security, and turn around with a policy going against the use of the thingy (unless it proves to be quite a useful piece of software) before it becomes widespread in your PC base. It also serves you by being on the floor for some periods of time (not necessary that much if your systems are knit tight), and you never know when you might fall on some unknown guy in a suit using someone else's PC. Oh! And one more thing, that can never be said to much: use your head! And learn about batch files, it's a life saver... 3. The batch file strategy We wanted to make sure that every PC connected to our network would be up to date with the McAfee software. We looked at some network-wide software distribution products, stuff with buzzwords that left me wondering if I should laugh or if I should quit computing (I chose the first option), and that costs at lot of pesos. We didn't have that kind of budget. (Bob, do we have a budget?) Well, to me it was obvious that everyone connecting to our network had a login script running to map shares (we made sure of that). It would be quite easy to call a batch file from there that could take a part of the chore with it. At that stade, McAfee was at their firsts versions for Windows 95. To update it signature files, it was only a matter to copy the unzipped .dat files in the McAfee directory (somewhere in Program Files). That was fairly easy to implement: net use x: \\servername\update echo updating antivirus software, please wait... xcopy x:*.* c:\Antivirus_Software_Path_\ /d /y >nul net use x: /delete This batch file is simple. It connects you to a previously created share on a server that holds the up to date .dat files. Then it proceeds to copy these files to the correct destination, assuming that all your stations are configured the same (they should if you really want to stand a chance). The /y switch is to overwrite files without prompting to eliminate interaction with users. The /d switch is what makes this little thing so useful, it will copy the files only if the source is newer than the destination. Since the version on the server will always be newer or equal to the version found on the stations, this makes sure that everyone gets updated while eliminating unnecessary downloads, thus reducing network traffic. Of course, common sense insists that the \\servername\share is set as read-only (and also your login scripts and your batch files). After a bit more research, I figured that all the configuration settings were held in ascii files (avconsol.ini, *.vs?), and assuring an homogeneous setup across my network was only a matter of forcing them on the stations the same way I did with the .dat files. Again, the /d switch was used to reduce unecessary downloads to reduce traffic, albeit that these files were actually small, but one takes pride on optimizing such things. The problem here though, is that if someone changes the configuration on his PC (the configurations weren't password protected, and even if they were, it could be bypassed by editing the file with an ascii editor or overwriting the file), then the destination file becomes newer than the target file, preventing copy. To circumvent this, I simply advanced the date on my machine one year forward before creating my standard configuration files. I could restore the real date afterwards. So now I am sure that the files on the server will always be newer than these on destops and laptops for the year to come. By then, every time I make a change to these files, I use the same technique, which makes the deadline date progress along with real time. OK, this isn't 100% foolproof, but bring me a user who can get around this, and you will have a man who should reconsider his carreer choices. At that time, McAfee anti-virus software didn't provide any auto-updating or auto-upgrading features, but some common sense helped us get away with some of it. We still had to install versions upgrade manually, because installation sometimes proved to be adventurous, but that wasn't so bad, these occured only each 3-4 months. Once we had a stable version deployed, we were sure that the machines would be in optimal state for a few months. Take note that we didn't automatically install new versions as they came out, because some of them proved to be very flaky. Remember my advice about testing before deploying (this advice should be applied to any software). 4. Then came Autoupdate When version 3.x came out, a long awaited feature was added to the product, Autoupdate and Autoupgrade. Our prayers has been answered. Almost... First, Autoupdate delivered the marchandise. You could connect with this utility trhough ftp or thrgough UNC (Universal Naming Covention, used widely in Windows NT to set shared resources), update the .dat files or just keep them as is (I never figured that option), to back up previous .dat files (just in case), and to schedule the time the updates occur. You could connect to McAfee ftp site or set up your own ftp or NT server for your distribution. As far as my previous setup goes, this does nothing more that my batch file was doing. The batch file was actually better, because the .dat files would be matched for a new version at every login, while autoupdate was set to once a week. I switched to Auto update anyway, only because I tought it was cool to use all the options the software offered me by literally playing in the guts of the software. What became interesting was not how to distribute the files anymore, it was to play with the various settings of the configuration ascii files that contained the server name, the schedule, ... And I knew that in the event of an emergency where I would want to rush files to prevent an infectious attack, I could implement quickly the old way, simple as it is. Which brings me to one point: the batch file is the quicker to get your files to spread on the network, but for it to work, you need your users to login. In the event of an emergency, set up your downloading batch file, along with the latests .dat files in the shared directory. Test it at least once just to make sure you didn't screw up. A typo happens sso quiclky. Then if you followed my advice about knowing your people, the rest is only a matter of human relations. Try to get good relationships with administrative assistants, or secretaries, these are the ones who really run the place, not their bosses. By reaching four, five or six of them will get a message in everybody's voice-mail in no time telling them to save their work and reboot at once. What? You think you can maintain your calling/mailing lists yourselves? Fine, go ahead, but then don't complain when you realize that you missed one machine used by a temporary employee that you didn't know about (Oh! She uses Carol password, she's only going to be with us for two weeks!). Trust me on this, these people are effective, use them wisely and they'll be your best friends. I doesn't hurt neither to go around in the offices to spread the word: Please reboot now, thank you. One sweep should be enough, the excitement of something potentially dangerous is enough to get people talking about it all afternoon, so you know that the ones you didn't catch will eventually hear about it sooner or later. Don't forget to send a quick e-mail reiterating the phone message they should have received (mailing lists are generally easier to maintain than phone lists). On the other hand, Autoupgrade proved to be a fluke. The GUI (graphical user interface) was quite similar to the Autoupdate GUI. It would connect thgough ftp or UNC to a distribution server, download installation files necessary to upgrade the software to the new version, and then run Setup.exe from the local machine and proceeding with a silent install (the command ran was actually setup /s). Nothing extraordinary again that I couldn't have done without a batch file, which is why I was disapointed. I expected the Autoupgrade to keep the current configuration after the upgrade, keeping the software fully autonomous. Far from it. Running Autoupgrade forced the default settings to be implemented, which caused McAfee to forget to which server get his next upgrade, or even when he was expected to do an update/upgrade. That was really dumb design from the part of McAfee, to which I had complained. They answered me with the typical answer: this will be resolved in the next release (and they actually did, in version 4.x). Add to this that I still thought it was too risky at the time to remotely install software, let's say I retired more academic knowledge from this more than actual improvement, but that would help me later on. By the way, my batch file stills force the configuration files to keep a homogenous environment. 5. Batch, batch and more batch... I was reading the alt.2600 newsgroup the other day and during a thread, someone mentionned how much he missed playing in batch files to automates his tasks, recollect info from PCs, ... Batch files are to Microsoft administrators what shell scripts are to UNIX hackers. Too bad this is now a lost art in the MS community. If you like batch files, it's only up to you to use them. You'll be surprised what you can do with them with some imagination. And to think they wanted me to purchase a multi-K $ system to take care of this chore. The antivirus product proved to be more and more stable, even if a small glitch was causing the current (live) virus scanner to jam while scanning the executables of the new version. I could not figure out a pattern to this problem, as it proved to happen randomly. This appeared first on a few IBM laptops, and I tought it was an IBM related problem, but I had several identical machines updated succesfully. Then the problem appeared on some Dell PCs. This was (or appeared to be) random, and even if this didn't happen that often, it was still annoying. It was happening too often. I decided to take a shot at it anyway. This was getting complicated, because the installation process required three reboots to be completed. Also, at around 3.5 or 3.6, McAfee moved some configuration information away from the ascii files to put it in the registry. Playing in the registry from a batch file! What was I on? Now, with the experience I have, this isn't probably as bad as it sounds. I'm pretty sure that there's a command out there that let's you add registry keys at the command prompt. But at the time, I was a bit puzzled. This needed condition handling, installation step recognition over several reboot cycles, and registry-fiddling. That didn't prove to be so hard. I created dummy flag files, namely called flag1.dum, flag2.dum and so on. I copied them on the machine at different steps in the process, and I could rely on the presence or absence of these files to determine where the batch file had to pick up after a reboot. As for the registry settings I needed to transplant, all I had to do was to export the corresponding registry hive containing the relevant information from a properly configured machine. That would give me a file called update.reg. Then I would copy that file to the Startup folder, which on reboot will merge the hive in the registry, putting it up to date. Only one drawback, minor in fact, this will present a dialog box to the user saying that the registry had succesfully been updated, with a OK button. Not much, but I would have liked to do it with zero-interaction. The usage of flag#.dum files made it so that I had to manage them. Since I was making this up on the run, I didn't take time to see if I could make it simpler. Here's what it looked like, along with comments. Note the OS version checking at the beginning, useful if your site is mixed with Win95/98/NT. Different Windows versions sometimes means different path or files, so you may need a batch file for each. Rem McAfee remote installer - No more one-by-one installs By Floydman :Vertest @echo off ver > c:\ver.txt Rem This works by matching the OS name string to the output of ver.exe Rem If there's a match, it is piped to ver(n).txt, if there's no match, the file is empty Rem Then ver(n).txt is copied to ver(n+1).txt Rem Copying an empty file produces no file copied at all Rem So if ver(n+1).txt exists, we know the OS, Rem If not check next version until all are checked :Windows NT type c:\ver.txt |find "Windows NT" > c:\ver4.txt copy c:\ver4.txt c:\ver5.txt if exist c:\ver5.txt goto Endfinal :Windows 95 type c:\ver.txt |find "Windows 95" > c:\ver2.txt copy c:\ver2.txt c:\ver3.txt if exist c:\ver5.txt goto Win95 :Unknown OS echo ERROR: OS cannot be identified. goto Endfinal :Win95 Rem some cleanup erase c:\ver*.txt >nul echo This will install the latest McAfee antivirus software echo with the proper configuration echo :Install Rem If this step is done, proceed to the next step if exist c:\flag3.dum goto cleanup Rem Display installation notice contained in a .txt file c:\windows\notepad.exe \\servername\misc\install.txt xcopy \\servername\misc\flag3.dum c:\ >nul Rem Copy installation directory on C: drive echo Copying installation files (this will take about 30 minutes on DialUp connections) Rem Create temp directory for installation c: cd\ md mcinst cd mcinst xcopy \\servername\upgrade\*.* >nul Rem Starts installation routine echo Silent installation begins... setup -s goto End :Cleanup Rem If this step is done, proceed to the next step if exist c:\flag4.dum goto Final Rem Back up the registry attrib -h -s c:\windows\system.dat >nul attrib -h -s c:\windows\user.dat >nul copy c:\windows\system.dat c:\windows\system._da >nul copy c:\windows\user.dat c:\windows\user._da >nul attrib +h +s c:\windows\system.dat >nul attrib +h +s c:\windows\user.dat >nul Rem Copy registry update to be executed at startup echo modifying registry setings... xcopy \\servername\config\update.reg c:\windows\startm~1\programs\startup\ >nul Rem Remove installation directory that is no longer required c:\windows\deltree /Y c:\mcinst >nul Rem Remove old flag files from previous installations if exist c:\flag1.dum erase c:\flag1.dum >nul if exist c:\flag2.dum erase c:\flag2.dum >nul xcopy \\servername\misc\flag4.dum c:\ >nul goto End :Final Rem Check if last step is done if exist flag5.dum goto End Rem Remove registry update from startup folder to avoid numerous registry updates erase c:\windows\startm~1\programs\startup\update.reg >nul xcopy \\servername\misc\flag5.dum >nul goto End :End Rem Updates the configuration files if needed. Rem Don't forget the files on server NEED to be a year or do forward xcopy \\servername\update\*.vs? c:\progra~1\McAfee\ /q /y /d xcopy \\servername\update\avconsol.ini c:\progra~1\McAfee\ /q /y /d :Endfinal echo People new to batch files, or to programming in general (as is often the case with NT system admins) will notice three principal characteristics about this program that makes it quite a good piece of batch file. First, it is well documented. Second, it tells the user what is happening as it happens, but actually hiding the real output to them. Third, it cleans itself up (unlike several commercial programs on the market). The rest is just logic applied. On top of it, I'd say I made it deadly accurate. There wasn't really a need to specify all path names when calling commands. Make sure you have your batch file to correspond your path structure on the server (upgrade, update, misc and config directories) along with the needed files. Install.txt is actually an ascii file containing a message explaining that a software install is currently going on, along with my coordinates if they have any questions or problems. Usually, system admins would like to hear as less as possible from their users, and would like having their phone number/pager number in circulation as less as possible. To some people, broadcasting their coordinates to their users would be like "advertising" their services, making people think "about that little quirk on my machine that is no really big deal, but since you reminded me..." I would say the opposite. The fact alone that my full name, title and phone number (instead of just "The support team") is listed is enough to inspire faith in the most fearsome user. I would point out to something else: remember when we used to have a dull presentation at school (or even at work) and the teacher would ask "Anyone have a question? Anyone?", and nobody raised their hand? Well that still work today, even if phone call or e-mail is more discreet to admit ignorance than raising a hand in front of people. So sign your message with "So if you have any questions, Anyone? Contact me at ###-####". You'd be surprised how effective this is to prevent annoying phone calls. 6. McAfee Customer support So my setup worked quite right, but I still had a problem with the random freeze problem. I decided to contact McAfee Customer Support once more to try to get a fix for this. In the end, McAfee didn't fix my problem. They stated they never heard of that problem before, although I communicated with one of their customers who had the same problem. We came down to figuring out that McAfee would randomly freeze on the execution of any instance of setup.exe (MS-Office, Netscape or... McAfee). But I still want to give high marks for the service McAfee provided with my problem. Customer support if often a issue when it comes to evaluating a company, many being shortstaffed to answer phone lines, resulting on long waiting time, other relying only on their web sites to provide any support. I had to deal with customer support departments before with other companies, and these problems would show up at times, and with varying results. But this time, even if McAfee didn't know why I had this problem, they said they tried to reproduce the same problem. That's ok, even I couldn't reproduce the dang thing at will! But here's the icing on the cake, after a few e-mails and phone calls on this, they sent me their ISeamLess scripting tool. Now that was fun. ISeamLess was a scripting language, similar in appearance to C or Pascal (but much simpler). It contains fewer commands, but useful ones indeed. Mostly file handling functions (copy, delete, ...), and easy access to system variables (%tempdir%, ...). An ISeamLess script is actually made out of several breakpoints, corresponding to the various steps of the installation process (PreInitial(), Prelicense(), PostLicense()...). So you could decide what action (display a message, copy a file, change a registry setting) at the precise moment you want it to happen in the installation process. Once you've been fiddling a while with McAfee's configuration file (which isn't that complicated, much like a .ini file), ISeamLess scripting was easy as 1, 2, 3. Once you made your script, you had to "compile" it, which produced a .iss file if my memory is correct. All you had to do was to include the .iss file in the installation directory, and run setup -s. The rest was automatic. Here is a sample, from the ISeamless documentation: PreInitial() { // VariableValue( // Uncomment if you want the debug back // "NCCONFIGDEBUG", // "TRUE" ); } PreLicense() { /* The next instruction extracts the licensing agreement from a zip file located in the same directory as the install. This way, the custom agreement is displayed in the setup instead of the standard one */ CopyFile( "%ISSRCDIR%License.zip", "LICENSE.TXT", "", "", "%ISSUPPORTDIR%", 0 ); } PostLicense() { // You don't have to do this but it was just to show how to erase a file DeleteFile( "LICENSE.TXT", "%ISSUPPORTDIR%", 0 ); That simplified a lot my batch file process, making the setup, the configuration updates and registry updates in one sweep, taking care of updating the .dat files on next login (.dat files included in upgrade releases were often one month behind than actual .dat file), which is not so bad. I tried to use this to issue a command that would disable the live scanner first thing in the process, but the bug made setup run awfully slow (about an hour for doing what should take five minutes at most, hence we tought they were frozen) even before it could be disabled. I was resigned, I had to disable it from the batch file prior to the execution of setup.exe. Some might say "So what, you got it working right?" Right. But I would have loved a fix for this because it caused me other problems, as I had to install a lot of software. I know that anti-virus documentations strongly suggest disabling their software before installing programs on a system. But I had installed many softwares (I insist on MANY) in the past, with antivirus loaded and fully active with no problem. Think of it in terms of hundreds and hundreds of time that I have to sit on a PC just to launch a installation program. I want to be relieved of the pain of having to disable the anti-virus program all the time, or be sorry about it if I forgot. And this is in my humble opinion how a good anti-virus software should work. If I recall correctly, the problem was gone with 4.x. When it became available, McAfee sent me an updated version of ISeamLess scripting, more powerful. Now this is something that could be useful if this could work with all setup.exe out there. In the hands of site administrators, this is the solution to remote distribution software, no need for config-o-matic, wininstall and the likes who costs a small fortune to acquire and often produces flaky results. But it could also be a very dangerous weapon if in the wrong hands. As you can see from the example above, one can easily delete files with ISeamLess. Imagine a script-kiddie who picks up any piece of shareware on the net, unzips it, adds a maliciously crafted .iss file, and re-zips the package in a self-extracting .exe, then distributes through on the internet(ftp upload, mail, newsgroup, web page, ...). Ouch. But still, it would be nice to have such a tool available for all software. But to effeciently use it, one condition must apply: know your software from the indide out. Security could be improved by adding built-in file-trace, allowing a IseamLess script to delete only files it has created itself. I have not tried new versions of this tool in a while, and I don't know if McAfee still provides it, but I really had fun with this gimmick. 7. My Web I was lucky enough to take care of both desktops and servers. Here's why: I had about 150 local users and 150 remote (taken care by people over there, but with personnel changes, I became at one point the sole in charge on antivirus management for them too) and about 4 servers local and 4 remote (same story). Taking care of the file servers wasn't as complicated as it was for the desktop. I figure it is still easier and shorter to make them manually since there's only 8 of them, besides since your servers are your balls, you want to keep an eye on it when you install software on it. But I did find several little things that would get me a real web where any known virus couldn't trip into without me knowing it. The server version of McAfee supported a feature called McAlert. Basically, this worked by setting up a share on a server where users have write-access. McAfee puts a file there that serves the purpose of alert queuing. The server "listens" to the file. You set up your McAfee clients simply by specifying the share path in UNC format in the McAlert configuration window. Whenever VShield (live scanner) or VirusScan (file scanner) trips on a virus or detects virus-like activity, it would pipe information to the file on the server. The message could then be sent to you through various ways (SMTP mail, alpha-numerical pager, network broadcast, ...). Take the time to look at what kind of message you can send to yourself. The default message is often futile and nearly useless. Look if you can customize it with variables, such as username, machine name, file infected, virus name, ... The more precise the message you get, and the more you know your users and your systems, the more easily you will be able to determine the severity of the threat. 90% of the time, it's no big deal, so you kind of get used to it. I coupled the effect of this by pointing all log files from the various stations to one single log file (actually 2, 1 for VShield and 1 for VirusScan). These log files can be in the same directory than the message-queue file, they do not interact. This is not fiction, this is quite easy if you apply yourself to it. I have tried to explain some of this to some of my co-workers, supposedly equal in our job descriptions and pre-requisites, only to be confronted to a gazed look. More about that later. So this is the kind of setup I had. All PCs up to date in terms of antivirus software, updated automatically, a quick mean to react to emergencies, instant knowledge of virus activity by e-mail and pager, and shortcuts on my desktop on my two log files. The log files actually provided me with more information than the e-mail/pager. The Alert would tell me the machine name, which would tell me what building is located the infected machine, along with the time and the virus name. The log files gave the time, the status (infected, cleaned, clean-error, ...)the user name and the full path of the infected file. Here are samples of the two files: VShield Master Log File Sample 2/19/98 3:50 PM Infected adrichar C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/19/98 3:51 PM Clean Error adrichar C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/19/98 3:51 PM Infected adrichar C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/19/98 4:23 PM Infected rwyllie C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/19/98 4:23 PM Clean Error rwyllie C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/19/98 4:23 PM Infected rwyllie C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/19/98 4:24 PM Clean Error rwyllie C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/19/98 4:24 PM Infected rwyllie C:\PROGRAM FILES\MCAFEE\VIRUSSCAN\EICAR.COM EICAR-STANDARD-AV-TEST-FILE 2/23/98 7:42 AM Infected fserrano C:\WINDOWS\TEMP\CONTRAT (1).TXT CAP 2/23/98 7:43 AM Cleaned fserrano C:\WINDOWS\TEMP\CONTRAT (1).TXT CAP 2/23/98 7:43 AM Infected fserrano C:\CONTRAT-FRANCE CAP 2/23/98 7:43 AM Cleaned fserrano C:\CONTRAT-FRANCE CAP 2/23/98 9:27 AM Infected fserrano C:\WINDOWS\TEMP\CONTRAT (1).TXT CAP 2/23/98 9:37 AM Cleaned fserrano C:\WINDOWS\TEMP\CONTRAT (1).TXT CAP 2/23/98 9:37 AM Infected fserrano C:\WINDOWS\TEMP\CONTRAT.TXT CAP 2/23/98 9:37 AM Cleaned fserrano C:\WINDOWS\TEMP\CONTRAT.TXT CAP 2/23/98 9:51 AM Infected wmsippel C:\MY DOCUMENTS\CONFID-1.DOC CAP 2/25/98 11:47 AM Infected smaloney C:\WINDOWS\TEMP\MOVINGPACK.DOC CAP 26/02/98 8:59 AM Infected sirianni A:\ NYB 26/02/98 8:59 AM Cleaned sirianni A:\ NYB 26/02/98 9:03 AM Infected sirianni A:\ NYB 26/02/98 9:07 AM Infected sirianni A:\ MONKEY_B VirusScan Master Log File Sample 2/23/98 1:31 PM Scan Started danfair On Demand Scan 2/23/98 1:31 PM Scan Complete danfair On Demand Scan 2/23/98 4:56 PM Scan Started rwyllie On Demand Scan 2/23/98 5:02 PM Scan Complete rwyllie On Demand Scan 2/24/98 9:35 AM Scan Started gbergogn On Demand Scan 2/24/98 9:35 AM Scan Complete gbergogn On Demand Scan 2/25/98 7:48 AM Scan Started ldollimo On Demand Scan 2/25/98 7:48 AM Scan Complete ldollimo On Demand Scan 2/25/98 9:55 AM Scan Started amorcom On Demand Scan 2/25/98 10:06 AM Scan Complete amorcom On Demand Scan 25/02/98 1:08 PM Scan Started KDEGRACE On Demand Scan 25/02/98 1:08 PM Infected KDEGRACE C:\PROGRAM FILES\CAERE\OMNIPAGEPRO80\OpFor70.Dot PROBABLE MACRO (No Remover Available) 25/02/98 1:08 PM Scan Complete KDEGRACE On Demand Scan 2/25/98 5:38 PM Scan Started lague On Demand Scan 2/25/98 5:38 PM Scan Complete lague On Demand Scan 2/26/98 7:22 AM Scan Started rwyllie On Demand Scan 2/26/98 7:28 AM Scan Complete rwyllie On Demand Scan 26/02/98 10:37 AM Scan Started KDEGRACE On Demand Scan 26/02/98 10:38 AM Infected KDEGRACE C:\PROGRAM FILES\CAERE\OMNIPAGEPRO80\OpFor70.Dot PROBABLE MACRO (No Remover Available) 26/02/98 10:38 AM Scan Complete KDEGRACE On Demand Scan 26/02/98 10:40 AM Scan Started KDEGRACE On Demand Scan 26/02/98 10:49 AM Infected KDEGRACE C:\Program Files\Caere\OmniPagePro80\OpFor70.Dot PROBABLE MACRO (No Remover Available) 26/02/98 10:51 AM Scan Complete KDEGRACE On Demand Scan This is as concise as you could get. When you know your people and what place they work at, and you see a particular virus spread in a particular location, it doesn't take time to launch a phone call with the concerned people right away. In the event that the virus is detected but not cleaned, you may be able to stop the infection right there, dealing with only one machine instead of 30 (or 300, or 3000). The ability to be able to pinpoint to the file location also helps a great deal. Even if the user is far away and never seen you before, the fact that you talk to him like you're right there over their shoulder protecting them removes any computer fear even in the most techno-impressionnable personae. Some sort of good Big Brother. You ask them about the origin of said file, and suggest proper action. If the file came by e-mail, then immediately mail back to the originator saying that attached file is infected. Reply to all if multiple recipients. Make sure to specify to upgrade to a more recent version of software. And when you do this, remind your users that they should send such messages on your request only. Users can actually do all that, explain to them over the phone, calmly, and let them reply to their own mail. You don't want to put your fingers in others people mail. That's like putting your fingers in the brown stuff. Ok, if the infected file came on a disk, or was already on hard disk before, of course you want to scan all hard drives on the infected PC. Scan all related diskettes (don't go on a scanning frenzy just for one diskette, but if the user says "It came with these", you want to scan "these"). This is a bit trickier. Even tough more and more people uses computer everyday as part of their jobs, that doesn't necessarily make them one bit more computer-literate. So giving instructions over the phone on how to start a scanning job can be done, but sometimes can be cumbersome. If the user happens to be on your floor, or your building, give yourself a chance and go do it yourself. And you better take the stairs too, I'm sure you won't mind the extra exercise. So we can see from these log files that I have been playing a few times with the EICAR test virus (this is provided to simulate virus presence to test antivirus configurations). We see that some people are stuck with some macro virus (Concept, Cap, and the Laroux are some of the most persistent viruses I have seen), and someone brought some old diskettes back to the office, infected with "oldies" like NYB and Monkey_B. VirusScan also detected possible macro infection, but couldn't match it with any such known virus. Turns out that the macro was legit. Also, some expected entries do not show, namely the Cleaned that should complete the pair with Detected for wmsippel, smaloney and sirianni. Some entries obviously gets lost in the void, probably colliding with it predecessor and dying on the shock. Networking can be so cruel. With experience, you get to know which viruses actually get cleaned even if you don't have full proof. Common sense here indicate that the second infected diskette infected with NYB should be cleaned if the first one was, right? So either a message is simply missing, either the second diskette was scanned on a different machine that was not fully up to date. From my e-mail/pager message, I knew that both diskettes were scanned on the same machine. So the message must have been lost. Just to make sure, I actually checked (I had plenty of time, since it was a quiet day, as usual), and all the diskettes were actually cleaned. The counters in VShield proved me right. These samples don't show any example of it, but I sometimes would get alert about a temp file in c:\windows\temp. That was telling me that the infected file is an attachment received in Lotus Notes. Instead, if the file was in c:\progra~1\netscape\..., then you know the attachment is in Netscape. This may sound trivial, but again it is really useful when dealing with people. Some people were using both mail clients on their machine, so when you call them and say:"Hi [username real name here], I am calling you about a file attachment you just received about two or three minutes ago in your Lotus Notes account, it is/was infected with a virus. Could you please contact...", to a lot of people this seems as if I had told them down to the exact penny how much money they have in their pockets. 8. Strategies to adopt In this section, I will treat about configurations that should be adopted. First, don't be afraid to play a bit with the software, learn its many options and on the different ways that they could help you do your job. Also, don't be cheap on security: today's oldest machines used in most Windows networks are Pentium 133 MHZ at worst. This is still more than enough power to handle the strictest security settings, so there's no reason not doing it. If you have something like a P3 processor running at 750Mhz and you're afraid to slow down your machine too much, you should be ashamed. In the setup I have described earlier, here are the options I implemented through the various configuration files: All scanning tasks present in the AntiVirus Console are set like this: Scan files on: Run, Create, Copy, Rename Scan floppies on: Access, Shutdown //scanning floppy on shutdown helps preventing to accidently boot with a boot-sector infected diskette Scan: All files, Compressed files Action: Clean automatically //viruses are so common these days, there's no need to collect them all, if you still want to take a look at it, then configure one machine that can accept it, and ask the originator if he could send it back to you . Make sure not to activate it and infect your own network. Network alert: \\servername\alert\ Log: \\servername\logs\ Log options: Virus detection, Virus cleaning, Infected file deletion, Infected file move, Date and Time, username Autoupdate was configured like this: Copy files from UNC server, at \\servername\update //Antivirus companies websites are flooded when new outbreaks happen. It is more effective to download one copy that you make available internally than have all your computer site trying to connect to an already overloaded site Log update activity: Local Backup existing DAT files //Why not? Autoupgrade was configured like this (with ISeamLess script enhancements) Copy files from UNC server, at \\servername\upgrade //Same as above Log upgrading activity: local Have a live process scanner load at startup (like VShield for example), and have it also configured with strict settings. It is a good idea to schedule one scan job per week, and the same frequency should be made for file downloads. Plan these activities on separate days (ie: upgrade if necessary on Mondays, update if necessary on Tuesdays, and Local drives scan on Wednesdays. If the business hours of your site are in the type 9-5, try to make it happen when you expect the most downtime, during lunch time (around 12:30 is the quietest). One could put the update task on the same day as the upgrade, 15 minutes later, and save a day in the process. I would advise against it: you never know what the result of an install will be, even more so if this is multi-remote installs. But then again, software evolves and current versions are more stable while upgrading themselves, and require fewer (if none at all) reboot processes than older versions. Still, this is the kind of things you want to find out before implementing rather than after realizing too late the potential problems you have with a particular bug. 9. Real-life crisis case study As job descriptions and personnel changes a lot due to a ongoing company merger, I was less and less involved in desktop support, and temporarily taking care of the servers I was already supporting until I migrate them to the new centralized servers they're trying to set up. As I had more free time (and could hardly charge all that "Documentation" time on company-time), I got assigned to projects, which brought me to a remote-city meeting for the whole day. That was in July, and I managed to escape the meeting early (I was replacing someone else, they didn't need my input anymore, I had to get the train back home, you know?). Wonderful afternoon writing lyrics in the train ride home. Since the train station is so near to the office, and since it was still early, about four, I decided to make a quick stop just to check my e-mail and leap-off. As one of my colleagues used to say "Do like Gordie Howe, and get the puck outta here!" Well I got there not a minute too soon. I have a desktop (only) support guy from the remote office who left me a voicemail message about disapearing files on one of their servers. He also mentions that this could be related to a new virus that appeared earlier in the morning, Worm.Explore.Zip. I didn't touch a computer all day, so I didn't get the news, but that snaped me out of my clouded lyrics from the train ride. This is a nasty. It replicates itself by replying to unread messages found in Exchange, Outlook Express and Outlook. That's 1-0 for me on the nasty. I don't know yet how many people are infected, but I know they won't cross-contaminate each-other, that's a relief. What else does it says... Copies itself in predictable places, that's 2-0 for me, because with this information, I figure I will be able to quickly craft a searching batch file to scan for it and remove the infected file (deletion was the cure). I would redirect the output of a "if exist" file check conditional delete to a log file, along with username and machine name, and have it run from the login script. What else does it do? Delete .doc, .xls, .ppt, .c, .cpp, .prg and some others like that, sneaking everywhere to find them, and attacking on brute-mode. Actually, it doesn't *erase* the files, rather it puts them to 0 size. NASTY! He definetely scored a goal on that one, 2-1 for me. Well, it's past 4:30 now, at least most systems will be down by an hour, this buys me time. OK, time to call the support guy back. John was anxiously waiting for my call, apparently overwhelmed by the situation. I try to calm him down and try to see if he could give me any help since he`s on-site, and at least he's not a user, right? All he knows is that he received that trouble ticket call by a guy who complained about a bunch of files in the departemental drive *ouch! the biggest repository of documents in our network environment*. And that's all he could tell me! OK kid (and I was only 24 at the time!), stand by and watch a pro in action. First, I open Windows Explorer and connect to the departmental shared drive. This drive is a mess. It was set up as a mean to keep the file servers to be used only by local users to reduce traffic, but still have a mean to exchange documents between offices. This was at the dawn of corporate internet mail, when only few privileged employees had it at the time, and the main corporate messaging system was through a mainframe system that we connected to with a TN3270 client. Not really called for file exchange. So this drive is the place where everyone in city A places a document a person in city B needs to read, and vice-versa. It is free for all. When there's no room, clean up. Don't put your single copy of a document here (but you can't rely on users for that one). So I initiate a Find files request with search criteria *.*, 0 bytes in size. WHOOAAAA!!! Over 200 files showing, and a few refreshes of the view shows me that it keeps going up. 2-2. Nasty! As I keep doing this, I am still on the phone with John, who is now in the server room. I tell him exactly what I'm doing, step-by-step, live to the minute, and I suggest him to do the same from the console. OK, so anybody who knows a little bit in networked operating systems and virus infections or computer security knows that for a malicious program to deliver effectively its payload, it relies on the access rights of the infected username. Crunching a file to 0 bytes in size leaves a trace, NT can tell me who owns files and who has made last modifications to it. I check a few files and this gives me one username that seems to be deleting the files. There's still more and more getting deleted as we speak over the phone (I told you there were a *lot* of files in that drive). As we make the investigation, we discuss about if this could have been due to the server being infected. That was rejected for obvious reasons:1) the file servers don't run any mail package, so could not receive the virus and execute it, 2)the fact that files on the server were being deleted was no indicator that the deletion process was executed on the server, 3)the username and the date of last change gave us an excellent view on the virus origin and progression. It was amazing to sort them by time to realize how steadily the files were going away. It is sad that I did not keep a snapshot of that to include it here. You could clearly see it plunder files in a stubborn and quite effective alphabetic fashion, putting in our imagination the image of a small furry electronic beast, with only one obsessive tought totally occupying his small but determined mind: Do { Crunch (*file) file++ } While (Files_exists()==1) "OK", I told John, "go to that guy's office and tell him to shut down is computer NOW! Disconnect it from the network and put it aside for further inspection." In the meanwhile, I know enough, now all I want is to stop the murder. I close all connections to the server and remove all share access to the attacked folder. Quick Find files request on other important shares (\\servername\users\*; \\servername\groupdata\*) for 0 bytes files. That doesn't bring up much information, gladly, because that means no damage there *yet*, so I keep these services live, but I keep an close eye on it. I do see two 0 bytes files in the same guys user directory, though, I'm pretty sure I'm on the right track. So tell me, Worm.Explore.Zip, can you feel my breath on your neck? I quickly craft a batch file that I launch from my faithful login scripts. In the batch, in put in a quick check for my previous idea, and I add some DIR to search local drives for 0 bytes sized files. I redirect all output to a writable file on a server that I keep around for testing (so if something bad happens to the machine, it's no big deal). In corporate environments, these machines are sometimes hard to get. Obtain them from when servers are migrated and upgraded, when older machines are released. When such machines get free, TRY to get your hand on it. Make up false requisition, invent fake project, or if it happens that one of your current server gets upgraded, it is even easier to keep the old machine also by invoking some "legacy-application" that can only be run from this machine, whatever. You're not stealing them, you're simply trying to do your job. You just have to play their game with their own rules if you want to stand a chance. OK, batch file ready, 4 phone calls to 4 wonderful ladies that have the ability to reach everyone with the contact of one single touch on a key on the phone pad. Words spreads fast: Logout now! Log back in if you really intend to... It's nearly 5 PM now, and I imagine that I won't get to see too many outputs to test the effienciency of the batch file. Well, it appears that some people are zealous tonight, because the results starts coming in already. So far, so good, all signs of further file destruction are gone, and I am pretty sure by now that I've singled out the culptrit. 3-2 for me after a nice breakaway goal! A quick look for last minute .dat files... not yet. Mental note to check first thing in the morning. And last thing before quiting tonight. I would have wished they all went home right now, but I think I have enough alarms to warn me quick enough if another breakout happens. Let's go back to the departemental drive to see if I can't squeeze a bit more info from it. This is where the virus has done the most damage, this is where I can learn most about it. What??? There's even more file getting deleted? But I thought I cut access from it??? IT'S BACK??? HOW THE HELL DID THAT SHARE RE-SHARED ITSELF ALL ALONE LIKE THAT??? #$%#!^%!^&*^&($*^&@$^!!! Trusting my instinct, I call John to see where he's getting at with our infected friend. He tells me that the guy is not at his office, and knowing he has a laptop, he could be connected anywhere. No way to reach him (he didn't respond to my URGENT mail). Things are not looking good: Nasty's got a powerplay with a few minutes left remaining on the clock. I then ask John if he would know anything about the departmental share being back on. "Well, yeah!" he says, cheerfully. "I had received a trouble ticket call from a user saying he couldn't access the shared drive, so I investigated the problem and I found that the share service was removed. So I put it back in place." -YOU DID WHAT!???!! YOU PUT BACK THE FILE SERVICE THAT YOU KNOW IS BEING SEVERELY ATTACKED RIGHT NOW AND IT DIDN'T COME TO YOUR INVESTIGATIVE WANDERINGS THAT IT WAS I WHO MIGHT HAVE DISCONNECTED THE DRIVE? YOU MORON! $$^*(%^($^(&**&!@^!!!! -But you should have told me you did that! (Remember about the obvious: Use your head? Here's the perfect example). (Dimwit) -OK, I am going to cut access again, John, please don't put it back until I say so. Better yet, let me put it back when I decide it so. If people ask you about it, explain that we're under severe virus attack and that the drive has been disconnected until further notice for protective reasons. Some people, you have to tell them everything. -OK, he says. Oh, by the way, did you know the backups aren't running on these machines? !!!!!!!!!!! -WHAT!!! TELL ME I HEARD WRONG!!! PLEASE TELL ME I HEARD WRONG!!! WHAT ARE YOU GUYS DOING UP THERE YOU BUNCH OF $#$^&% MORONS??? STOP PLAYING WITH YOUR $%^@$& A$$E$ AND TRY TO BE USEFUL FOR A CHANGE!!!!!&%*&%*$^!@^%!^&@!!!! -Hey, don't blame me, I didn't even know there were servers in that closet before I received that ticket this afternoon. Looks like some information went down the drain during the exchange of responsabilities between staff members. After I calm down, I get a better picture of the real situation. The normal file server have backup running, but nobody changed the tape since who-knows-when, so all I got on this one is a 24 hours old backup. Better make sure this doesn't get overwriten. The departemental server, on the other hand, have its backup software crucified on the screen console: General Protection Fault, since who-knows-when for that one too. Kick in the groin goal! 3-3, Nasty evens the score. It will be impossible to recover any erased files, hundreds of them. This story is true to the bone. The worst is this shouldn't have happened. Backup tapes is the first daily chore of server administration, and it's last resort solution if the absolute worst happens. Well, bad things happened, and this is one more proof that the chain can be only as strong as its weakest link. I just thank the Lord this didn't happened at 8 in the morning, when I was on the train on my way to meeting. Epilogue: Overtime I spent the next few days closely monitoring my output files to see if the batch files would catch anything new. I get the newest .dat files as soon as they're available. I ask people to reboot this once to activate as many logins as soon as possible, in order to have something a bit more reliable that batch-files for protection. Having been a witness of another IT fiasco in the same department, I was expecting some directors to go out for blood on this one. Surprisingly enough, I had ZERO complaints about the lost files. I will never be sure of why, but I will put to the credit that they followed our advice and not put the single copy of an important document on that drive. So the games end in a tie, it was a fierce matchup. It could have been worse. I could have been beaten bad by Worm.Explore.Zip. Many others faced many days of reconstrucing and ensuring sites. Let's just say that I got a bit lucky, but like every good goalie, I made my luck. 10. The brown stuff If you haven't figured what this section will be about, I will refer you back to the passages containing the asterisks, ampersands and dollar signs. Yes, this is dedicated to the brown, sticky, smelly stuff. You can avoid it, if you don't make the same mistakes as I do, If you can predict the unpredictable. I had different encounters with the brown stuff, some more serious than others, and it will strike you when you expect it least (aaahhh the nice train ride). Let's get real, accidents do happen, but that is not valid ground to excuse everything. The backup incident should never have happened. I should have had at least a week old backup (even that would have been great). Other brown encounters happened like this: - When version 4.x came out, I resisted the temptation to go and install it on all my PC base. After all, my current 3.x offered me all the protection and stability I wanted, and McAfee would continue to supply up to date .dat files for 3.x for the coming months. That would give me all the time I needed to play with 4.x to see what new features/headaches this would procure me. So I get the installation files, install the new version on my PC, and to my great pleasure, it kept my previous configuration. That meant that it still remembered to look at \\servername\updates for the current patches. So I dutifuly put the bone where my dog expects it, then I click Autoupdate, and there it was, what I always wanted. Time to go to lunch. Brown stuff happens most of the time just because of stupid things. When I came back from lunch, I realized it was the day that I proceeded with the updates. All of a sudden, no antivirus works at all, all versions 3.x being updated with incompatible 4.x .dat files. I would have thought that the software would have provided some kind of failsafe to prevent this, but apparently not. By the time I realized what I have done and figure out a way to get out of this mess quick before it gets worst, I came up with (you guessed it) a batch file that would overwrite no matter what the current .dat files with version 3.x .dat files. A few magic phone calls later, I have everyone rebooting and having my fix applied. Except that now support calls are now sent to phone-tech-center who tries his best (instead of directly to me, like in the past), but fails to grasp the nature of the problem (I should have thought to mention them right away about my bug and my current work on it, this would probably would have blocked the calls). So a few users had the time to reach phone-tech-support and complain about a non-working anti-virus software. So the techs dutifuly proceeded over the phone to install version 4.x on client's PC. That means that when my batch file hits these machines, it will put incompatible version 3.x .dat files on a 4.x installation. Same problem, reverse. At time like this, it is very valuable to be on the floor. You make a batch file to stop a gap, you enable it and then you get to see it in action right in front of your eyes only by looking at the cubicules. When you see one that stands apart from the rest of the machines, you know something's wrong. I actually resolved the reverse problem by manually des-intalling (only 3 machines) version 4.x, and re-installing version 3.x, for homogeneity's sake. But I had learned a valuable lesson. - Another time I had brown stuff, it happened again during lunch time (I guess it doesn't matter how many people is logged at that time, but who; I wish I had a lab to make experiments on these things instead of the real network). So I didn't realize *again* that it was upgrade day today, even more so because I wasn't using it at the time. But I was still making tests on my machine with my test server (which happened to be also the real server for antivirus distribution, since this all started out as a side-project and evoluted to what it had became. So I accidentally copie test-purpose only configuration file with auto-upgrade enabled (I was investigating the random freeze bug) in the distribution directory. I realized my mistake a mere 10 minutes after, as I started seeing unnexpected machines connecting to the server. ooopps! Seems like some people actually logs on at 12:20-12:30 timeframe. Fix the file back, then take a look who's got the leak, call them back to make sure everything's running smooth. The first three calls confirms current installation of the software, but no problems arose. All right, I said to myself, this might turn out for good after all, free guinea pigs from the wild. Aaahh, the sad mistake that someone makes when he pats himself in the back before the completion of his task. Caller number four, which happened to be the last leaked guinea pig, turned out to be a real pig. His computer froze all right. I could still make it by killing the AVSHIELD.EXE task, which would release the CPU to perform the installation procedure. But to do so required patience that my interlocutor didn't seem to have:"OK, put your mouse over the small shield icon, and wait for the label to appear. This may take as long as 30 seconds. Then, right-click on it, then in another 30 seconds or so, a menu will pop-up, then click on Disable, then wait..." You get the picture. He couldn't endure that situation because he had files opened that he didn't save before going to lunch (that was HIS mistake, I have enough with mines already). So of course he's all hurried over the screen, going in a clicking frenzy when the last thing you want if you want to recover from this is to avoid a buffer overflow. He's brought the machine to a point where we couldn't even call the task list with Alt-Ctrl-Del. There's no choice left, power cycle. The guy was pissed off at me because he lost some of his morning work. He considered this as an intolerable situation and threathened to take this further. After getting a reality check from one of my contacts on site, it turns out to be that this guy was of the brown-nosed type. The type who accepts gladfully a "promotion" to entry-level management, removing him from union, but with no pay raise or new responsabilities, actually doing the exact same work as before, but with unpaid overtime. The guy probably wished he could get a promotion by nailing me. I guess he failed, because I never heard of that incident afterwards. (By the way, even if I may sound harsh, I did not deal with this guy in the manner of the BOFH. But I have to admit it was tempting :-)BOFH:Bastard Operator From Hell, http://members.iinet.net.au/~bofh/ ) - Some places don't take security seriously, even if they think they do, and that is serious brown stuff. I have seen computer security personnel investigating on computer equipment theft who couldn't tell the difference between RAM, a CPU or a hard-disk. I have seen personnel from same department that were specialized in viruses. They ran up a monthly report on the number of virus incidents. Any single file found with a virus counted as an incident. Known incidents were those reported by company employees to the computer security department (this was mandatory). It averaged around 50-150 incidents per months for the whole company (about 30 000 people). I actually never counted how many virus detections I had in my logs on a monthly basis, but I know it was easily in the 50-100 range on a regular basis (for about 250 users). I didn't report these cases to computer security dept. for two reasons: 1) I knew the files were cleaned even before I knew about it, and I was pretty quick at knowing about it; 2) it was obvious that my input alone would have doubled their figures, and they already seemed so busy at investigating deeply each infections that they would probably be overwhelmed by the numbers, added to the fact that there was not much more that they could do that I hadn't done before. The fact that my numbers compete with theirs even if I only have a small subset of the user base doesn't mean my network was attacked more often than others. This was due to poor knowledge about technology amongst the computer security people, and their lack of leadership in regards to virus protection. Virus protection software was installed on every machine in the company, be it running DOS, Win 3.1, Win98, Win NT4, or OS/2 or other appliable. But it was the responsability of each system administrator/desktop support departments to take care of keeping them updated, usually manually on each machine. This is inneficient and may leave some "forgotten machine in the corner" to quickly become outdated. At one point, it became company policy to handle the responsability of updating antivirus software to the users themselves. Either they did it themselves, of if they weren't confident enough in doing it they could call to open a service ticket to get the machine updated by a tech support. Of course, tech support can't take on himself to update all machines in the office, because the ticket specifies only one. And he has to work on the other tickets too. So the few users who actually think about updating the software have to specify how many machines (usually only subsets of departments). Besides, default settings are applied, so even if the newest .dat files are available on the company intranet web site, the software don't actually go and pick it up itself, even if it is capable of that. In this kind of environment (entirely true), you get quickly outdated (useless) antivirus software, poorly configured (in many versions, default configurations specify to continue scanning on virus detection, without actually cleaning or isolating the file), and absolutely no way to have a clear picture of what's going on (poor logs on users machines, viruses that stay undiscovered for months, personnel who don't bother to report to them (I wasn't alone to do this), except that I was probably the one with the clearer picture). Had they taken the time to take care of this aspect by specifying configuration standards on the desktop configurations (implemented in part of a large company-wide PC upgrade), they would have both a clear idea of what's going on AND quicker investigations, which would leave them with time to apply to more useful tasks. I actually reported to the computer security department the Worm.Explore.Zip incident, to fill them in on what to expect if they have to deal with it in other departments. I also gave them a glimpse of the kind of setup I had, in order to explain how I got to finding who was infected. That was a bit of risky, because they could figure out that I didn't report previous virus infections (tons of them actually, plenty for them to "investigate"). You can get slapped on your hands in these companies for actually doing too much a good job. They actually seemed impressed by it, and thanked me for the info about the virus, but no one actually got back to me to see how I set up my Web. I had showed my Web also to my new boss, and to her boss, and the boss that took it's place, along with some peers network administrators. They were all impressed and liked it, but nobody asked me how I did it or if it could be implemented on a larger scale (or any scale!). I guess that with a system like this, you can't charge as many man-months of work to the customer as with manual installs. Someone from security actually phoned me a while later, because thay had my phone number from my Worm.Explore.Zip reporting. Since I actually knew the guy who had the .dat files on an intranet web server (the guy started this out of his own initiative), they wondered if I actually knew the guy who was taking care of *their* official way of distribution via mainframe TN3270 emulators. Apparently, the service wasn't running anymore for a couple of months and they wanted it back, since it was the only mean that could rech everybody in the company (some legacy machines were running on DOS/Banyan Vines, so they could download via TN3270 the updated DOS .dat files). Unfortunately for them, I didn't have a clue, since this was concerning arrangements that they made with an old mainframe employee who probably left the company before I was even hired there. Tough luck. This is serious brown stuff because the people who hired these people think these guys are gurus. They rely on them to protect their bat and their balls, namely their data. I remember a time when I was on coop term in a federal agency, we had to deal with the then famed Stone virus. We scanned all our dikettes that could have been potentially infected, finding the sole copy of it on a manufacturer's boot diskette. Then we reported it to the RCMP who sent a "virus expert" on site. As soon as he gets in, I try to explain him what we had done so far, why we know this is the Stone virus, and what the damages were (relatively small, the best part of a day watching the hardware tech trying to install a bigger hard-disk that wouldn't boot, until we saw the message "Your PC is Stoned!"). But no, mister Big Agent had to make a show. So he takes a hex editor and proceeds to read the boot sector of the suspicious diskette. "Ah-Ah!, here it is!" he says joyfully while showing us the word STONE clearly stated in the middle of hexadecimal numbers. "I have found the virus!", he says, acting like he didn't take into consideration any word of what I had told him so far. Had he asked me before he checked the diskette, I could have told him that this virus is a boot-sector virus, that this is where it leaves his signature (read that in a magazine), and that this is how the virus scanner detected and identified the virus. I didn't see the need to actually go see by ourselves if it was really Stone. If these guys perform that kind of voodoo to impress their bosses, then I am wondering about the relative safety of North-American online businesses and public networks. These guys are only good cops that knows a bit on computers. (Make it clear that I am not talking here about real computer security experts finds and plugs holes in software as a hobby or people that goes up to even put cryptography in their spoken language ;-)) They actually caught the guy stealing computer parts, but only because the robber was astoundishingly stupid. The robber stole 6 times in my cubicle (I had to work in a cubicule with all the equipment I had to deal with; at least the servers where in a closed ventilated room). They put a camera in the floor after the third time, and he saw them do it (he was doing night system monitoring, and was close from my cube). The fourth time, the camera wasn't running, the fifth time the lights were shut. Then he stole the PC next to his cubicule. The sixth time, he was on tape. When they first questionned him about that PC, he pretended he sat at his desk all night and that he heard nothing suspicious! In the words of Forrest Gump, "Stupid is as stupid does". He actually opened the case, took out all the parts and the motherboard out of the case, put it in his bag and to the car, back to the office, then *get this*, fold the metal case so it would fit in his bag then he went for another "cigarette break" to his car, and came back to the office to finish the night. He said that he did all this so we wouldn't notice that there would be an empty machine. Instead I noticed an unplugged monitor, keyboard mouse and network cable,all assembled as if there was actually a desktop under that monitor! I told you he was not Brainiac, and I insist that all this is 100% true. 11. The sad thruth It does get sadder indeed. As company mergers went along, I actually winded up with less and less control on my computer base. I was forbidden to do hardware support, even with my long experience in the field. They preffered "certified" rent-a-techs support from machine suppliers. I had less and less to do on support also, as they tried to implement nearly-experimental-like products to remotely install software company-wide, with poor configurations (not just antivirus-software) and bugs all over the place, all in the sake to achieve some irrealistic objectives in an equally irrealistic deadline. The support I couldn't deliver was done by newbies, going with default or company-wide approved standards, thus sabotaging the efficiency and the reliability of my Web. It came down to a point where I just decided to shut it down. I pulled the plug, which wasn't as hard as setting this all up: I just turned off the alarm and removed the shared directories used for logging and distributing. I had previously ran a batch file that forced a more "standard" configuration on the PCs, and that was it. But it got worse. I had changed job and went to a multimedia development shop, where they actually seemed to care enough about virus protection. Their Symantec software was pulling updates regularly (weekly), but from Symantec's web site. Did you try to connect to these sites on the day I love You.txt.vbs came out? If the web was like a telephone, you would have received busy signals. But I had learn the hard way that multimedia shops aren't the most security-conscious places around, and I found myself picking up a bunch of unstable poorly configured systems, using old (and probably holed) software. The systems crached often and I saw some very weird things that left me wondering of what was really going on. Although their net was relatively safe from outside attack (thanks to a now moved-on-elsewhere UNIX admin), the inside was severely crippled. But the mix of delayed-for-too-long urgent projects and constant fire extinguishing brought me to a burnout before I could even take a look at stabilizing the systems. I actually quit that job before losing anymore of my health in that place (they had plans to migrate from Lotus Notes to Outlook ~shivers~). 12. In conclusion Things are not so sad after all, because this break gave me the time to come up with this document. I hope this will stand as a must read for network administrators and security experts out there. As new technologies and new viruses will emerge, some of the information may soon be obselete, but I think that above the batch code presented here, the lesson to be learned is that it makes sense to look at what you got and what you can do with it before going for expensive software purchases that can deliver most of what you want, not always the way you want, and the possible bugs and flaws that are likely to come with it. You benefit from having knowledge of what's going on on peoples PC, and no remote-monitoring software can beat the human approach (and the human approach will never crash due to system failure). A few early extra steps can save you a lot of steps in the long run (especially the steps you have to make to go from one PC to another). As for me, I have started to refine my knowledges in computer security, and I am getting anxious at trying some new ideas on my next workplace. I intend to implement a UNIX way of dealing with security in Windows NT networks. This will surely prove to be fertile grounds for new documents like this one. Appendice A: Something extra What I am about to cover here is not exactly about virus detection, well at least not in the traditional way. This is about other steps that you can make to give you a chance. Do occasionnal search on your file servers (and especially the user directories) for certain pattern files that could call for bad luck. This would include .exe, .com, .scr (not necessarily a virus, but potential software instability), .vbs, .dot and the like. If your site policies are strict about drive space usage (for business purpose only), you can include .jpg, .gif, .mpg, ... It is also a good idea to make a search for known trojan horses, password crackers and stuff like that (you'd be surprised to see that some other people shares the same hobbies as you do). I used to do it with a Find files request sent over my user directories, then capture the graphical output with a automatic scrolling window capture utility, then pasted it in a .BMP file. This was ugly, but I had the job done. But this could probably be more convenient, and more easy to schedule such searches with batch files. I let you write this one. One last word of advice about this, especially if you're consulting: depending on where you work, some places are pretty uptight about what can actually be done by a system admin. I have seen a lawyer department referring to as "unethical" to perform such searches over the file servers, as it could reveal confidential information to prying eyes (corrupted network admins). They claimed that in some cases, the filename alone could reveal too much. I wouldn't debate about that last claim, but when you conduct a search for specific patterns for anomalies, you expect to only rake anomalies (or most of it anyway). While it is true that a business having contracts with the filename containing all the references needed to clearly identify the files content may want to over-protect such information, these files are expected to have .doc, .wks, .wp, .ppt, ... extensions. You don't expect that much company-related info when searching for executable files (which should be considered as your exclusive privilege, the same way they want to take privilege from you about what you should do with their data files). Unless management are affraid that XXX .jpg files show up at unconvenient places. So document these procedures, have them approved by whoever wants to take the responsability of approving it, and keep secured logs of the information obtained that way and which files were discarded, and for what reason. In three words: Cover Your Ass!