11 October 2004

(Almost) Never Be Owned!

Gogoulos Markos, cs00013 at icsd.aegean.gr

Ancient Greeks used to say this:

"Nothing more permanent than the temporal"

In our case, most administrators are too busy to make any changes to the systems, so they remain vulnerable.

Introduction

Patching systems, using high-tech firewalls and intrusion detection systems, data encryption, and even strong passwords will not prevent security breaches from occurring. No matter how much is invested, breaches are inevitable and will happen sooner than later.

In order to keep a system -- let alone an entire network -- safe, a security plan must be in place, carefully revised and designed to be followed by everyone. The administrator must always be aware of the kinds of threats that exist and how to protect the systems against them. Additionally, he must be in position to keep the system's users informed about the damage they can create unwittingly, if they overlook the security policy.

(The word system is used numerous times on this article. It has the sense of a server, a whole network, or a complete site, comprising hardware, software, users, and procedures.)

Security is a matter of procedures and not solely of actions. If Procedures are followed, the probability for a successful break-in and penetration is significantly lessened though in nearly all cases will never vanish.

One important issue in the systems' use is the functionality and freedom that users enjoy to do what the system is designed to do. The skills and knowledge of the admin in conjunction with his awareness will judge whether or not the system will end either less functional (which would lead to a plethora of unremitting problems) or user capabilities will be maintained and in parallel the security of the system will be held to the highest standards.

Security as implemented by sysadmins can be categorized into three main sectors. It is necessary to underline the fact that these three sectors complete each other and must be taken into account as one entity.

Skills/knowledge

The knowledge and skills of the administrator will impact not only the security of the system but also its functionality and usability. On the contrary, ignorance, low skills, and shortage of good will for acquiring new knowledge will certainly lead to the security breaches of the systems.

A knowledgeable admin will first make sure to document the reason each system exists and install only the necessary services on each system. Security of the systems, in parallel with functionality, will be considered a primary goal to achieve from the initial installation. Setting up a system and adjusting it later in order to improve its security doesn't work in most cases.

System and OS and services configuration

The administrator must be current as far as a system's OS which is is concerned, including its capabilities and the purposes the system will serve. For example, if a system is going to be used as a standalone e-mail server, there is no reason to run an NFS daemon on that system. The administrator must make sure to install or enable only the necessary services and then close all unneeded network ports, which obviously constitute a gateway to the system. Additionally, a correct choice of options for kernel tuning will aid in the correct functionality of the system to the best of its ability, conserving resources and averting many problems, i.e., some kinds of DOS attacks, system crashes, etc.

Tight configuration and use of firewalls

Generally, firewalls are compulsory to protect any network. One of the most damaging security slogans one can hear is that "firewalls defend against every possible threat and attack." This is certainly false because first of all, if a firewall is not configured correctly, then not only it does not enhance security but it create a threat, since administrators believe that everything is OK so they stop bothering about other security matters.

Even a properly configured firewall will definitely not solve all problems. For example, if a website runs an authentication system that asks for a password is not configured to work only over httpsd (using insecure http instead), the firewall can't really do anything to prevent compromise of that password.

Firewall are generally necessary to prevent a great majority of attacks from external networks. Unfortunately, many successful attacks originate from inside the network, aided by nonexistent or incorrect access control and improper trust relationships. Firewalls do prevent novice and low-level attacks and potentially will protect the network from certain penetrations.

Consider a Samba server created to serve a local network and not remote connections/user. In the case of a zero-day exploit (where crackers have code to compromise security but vendors have not yet released a patch), the lack of a firewall blocking improper incoming connections can potentially lead to the compromise of an entire internal network.

Firewalls can also prevent the spread of worms and viruses on Microsoft's Windows OS's networks. Most Windows computers typically use Netbios protocols and have misconfigured 'shares'. Password guessing, brute force attacks, and simple enumeration can enable compromise. All of the current worms exploit the various problems on the security of Windows, attacking shares, DCOM, LSAS, Microsoft SQL server, etc. Combining a properly configured global firewall with a local firewall in every system will avert the infection and spreading of these viruses/worms. Additionally, if the domain controller of a windows network is penetrated and shares are accessible from everywhere, then all the files located on the server (and probably on the network) might also be accessed. With correct blocking of those specific ports, that won't be feasible.

Administrators must always bear in mind that firewalls will never solve every security problem.

Correct use of cryptography

Proper use of cryptography has become necessary for maintenance of a system's or a network's security. Every service that sends passwords and data in plain text must be replaced with one that encrypts that data when it is on the network. Very few administrators allow connections using telnet protocol these days. Users, however, read their daily e-mail through POP3 or IMAP protocols, or even through a webmail system over HTTP! The gap is clear.

HTTPS should always be used for reading webmail. Likewise, for those who prefer to work with e-mail management programs, only the secure protocols pop3s and imaps should be used.

A skilled administrator should acquire a powerful, theoretical background in respect to networks, the TCP/IP model, public/private key encryption, and thus understand the functionality and operation of all related protocols.

For example, some administrators have been victims of attacks against ssh and, as a consequence, have inadvertently disclosed the root password because they hadn't heard of ARP poisoning, ssh sniffing, and man-in-the-middle attacks, and thus didn't understand that their network harbored an ssh sniffer.

Security Policy

A skilled administrator will designate a correct security policy, which describes the security of the system starting with its creation and set up and not as a set of additions/improvements to a current system. He will define the policies for the selection, usage, and safe storage of passwords. Periodically, he will audit passwords and inform the users for weak passwords. He will audit the system in order to discover problems that have been created and malfunctions.

Naturally, he will document that policy, in a direct and understandable way in addition to informing the system's users of the various problems that arise if, e.g., they share their password with others, perform sensitive transactions from insecure networks, etc. Backup strategies and retention policies need to be complemented by frequent tests to ensure backups work properly. Similarly, the administrator must specify policies for rotating backups through offsite (or some other physically secure)storage.

The security policy should specify what must be protected and by whom. The skilled administrator will combine, in the best feasible way, the security of the system with the concomitant usability and functionality. He understands well that human factor is unpredictable and can't really be treated as if it is one more technical factor with a predictable conduct that can be dealt with conventional standard technical methods.

Overcommitment and laziness

Many problems and break-ins take place because administrators are too busy or too lazy to update their systems. Either they consider the updates superfluous and futile or they believe that they would consume enormous time (leaving other tasks unfinished) There is also the case of relentlessly postponing actions 'until next week', which never comes.

Programs, services and patches

It is of vital importance for every system to have up-to-date versions of services and programs. All Linux/Unix distributions and vendors (as well as router companies) publish patches for every exploit that is discovered.

Most Linux patches are small pieces of easily executed code that are added to already-installed systems. and most of the times. Less frequently, a service must be re-installed from scratch. Vendors have done all they can to make patches easy and quick to install. Fast patch installation reduces the probabilities of a system intrusion.

One well-tested practice is the registration in the security list that the Linux or Unix vendor maintains. Every time a new security vulnerability is discovered, the vendor immediately sends notices of a patch's location with instructions for installing it.

Additionally, visiting websites such as securityfocus.com and packetstormsecurity.nl is a task of fundamental importance for an administrator who wants to keep on top of the security of his network or system to be compromised and wants to stay informed about the latest. It's very difficult to keep up (on, say, a daily basis) with the sites that monitor all services and operating systems one might use. Thus, it is highly advisable to register with related security mailing lists with early notification about new vulnerabilities and threats. Again, the sooner a patch is applied, the smaller the probabilities that systems are compromised.

Sometimes, however, exploits are distributed 'underground' for a period of time before vendors are notified. In that case, almost nothing can be done. Properly configured firewalls, access control, correct security policy, and probably a strong IDS system might avert a successful intrusion, but if not, it is certain that the administrator will believe that a successful intrusion has been accomplished and will have to proceed to the disaster recovery procedures.

An example of such a case is the attack on the Debian Linux distribution, in which its network was compromised with the use of a kernel root exploit. This exploit was only distributed through the cracker underground and was not publicly disclosed until after a damaging attack.

Many administrators are aware of the existence of a problem on their system but, for whatever reason, are certain that no one is going to exploit it. Maybe they believe that their system is uninteresting to anyone because it doesn't contain any sensitive data. If only they knew that their system is very possibly known to dozens of attackers around the world who are waiting to penetrate it when a new exploit is discovered.

Event logging

Event logging unfortunate constitutes one of the lesser-valued aspects of security. One security maxim says, "Prevention is ideal but detection is a must!". Unix and Linux operating systems provide many efficient logging tools, whereas in other OS's (such as Windows), most probably it is not considered necessary. Many commercial tools are available for logging in addition to a plethora of very effective open source tools.

Logs provide information about normal system operation, operations or patterns that deviate from the norm, notes about resource usage patterns, and sometimes even information about a system attack. Careful examination of related log files can reveal that an attack did actually occur, unless of course the intruder altered the log files so that his presence can't be traced. Even in this case, though, it is very difficult to leave absolutely no trace.

Additionally, file integrity checking scanners such as Tripwire, samhaim, or AIDE will reveal system changes that an intruder has made. Those programs, since they operate with the creation of an signature database, can't really be manipulated by an intruder (provided that the database is kept on a safe place).

In order for an attack to be traced in real time or to understand whether or not an attack has taken place, conventional tools provided by the operating system can be used, thus obviating the need for IDS systems or more complicated service tools.

A true story

A friend from a university was the long-time administrator for a Unix server dedicated to students working their projects. When I notified him that his server was running a variety of unnecessary services that could lead to a system compromise and furthermore that he hadn't maintained the security of the system since its initial installation, he responded that no one could be interested in that machine since it didn't contain any valuable information.

He also told me that even if someone were to format the hard disks at any time, he was confident that there would be no problem, since he was periodically keeping backups through automated scripts. I told him that the bandwidth that university networks provide is by itself sufficient for the server to be a high-profile target! My friend didn't heed of my advice to deal with the system's security.

Later on, we were notified by the network operation center that his server had been performing port scans and distributed attacks for an entire week, paralyzing many networks and as a consequence, the university had received hundreds of e-mails from other networks. Each of the e-mails asked at minimum that the university stop the attacks; others claimed that advocates (lawyers) had been notified and lawsuits were about to be issued to the university. After this incident, my friend better understood that a system's data is not the only target of intruders.

"Paranoia"/awareness

In system security, we adopt the word paranoia to express high awareness, as to what an administrator can do to keep his systems safe.

Passwords

Anyone interested in computer security has faced extreme cases, where even computers with critical -- and well secured -- data has still had serious problem with the access codes. People tend to choose easy or simple-to-guess passwords, so that they can quickly and easily remember them. They might use the same password in each system they control or have access to. These constitute huge holes in the security chain waiting for someone to exploit them. This sort of exploit regrettably happens with tremendous regularity.

Administrators, particularly, should choose a hard-to-guess password and remember it without writing it down. But that alone is still not sufficient: If he doesn't change his password, as time progresses, it is likely that the password would leak. It is also important that passwords not be related (e.g., "pw1", "pw2", "pw3", ...). The knowledge of one password must never lead to the discovery or recovery of other passwords.

A sad story...

One acquaintance of mine told me that he had penetrated a network with approximately 20 Unix servers. Long ago he had set up a sniffer on the network and had gained user level access to one of those servers.

He had been logging onto the system for two years without anyone suspecting anything, although he knew that people that used the account he was using were logging to the system only from one fixed IP address! Furthermore, the password was never changed during the entire two year time period.

Finally, as happens so frequently, a vulnerability that affected a network service on the server led to privilege escalation and thus he gained root access. Since he had access to the password hashes, he ran a password cracker on the same Unix box being cracked. He made it look like a system service, just in case someone looked at the process table. He obtained the password of the root user.

The password was comprised of a letter on the beginning and one word appended in the end. Noting that letter was the initial letter of the domain name of the server, he tried connecting to the rest of the network's servers, since he knew that the first four letters of the domain names were the same as the name of the city to which the server was located.

The task of finding the list with the server's domain names was not what one would classify as strenuous. but he was 'lucky', because he discovered that the administrators of that network had used as root password for each server the initial letter of the city in which it was located followed by that common word (which was hard to guess but was nevertheless cracked). These root passwords were not changed since the network's initial installation. That policy was convenient for system administrators since they should have access to all servers, but undoubtedly insecure.

It is imperative to pick a proper priority. Do we want easy password management which heightens the possibilities for the passwords to leak or do we want to keep the probability of intrusion as low as possible?

A really scary story

Another illuminating example comes from the Internet. It concerns a guy that exploited the security problems that exist in almost all Windows versions to break into other computers. He didn't search for a long time; he was looking for anything interesting and upon acquiring it he would disconnect. From the ethical side, it is certainly immoral and inappropriate, though technically not much damage was done.

At one of those computers, he came across a file with the name passwords.doc! He quickly downloaded the file to his computer. Upon trying to open it, he observed that it was locked with a password. That was his least concern since there exist a variety of programs to crack the security of Word documents. After successful cracking of that file, he surprisingly noticed that the file was not named by accident. In front of his eyes, he had a password file with myriads of passwords that belonged to systems managed by one administrator! It was stuffed with hundreds of root passwords from unix, linux, and windows systems, admin level passwords for a vast network's routers, mail account's passwords, and -- last but not least -- codes for web banking, credit card numbers, and utterly everything that shouldn't at all costs be stored on a desktop computer. Talk about scary!

Obvious conclusions

Never, ever store critical passwords in files accessible from the Internet! There exist some password management tools, but my personal experience in those issues is nonexistent.

One good practice is to define in the security policy how frequently passwords must be changed and follow that policy at all times and with accurate precision. Never use same passwords for two or more systems, since it is very likely that we would have more accounts cracked. On systems that many people have physical access to, it is absolutely necessary to issue a password for the BIOS. In case of a password-less BIOS, it is the easiest thing for someone to boot the computer with a floppy disk or CD and mount whichever partition of the hard disk he wants (unless the filesystem is encrypted). Many tools are widely distributed both for windows and unix that automate the procedure of acquiring administrator or root privileges.

Use of impossible-to-guess passwords is advisable. Train users to use such passwords that comprise of words and phrases that can't be guessed and even password cracking programs cannot break.

Security is a matter of "trust"

Can anyone say that after many hours of diligent work, the systems are safe? The answer is positively NO! Let' s consider the following paradigm:

The administrator of a, until then, safe system found himself on a university network with high-bandwidth internet connection and decided to check his e-mails. However, the computer he used had a keylogger program installed and that resulted in the exposure of his password to malicious hands.

Keyloggers are very sneaky programs that run in the background and log every key the user pushes on the keyboard. Additionally, besides software keyloggers, hardware keyloggers exist, though they require physical access to the keyboard to which they are going to be installed.

Some times, the use of ssh public/private key authentication is feasible and advisable, replacing the need for passwords.

But above all, keep in mind that we must not blindly trust computers! Even a computer that looks quite harmless,as of the above example, a simple desktop computer on a university's network, can be really dangerous, after all.

Final Outcome

The administrator determined to prevent successful attacks to his systems from taking place knows that there exist a huge community of skilled attackers, who will try to take advantage to the full extent of EVEN THE SIMPLEST weakness. He must not only know how attackers think and act but he must constantly be one step ahead from them.