How to identify world-writable files in a Unix / Linux system

World writable files are those that all users of the system could write to.  In Unix / Linux systems, few of the files and directories are world writable, and for a specific purpose.  /tmp and /var/tmp directories are world writable, and sticky bit is set for them.  A directory that is world writable should have sticky bit set as well.  If stick bit is not set for a world writable directory, that is a cause for an alarm.  World writable files pose even more risk when they are executable by all as well.  Because those are scripted disasters that are waiting to happen.

Here is a command to list files that are world-writable.

# find /  -perm -777  ! -type l  ! \( -type d -and -perm -1000 \)  -exec ls -ld {} \;  2>/dev/null

Symbolic links should be excluded from this list, since their permission bits are ignored.  We achieved this using   ! -type l

Directories that have sticky bit set should be excluded from this list.  We got this done using   ! \( -type d -and -perm -1000 \)

If you are not familiar with the find command, do check the manual page of find command.  And about the options I have used in the above command, here is what the manual says about them :

       -perm mode
              File's  permission  bits  are exactly mode (octal or symbolic). 
             
       -type l      symbolic link

       -type d      directory

       -exec command ;
              Execute command; true if 0 status is returned.  All following arguments to find are taken to be arguments to the command until an argument consisting of ; is encountered.  The string {} is replaced by the current file name being processed everywhere it occurs in the arguments to the command, not just in arguments where it is alone, as in some versions of find.  Both of these constructions might need to be escaped (with a \) or quoted to protect them from expansion by the shell.

Careful examination should be done to identify whether the world writable files and directories found are actually necessary for the functionality of the system.  Permission should be updated appropriately for all the files and directories that are not necessary for the functionality of the system.

How to identify suid and sgid files in a Unix / Linux system

When a file is executed in a Unix / Linux system, execution is done using the privileges of the user who started execution.  This usual behavior could be altered using setuid and setgid flags.  Setuid and setgid flags provide elevated access rights while executing a file.  When setuid flag is set, execution is done using the privileges of the owner of the file.  Similarly, when setgid flag is set, privileges of the group owner of the file are used for executing the file.  This facility of executing files with elevated privileges is necessary for certain tasks of the operating system.  However, this facility poses risk when exploited by an attacker.

To mitigate the risk posed by presence of files having suid and sgid files set, careful examination should be done to identify whether the setuid and setgid flags are actually necessary for the functionality of the system.  All those that are not necessary should be removed.

Here is a command to list files that have suid flag set.

$ find /  -perm -4000  -exec ls -l {} \; 2>/dev/null

Here is a command to list files that have sgid flag set.

$ find /  -perm -2000  -exec ls -l {} \; 2>/dev/null

Here is a command to list files that have both suid and sgid flag set.

$ find /  -perm -6000  -exec ls -l {} \; 2>/dev/null

Are you wondering how is this facility is used by the Unix / Linux operating system.  Well, check the lists obtained from commands listed above.  You will see passwd which is used to update passwords of system users.  You will see sudo that is useful for executing a command as superuser or another user.  You will see ping, the useful networking command.  And many others.  So, before taking action on files that have suid or sgid flag set, careful examination should be done to ensure that we're doing what is right.  In an attempt of hardening the system, we don't want to halt some legitimate functionality of the system.

And how could an attacker exploit this facility.  Well, if an attacker gains access to the system, they could write a program to make a copy of /etc/shadow file.  Then set suid flag on that file.  Then change ownership of that file to superuser, who has access to /etc/shadow file.  Thus, /etc/shadow file is obtained.  Then remove the program that helped in this endeavor.  Now passwords could be obtained using John the Ripper.  Of course, these actions would be logged, and could be traced.  But why let the damage take place, when you could avoid it.

How to identify orphaned or unowned files in a Unix / Linux system

In a Unix / Linux system, files that are not owned by any user or group are known as orphaned files.  Having such files in the system is not a good idea.  Although these files themselves are not a risk, they could be used by an attacker in case of a breach. 

Here is a command to list files that are not owned by any user or group.

# find /  -path /proc -prune   -nouser -o -nogroup  -exec ls -l {} \; 2>/dev/null

For the files in this list, you could determine and implement appropriate actions, for the purpose of hardening the system.  Appropriate action could be to assign a user or group owner to the file, which was missing.  Or to remove the file if it is not required for the functionality of the system.  Or may be some other action.  Making these informed choices is better than not being aware of the situation.

If you are not familiar with the find command, do check the manual page of find command.  And about the options I have used in the above command, here is what the manual says about them :

       -path pattern
              File name matches shell pattern pattern. 

       -prune True;  if  the  file  is  a  directory,  do not descend into it.

       -nouser
              No user corresponds to file's numeric user ID.

       -nogroup
              No group corresponds to file's numeric group ID.

      -exec command ;
              Execute command; true if 0 status is returned.  All following arguments to find are taken to be arguments to the command until an argument consisting of ; is encountered.  The string {} is replaced by the current file name being processed everywhere it occurs in the arguments to the command, not just in arguments where it is alone, as in some versions of find.  Both of these constructions might need to be escaped (with a \) or quoted to protect them from expansion by the shell.

How do I extract files from ISO

I had created .iso file.  Then I wanted to check if a particular file is included in this image.

In the Windows world, a third party software is required to do this.  Linux has built-in support to do this.

I created a directory in which to mount the image.
mkdir /mnt/iso-check/

Then mount the .iso file.
mount -o loop  /disk1/iso/SONAS-1.6.0.0-29-1429.iso  /mnt/iso-check/

Now I could traverse to /mnt/iso-check/ and check that the required file is in there or not.

After I am done with whatever I was looking for, I unmounted the image.
umount /mnt/iso-check

How do I calculate sum of a column of numbers

I executed a command at a Linux machine, and it produced a long output.  I did the filtering, and now I have a column of numbers.  Now, how do I calculate sum of these numbers?  If you have awk available at your machine, here's how to do it.

$ your_long_list_of_command | awk '{ sum+=$1 } END { print sum}'

How do I get difference between two dates that are in yyyy-mm-dd format

A few days ago one of my friends asked me - How to get the difference in terms of number of days from two dates stored in yyyy-mm-dd format. I was in a hurry and told him how to go about, but couldn't show him the exact steps.

Yesterday I posed this questions to the interns in our company, thinking they would benefit from solving it. I told them to read the date values from shell environment variables, and use the programming language /tools of their choice to prepare the solution.

One guy wrote a neat C code to get the answer. He had to maintain a data structure to store how many days are present in all the months of the year, and also a function to find if the given year is a leap year or not. That resulted in the code growing to about 100 lines.

I asked them if they could think of some other approach, and if they were aware of how time is maintained in UNIX systems.

A perl script that uses package Time::Local makes the task easier to accomplish.

#!/bin/perl
use strict;
use warnings;
use Time::Local;

my $date1 = $ENV{'date1'};
my $date2 = $ENV{'date2'};

my @date1_breakup = split(/-/, $date1);
my @date2_breakup = split(/-/, $date2);

my $date1_unix = timelocal(0, 0, 0, $date1_breakup[2], $date1_breakup[1], $date1_breakup[0]);
my $date2_unix = timelocal(0, 0, 0, $date2_breakup[2], $date2_breakup[1], $date2_breakup[0]);

my $diffSeconds = $date2_unix - $date1_unix;
my $diffDays = $diffSeconds / (60 * 60 * 24);
print "difference in days : $diffDays\n";


A plain simple bourne shell script can also do the job for us, if the date command supports -d option.

#/bin/sh
date1_unix=`date -d $date1 +%s`
date2_unix=`date -d $date2 +%s`
diff=`expr $date2_unix - $date1_unix`
diff_days=`expr $diff / 86400` # Number of seconds in a day are 86400
echo $diff_days

yogeshs@yogesh-laptop:~/temp$ export date1=2010-04-22
yogeshs@yogesh-laptop:~/temp$ export date2=2010-05-23
yogeshs@yogesh-laptop:~/temp$ ./date_diff.sh
31
yogeshs@yogesh-laptop:~/temp$

Here's what happened inside the shell script.
yogeshs@yogesh-laptop:~/temp$ sh -x date_diff.sh
+ date -d 2010-04-22 +%s
+ date1_unix=1271874600
+ date -d 2010-05-23 +%s
+ date2_unix=1274553000
+ expr 1274553000 - 1271874600
+ diff=2678400
+ expr 2678400 / 86400
+ diff_days=31
+ echo 31
31
yogeshs@yogesh-laptop:~/temp$


You still want to see how this is done using C? Do let me know and I'll share that piece of code with you.

Zenity

Its been ten years since I was introduced to UNIX, and there are still so many things to learn. Today while browsing at my favorite UNIX forums, I saw a question: help - something doesn't work with zenity.

I had never heard of zenity, and the question aroused my interest to know what it was about. After following the rule do a web search before you ask, I discovered that zenity was present at my system! And I didn't knew that. Poor me.

I never knew that we could pop up a dialog box from a shell script. Here's how.

zenity --info --title "Hi" --window-icon=none --text "Howdy world\!"

A look at the man page and I got answer to that question

screen 'em

Some of the tasks that I do involve running commands that take way long to finish - some take a few hours, others take a couple of days. Obviously, you can't be staring at the terminal till the task is done. And if you close the terminal, the command dies away. So how to run these commands? Simply running them in background won't help, as they would be terminated when you log out.

One idea is to run them in background, and using nohup.

$ nohup my_cmd &
$ exit

This would start the command in the background in such a way that it won't be stopped even if you log out. But you won't see the output at your terminal. nohup would redirect it to a file named nohup.out

Once I start the command using nohup, there is no way I can interact with it. And I can't see the output at the terminal. Sure, there is a way if I really want to. Keep watching: $ tail -f nohup.out

A better alternative in these scenarios is to use the GNU Screen utility. It offers many useful features.

* I can start the command and leave it alone. Later, I can see how it is progressing whenever I want to.

* I can start the command from one computer, and then see how it is progressing from a different computer. So I can start the task at office, then go home, and when I find time, check how it is going.

* More than one users can share a screen. So if my colleague wants to check, he can also connect to the same screen and see the progress.

* Multiple terminal sessions can be created in a screen session. So I can keep running the command in one of them, and do something else in the other.

* Screen is very helpful when the network connection is unreliable. I would still have the task running even if connection breaks.

Enough. Tel me how do I start.

Check if a screen session is already running
$ screen -ls

Attach to a not detached session
$ screen -x

Start a new screen session
$ screen

Create a new terminal session
CTRL+a c

Toggle between two terminal sessions
CTRL+a CTRL+a

Go to the nth terminal session
CTRL+a n

Send the command character CTRL+a to a window
CTRL+a a

Detach from a screen
CTRL+a d

Terminate the screen
$ exit

man screen for more information

How to remove duplicate lines from a file

Our on-line publishing system has a text file that contains certain entries, one per line. Some entries were duplicate, and we wanted to remove them.

This can be done using sort and uniq commands.
sort /foo/bar | uniq > /new/bar

But we wanted to retain the order of lines, and so didn't want to sort the file. I found a solution using awk.
awk '!x[$0]++' /foo/bar > /new/bar

And how do I check if the file contains duplicate lines or not? The -d option of uniq command is helpful in this case.
sort /foo/bar | uniq -d

How to convert Unix time to human readable format

Unix-like operating systems maintain time as the number of seconds elapsed since midnight UTC of January 1, 1970. For example, 1217646573 represents Sat Aug 2 08:39:33 2008.

How do you convert time mentioned in the Unix time format to human readable format? Perl is handy in this situation.
% perl -e 'print scalar localtime(1217646573), "\n";'

I used to work with a network monitoring software that produced logs containing time stamps in Unix time. Analyzing the logs would be difficult unless the time stamps were replaced by a human readable format.

The logs were as shown below.
STATUS [1217650382] Event: Description

Here's a perl one-liner that does the job.
% perl -pi -e 's#(?<=\[)(\d+)(?=])#scalar localtime($2)#e' /foo/bar