Just Another Perl Hacker

Sorting techniques in Perl

2011-01-05T20:53:00.001+05:30

Check out this SlideShare Presentation:

Sorting techniques in Perl

View more presentations from Yogesh Sawant.

Validating numbers in a perl script

2010-09-02T12:01:00.003+05:30

This morning one of my perlish friends asked me for help. He had a regular expression that was used for validating a string which contained numbers. The regex was failing and he wanted to know how to fix it. The strings he had were "12.3%", "1.25%" and similar. Numbers in it came from disk space usage, in percentages varying from 0% to 100% Similar to what we see in the output of df command.
And he wanted to extract the numeric part of it, to be used in numeric calculations.

His regex was not taking into account that these numbers could have "optional" parts. They could be "100%", "12%", "0.8%", "0%" and so on. Here is the regex I gave him.

[root@sonash5 tmp]# perl -e '$str = "1.23%"; print "matches" if ($str =~ m/^((\d{1,3})?(\.\d+)?)\s*%$/); print ", and the match is $1 \n"'
matches, and the match is 1.23
[root@sonash5 tmp]#

Are you thinking, why not just remove the % character from end of the string and use the rest of it as numeric entity?

That's a bad idea. What if some machine gives you "75 %"? Or even "75", assuming that they are percentages? And the string obtained could even be " 34.5%", resulting in the numeric calculations go wrong. When those string are coming from a source that you don't know, or may vary based on the operating system and the hardware and other things, it is a good idea to validate the string for the numeric pattern that you are looking for. And if the pattern is found, extract it and use it further.

How do I I find doubled words in a file

2010-05-25T11:01:00.000+05:30

Noticed that mistake in the title? Or missed it? Here's how to catch those in your text files.

perl -nl -e 'print if m/\b(\w+)\s+\1/' filename

Instead if the whole line, you just want to see the line number and the word that is doubled?

perl -nl -e 'print "$. : $1" if m/\b(\w+)\s+\1/' filename

And you want to correct these?

perl -pi -e 's/\b(\w+)\s+\1/\1/g' filename

How do I get difference between two dates that are in yyyy-mm-dd format

2010-05-21T12:34:00.001+05:30

A few days ago one of my friends asked me - How to get the difference in terms of number of days from two dates stored in yyyy-mm-dd format. I was in a hurry and told him how to go about, but couldn't show him the exact steps.

Yesterday I posed this questions to the interns in our company, thinking they would benefit from solving it. I told them to read the date values from shell environment variables, and use the programming language /tools of their choice to prepare the solution.

One guy wrote a neat C code to get the answer. He had to maintain a data structure to store how many days are present in all the months of the year, and also a function to find if the given year is a leap year or not. That resulted in the code growing to about 100 lines.

I asked them if they could think of some other approach, and if they were aware of how time is maintained in UNIX systems.

A perl script that uses package Time::Local makes the task easier to accomplish.

#!/bin/perl
use strict;
use warnings;
use Time::Local;

my $date1 = $ENV{'date1'};
my $date2 = $ENV{'date2'};

my @date1_breakup = split(/-/, $date1);
my @date2_breakup = split(/-/, $date2);

my $date1_unix = timelocal(0, 0, 0, $date1_breakup[2], $date1_breakup[1], $date1_breakup[0]);
my $date2_unix = timelocal(0, 0, 0, $date2_breakup[2], $date2_breakup[1], $date2_breakup[0]);

my $diffSeconds = $date2_unix - $date1_unix;
my $diffDays = $diffSeconds / (60 * 60 * 24);
print "difference in days : $diffDays\n";

A plain simple bourne shell script can also do the job for us, if the date command supports -d option.

#/bin/sh
date1_unix=`date -d $date1 +%s`
date2_unix=`date -d $date2 +%s`
diff=`expr $date2_unix - $date1_unix`
diff_days=`expr $diff / 86400` # Number of seconds in a day are 86400
echo $diff_days

yogeshs@yogesh-laptop:~/temp$ export date1=2010-04-22
yogeshs@yogesh-laptop:~/temp$ export date2=2010-05-23
yogeshs@yogesh-laptop:~/temp$ ./date_diff.sh
31
yogeshs@yogesh-laptop:~/temp$

Here's what happened inside the shell script.
yogeshs@yogesh-laptop:~/temp$ sh -x date_diff.sh
+ date -d 2010-04-22 +%s
+ date1_unix=1271874600
+ date -d 2010-05-23 +%s
+ date2_unix=1274553000
+ expr 1274553000 - 1271874600
+ diff=2678400
+ expr 2678400 / 86400
+ diff_days=31
+ echo 31
31
yogeshs@yogesh-laptop:~/temp$

You still want to see how this is done using C? Do let me know and I'll share that piece of code with you.

Zenity

2009-03-25T20:06:00.003+05:30

Its been ten years since I was introduced to UNIX, and there are still so many things to learn. Today while browsing at my favorite UNIX forums, I saw a question: help - something doesn't work with zenity.

I had never heard of zenity, and the question aroused my interest to know what it was about. After following the rule do a web search before you ask, I discovered that zenity was present at my system! And I didn't knew that. Poor me.

I never knew that we could pop up a dialog box from a shell script. Here's how.

zenity --info --title "Hi" --window-icon=none --text "Howdy world\!"

A look at the man page and I got answer to that question

Problems galore, Solutions aplenty

2008-12-26T23:29:00.004+05:30

A few years ago, one of my friends asked me for advice. He was frustrated, mostly out of his job. I wanted to help, but he was at the other side of the globe! I couldn't give him a ready solution. But instead, I sent him my recepie of solving a problem. Here it is.

1. Identify the problem
This is the first step. Unless we identify the exact problem, with all its details, we won't be able to prepare a solution for it. If you could prepare a problem statement, not more than a few lines, that would help much to make things clear.

2. Make a list of all possible solutions
When you are aware what the problem is or what is bothering you or what is it that you are not satisfied with, think of all the possible ways in which the situation could be improved. Even if a solution looks simple, or complex, far fetched, or lengthy, do add it to your list. Include only practical solutions.

3. Choose the best solution from the list
When you know the all possible ways, choose the best one. Depends on what is best for you in that situation. Choosing the best one would require comparisons and judgement on the criteria that you have.

4. Implement the best solution that we now know
Go ahead, its time for action.

When I have a problem, I tend to think in this way. And it works for me. And who doesn't have problems in this world? Sometimes I think that we are here to solve problems. We are all problem-solvers.

screen 'em

2008-12-21T14:13:00.009+05:30

Some of the tasks that I do involve running commands that take way long to finish - some take a few hours, others take a couple of days. Obviously, you can't be staring at the terminal till the task is done. And if you close the terminal, the command dies away. So how to run these commands? Simply running them in background won't help, as they would be terminated when you log out.

One idea is to run them in background, and using nohup.

$ nohup my_cmd &
$ exit

This would start the command in the background in such a way that it won't be stopped even if you log out. But you won't see the output at your terminal. nohup would redirect it to a file named nohup.out

Once I start the command using nohup, there is no way I can interact with it. And I can't see the output at the terminal. Sure, there is a way if I really want to. Keep watching: $ tail -f nohup.out

A better alternative in these scenarios is to use the GNU Screen utility. It offers many useful features.

* I can start the command and leave it alone. Later, I can see how it is progressing whenever I want to.

* I can start the command from one computer, and then see how it is progressing from a different computer. So I can start the task at office, then go home, and when I find time, check how it is going.

* More than one users can share a screen. So if my colleague wants to check, he can also connect to the same screen and see the progress.

* Multiple terminal sessions can be created in a screen session. So I can keep running the command in one of them, and do something else in the other.

* Screen is very helpful when the network connection is unreliable. I would still have the task running even if connection breaks.

Enough. Tel me how do I start.

Check if a screen session is already running
$ screen -ls

Attach to a not detached session
$ screen -x

Start a new screen session
$ screen

Create a new terminal session
CTRL+a c

Toggle between two terminal sessions
CTRL+a CTRL+a

Go to the nth terminal session
CTRL+a n

Send the command character CTRL+a to a window
CTRL+a a

Detach from a screen
CTRL+a d

Terminate the screen
$ exit

man screen for more information

bad command or file name

2008-12-20T17:35:00.004+05:30

Really? Is my command so bad? Then what would a good command look like?

Long gone are the days when I last saw the (in)famous error message. Most of us would have forgotten it by now. One of the blogs that I was reading today mentioned about it. And in a rhetoric manner.

Why would you want to tell someone that something they entered is bad? Or something bad has happened? Isn't that really foolish? Didn't Mom tell them: "If you can't say anything nice, don't say anything at all." At least, don't say a bad word.

A major reason why software is not easy to learn is that it doesn't give positive feedback. We learn better from positive feedback, rather than from negative feedback. Would you prefer a hired ski instructor who yells at you? Or a restaurant host who loudly announces to other patrons that your credit card was rejected? Certainly not. Then why the should software tell me that I failed?

In my opinion, to be given negative feedback by software - any software - is an insult to human. Yes, I've read the three laws of robotics, and can recite them for you even during sleep.

Forget robotics and the mean machines, but for me, to be told by software that you have failed is degrading. And the same should be for everyone. There is nothing so important inside that dumb box that you can justify degrading a human user. If you want to protect you files, do it. But do so without bothering me. It's all right to protect the computer but not at the cost of bothering the user.

If I mistype a command, with bash, csh, sh, tcsh, zsh - they all tell me:
command not found

Well, that's better - it tells me without insulting me that my input could not be processed. And it even hints that what I entered was treated as a command, without misleading me to file names.

An error message is the program reporting on its failure to do the job, and it is interrupting the user to do this. So it must be polite, illuminating, and helpful.

Have you never heard - The customer is always right

How to count occurences of a tag in an xml file

2008-11-17T23:14:00.020+05:30

How can you find out total occurrences of a particular tag in an XML file? Say, I want to find out how many property tags are present in an XML file.

grep is not sufficient for this task. Here is a perl script that does the job.

#!/usr/bin/perl
# count_xml_tags.pl
my $xml_tag = shift;
my $filename = shift;
my $count = 0;
open (X_FILE, '<', $filename) or die "Failed to read file $filename : $!";
{
local $/;
while (<X_FILE>) {
while (m#<$xml_tag>(.*?)</$xml_tag>#gs) {
$count++;
}
}
}
close (X_FILE);
print "$count $xml_tag tag(s) found.\n";

Run this script as:
% perl count_xml_tags.pl some_tag filename.xml

How to remove duplicate lines from a file

2008-11-17T22:27:00.007+05:30

Our on-line publishing system has a text file that contains certain entries, one per line. Some entries were duplicate, and we wanted to remove them.

This can be done using sort and uniq commands.
sort /foo/bar | uniq > /new/bar

But we wanted to retain the order of lines, and so didn't want to sort the file. I found a solution using awk.
awk '!x[$0]++' /foo/bar > /new/bar

And how do I check if the file contains duplicate lines or not? The -d option of uniq command is helpful in this case.
sort /foo/bar | uniq -d

How to convert Unix time to human readable format

2008-08-02T14:22:00.005+05:30

Unix-like operating systems maintain time as the number of seconds elapsed since midnight UTC of January 1, 1970. For example, 1217646573 represents Sat Aug 2 08:39:33 2008.

How do you convert time mentioned in the Unix time format to human readable format? Perl is handy in this situation.
% perl -e 'print scalar localtime(1217646573), "\n";'

I used to work with a network monitoring software that produced logs containing time stamps in Unix time. Analyzing the logs would be difficult unless the time stamps were replaced by a human readable format.

The logs were as shown below.
STATUS [1217650382] Event: Description

Here's a perl one-liner that does the job.
% perl -pi -e 's#(?<=\[)(\d+)(?=])#scalar localtime($2)#e' /foo/bar

My Geek Code

2008-08-01T14:12:00.000+05:30

-----BEGIN GEEK CODE BLOCK-----
Version 1.0
GO d-@ s: a- C++$ UBL++>$ P+++>$ L+++>$
E- W++ N+ o K w-- !O- !M- V-
@PS+ @PE++ Y+ !PGP
t 5 X R+++ tv b+> DI++ D G
e++ h--- r+++ y+++
------END GEEK CODE BLOCK------

How to find out if a given year is a leap year or not?

2008-07-31T15:08:00.016+05:30

Well, for that, how do you define a leap year?
Leap year is every year the number of which is a multiple of four, except those devisible by 100 and not by 400.

Here are the steps to determine if a given year is a leap year or not:
if year modulo 400 is 0 then leap
else if year modulo 100 is 0 then no_leap
else if year modulo 4 is 0 then leap
else no_leap

Here is one Perl implementation.

#!/usr/bin/perl
use strict;
use warnings;
###### is_leap_year ######
# Purpose : To determine whether a given year is a leap year or not
# Arguements : One
# First (Number): Year to be checked
# Returns : Boolean value indicating whether given year is a leap year or not
# true = given year is a leap year
# false = given year is not a leap year
############################
sub is_leap_year ($)
{
my $year = shift;
return unless ($year =~ m/^\d+$/);

if (($year % 400) == 0) {
# This year is completely divisible by 400
return 1; # This is a leap year
}
elsif (($year % 100) == 0) {
# This year is completely divisible by 100
return 0; # This is not a leap year
}
elsif (($year % 4) == 0) {
# This year is completely divisible by 4
return 1; # This is a leap year
}
return 0; # This is not a leap year
}

my $this_year = 2040;
if (is_leap_year ($this_year)) {
print $this_year, " is a leap year\n";
}
else {
print $this_year, " is a not a leap year\n";
}

Count number of occurrences of a word

2008-07-30T16:57:00.016+05:30

How can you count number of occurrences of a word in a file? Say, I want to count how many times ring is present in a file.

How about using the -c option of grep?
% grep -c ring /foo/bar

And what if the file contains words such as string and rings? That would break our count of ring.
Using the -w option of grep solves this problem
% grep -cw ring /foo/bar

What if the word ring is present more than once in a line? grep would produce incorrect count in this case, since it counts the number of lines in which the pattern is found. grep is not sufficient for this job. We need something that can count multiple occurrences of a word in a line.

To get the exact count:
#!/usr/bin/perl
# search_word.pl
my $search_this = shift or exit 1;
my $count = 0;
while (<>) {
while (m/\b$search_this\b/g) {
$count++;
}
}
if ($count == 0) {
print $ARGV . "does not contain " . $search_this . "\n";
}
else {
print $ARGV . "contains " . $search_this . " " . $count . (($count == 1) ? " time\n" : " times\n");
}

Run this perl script as:
% perl search_word.pl ring filename

And what if I don't want to use Perl? Well, then this should work for you:
% grep -w -o ring /foo/bar | wc -l

Why not to use awk when sed can do the job?

2008-07-29T17:15:00.005+05:30

Using awk instead of sed has the price of performance and size
compared to sed and ed, awk takes a substantially longer time to load, and does its job at a considerably slower pace
The real distinguishing point between sed and awk as a text processor is that awk is able to work with a persistent context, whereas capabilities of sed in this area are limited to non-existent. If you - for instance - would have to sum one field to a total you would do it with awk (it would be possible to do it with sed, but would be a nightmare - poorly suited tool for the job)

Changing size of a file

2008-07-28T15:08:00.007+05:30

How do you change the size of a file in Linux? If you are not concerned about the contents of the file, use the dd command.

dd if=/dev/zero of=/foo/bar bs=1024 count=1

This would change the size of a file to 1 KB. inode number of the file would be preserved.

So if you want to change the size to 2 MB

dd if=/dev/zero of=/foo/bar bs=1024 count=2048

Note that 2 MB is 2048 KB, and not 2000 KB.

Epigram on a Software Laird

2008-07-27T17:06:00.002+05:30

Bless Linus Torvalds, O Engineer,
With grateful, lifted eyes,
Who taught that not the windows alone,
But doors too shall open;
For had He said "the doors alone
From source I will deliver,"
Alas, alas! O Engineer,
Then hadst thou lain for ever.

Original work: Epigram on a Country Laird by Robert Burns

Sorting alphanumeric strings in Perl

2008-07-22T15:21:00.002+05:30

One of my friends had trouble sorting alphanumeric strings in Perl. Strings he had were similar to:

child1
child3
child11
child51
child5
child24

He wanted to sort them based on the numbers that follow after the alphabtic part. So, the order he wanted:

child1
child3
child5
child11
child24
child51

I extracted the numeric part from the alphanumeric string and used it for comparisons

@alphanumeric_sorted = sort { substr($a, 5) <=> substr($b, 5)} @alphanumeric;

Apache Lyrics

2008-07-21T14:12:00.002+05:30

Sometime last year I wrote parody of a famous song. Some of the lines of this song are in Spanish, and you need to know a bit of Spanish to understand them.

"Ports Won't Die"
(feat. Office Assistant)

Macros up in here tonight
No fighting, no fighting
We got the fixes up in here
No fighting, no fighting

Apache, Apache

I never really knew that it could serve so fast
It makes a man wants to HTTP
Como se llama (si), bonita (si), mi OS (si, Apache Apache), su OS
Apache, Apache

Oh baby when you talk like that
You make IIS go mad
So be wise and keep on
Reading the signs of my Ballmer

And I'm on tonight
You know my ports won't die
And I'm starting to feel it's right
All the attraction, the tension
Don't you see baby, this is perfection

Hey Girl, I can see your traffic moving
And it's driving me crazy
And I didn't have the slightest idea
Until I saw you serving

And when you take up a bad request
Nobody cannot ignore the way you check it out, girl
And architecture so perfected - the way you right and left it
So you can keep on excelling

I never really knew that it could serve so fast
It makes a man want to HTTP
Como se llama (si), bonita (si), mi OS (si, Apache Apache), su OS
Apache, Apache

Oh baby when you talk like that
You make IIS go mad
So be wise and keep on
Reading the signs of my Ballmer

And I'm on tonight
You know my ports won't die
And I am starting to feel you boy
Come on lets go, real slow
Don't you see baby asi es perfecto

Oh I know I am on tonight my ports won't die
And I am starting to feel it's right
All the attraction, the tension
Don't you see baby, this is perfection
Apache, Apache

Oh boy, I can see your traffic moving
Half filtered, half cached
I don't, don't really know what I'm doing
But you seem to have a plan
My will and self restraint
Have come to fail now, fail now
See, I am doing what I can, but I can't so you know
That's a bit too hard to explain

Roca en el Internet toda la noche
Roca en el Internet toda el día

Roca en el Internet toda la noche
Roca en el Internet toda el día

I never really knew that it could serve so fast
It makes a man want to HTTP
Como se llama (si), bonita (si), mi OS (si, Apache Apache), su OS
Apache, Apache

Oh baby when you talk like that
You know you got me hypnotized
So be wise and keep on
Reading the signs of my Ballmer

Senorita, feel the conga, let me see you serve like you come from Redmond

Mira en Maryland se baila así, say it!
Mira en Maryland se baila así

Yeah
It's so popular, macro shaft's jealousy a refugee like me back with the Fugees from a buggy territory
I go back like when 'pac carried crates for Humpty Humpty
I need a whole club dizzy
Why the Court wanna break us?
Explorers and Windows
I ain't guilty, it's no monopoly transaction
No more do we snatch ropes
Refugees run the seas 'cause we own our own browser

I'm on tonight, my ports won't die
And I'm starting to feel you boy
Come on let's go, real slow
Baby, like this is perfecto

Oh, you know I am on tonight and my ports won't die
And I am starting to feel it's right
The attraction, the tension
Baby, like this is perfection

No fighting
No fighting