How to install Perl modules that are available from CPAN

Last month I conducted Perl training for IBM Pune team.  During conducting the training, I mentioned about modules that are available at CPAN.  The obvious question from course participants was, how to install the modules available from CPAN.  Here's how.

If you don't have time for the details, here's what to do.
Download the tarball (the .tar.gz file) from CPAN.  Unzip and untar the tarball.
# make
# make test
# make install
Or if you have Internet connection available at your system :

# perl -MCPAN -e "install Your::Module"

Well, if you're interested in the full story, then read on.

The Comprehensive Perl Archive Network (CPAN) is a repository of over 250,000 software modules written in the Perl programming language by over 12,000 contributors.

Like all other modern programming languages, Perl provides a way to use external libraries of code.  The common subroutines could be then used by several programs.  In the Perl culture, these are called modules.

Before we begin with installing a CPAN module, why not check whether it is already installed at the system?  Say, we want to check whether module Data::Dumper is installed or not.  To do that, we could make an attempt to use the Data::Dumper module.  If that turns successful, we have the module installed.  If we see an error, that means the module is not present at our system.

$ perl -e "use Data::Dumper;"
$
No error, so module Data::Dumper is already installed! 

$ perl -e "use XML::Reader;"
Can't locate XML/Reader.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at -e line 1.
BEGIN failed--compilation aborted at -e line 1.
$
My attempt to use XML::Reader module resulted in an error, indicating that the module is not installed at my system.

Please note : For installing CPAN modules at Linux system, our user account should have root privilege.

There are two ways to install a CPAN module at a Linux system. 
Method 1 : Manual installation by downloading the tarball from CPAN and build from source code using make
Method 2 : Automated installation using the CPAN shell.  This requires connection to Internet.

To begin with, let's see how to install a module using the first (manual installation) method.

Download the tarball from CPAN.  A tarball is a gzip compressed tar file, usually ending with .tar.gz in its name.  For example, .tar.gz file of XML::Reader module is available at https://cpan.metacpan.org/authors/id/K/KE/KEICHNER/XML-Reader-0.65.tar.gz

Here's how I download the tarball using wget.

# wget https://cpan.metacpan.org/authors/id/K/KE/KEICHNER/XML-Reader-0.65.tar.gz  --no-check-certificate
--2018-07-19 13:13:18--  https://cpan.metacpan.org/authors/id/K/KE/KEICHNER/XML-Reader-0.65.tar.gz
Resolving cpan.metacpan.org... 151.101.158.217, 2a04:4e42:25::729
Connecting to cpan.metacpan.org|151.101.158.217|:443... connected.
WARNING: certificate common name e.sni.fastly.netâcpan.metacpan.org
                                                                     HTTP request sent, awaiting response... 200 OK
Length: 75978 (74K) [application/x-gzip]
Saving to: XML-Reader-0.65.tar.gz

100%[===================================================================================================================>] 75,978      --.-K/s   in 0.08s

2018-07-19 13:13:20 (931 KB/s) - XML-Reader-0.65.tar.gz
#

Now unzip and untar XML-Reader-0.65.tar.gz file that we have obtained.

# tar -zxvf XML-Reader-0.65.tar.gz
XML-Reader-0.65
XML-Reader-0.65/README
XML-Reader-0.65/Changes
XML-Reader-0.65/LICENSE
XML-Reader-0.65/dist.ini
XML-Reader-0.65/META.yml
XML-Reader-0.65/MANIFEST
XML-Reader-0.65/Makefile.PL
XML-Reader-0.65/lib/XML
XML-Reader-0.65/lib/XML/Reader.pm
XML-Reader-0.65/lib/XML/Reader_de.pod
XML-Reader-0.65/lib/XML/Reader_fr.pod
XML-Reader-0.65/t
XML-Reader-0.65/t/release-pod-syntax.t
XML-Reader-0.65/t/0010_test_functions.t
XML-Reader-0.65/lib/XML/Reader
XML-Reader-0.65/lib/XML/Reader/Testcases.pm
#

What are these z x v f options of tar command?
From the manual page of tar command :
       -z, --gzip
              filter the archive through gzip
       -x, --extract, --get
              extract files from an archive
       -v, --verbose
              verbosely list files processed
       -f, --file=ARCHIVE
              use archive file or device ARCHIVE         

A directory named XML-Reader-0.65 would be created in the current working directory.
             
# cd XML-Reader-0.65
#
# ls -l
total 64
-rw-r--r-- 1 root root 14247 Dec 28  2014 Changes
-rw-r--r-- 1 root root   469 Dec 28  2014 dist.ini
drwx------ 3 root root  4096 Jul 19 13:14 lib
-rw-r--r-- 1 root root 18352 Dec 28  2014 LICENSE
-rw-r--r-- 1 root root  1321 Dec 28  2014 Makefile.PL
-rw-r--r-- 1 root root   280 Dec 28  2014 MANIFEST
-rw-r--r-- 1 root root   693 Dec 28  2014 META.yml
-rw-r--r-- 1 root root   402 Dec 28  2014 README
drwxr-xr-x 2 root root  4096 Dec 28  2014 t
#

While in XML-Reader-0.65 directory, execute `perl Makefile.PL`  This will check if all the dependencies are installed, and if all the files listed in the MANIFEST file were included.  If successful, a Makefile will be generated, without any extension.

# perl Makefile.PL
Checking if your kit is complete...
Looks good
Warning: prerequisite Acme::HTTP 0 not found.
Warning: prerequisite XML::Parsepp 0 not found.
Writing Makefile for XML::Reader
[root@.localhost XML-Reader-0.65]#

In this case, two prerequisite modules Acme::HTTP and XML::Parsepp are not found.  So I should install them first and then proceed further.  And what if I continue further without installing the prerequisite modules?  Okay, let's try that.  Next step is to execute `make` in order to compile the source code.

# make
cp lib/XML/Reader_de.pod blib/lib/XML/Reader_de.pod
cp lib/XML/Reader.pm blib/lib/XML/Reader.pm
cp lib/XML/Reader_fr.pod blib/lib/XML/Reader_fr.pod
cp lib/XML/Reader/Testcases.pm blib/lib/XML/Reader/Testcases.pm
Manifying blib/man3/XML::Reader_de.3pm
Manifying blib/man3/XML::Reader::Testcases.3pm
Manifying blib/man3/XML::Reader_fr.3pm
Manifying blib/man3/XML::Reader.3pm
#

Optionally, execute `make test` to run the automated tests.
Although we could skip executing `make test`, this is a recommended step.  Result coming from executing the tests is usually a good indicator of the road ahead.

# make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/0010_test_functions.t .. 1/54
#   Failed test 'use XML::Reader;'
#   at t/0010_test_functions.t line 6.
#     Tried to use 'XML::Reader'.
#     Error:  Can't locate Acme/HTTP.pm in @INC (@INC contains: /root/XML-Reader-0.65/blib/lib /root/XML-Reader-0.65/blib/arch /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /root/XML-Reader-0.65/blib/lib/XML/Reader.pm line 7.
# BEGIN failed--compilation aborted at /root/XML-Reader-0.65/blib/lib/XML/Reader.pm line 7.
# Compilation failed in require at t/0010_test_functions.t line 6.
# BEGIN failed--compilation aborted at t/0010_test_functions.t line 6.

#   Failed test 'Test-001 sub XML::Reader::import is defined'
#   at t/0010_test_functions.t line 69.

#   Failed test 'Test-002 sub XML::Reader::activate is defined'
#   at t/0010_test_functions.t line 69.

...
...
...


#   Failed test 'Test-052 sub XML::Reader::Token::extract_proc is defined'
#   at t/0010_test_functions.t line 69.

#   Failed test 'Test-053 sub XML::Reader::Token::extract_decl is defined'
#   at t/0010_test_functions.t line 69.
# Looks like you failed 54 tests of 54.
t/0010_test_functions.t .. Dubious, test returned 54 (wstat 13824, 0x3600)
Failed 54/54 subtests
t/release-pod-syntax.t ... skipped: these tests are for release candidate testing

Test Summary Report
-------------------
t/0010_test_functions.t (Wstat: 13824 Tests: 54 Failed: 54)
  Failed tests:  1-54
  Non-zero exit status: 54
Files=2, Tests=54,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.13 cusr  0.01 csys =  0.18 CPU)
Result: FAIL
Failed 1/2 test programs. 54/54 subtests failed.
make: *** [test_dynamic] Error 255
#
Executing the automated tests usually produces a long output, which I have cut short in the above listing.  In this case, all tests failed.  Possibly because the prerequisite modules are not available.

Although tests have failed, I could execute `make install` to install the module in the correct place.

# make install
Installing /usr/local/share/perl5/XML/Reader_de.pod
Installing /usr/local/share/perl5/XML/Reader.pm
Installing /usr/local/share/perl5/XML/Reader_fr.pod
Installing /usr/local/share/perl5/XML/Reader/Testcases.pm
Installing /usr/local/share/man/man3/XML::Reader_de.3pm
Installing /usr/local/share/man/man3/XML::Reader_fr.3pm
Installing /usr/local/share/man/man3/XML::Reader.3pm
Installing /usr/local/share/man/man3/XML::Reader::Testcases.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
#

Time to check whether XML::Reader module is now available.

# perl -e "use XML::Reader;"
Can't locate Acme/HTTP.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/share/perl5/XML/Reader.pm line 7.
BEGIN failed--compilation aborted at /usr/local/share/perl5/XML/Reader.pm line 7.
Compilation failed in require at -e line 1.
BEGIN failed--compilation aborted at -e line 1.
#

So we must install the prerequisite module Acme::HTTP
Then another error would tell that prerequisite module XML::Parsepp can't be located.  I will have to install XML::Parsepp as well, so that all required prerequisite modules are available.

This could be a long way to home.  Before I install the module I intended, I must identify all prerequisite modules and install all of them.  A prerequisite module may have its own prerequisite(s).  This could be a really long way to home, depending upon how many prerequisites we run into.

There is a way available to make this task easier.  By opening a CPAN shell in perl, we could install the required module and all of its prerequisites.  The CPAN shell connects to cpan.org, identifies all prerequisites, downloads all prerequisites and the module we want to install, and installs them in appropriate sequence.  Voila!  All we need is Internet connection to connect to cpan.org

# perl -MCPAN -e "install XML::Reader"

No Internet connection to connect to cpan.org?  Well, the only option left is to get the tarballs (the .tar.gz files) from somewhere else and install each of them in the right sequence, as per the dependencies.

And now if you insist on looking at official documentation, why don't you get a terminal and type `perldoc perlmodinstall` or point your web browser to http://perldoc.perl.org/perlmodinstall.html

Perl : How to obtain a past date in YYYY-MM-DD format

In one particular perl script, I wanted to obtain the date that was 30 days ago, in YYYY-MM-DD format.  Here is the perl code I wrote.

#!/usr/bin/perl
use strict;
use warnings;

# 1 day = 86400 seconds
# 30 days 2592000 seconds

my ($day, $month, $year) = (localtime(time() - 2592000))[3,4,5];
$month++;
$year += 1900;
print "$year-$month-$day \n";

__END__


Perl's built-in function time returns the number of seconds since the epoch.  The epoch is 00:00:00 UTC, January 1, 1970.
If we subtract 2592000 from the value returned by time, the result we get is useful for obtaining a date from 30 days ago.

Using the value obtained from time function, Perl's built-in function localtime is useful in obtaining day of month, month, and year.
Value of year returned by localtime function contains the number of years since 1900.  So we have to add 1900 to it, to obtain the year we were looking for.
Value of month returned by localtime function is in the range 0 to 11, representing January to December.  So adding 1 to it gives the month of year that we are interested in.

Validating numbers in a perl script

This morning one of my perlish friends asked me for help. He had a regular expression that was used for validating a string which contained numbers. The regex was failing and he wanted to know how to fix it. The strings he had were "12.3%", "1.25%" and similar. Numbers in it came from disk space usage, in percentages varying from 0% to 100% Similar to what we see in the output of df command.
And he wanted to extract the numeric part of it, to be used in numeric calculations.

His regex was not taking into account that these numbers could have "optional" parts. They could be "100%", "12%", "0.8%", "0%" and so on. Here is the regex I gave him.

[root@sonash5 tmp]# perl -e '$str = "1.23%"; print "matches" if ($str =~ m/^((\d{1,3})?(\.\d+)?)\s*%$/); print ", and the match is $1 \n"'
matches, and the match is 1.23
[root@sonash5 tmp]#

Are you thinking, why not just remove the % character from end of the string and use the rest of it as numeric entity?

That's a bad idea. What if some machine gives you "75 %"? Or even "75", assuming that they are percentages? And the string obtained could even be " 34.5%", resulting in the numeric calculations go wrong. When those string are coming from a source that you don't know, or may vary based on the operating system and the hardware and other things, it is a good idea to validate the string for the numeric pattern that you are looking for. And if the pattern is found, extract it and use it further.

How do I I find doubled words in a file

Noticed that mistake in the title? Or missed it? Here's how to catch those in your text files.

perl -nl -e 'print if m/\b(\w+)\s+\1/' filename

Instead if the whole line, you just want to see the line number and the word that is doubled?

perl -nl -e 'print "$. : $1" if m/\b(\w+)\s+\1/' filename

And you want to correct these?

perl -pi -e 's/\b(\w+)\s+\1/\1/g' filename

How do I get difference between two dates that are in yyyy-mm-dd format

A few days ago one of my friends asked me - How to get the difference in terms of number of days from two dates stored in yyyy-mm-dd format. I was in a hurry and told him how to go about, but couldn't show him the exact steps.

Yesterday I posed this questions to the interns in our company, thinking they would benefit from solving it. I told them to read the date values from shell environment variables, and use the programming language /tools of their choice to prepare the solution.

One guy wrote a neat C code to get the answer. He had to maintain a data structure to store how many days are present in all the months of the year, and also a function to find if the given year is a leap year or not. That resulted in the code growing to about 100 lines.

I asked them if they could think of some other approach, and if they were aware of how time is maintained in UNIX systems.

A perl script that uses package Time::Local makes the task easier to accomplish.

#!/bin/perl
use strict;
use warnings;
use Time::Local;

my $date1 = $ENV{'date1'};
my $date2 = $ENV{'date2'};

my @date1_breakup = split(/-/, $date1);
my @date2_breakup = split(/-/, $date2);

my $date1_unix = timelocal(0, 0, 0, $date1_breakup[2], $date1_breakup[1], $date1_breakup[0]);
my $date2_unix = timelocal(0, 0, 0, $date2_breakup[2], $date2_breakup[1], $date2_breakup[0]);

my $diffSeconds = $date2_unix - $date1_unix;
my $diffDays = $diffSeconds / (60 * 60 * 24);
print "difference in days : $diffDays\n";


A plain simple bourne shell script can also do the job for us, if the date command supports -d option.

#/bin/sh
date1_unix=`date -d $date1 +%s`
date2_unix=`date -d $date2 +%s`
diff=`expr $date2_unix - $date1_unix`
diff_days=`expr $diff / 86400` # Number of seconds in a day are 86400
echo $diff_days

yogeshs@yogesh-laptop:~/temp$ export date1=2010-04-22
yogeshs@yogesh-laptop:~/temp$ export date2=2010-05-23
yogeshs@yogesh-laptop:~/temp$ ./date_diff.sh
31
yogeshs@yogesh-laptop:~/temp$

Here's what happened inside the shell script.
yogeshs@yogesh-laptop:~/temp$ sh -x date_diff.sh
+ date -d 2010-04-22 +%s
+ date1_unix=1271874600
+ date -d 2010-05-23 +%s
+ date2_unix=1274553000
+ expr 1274553000 - 1271874600
+ diff=2678400
+ expr 2678400 / 86400
+ diff_days=31
+ echo 31
31
yogeshs@yogesh-laptop:~/temp$


You still want to see how this is done using C? Do let me know and I'll share that piece of code with you.

How to count occurences of a tag in an xml file

How can you find out total occurrences of a particular tag in an XML file? Say, I want to find out how many property tags are present in an XML file.

grep is not sufficient for this task. Here is a perl script that does the job.

#!/usr/bin/perl
# count_xml_tags.pl
my $xml_tag = shift;
my $filename = shift;
my $count     = 0;
open (X_FILE, '<', $filename) or die "Failed to read file $filename : $!";
{
    local $/;
    while (<X_FILE>) {
        while (m#<$xml_tag>(.*?)</$xml_tag>#gs) {
            $count++;
        }
    }
}
close (X_FILE);
print "$count $xml_tag tag(s) found.\n";

Run this script as:
% perl count_xml_tags.pl some_tag filename.xml

How to convert Unix time to human readable format

Unix-like operating systems maintain time as the number of seconds elapsed since midnight UTC of January 1, 1970. For example, 1217646573 represents Sat Aug 2 08:39:33 2008.

How do you convert time mentioned in the Unix time format to human readable format? Perl is handy in this situation.
% perl -e 'print scalar localtime(1217646573), "\n";'

I used to work with a network monitoring software that produced logs containing time stamps in Unix time. Analyzing the logs would be difficult unless the time stamps were replaced by a human readable format.

The logs were as shown below.
STATUS [1217650382] Event: Description

Here's a perl one-liner that does the job.
% perl -pi -e 's#(?<=\[)(\d+)(?=])#scalar localtime($2)#e' /foo/bar

How to find out if a given year is a leap year or not?

Well, for that, how do you define a leap year?
Leap year is every year the number of which is a multiple of four, except those devisible by 100 and not by 400.

Here are the steps to determine if a given year is a leap year or not:
if year modulo 400 is 0 then leap
    else if year modulo 100 is 0 then no_leap
    else if year modulo 4 is 0 then leap
    else no_leap

Here is one Perl implementation.

#!/usr/bin/perl
use strict;
use warnings;
###### is_leap_year ######
# Purpose : To determine whether a given year is a leap year or not
# Arguements : One
# First (Number): Year to be checked
# Returns : Boolean value indicating whether given year is a leap year or not
# true = given year is a leap year
# false = given year is not a leap year
############################
sub is_leap_year ($)
{
       my $year = shift;
       return unless ($year =~ m/^\d+$/);

       if (($year % 400) == 0) {
       # This year is completely divisible by 400
               return 1;     # This is a leap year
       }
       elsif (($year % 100) == 0) {
       # This year is completely divisible by 100
               return 0;     # This is not a leap year
       }
       elsif (($year % 4) == 0) {
       # This year is completely divisible by 4
               return 1;     # This is a leap year
       }
       return 0;     # This is not a leap year
}

my $this_year = 2040;
if (is_leap_year ($this_year)) {
       print $this_year, " is a leap year\n";
}
else {
       print $this_year, " is a not a leap year\n";
}

Count number of occurrences of a word

How can you count number of occurrences of a word in a file? Say, I want to count how many times ring is present in a file.

How about using the -c option of grep?
% grep -c ring /foo/bar

And what if the file contains words such as string and rings? That would break our count of ring.
Using the -w option of grep solves this problem
% grep -cw ring /foo/bar

What if the word ring is present more than once in a line? grep would produce incorrect count in this case, since it counts the number of lines in which the pattern is found. grep is not sufficient for this job. We need something that can count multiple occurrences of a word in a line.

To get the exact count:
#!/usr/bin/perl
# search_word.pl
my $search_this = shift or exit 1;
my $count = 0;
while (<>) {
    while (m/\b$search_this\b/g) {
        $count++;
    }
}
if ($count == 0) {
    print $ARGV . "does not contain " . $search_this . "\n";
}
else {
    print $ARGV . "contains " . $search_this . " " . $count . (($count == 1) ? " time\n" : " times\n");
}

Run this perl script as:
% perl search_word.pl ring filename

And what if I don't want to use Perl? Well, then this should work for you:
% grep -w -o ring /foo/bar | wc -l

Sorting alphanumeric strings in Perl

One of my friends had trouble sorting alphanumeric strings in Perl. Strings he had were similar to:

child1
child3
child11
child51
child5
child24

He wanted to sort them based on the numbers that follow after the alphabtic part. So, the order he wanted:

child1
child3
child5
child11
child24
child51

I extracted the numeric part from the alphanumeric string and used it for comparisons

@alphanumeric_sorted = sort { substr($a, 5) <=> substr($b, 5)} @alphanumeric;