AWK to take last previous column

You can use the below command to print last previous column.

awk '{print $(NF-1)}' access.log | sort | uniq | wc -l

Visual block in VI

Do you need to perform multiple line edits at a single stretch.

I came across VISUAL BLOCK in vi editor.

   CTRL + v makes you to enter the visual mode for your file.
   select multiple lines
   SHIFT + i gives you the insert mode
   Enter the text at beginning of the block
   Press ESC twice to apply changes to all lines.

Blogged with the Flock Browser

PERL script for extracting log file based on time

Below code extracts the huge log file based on time frame passed.

#!/usr/bin/perl

use strict;

my $file;
my $times = 0;
my $needed_response_time = 0;
my $start;
my $end;

sub Usage
{
my ($msg) = $_[0];
if ($msg)
{
print STDERR "$msg \n";
}
print STDERR << "EOHELP"
Usage: split_access_log_by_time

options:

-t - :: time interval for which the log is needed
-g :: filter requests greater than the specified response time

example : split_access_log_by_time -t 10:30:32-12:20:22 access_log
example : split_access_log_by_time -g 10 -t 10:30:32-12:20:22 access_log

EOHELP

}

my $i = 0;
while(defined( (my $arg = $ARGV[$i++]) ))
{
if($arg eq '-g')
{
$needed_response_time = $ARGV[$i++];
}
elsif($arg eq '-t')
{
$times = $ARGV[$i++];
}
elsif($arg eq '-h')
{
Usage();
exit 1;
}
else
{
$file = $arg;
last;
}
}

if(!defined($file))
{
Usage("No input file specified");
exit 1;
}

($start, $end) = split('-', $times);
$start = make_seconds(split(':', $start));
$end = make_seconds(split(':', $end));

if ($end < $start)
{
Usage("end time should be greater than start time");
exit 1;
}

# Group requests based on users
open(FILE, $file);
while()
{
my $line = $_;

my @split = split(' ', $line);
my $time = $split[3];
my $response_time = $split[-1];
$time = substr($time, 1);

my @timefields = split(':', $time);

my $hour = $timefields[1];
my $minute = $timefields[2];
my $second = $timefields[3];

#my $time_in_seconds = ($hour * 60 * 60) + ($minute * 60) + $second;
my $time_in_seconds = make_seconds($hour, $minute, $second);
if ($start<=$time_in_seconds && $end>=$time_in_seconds && $response_time >= $needed_response_time)
{
print $line;
}
}

sub make_seconds()
{
(my $hour, my $minute, my $second) = @_;
return ($hour * 60 * 60) + ($minute * 60) + $second;
}

PERL script to combine two csv files

Below script is used to combine two csv files based on the first column. For example, it is assumed both the file has first column related to project names.

#!/usr/bin/perl -w

$FILE1 = $ARGV[0];
$FILE2 = $ARGV[1];
$PROJECTFILE = $ARGV[2];

my %File1Map = ();
my %File2Map = ();

open(FILE1) or die("Could not open $FILE1 file.");
foreach $line () {
# ($Project,$Attachments,$Announcements,$Documents,$Discussions,$Users,$ArtifactTypeCount,$Alm) = split(',',$line);
($Project,$values) = split(',',$line,2);
# my $tempval="$Attachments,$Announcements,$Documents,$Discussions,$Users,$ArtifactTypeCount,$Alm";
my $tempval = $values;
# print "$Project,$values";
$tempval =~ s/\r|\n//g;
$File1Map{"$Project"} = "$tempval";
}

close(FILE1);

open(FILE2) or die("Could not open $FILE2 file.");
foreach $subline() {
($proj,$izcount) = split(',',$subline);
$File2Map{"$proj"}="$izcount";
}

close(FILE2);

open(PROJECTFILE) or die("Could not open $PROJECTFILE file.");
foreach $projname() {
($name,$dummy)=split('\n',$projname);
my $existingValue=$File1Map{"$name"};
my $newValue=$File2Map{"$name"};

if (! defined $existingValue)
{
$existingValue = ",,,,,,,,";
}
if (! defined $newValue)
{
$newValue = "\n";
}

print "" . $name . "," . $existingValue . "," . $newValue;
}

Script to copy contents of file between two specific line patterns

Here is the script code to copy contents of files between two line patterns.

For example:
To copy the contents of file starting between the particular date and end date.

if [ $# -ne 3 ]; then
echo 1>&2 Usage: $0 file_to_parsed starting_pattern ending_pattern
exit 127
fi

startline=`grep -ni -m1 $2 $1 | awk -F ":" '{print $1}'`
endline=`grep -ni $3 $1 | tail -n1 | awk -F ":" '{print $1}'`

totallineno=`expr $endline - $startline + 1`
echo "Extracting contents between" $2 " and " $3
echo "No of lines found: " $totallineno
echo "Copying contents to temp file: " temp_file
echo "`head -n$endline $1 | tail -n $totallineno`" > temp_file
echo "Done copying."

AWK to calculate sum, min, max of particular field in txt

Recently, I came across situation to calculate the sum, min, max of particular field using awk.
Below is the snippet I've used to calculate the field values.

awk 'BEGIN{
FS=OFS=",";
max = -999999999; min = 9999999999;
print "----------------------------"
}
{
sub(" Total requests: ","",$2);
val = int($2);
lines+=1;sum+=val;
if( val > max ) max = val;
if ( val < min ) min = val;
}
END{
print "| Total No of Lines: "lines " Sum: " sum " Average: " sum/lines " Max: " max " Min: " min
}' with_out.txt

Text File Contents for parsing:

09/Sep/2008:03:57:09, Total requests: 36, New requests: 34
09/Sep/2008:03:57:10, Total requests: 61, New requests: 58
09/Sep/2008:03:57:11, Total requests: 40, New requests: 38
09/Sep/2008:03:57:12, Total requests: 101, New requests: 97
09/Sep/2008:03:57:13, Total requests: 33, New requests: 35