[Mono-list] Mono Issue

Garikai Tachiwona (DSL AK) GarikaiT@datacom.co.nz
Tue, 12 Apr 2005 12:11:16 +1200


Hi Miguel

This is what we have:

Slackware Linux v10, with a 2.6.10 kernel
Mod_mono =3D 1.0.5
Xsp =3D 1.0.5

Process tree:

init-+-6*[agetty]
     |-crond
     |-events/0-+-aio/0
     |          |-kblockd/0
     |          |-khelper
     |          |-2*[pdflush]
     |          `-reiserfs/0
     |-httpd---7*[httpd]
     |-inetd
     |-klogd
     |-kseriod
     |-ksoftirqd/0
     |-kswapd0
     |-migration/0
     |-mono-+-mono
     |      `-mono---7*[mono]
     |-mysqld_safe---mysqld---mysqld---12*[mysqld]
     |-rpc.portmap
     |-sshd-+-sshd---sftp-server
     |      |-sshd---bash---pstree
     |      `-sshd---bash
     `-syslogd

TOP gives the following output:

top - 11:45:32 up 20:14,  2 users,  load average: 0.01, 0.07, 0.04
Tasks:  63 total,   1 running,  62 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.3% us,  0.7% sy,  0.0% ni, 98.0% id,  0.0% wa,  0.0% hi,
0.0% si
Mem:    515320k total,   443292k used,    72028k free,   209908k buffers
Swap:  1995760k total,        0k used,  1995760k free,   109556k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4631 nobody    16   0 61696  27m 7516 S  1.3  5.5   0:04.97 mono
 4621 nobody    15   0  5400 2136 1712 S  0.3  0.4   0:01.48 mono
 4739 root      16   0  1928 1008  796 R  0.3  0.2   0:00.21 top
    1 root      16   0   480  240  212 S  0.0  0.0   0:00.17 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
    4 root       5 -10     0    0    0 S  0.0  0.0   0:00.00 events/0
    5 root       5 -10     0    0    0 S  0.0  0.0   0:00.00 khelper
    6 root       5 -10     0    0    0 S  0.0  0.0   0:00.85 kblockd/0
    7 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
    8 root      15   0     0    0    0 S  0.0  0.0   0:07.21 pdflush
   10 root      13 -10     0    0    0 S  0.0  0.0   0:00.00 aio/0
    9 root      25   0     0    0    0 S  0.0  0.0   0:00.00 kswapd0
   11 root      25   0     0    0    0 S  0.0  0.0   0:00.00 kseriod
   12 root       5 -10     0    0    0 S  0.0  0.0   0:00.04 reiserfs/0
   70 root      16   0  1536  596  512 S  0.0  0.1   0:00.41 syslogd
   73 root      19   0  1468  452  392 S  0.0  0.1   0:00.00 klogd
  119 bin       24   0  1480  540  468 S  0.0  0.1   0:00.00 rpc.portmap
  125 root      24   0  1380  524  464 S  0.0  0.1   0:00.00 inetd
  128 root      17   0  3140 1408 1192 S  0.0  0.3   0:00.19 sshd
  140 root      16   0  1456  560  496 S  0.0  0.1   0:00.01 crond
  144 root      19   0  2160 1100  936 S  0.0  0.2   0:00.01 mysqld_safe
  145 root      16   0  1460  472  416 S  0.0  0.1   0:00.00 agetty
  146 root      16   0  1460  472  416 S  0.0  0.1   0:00.00 agetty
  147 root      16   0  1460  472  416 S  0.0  0.1   0:00.00 agetty
  148 root      16   0  1460  472  416 S  0.0  0.1   0:00.00 agetty
  149 root      16   0  1460  472  416 S  0.0  0.1   0:00.00 agetty
  150 root      16   0  1460  472  416 S  0.0  0.1   0:00.00 agetty
  167 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:00.15 mysqld
  168 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:00.00 mysqld
  169 mysql     20   0 49932  15m 3740 S  0.0  3.0   0:00.00 mysqld
  170 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:00.00 mysqld
  171 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:00.00 mysqld
  172 mysql     20   0 49932  15m 3740 S  0.0  3.0   0:00.00 mysqld
  173 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:04.79 mysqld
  174 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:00.22 mysqld
  175 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:00.00 mysqld
  176 mysql     16   0 49932  15m 3740 S  0.0  3.0   0:00.10 mysqld
  186 root      16   0  6476 2892 2024 S  0.0  0.6   0:00.90 httpd
 4534 root      15   0  5756 1656 1384 S  0.0  0.3   0:00.02 sshd
 4536 root      16   0  2964  952  748 S  0.0  0.2   0:00.01 sftp-server
 4540 root      16   0  5832 1696 1408 S  0.0  0.3   0:03.97 sshd
 4542 root      16   0  2656 1544 1128 S  0.0  0.3   0:00.04 bash
 4555 root      15   0  5752 1676 1408 S  0.0  0.3   0:00.05 sshd
 4557 root      16   0  2656 1540 1124 S  0.0  0.3   0:00.04 bash
 4613 nobody    15   0  6644 3140 2120 S  0.0  0.6   0:00.03 httpd
 4614 nobody    16   0  6592 3084 2120 S  0.0  0.6   0:00.00 httpd
 4615 nobody    15   0  6632 3128 2120 S  0.0  0.6   0:00.02 httpd

As for the strace log, we will provide this when mono next fails.

Regards

Garikai

-----Original Message-----
From: Miguel de Icaza [mailto:miguel@ximian.com]=20
Sent: Tuesday, 12 April 2005 10:19 a.m.
To: Garikai Tachiwona (DSL AK)
Cc: mono-list@lists.ximian.com; Paolo Molaro; Mark Prins (DSL AK); Wayne
Opai (DSL AK)
Subject: Re: [Mono-list] Mono Issue

Hello,

> The background to our problem is that we are deploying a large website
> on mono for a customer, and have the server 'freeze' about 1-2  times
> a day. At the freeze points mono takes up between 15-55% of  memory on
> the web server, typically consumes around 3% of CPU, and the  website
> becomes responsive again if the mono processes are restarted.
>=20
> =20
>=20
> The freeze is completely random. When we run strace we get copius
> output into the trace logs resulting in a large build-up in the log
> size. We are therefore forced to stop the trace because of this. Gdb
> kills the running application, so we don't use it.
>=20
> =20
>=20
> Is there another way of running the trace for days on-end without
> producing voluminous (and perhaps unnecessary) data?

Could you provide the following information:

	* Operating system.

	* Mono version, xsp version, mod_mono version.

	* Process list, process sizes.
=09
	* If you strace the hung process, what do you get?  A few lines
	  will be fine.

Miguel.