 |
 |
*****************************************************************************
* Description: Troubleshoot SQL*Plus login hanging via truss on AIX platform
* Date: 11:56 PM EST, 04/01/2018
*****************************************************************************
<1> After launching "sqlplus username/password" on AIX command line, the interface hanging for a long time before logging in:
|
|__ o. In addition, "tnsping" and "lsnrctl" hanging for a long period as same.
<2> Using "truss" to trace the system call for command "tnsping database_tns" in AIX system, and review dump file "truss_tnsping.lis":
|
|__ $ truss -o /home/oracle/truss_tnsping.lis -f -e -a -d tnsping database_tns
|
|__ $ strace lsnrctl status ....... This is the command for RHEL.
|
|__ o. Within the output, we can see process sleep at "/etc/netsvc.conf" for 13 seconds. The cause is DNS naming resolving method is not optimized.
|
|__ o. The /etc/netsvc.conf file is specifying the ordering of name resolution for gethostbyname, gethostbyname2, gethostbyaddr routines and alias resolution for the "sendmail".
38797490: 64815333: 0.6711: kopen("/etc/resolv.conf", O_RDONLY) = 8
38797490: 64815333: 0.6713: kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
38797490: 64815333: 0.6714: kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
38797490: 64815333: kread(8, " n a m e s e r v e r ".., 4096) = 150
38797490: 64815333: kread(8, " n a m e s e r v e r ".., 4096) = 0
38797490: 64815333: 0.6718: statx("/etc/resolv.conf", 0x0FFFFFFFFFFF66E8, 176, 0) = 0
38797490: 64815333: 0.6719: close(8) = 0
38797490: 64815333: 0.6721: socket(1, 1, 0) = 8
38797490: 64815333: 0.6723: kfcntl(8, F_SETFD, 0x0000000000000001) = 0
38797490: 64815333: 0.6724: connext(8, 0x0FFFFFFFFFFF61B8, 1025) Err#2 ENOENT
38797490: 64815333: 0.6725: close(8) = 0
38797490: 64815333: 0.6727: kopen("/etc/netsvc.conf", O_RDONLY) = 8
38797490: 64815333: 0.6729: kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
38797490: 64815333: 0.6730: kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
38797490: 64815333: kread(8, " # @ ( # ) 4 3 ".., 4096) = 4096
38797490: 64815333: kread(8, " o n a n d r e s o l".., 4096) = 620
38797490: 64815333: kread(8, " o n a n d r e s o l".., 4096) = 0
38797490: 64815333: 0.6735: close(8) = 0
38797490: 64815333: 0.6737: kopen("/etc/irs.conf", O_RDONLY) Err#2 ENOENT
38797490: 64815333: 0.6738: _thread_self() = 64815333
38797490: 64815333: 0.6740: getdomainname(0x09001000A00DD7D0, 1024) = 0
38797490: 64815333: 0.6741: _thread_self() = 64815333
38797490: 64815333: 0.6743: _thread_self() = 64815333
38797490: 64815333: 0.6744: __libc_sbrk(0x0000000000010020) = 0x0000000110389980
38797490: 64815333: 0.6746: _thread_self() = 64815333
38797490: 64815333: 0.6748: getdomainname(0x09001000A00DD7D0, 1024) = 0
38797490: 64815333: 0.6749: _thread_self() = 64815333
38797490: 64815333: 0.6750: _thread_self() = 64815333
38797490: 64815333: 0.6752: _thread_self() = 64815333
38797490: 64815333: 0.6753: kopen("/etc/hesiod.conf", O_RDONLY) Err#2 ENOENT
38797490: 64815333: 0.6755: _thread_self() = 64815333
38797490: 64815333: 0.6756: getdomainname(0x09001000A00DD7D0, 1024) = 0
38797490: 64815333: 0.6757: _thread_self() = 64815333
38797490: 64815333: 0.6759: _thread_self() = 64815333
38797490: 64815333: 0.6760: _thread_self() = 64815333
38797490: 64815333: 0.6761: getdomainname(0x09001000A00DD7D0, 1024) = 0
38797490: 64815333: 0.6763: _thread_self() = 64815333
38797490: 64815333: 0.6764: _thread_self() = 64815333
38797490: 64815333: 0.6765: _thread_self() = 64815333
38797490: 64815333: 0.6767: getdomainname(0x09001000A00DD7D0, 1024) = 0
38797490: 64815333: 0.6768: _thread_self() = 64815333
38797490: 64815333: 0.6769: _thread_self() = 64815333
38797490: 64815333: 0.6771: __libc_sbrk(0x0000000000010020) = 0x00000001103999A0
38797490: 64815333: 0.6772: socket(2, 2, 0) = 8
38797490: 64815333: 0.6774: getsockopt(8, 65535, 4104, 0x0FFFFFFFFFFF51C4, 0x0FFFFFFFFFFF51C0) = 0
38797490: 64815333: 0.6775: connext(8, 0x09001000A0023838, 16) = 0
38797490: 64815333: 0.6777: _esend(8, 0x0FFFFFFFFFFF60D0, 47, 0, 0x0000000000000000) = 47
38797490: 64815333: 0.6778: _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 1
38797490: 64815333: 0.6857: _enrecvfrom(8, 0x0FFFFFFFFFFF7400, 1024, 0, 0x0FFFFFFFFFFF5A50, 0x0FFFFFFFFFFF5278, 0x0000000000000000) = 123
38797490: 64815333: 0.6859: _esend(8, 0x0FFFFFFFFFFF60D0, 31, 0, 0x0000000000000000) = 31
38797490: 64815333: 0.6860: _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 1
38797490: 64815333: 0.6931: _enrecvfrom(8, 0x0FFFFFFFFFFF7400, 1024, 0, 0x0FFFFFFFFFFF5A50, 0x0FFFFFFFFFFF5278, 0x0000000000000000) = 31
38797490: 64815333: 0.6932: close(8) = 0
38797490: 64815333: 0.6934: socket(2, 2, 0) = 8
38797490: 64815333: 0.6935: _esendto(8, 0x0FFFFFFFFFFF60D0, 31, 0, 0x09001000A0023848, 16, 0x0000000000000000) = 31
24051790: 56754361: 2.6656: kread(8, "\0 -\0\006\0\0\0\0\003 ;".., 8208) (Sleeping...)
38797490: 64815333: 2.6937: _poll(0x0FFFFFFFFFFF5290, 1, 5000) (sleeping...)
38797490: 64815333: 2.6937: _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 0
38797490: 64815333: 5.6939: close(8) = 0
38797490: 64815333: 5.6940: socket(2, 2, 0) = 8
38797490: 64815333: 5.6942: _esendto(8, 0x0FFFFFFFFFFF60D0, 31, 0, 0x09001000A0023858, 16, 0x0000000000000000) = 31
38797490: 64815333: 5.6943: _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 1
38797490: 64815333: 5.6949: _enrecvfrom(8, 0x0FFFFFFFFFFF7400, 1024, 0, 0x0FFFFFFFFFFF5A50, 0x0FFFFFFFFFFF5278, 0x0000000000000000) = 106
38797490: 64815333: 5.6950: close(8) = 0
<3> Solution:
|
|__ o. Adding following line within "/etc/netsvc.conf".
|
|
| hosts = local,bind4
|
|
|__ o. This parameter will tell db connection to seek local /etc/hosts to resolve IPv4 only, instead of going to DNS server to resolve the name, which takes time.
Your Comments