*****************************************************************************
* Description: Troubleshoot SQL*Plus login hanging via truss on AIX platform
* Date: 11:56 PM EST, 04/01/2018
*****************************************************************************

		 
<1> After launching "sqlplus username/password" on AIX command line, the interface hanging for a long time before logging in:
     |
     |__ o. In addition, "tnsping" and "lsnrctl" hanging for a long period as same.



	 

<2> Using "truss" to trace the system call for command "tnsping database_tns" in AIX system, and review dump file "truss_tnsping.lis":
     |
     |__ $ truss -o /home/oracle/truss_tnsping.lis -f -e -a -d tnsping database_tns
     |
     |__ o. Within the output, we can see process sleep at "/etc/netsvc.conf" for 13 seconds. The cause is DNS naming resolving method is not optimized.
     |
     |__ o. The /etc/netsvc.conf file is specifying the ordering of name resolution for gethostbyname, gethostbyname2, gethostbyaddr routines and alias resolution for the "sendmail".

	 
	 

                 38797490: 64815333: 0.6711:        kopen("/etc/resolv.conf", O_RDONLY) = 8
                 38797490: 64815333: 0.6713:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
                 38797490: 64815333: 0.6714:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
                 38797490: 64815333: kread(8, " n a m e s e r v e r    ".., 4096) = 150
                 38797490: 64815333: kread(8, " n a m e s e r v e r    ".., 4096) = 0
                 38797490: 64815333: 0.6718:        statx("/etc/resolv.conf", 0x0FFFFFFFFFFF66E8, 176, 0) = 0
                 38797490: 64815333: 0.6719:        close(8)             = 0
                 38797490: 64815333: 0.6721:        socket(1, 1, 0)      = 8
                 38797490: 64815333: 0.6723:        kfcntl(8, F_SETFD, 0x0000000000000001) = 0
                 38797490: 64815333: 0.6724:        connext(8, 0x0FFFFFFFFFFF61B8, 1025) Err#2  ENOENT
                 38797490: 64815333: 0.6725:        close(8)             = 0
                 38797490: 64815333: 0.6727:        kopen("/etc/netsvc.conf", O_RDONLY) = 8
                 38797490: 64815333: 0.6729:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
                 38797490: 64815333: 0.6730:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
                 38797490: 64815333: kread(8, " #   @ ( # ) 4 3        ".., 4096) = 4096
                 38797490: 64815333: kread(8, " o n   a n d   r e s o l".., 4096) = 620
                 38797490: 64815333: kread(8, " o n   a n d   r e s o l".., 4096) = 0
                 38797490: 64815333: 0.6735:        close(8)             = 0
                 38797490: 64815333: 0.6737:        kopen("/etc/irs.conf", O_RDONLY) Err#2  ENOENT
                 38797490: 64815333: 0.6738:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6740:        getdomainname(0x09001000A00DD7D0, 1024) = 0
                 38797490: 64815333: 0.6741:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6743:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6744:        __libc_sbrk(0x0000000000010020) = 0x0000000110389980
                 38797490: 64815333: 0.6746:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6748:        getdomainname(0x09001000A00DD7D0, 1024) = 0
                 38797490: 64815333: 0.6749:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6750:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6752:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6753:        kopen("/etc/hesiod.conf", O_RDONLY) Err#2  ENOENT
                 38797490: 64815333: 0.6755:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6756:        getdomainname(0x09001000A00DD7D0, 1024) = 0
                 38797490: 64815333: 0.6757:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6759:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6760:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6761:        getdomainname(0x09001000A00DD7D0, 1024) = 0
                 38797490: 64815333: 0.6763:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6764:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6765:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6767:        getdomainname(0x09001000A00DD7D0, 1024) = 0
                 38797490: 64815333: 0.6768:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6769:        _thread_self()               = 64815333
                 38797490: 64815333: 0.6771:        __libc_sbrk(0x0000000000010020) = 0x00000001103999A0
                 38797490: 64815333: 0.6772:        socket(2, 2, 0)      = 8
                 38797490: 64815333: 0.6774:        getsockopt(8, 65535, 4104, 0x0FFFFFFFFFFF51C4, 0x0FFFFFFFFFFF51C0) = 0
                 38797490: 64815333: 0.6775:        connext(8, 0x09001000A0023838, 16) = 0
                 38797490: 64815333: 0.6777:        _esend(8, 0x0FFFFFFFFFFF60D0, 47, 0, 0x0000000000000000) = 47
                 38797490: 64815333: 0.6778:        _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 1
                 38797490: 64815333: 0.6857:        _enrecvfrom(8, 0x0FFFFFFFFFFF7400, 1024, 0, 0x0FFFFFFFFFFF5A50, 0x0FFFFFFFFFFF5278, 0x0000000000000000) = 123
                 38797490: 64815333: 0.6859:        _esend(8, 0x0FFFFFFFFFFF60D0, 31, 0, 0x0000000000000000) = 31
                 38797490: 64815333: 0.6860:        _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 1
                 38797490: 64815333: 0.6931:        _enrecvfrom(8, 0x0FFFFFFFFFFF7400, 1024, 0, 0x0FFFFFFFFFFF5A50, 0x0FFFFFFFFFFF5278, 0x0000000000000000) = 31
                 38797490: 64815333: 0.6932:        close(8)             = 0
                 38797490: 64815333: 0.6934:        socket(2, 2, 0)      = 8
                 38797490: 64815333: 0.6935:        _esendto(8, 0x0FFFFFFFFFFF60D0, 31, 0, 0x09001000A0023848, 16, 0x0000000000000000) = 31
                 24051790: 56754361: 2.6656:        kread(8, "\0 -\0\006\0\0\0\0\003 ;".., 8208) (Sleeping...)
                 38797490: 64815333: 2.6937:        _poll(0x0FFFFFFFFFFF5290, 1, 5000) (sleeping...)
                 38797490: 64815333: 2.6937:        _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 0
                 38797490: 64815333: 5.6939:        close(8)             = 0
                 38797490: 64815333: 5.6940:        socket(2, 2, 0)      = 8
                 38797490: 64815333: 5.6942:        _esendto(8, 0x0FFFFFFFFFFF60D0, 31, 0, 0x09001000A0023858, 16, 0x0000000000000000) = 31
                 38797490: 64815333: 5.6943:        _poll(0x0FFFFFFFFFFF5290, 1, 5000) = 1
                 38797490: 64815333: 5.6949:        _enrecvfrom(8, 0x0FFFFFFFFFFF7400, 1024, 0, 0x0FFFFFFFFFFF5A50, 0x0FFFFFFFFFFF5278, 0x0000000000000000) = 106
                 38797490: 64815333: 5.6950:        close(8)             = 0
	 
	 
	 
	 
	 
<3> Solution:
     |
     |__ o. Adding following line within "/etc/netsvc.conf". 
     |
     |
     |                 hosts = local,bind4
     |
     |
     |__ o. This parameter will tell db connection to seek local /etc/hosts to resolve IPv4 only, instead of going to DNS server to resolve the name, which takes time.