I’ve decided to write down a few useful tips on observing binary files in Solaris. The commands used are available since Solaris 8, so hopefully you’ll find these tricks useful.
Quite often it happens that the programs we run don’t work as they should, showing us errors like “referenced symbol not found” or this:
ld.so.1: ./thunderbird-bin: fatal: libmozjs.so: open failed: No such file or directory
So in order for us to find out what exact librabries are being used by a given binary, or where it expects these libraries to be found, and what libraries are not found at all, we have to take the following steps:
1. Determine what libraries are needed
First we have to see what libraries the ld linker is going to use for our file. For instance, this is how you would find our what libraries are used by the main executable of the Thunderbird mail client:
solaris$ ldd thunderbird-bin
libmozjs.so => (file not found)
libgtk-x11-2.0.so.0 => /usr/lib/libgtk-x11-2.0.so.0
libgdk-x11-2.0.so.0 => /usr/lib/libgdk-x11-2.0.so.0
libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0
libgdk_pixbuf-2.0.so.0 => /usr/lib/libgdk_pixbuf-2.0.so.0
libm.so.2 => /lib/libm.so.2
...
As always in my examples, the command output is abridged, but you get the idea. It’s clearly seen that one of the files – the libmozjs.so library – isn’t found. This means, there is no such file in either of the standard library paths or in the current working directory.
2. Confirm where dynamic libraries are expected
Now it’s about time we find our where the linker expects this libmozjs.so to be found. To do this, we’re going to use the truss command. All we’re interested in at this stage are the stat and open system calls: stat is used to verify whether some file exists, and open is used to open this file for the later access.
bash-3.00$ truss -f -t stat,open ./thunderbird-bin
13070: stat("/export/soft/thunderbird/thunderbird-bin", 0xFFBFFAD0) = 0
13070: open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT
13070: stat("/export/soft/thunderbird/libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070: stat("/export/soft/thunderbird/../libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070: stat("/usr/sfw/lib/libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070: stat("/opt/sfw/lib/libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070: stat("/usr/local/lib/libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070: stat("/usr/openwin/lib/libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070: stat("/lib/libc.so.1", 0xFFBFF600) = 0
13070: open("/lib/libc.so.1", O_RDONLY) = 3
13070: stat("/export/soft/thunderbird/libxpcom.so", 0xFFBFF418) = 0
13070: open("/export/soft/thunderbird/libxpcom.so", O_RDONLY) = 3
13070: stat("/export/soft/thunderbird/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070: stat("/export/soft/thunderbird/../libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070: stat("/usr/sfw/lib/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070: stat("/opt/sfw/lib/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070: stat("/usr/local/lib/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070: stat("/usr/openwin/lib/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070: stat("/lib/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070: stat("/usr/lib/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
ld.so.1: ./thunderbird-bin: fatal: libmozjs.so: open failed: No such file or directory
13070: stat("/export/soft/thunderbird/libgtk-x11-2.0.so.0", 0xFFBFF398) Err#2 ENOENT
...
From this output you can see very well how linker goes through one standard libraries directory to another in a cyclic search for ever library our binary needs. And for our libmozjs.so we can see that logically this library would be found in the current directory where I’m running the thunderbird-bin from, /export/soft/thunderbird. There is no such file there, simply because I’ve renamed it for this demo.
And this is how the same ldd command would look when all the files are in place:
bash-3.00$ ldd thunderbird-bin
libmozjs.so => /export/soft/thunderbird/libmozjs.so
libgtk-x11-2.0.so.0 => /usr/lib/libgtk-x11-2.0.so.0
libgdk-x11-2.0.so.0 => /usr/lib/libgdk-x11-2.0.so.0
libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0
libgdk_pixbuf-2.0.so.0 => /usr/lib/libgdk_pixbuf-2.0.so.0
libm.so.2 => /lib/libm.so.2
...
So you can be assured now that our library is found and loaded from the /export/soft/thunderbird/libmozjs.so location.
Thus, by using this simple yet very powerful combination of ldd and truss commands, you can stop guessing and find out exactly which libraries are loaded in a particular order and where they are loaded from.
3. Resolve the referenced symbols errors
For the “referenced symbol not found” errors, we have to take one more step. These errors happen mostly because the linker finds the library our binary needs, but this library somehow doesn’t have the functions we’re interested in. So, once we know the exact library file after following the steps 1 and 2 of this manual, it’s time for us to look at this particular file and see what functions are available. Many libraries have thousands of functions accessible, so it would make sense to use grep if you know the function you’re looking for.
For instance, the following command shows all the functions of the libgdk-x11-2.0 library, which have gdk_threads in their names:
solaris$ nm /usr/lib/libgdk-x11-2.0.so.0 | grep gdk_threads
[3632] | 86512| 68|FUNC |GLOB |0 |11 |gdk_threads_enter
[219] | 86648| 104|FUNC |LOCL |0 |11 |gdk_threads_impl_lock
[220] | 86752| 104|FUNC |LOCL |0 |11 |gdk_threads_impl_unlock
[4329] | 86856| 212|FUNC |GLOB |0 |11 |gdk_threads_init
[3507] | 86580| 68|FUNC |GLOB |0 |11 |gdk_threads_leave
[4095] | 532720| 4|OBJT |GLOB |0 |17 |gdk_threads_lock
[3579] | 532716| 4|OBJT |GLOB |0 |17 |gdk_threads_mutex
[4391] | 87068| 144|FUNC |GLOB |0 |11 |gdk_threads_set_lock_functions
[4443] | 532724| 4|OBJT |GLOB |0 |17 |gdk_threads_unlock
As you understand, if you get a “referenced symbol not found” error, this is quite often because an incorrect (incompatible) version of some library is used – it could be very old or just too new a version of the library, so the functions your binary is looking for are simply not there anymore or called differently.
Well, that’s all for today. If you have any additional info to add to this short manual, you’re very welcome to leave comments. I’ll be happy to hear your opinion.
I wish the truss command was easier to obtain debug information and that the output was easier to read.
From your above truss output , I can see that there is a permission issue
hi i am facing similar problem, while using “find” command for a particular user. even afer bounced the server..
i have put
truss find . 2>&1 | tail -20
>truss find . 2>&1
execve(“/usr/bin/find”, 0xFFBEF764, 0xFFBEF770) argc = 2
resolvepath(“/usr/lib/ld.so.1”, “/usr/lib/ld.so.1”, 1023) = 16
open(“/var/ld/ld.config”, O_RDONLY) Err#2 ENOENT
stat(“/apps/oracle/product/9.2.0/lib/libc.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/ljcms650/cognos/cer2/bin/libc.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/usr/dt/lib/libc.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/usr/openwin/lib/libc.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/usr/lib/libc.so.1”, 0xFFBEF010) = 0
resolvepath(“/usr/lib/libc.so.1”, “/usr/lib/libc.so.1”, 1023) = 18
open(“/usr/lib/libc.so.1”, O_RDONLY) = 3
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF3A0000
mmap(0x00000000, 802816, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF280000
mmap(0xFF280000, 703044, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF280000
mmap(0xFF33C000, 24780, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 704512) = 0xFF33C000
munmap(0xFF32C000, 65536) = 0
memcntl(0xFF280000, 113492, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat(“/apps/oracle/product/9.2.0/lib/libdl.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/ljcms650/cognos/cer2/bin/libdl.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/usr/dt/lib/libdl.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/usr/openwin/lib/libdl.so.1”, 0xFFBEF010) Err#2 ENOENT
stat(“/usr/lib/libdl.so.1”, 0xFFBEF010) = 0
resolvepath(“/usr/lib/libdl.so.1”, “/usr/lib/libdl.so.1”, 1023) = 19
open(“/usr/lib/libdl.so.1”, O_RDONLY) = 3
mmap(0xFF3A0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3A0000
mmap(0x00000000, 8192, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF390000
mmap(0xFF390000, 2638, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF390000
close(3) = 0
stat(“/usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1”, 0xFFBEED30) = 0
resolvepath(“/usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1”, “/usr/platform/sun4u-us3/lib/libc_psr.so.1”, 1023) = 41
open(“/usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1”, O_RDONLY) = 3
mmap(0xFF3A0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3A0000
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF380000
close(3) = 0
brk(0x000261C8) = 0
brk(0x000281C8) = 0
stat(“/usr/lib/locale/en_US.ISO8859-15/en_US.ISO8859-15.so.2”, 0xFFBEE968) = 0
resolvepath(“/usr/lib/locale/en_US.ISO8859-15/en_US.ISO8859-15.so.2”, “/usr/lib/locale/en_US.ISO8859-15/en_US.ISO8859-15.so.2”, 1023) = 54
open(“/usr/lib/locale/en_US.ISO8859-15/en_US.ISO8859-15.so.2”, O_RDONLY) = 3
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF370000
mmap(0x2F706C61, 90112, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF350000
mmap(0xFF350000, 16366, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF350000
mmap(0xFF362000, 10322, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 8192) = 0xFF362000
munmap(0xFF354000, 57344) = 0
memcntl(0xFF350000, 8416, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
munmap(0xFF370000, 8192) = 0
time() = 1271446665
pathconf(“.”, _PC_PATH_MAX) = 1024
stat64(“./”, 0xFFBEED08) = 0
stat64(“/”, 0xFFBEEC70) = 0
open64(“./../”, O_RDONLY|O_NDELAY) = 3
fcntl(3, F_SETFD, 0x00000001) = 0
fstat64(3, 0xFFBEE288) = 0
fstat64(3, 0xFFBEED08) = 0
getdents64(3, 0x000272E0, 1048) = 80
close(3) = 0
open64(“./../../”, O_RDONLY|O_NDELAY) = 3
fcntl(3, F_SETFD, 0x00000001) = 0
fstat64(3, 0xFFBEE288) = 0
fstat64(3, 0xFFBEED08) = 0
getdents64(3, 0x000272E0, 1048) = 240
close(3) = 0
open64(“./../../../”, O_RDONLY|O_NDELAY) Err#13 EACCES
brk(0x000281C8) = 0
brk(0x0002A1C8) = 0
open(“/usr/lib/locale/C/LC_MESSAGES/SUNW_OST_OSCMD”, O_RDONLY) Err#2 ENOENT
open(“/apps/teradata/messages”, O_RDONLY) Err#2 ENOENT
open(“/usr/lib/locale/C/LC_MESSAGES/SUNW_OST_OSCMD.mo”, O_RDONLY) Err#2 ENOENT
open(“/usr/lib/locale/C/LC_MESSAGES/SUNW_OST_OSLIB”, O_RDONLY) Err#2 ENOENT
open(“/usr/lib/locale/C/LC_MESSAGES/SUNW_OST_OSLIB.mo”, O_RDONLY) Err#2 ENOENT
findwrite(2, ” f i n d”, 4) = 4
: cannot open write(2, ” : c a n n o t o p e”.., 14) = 14
.: write(2, ” . : “, 3) = 3
Permission deniedwrite(2, ” P e r m i s s i o n d”.., 17) = 17
write(2, “\n”, 1) = 1
llseek(0, 0, SEEK_CUR) = 88586
_exit(1)
pls give some solution for the problem
Thanks!!! This tip just made my week. I was able to fix a problem that had the team stalled for over a month in less than a day. I’ve been soaking up the Internals book etc but this tip was clean, quick, and easy to understand. Thanks much!!!
This is howto add library with `crle`
https://solaris.reys.net/using-crle/
Thanks. It helps me a lot. BTW, *truss* in Solaris is the same tool as *strace* in Linux.