Profiling and Tracing Dynamic Library Usage Via Interposition

Run-time resolution of library functions provides a rich and powerful opportunity to collect workload profiles and function/parameter trace information without source, special compilation, or special linking. This can be accomplished by having the linker resolve library functions to special wrapper functions that collect statistics before and after calling the real library function, leaving both the application and real library unaltered. The set of dynamic libraries is quite large including interesting libraries like libc (the C library and Operating System interface), graphics, database, network interface, and many more. Coupling this with the ability to simultaneously trace multiple processes on multiple processors covering both client and server processes yields tremendous feedback. We have found the amount of detailed information that can be gathered has been useful in many stages of the project lifecycle including the design, development, tuning, and sustaining of hardware, libraries, and applications. This paper first contrasts our extended view of interposition to other profiling, tracing, and interposing techniques. This is followed by a description and sample output of tools developed around this view; a discussion of obstacles encountered developing the tools; and finally, a discussion of anticipated and unanticipated ways those tools have been applied.