It's also worth looking at SystemTap or DTrace, depending on what OS you're running. While strace will allow you to look at an individual process and its children, SystemTap/DTrace will allow you to gather data on system call (and then some) usage system wide. Some examples:
- monitor execve() calls system-wide
- monitor I/O to a specific file, from any process
- measure per-process network usage
(note that newer Linux kernels may have other ways of accomplishing some of these tasks that I'm not aware of).
I've had a lot of success using SystemTap to look at low-level filesystem performance issues in the kernel. We've run SystemTap scripts on our production filesystem servers for over a year with no problems whatsoever.
- monitor execve() calls system-wide
- monitor I/O to a specific file, from any process
- measure per-process network usage
(note that newer Linux kernels may have other ways of accomplishing some of these tasks that I'm not aware of).
I've had a lot of success using SystemTap to look at low-level filesystem performance issues in the kernel. We've run SystemTap scripts on our production filesystem servers for over a year with no problems whatsoever.
Edit: formatting