Monitoring User and Application Activity with psacct

Monitoring User and Application Activity with psacct
One of the big advantages of using psacct on your server is that it provides excellent logging for activities of applications and users. When you are running scripts one of the important aspects of that script is how much resources it may be using and are there any resource limitations that may exist with the application. In addition, there may be times when you run a script as a user. In other words, you create a user with specific rights, maybe even using visudo. You will likely use this to reduce the security risks of a user who must issue a command with root privileges.

Install Process Accounting

# yum install psacct

Start Process Accounting

# /etc/init.d/psacct start
Starting process accounting:                               [  OK  ]

Connect Time
The connect time in hours is based on logins and logouts. The ac command provides a total.

# ac
total     1268.26

Accounting By Day
The system’s default login accounting file is /var/log/wtmp.

# ac -d

Oct 30  total        2.87
Oct 31  total        4.52
Nov  2  total        0.04
Nov  5  total        3.37
Nov  6  total       10.39
Nov  7  total       11.65
Nov  8  total        5.09
Nov 10  total        0.89
Nov 11  total        7.02
Nov 12  total        5.16
Nov 13  total        0.30
Nov 18  total       11.65
Nov 19  total        1.58
Nov 20  total        8.20
Nov 23  total        2.34
Nov 26  total        0.25
Nov 27  total        3.49
Dec  2  total        0.93
Today   total        2.45

Time Totals for Users

# ac -p
        yak                             8.09
        nagios                               0.04
        haywire                              33.76
        hatti                             12.93
        hacker                             334.98
        geddy                            30.89
        usayg                             198.59
        amar                                 0.12
        langoor                             13.82
        aanta                               18.00
        nildana                            105.30
        batley                                 0.00
        maka                              7.94
        hunter                               85.02
        gai                             416.38
        dhon                              2.42
        total     1268.27

Commands of Users
You can search out the commands of users with the lastcomm command which prints out the previously executed commands.

Process  Flag    Username    Terminal   Time
ping     S       dhon        pts/3      0.00 secs Thu Nov 30 18:09

# lastcomm dhon

hostname                dhon   pts/1      0.00 secs Mon Dec  3 18:41
bash               F    dhon   pts/1      0.00 secs Mon Dec  3 18:41
id                      dhon   pts/1      0.00 secs Mon Dec  3 18:41
su                S     dhon   __         0.02 secs Mon Dec  3 10:58
bash                  X dhon   __         0.04 secs Mon Dec  3 10:58
sshd              SF    dhon   __         0.04 secs Mon Dec  3 10:58

Search Logs for Commands
Using the lastcomm command you will be able to view each use of an individual command.

# lastcomm grep

grep                    aanta     pts/6      0.00 secs Thu Nov 30 13:28
grep                    aanta     pts/6      0.00 secs Thu Nov 30 13:28
grep                    aanta     pts/5      0.00 secs Thu Nov 30 12:57
grep                    aanta     pts/5      0.00 secs Thu Nov 30 12:57

Print Summary
The sa command will print a summary of commands that were executed. It will also condense the information into a summary file called savacct which contains the number of times that the command was executed. The useracct file keeps a summary of the commands by user.

Output Fields
cpu   -  sum of system and user time in cpu minutes
re    -  actual time in minutes
k     -  cpu-time averaged core usage, in 1k units
k*sec -  cpu storage integral (kilo-core seconds)
u     -  user cpu time in cpu minutes
s     -  system time in cpu minutes

# /usr/sbin/sa

Print User Information
Use the -u option to provide information on individual users.

# /usr/sbin/sa -u

root       0.00 cpu      598k mem accton
root       0.00 cpu     1081k mem initlog
root       0.00 cpu      920k mem initlog
root       0.00 cpu     1172k mem touch
root       0.00 cpu     1402k mem psacct
bomb       0.01 cpu     7282k mem kdeinit          *
bomb       0.00 cpu     6232k mem gnome-panel      *
bomb       0.02 cpu     4848k mem gnome-terminal

Display Number of Processes
An increase in these fields indicates a problem. This prints the number of processes and the number of CPU minutes. If these numbers continue to increase it is time to look into what is happening.

# /usr/sbin/sa -m

195         220.31re           0.09cp     2220k
aanta                                  65          198.37re           0.08cp     2135k
root                                  88          21.86re           0.00cp     1084k
postgres                              40          0.09re           0.00cp     4879k
smmsp                                 2           0.00re           0.00cp     1827k

Display All Names
This option will show each of the programs on your server so you may evaluate, real time, memory usage and which programs are running.

# /usr/sbin/sa -a

221      83.36re       0.01cp     1414k
1       0.01re       0.00cp     1471k   rpmq
7       0.33re       0.00cp     2465k   sendmail*
1      40.78re       0.00cp     1844k   sshd
37       0.00re       0.00cp      964k   bash*
32       0.00re       0.00cp      604k   tmpwatch
27       0.00re       0.00cp     4984k   postmaster*
26       0.00re       0.00cp     1116k   df
15       0.00re       0.00cp      959k   id
11       0.00re       0.00cp      709k   egrep
8       0.00re       0.00cp      636k   sa
7       0.00re       0.00cp      817k   grep
6       0.00re       0.00cp      562k   ac
5       0.01re       0.00cp      789k   awk
3       0.41re       0.00cp     1219k   crond*
3       0.40re       0.00cp      674k   run-parts
3       0.00re       0.00cp      774k   dircolors
3       0.00re       0.00cp      673k   consoletype
2      40.98re       0.00cp     1344k   bash
2       0.14re       0.00cp     1628k   sshd*
2       0.00re       0.00cp      914k   logrotate

# /usr/sbin/sa -a  It will sort the programs in percentage distributions.