Monitoring User and Application Activity with psacct


Monitoring User and Application Activity with psacct
One of the big advantages of using psacct on your server is that it provides excellent logging for activities of applications and users. When you are running scripts one of the important aspects of that script is how much resources it may be using and are there any resource limitations that may exist with the application. In addition, there may be times when you run a script as a user. In other words, you create a user with specific rights, maybe even using visudo. You will likely use this to reduce the security risks of a user who must issue a command with root privileges.

Install Process Accounting

# yum install psacct

Start Process Accounting

# /etc/init.d/psacct start
Starting process accounting:                               [  OK  ]

Connect Time
The connect time in hours is based on logins and logouts. The ac command provides a total.

# ac
total     1268.26

Accounting By Day
The system’s default login accounting file is /var/log/wtmp.

# ac -d

Oct 30  total        2.87
Oct 31  total        4.52
Nov  2  total        0.04
Nov  5  total        3.37
Nov  6  total       10.39
Nov  7  total       11.65
Nov  8  total        5.09
Nov 10  total        0.89
Nov 11  total        7.02
Nov 12  total        5.16
Nov 13  total        0.30
Nov 18  total       11.65
Nov 19  total        1.58
Nov 20  total        8.20
Nov 23  total        2.34
Nov 26  total        0.25
Nov 27  total        3.49
Dec  2  total        0.93
Today   total        2.45

Time Totals for Users

# ac -p
        yak                             8.09
        nagios                               0.04
        haywire                              33.76
        hatti                             12.93
        hacker                             334.98
        geddy                            30.89
        usayg                             198.59
        amar                                 0.12
        langoor                             13.82
        aanta                               18.00
        nildana                            105.30
        batley                                 0.00
        maka                              7.94
        hunter                               85.02
        gai                             416.38
        dhon                              2.42
        total     1268.27

Commands of Users
You can search out the commands of users with the lastcomm command which prints out the previously executed commands.

Process  Flag    Username    Terminal   Time
ping     S       dhon        pts/3      0.00 secs Thu Nov 30 18:09

# lastcomm dhon

hostname                dhon   pts/1      0.00 secs Mon Dec  3 18:41
bash               F    dhon   pts/1      0.00 secs Mon Dec  3 18:41
id                      dhon   pts/1      0.00 secs Mon Dec  3 18:41
su                S     dhon   __         0.02 secs Mon Dec  3 10:58
bash                  X dhon   __         0.04 secs Mon Dec  3 10:58
sshd              SF    dhon   __         0.04 secs Mon Dec  3 10:58

Search Logs for Commands
Using the lastcomm command you will be able to view each use of an individual command.

# lastcomm grep

grep                    aanta     pts/6      0.00 secs Thu Nov 30 13:28
grep                    aanta     pts/6      0.00 secs Thu Nov 30 13:28
grep                    aanta     pts/5      0.00 secs Thu Nov 30 12:57
grep                    aanta     pts/5      0.00 secs Thu Nov 30 12:57

Print Summary
The sa command will print a summary of commands that were executed. It will also condense the information into a summary file called savacct which contains the number of times that the command was executed. The useracct file keeps a summary of the commands by user.

Output Fields
cpu   -  sum of system and user time in cpu minutes
re    -  actual time in minutes
k     -  cpu-time averaged core usage, in 1k units
k*sec -  cpu storage integral (kilo-core seconds)
u     -  user cpu time in cpu minutes
s     -  system time in cpu minutes

# /usr/sbin/sa

Print User Information
Use the -u option to provide information on individual users.

# /usr/sbin/sa -u

root       0.00 cpu      598k mem accton
root       0.00 cpu     1081k mem initlog
root       0.00 cpu      920k mem initlog
root       0.00 cpu     1172k mem touch
root       0.00 cpu     1402k mem psacct
bomb       0.01 cpu     7282k mem kdeinit          *
bomb       0.00 cpu     6232k mem gnome-panel      *
bomb       0.02 cpu     4848k mem gnome-terminal

Display Number of Processes
An increase in these fields indicates a problem. This prints the number of processes and the number of CPU minutes. If these numbers continue to increase it is time to look into what is happening.

# /usr/sbin/sa -m

195         220.31re           0.09cp     2220k
aanta                                  65          198.37re           0.08cp     2135k
root                                  88          21.86re           0.00cp     1084k
postgres                              40          0.09re           0.00cp     4879k
smmsp                                 2           0.00re           0.00cp     1827k

Display All Names
This option will show each of the programs on your server so you may evaluate, real time, memory usage and which programs are running.

# /usr/sbin/sa -a

221      83.36re       0.01cp     1414k
1       0.01re       0.00cp     1471k   rpmq
7       0.33re       0.00cp     2465k   sendmail*
1      40.78re       0.00cp     1844k   sshd
37       0.00re       0.00cp      964k   bash*
32       0.00re       0.00cp      604k   tmpwatch
27       0.00re       0.00cp     4984k   postmaster*
26       0.00re       0.00cp     1116k   df
15       0.00re       0.00cp      959k   id
11       0.00re       0.00cp      709k   egrep
8       0.00re       0.00cp      636k   sa
7       0.00re       0.00cp      817k   grep
6       0.00re       0.00cp      562k   ac
5       0.01re       0.00cp      789k   awk
3       0.41re       0.00cp     1219k   crond*
3       0.40re       0.00cp      674k   run-parts
3       0.00re       0.00cp      774k   dircolors
3       0.00re       0.00cp      673k   consoletype
2      40.98re       0.00cp     1344k   bash
2       0.14re       0.00cp     1628k   sshd*
2       0.00re       0.00cp      914k   logrotate

# /usr/sbin/sa -a  It will sort the programs in percentage distributions.

How To Capture Packets with TCPDUMP?

See the list of interfaces on which tcpdump can listen
# /usr/sbin/tcpdump -D

Listen on any available interface
# /usr/sbin/tcpdump -i any

Verbose Mode
# /usr/sbin/tcpdump -v
# /usr/sbin/tcpdump -vv
# /usr/sbin/tcpdump -vvv
# /usr/sbin/tcpdump -q

Limit the capture to an number of  packets N
# /usr/sbin/tcpdump -c N

Display IP addresses and port numbers when capturing packets
# /usr/sbin/tcpdump -n

Capture any packets where the destination host is 192.168.0.1, display IP addresses and port numbers
# /usr/sbin/tcpdump -n dst host 192.168.0.1

Capture any packets where the source host is 192.168.0.1, display IP addresses and port numbers
# /usr/sbin/tcpdump -n src host 192.168.0.1

Capture any packets where the source or destination host is 192.168.0.1, display IP addresses and port numbers
# /usr/sbin/tcpdump -n host 192.168.0.1

Capture any packets where the destination network is 192.168.10.0/24, display IP addresses and port numbers
# /usr/sbin/tcpdump -n dst net 192.168.10.0/24

Capture any packets where the source network is 192.168.10.0/24, display IP addresses and port numbers
# /usr/sbin/tcpdump -n src net 192.168.10.0/24

Capture any packets where the source or destination network is 192.168.10.0/24,display IP addresses and port numbers
# /usr/sbin/tcpdump -n net 192.168.10.0/24

Capture any packets where the destination port is 23, display IP addresses and port numbers
# /usr/sbin/tcpdump -n dst port 23

Capture any packets where the destination port is is between 1 and 1023 inclusive, display IP addresses and port numbers
# /usr/sbin/tcpdump -n dst portrange 1-1023

Capture only TCP packets where the destination port is is between 1 and 1023 inclusive,display IP addresses and port numbers
# /usr/sbin/tcpdump -n tcp dst portrange 1-1023

Capture only UDP packets where the destination port is is between 1 and 1023 inclusive, display IP addresses and port numbers
# /usr/sbin/tcpdump -n udp dst portrange 1-1023

Capture any packets with destination IP 192.168.0.1 and destination port 23,display IP addresses and port numbers
# /usr/sbin/tcpdump -n "dst host 192.168.0.1 and dst port 23"

Capture any packets with destination IP 192.168.0.1 and destination port 80 or 443,display IP addresses and port numbers
# /usr/sbin/tcpdump -n "dst host 192.168.0.1 and (dst port 80 or dst port 443)"

Capture any ICMP packets
# /usr/sbin/tcpdump -v icmp

Capture any ARP packets
# /usr/sbin/tcpdump -v arp

Capture either ICMP or ARP packets
# /usr/sbin/tcpdump -v "icmp or arp"

Capture any packets that are broadcast or multicast
# /usr/sbin/tcpdump -n "broadcast or multicast"

Capture 500 bytes of data for each packet rather than the default of 68 bytes
# /usr/sbin/tcpdump -s 500

Capture all bytes of data within the packet
# /usr/sbin/tcpdump -s 0

Monitor all packets on eth1 interface
# /usr/sbin/tcpdump -i eth1

Monitor all traffic on port 80 ( HTTP )
# /usr/sbin/tcpdump -i eth0 'port 80'

Monitor all traffic on port 25 ( SMTP )
# /usr/sbin/tcpdump -vv -x -X -s 1500 -i eth0 'port 25'

Capture only N number of packets using tcpdump -c
# /usr/sbin/tcpdump -c 2 -i eth0

Display Captured Packets in ASCII using tcpdump -A
# /usr/sbin/tcpdump -A -i eth0

Display Captured Packets in HEX and ASCII using tcpdump -XX
# /usr/sbin/tcpdump -XX -i eth0

Capture the packets and write into a file using tcpdump -w
# /usr/sbin/tcpdump -w data.pcap -i eth0
.pcap is extension

Reading the packets from a saved file using tcpdump -r
# /usr/sbin/tcpdump -tttt -r data.pcap

Capture packets with IP address using tcpdump -n
# /usr/sbin/tcpdump -n -i eth0

Capture packets with proper readable timestamp using tcpdump -tttt
# /usr/sbin/tcpdump -n -tttt -i eth0

Read packets longer than N bytes
# /usr/sbin/tcpdump -w data.pcap greater 1024

Read packets lesser than N bytes
# /usr/sbin/tcpdump -w data1024.pcap  less 1024

Receive only the packets of a specific protocol type
# /usr/sbin/tcpdump -i eth0 arp

Receive packets flows on a particular port using tcpdump port
# /usr/sbin/tcpdump -i eth0 port 22

Capture packets for particular destination IP and Port
# /usr/sbin/tcpdump -w data.pcap -i eth0 dst 10.181.140.216 and port 22

Capture TCP communication packets between two hosts
# /usr/sbin/tcpdump -w data.pcap -i eth0 dst 16.181.170.246 and port 22

Tcpdump Filter Packets – Capture all the packets other than arp and rarp
# /usr/sbin/tcpdump -i eth0 not arp and not rarp

How to change the linux hostname?


# hostname
test.com

# hostname server.com

# hostname
server.com

# vi /etc/hostname
server.com

Now restart and see the changes.

How to change MySql root password?

For every database, you should set the root or sa passwords to something other than the default, unless you want to get hacked. For mysql, the system administrator user is called root. You will use the mysqladmin utility from a command line to set the new password.

Syntax:

# mysqladmin -u root password “new_password”

# mysqladmin -u root -h host_name password “new_password”

Example:

# mysqladmin -u root password Pa55w0rD

# mysqladmin -u root -h localhost password linuxgEEks

You need to restart the database server after this change

# /etc/init.d/mysql restart

How To Backup MySQL Database to a file?

Backing up your database is a very important system administration task, and should generally be run from a cron job at scheduled intervals. We will use the mysqldump utility included with mysql to dump the contents of the database to a text file that can be easily re-imported.

Syntax:

# mysqldump -h localhost -u root -pmypassword database_name > dumpfile_name.sql

Example:

# mysqldump -h localhost -u root -pPa55w0rD database110 > backup_file.sql

This will give you a text file containing all the commands required to re-create the database.

How To Set SSH Login Message?

To set ssh login message, its very easy and interesting.
# vi /etc/motd
write the message of your own

######### Welcome to the SSH World #########
### This is the Email Server, please exit properly ###
########################################

Save and Quit

// To check quit the ssh terminal and re-login...


######### Welcome to the SSH World #########
### This is the Email Server, please exit properly ###
########################################

Thats all, Enjoy!!!!!

nmap in details

nmap is a tool to check the status of ports in any machine

Example1 : To scan a particular system for open ports
#nmap hostname

Example2 : Scanning for a single port on a machine
#nmap –p 22 hostname
–p indicates port.

Example3 : For scanning only ports
#nmap –F hostname

-F is for fast scan and this will not do any other scanning like IP address, hostname, operating system, and uptime etc.

Example4 : Scanning only TCP ports
#nmap –sT hostname
-s is for scanning and T is for only scanning of TCP ports

Example5 : Scanning only UDP ports
#nmap –sU hostname
-U indicates UDP port scanning

Exmaple6 : Scan for ports and get the version of different services running on that machine
#nmap –sV hostname
-V indicates version of each network service running on that host

Example7 : Check which protocol is supported by the remote machine
#nmap –sO hostname

Example8 : Scan a system for operating system and uptime details
# nmap -O hostname
-O is for operating system scan along with default port scan

Example9 : Scan a network
#nmap networkID/subnetmask
For the above command you can try in this way
#nmap x.x.x.x/24

Netstat in Linux

List all ports
# netstat -a | more

List all tcp ports using netstat -at
# netstat -at

List all udp ports using netstat -au
# netstat -au

List only listening ports
# netstat -l

List only listening TCP Ports using netstat -lt
# netstat -lt

List only listening UDP Ports using netstat -lu
# netstat -lu

List only the listening UNIX Ports using netstat -lx
# netstat -lx

Show statistics for all ports
# netstat -s

Show statistics for TCP/UDP ports
# netstat -st
# netstat -su

Display PID and program names
# netstat -pt

Don’t resolve host, port and user name
# netstat -an

Print netstat information continuously
# netstat -c

Find the non supportive Address families in your system
# netstat --verbose

Display the kernel routing information
# netstat -r

Find out on which port a program is running
# netstat -ap | grep ssh

Find out which process is using a particular port
# netstat -an | grep ':80'

Show the list of network interfaces
# netstat -i

Display extended information on the interfaces
# netstat -ie


How to install SendMailAnalyser in linux, centOS?

SendmailAnalyzer can work in any platform where Sendmail and Perl could run. What you need is a modern Perl distribution 5.8.x or more is good but older version should also work.

Download sendmailanalyzer-x.x.tar.gz and perform the following opertaions

# tar -zxvf sendmailanalyzer-x.x.tar.gz
# cd sendmailanalyzer-x.x/
# perl Makefile.PL
# make && make install

Start SendmailAnalyzer daemon:

# /usr/local/sendmailanalyzer/sendmailanalyzer -f

Add httpd configuratiosn for SendmailAnalyzer

Alias /sareport /usr/local/sendmailanalyzer/www

<Directory /usr/local/sendmailanalyzer/www>
Options ExecCGI
AddHandler cgi-script .cgi
DirectoryIndex sa_report.cgi
Order deny,allow
Deny from all
Allow from 127.0.0.1
Allow from ::1
# Allow from .example.com
</Directory>

Test:

http://server_ip_address/sareport

Additional tasks to be added in crontab

# SendmailAnalyzer log reporting daily cache
0 1 * * * /usr/local/sendmailanalyzer/sa_cache > /dev/null 2>&1
# On huge MTA you may want to have five minutes caching
#*/5 * * * * /usr/local/sendmailanalyzer/sa_cache -a > /dev/null 2>&1

Logrotate:

/etc/logrotate.d/syslog to restart SendmailAnalyzer when maillog is rotated or create a cron job.

For example:

/var/log/cron /var/log/debug /var/log/maillog /var/log/messages /var/log/secure /var/log/spooler /var/log/syslog
{
   sharedscripts
   postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2>/dev/null` 2>/dev/null || true
/PATH_TO/rc.sendmailanalyzer restart >/dev/null 2>&1 || true
# or /etc/rc.d/init.d/sendmailanalyzer restart >/dev/null 2>&1 || true
   endscript
}

How to install darkstat in linux, centOS?

Darkstat - Web Based Network Traffic & Bandwidth Monitoring Tool on Linux

# yum install darkstat

# darkstat -i eth0

Test:

http://ip-address:667