Better Logging

I like to see more of what is going on than what CWP puts out by default.

Sitting behind Varnish I only see my shared IP being logged, to fix that I followed http://www.techstacks.com/howto/log-client-ip-and-xforwardedfor-ip-in-apache.html.

Here is the section of Apache Configuration I had to change, the existing settings I needed to comment out are led with ## and what i added are within ###:

Code: [Select]

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    ##LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    ##LogFormat "%h %l %u %t \"%r\" %>s %b" common

### Log format changes for X-Forwarded-For
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
CustomLog "logs/access_log" combined env=!forwarded
CustomLog "logs/access_log" proxy env=forwarded
###

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    ##CustomLog "logs/access_log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog "logs/access_log" combined
</IfModule>

Remember to restart Apache:

Code: [Select]

service httpd restart

Now if I tail '/usr/local/apache/logs/access_log' I see the client IPs.


Up next, named logs.

I like to see DNS queries against my servers. Even if there is nothing replied/served.

Following http://stackoverflow.com/a/12114139 I made these changes to '/etc/named.conf'

Code: [Select]

//logging {
//        channel default_debug {
//                file "data/named.run";
//                severity dynamic;
//        };
//};

logging {
    channel default_file {
        file "/var/log/named/default.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel general_file {
        file "/var/log/named/general.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel database_file {
        file "/var/log/named/database.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel security_file {
        file "/var/log/named/security.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel config_file {
        file "/var/log/named/config.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel resolver_file {
        file "/var/log/named/resolver.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel xfer-in_file {
        file "/var/log/named/xfer-in.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel xfer-out_file {
        file "/var/log/named/xfer-out.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel notify_file {
        file "/var/log/named/notify.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel client_file {
        file "/var/log/named/client.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel unmatched_file {
        file "/var/log/named/unmatched.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel queries_file {
        file "/var/log/named/queries.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel network_file {
        file "/var/log/named/network.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel update_file {
        file "/var/log/named/update.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel dispatch_file {
        file "/var/log/named/dispatch.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel dnssec_file {
        file "/var/log/named/dnssec.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };
    channel lame-servers_file {
        file "/var/log/named/lame-servers.log" versions 3 size 5m;
        severity dynamic;
        print-time yes;
    };

    category default { default_file; };
    category general { general_file; };
    category database { database_file; };
    category security { security_file; };
    category config { config_file; };
    category resolver { resolver_file; };
    category xfer-in { xfer-in_file; };
    category xfer-out { xfer-out_file; };
    category notify { notify_file; };
    category client { client_file; };
    category unmatched { unmatched_file; };
    category queries { queries_file; };
    category network { network_file; };
    category update { update_file; };
    category dispatch { dispatch_file; };
    category dnssec { dnssec_file; };
    category lame-servers { lame-servers_file; };
};

Then create the directory that will receive the logs and give proper permissions:

Code: [Select]

#mkdir /var/log/named
#chown -R named /var/log/named

Restart bind/named:

Code: [Select]

service named restart


Up next, log rotation.

I like to have busy logs rotated daily with a 30 day retention. Having to grep through a log that is a week old and a few GB can get painful.

In '/etc/logrotate.conf' I change 'weekly' to 'daily' and 'rotate 4' to 'rotate 30'

The configs I changed:

'/etc/logrotate.d/lfd'

Code: [Select]

/var/log/lfd.log {
    rotate 30
    daily
    missingok
    notifempty
    compress
    delaycompress
}

'/etc/logrotate.d/pure-ftpd'

Code: [Select]

/var/log/pureftpd.log {
    rotate 30
    daily
    missingok
    notifempty
    compress
    delaycompress
}

'/etc/logrotate.d/syslog'

Code: [Select]

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    rotate 30
    daily
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

This one I created for Apache.
'/etc/logrotate.d/apache'

Code: [Select]

/usr/local/apache/logs/*_log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        notifempty
        sharedscripts
        postrotate
                service httpd restart > /dev/null
        endscript
}

That's it for now.
Let me know if I screwed something up because I wrote this as I made the changes to a new/fresh install.

That sounds cool. I have not need about log others IP yet, but glad to know about Varnish proxy IP bypass.

Thank you for share.

I need to figure out how to get the client IP logged in mod-security. I'll post it here when I get it.

Hey thanks for this, that website was down so you can use : https://web.archive.org/web/20171128175855/http://www.techstacks.com/howto/log-client-ip-and-xforwardedfor-ip-in-apache.html

ty

After changing the logs to what you have (for awstats) I just had to run :
echo "" > /usr/local/apache/logs/access_log (to clear awstats for the new logs)