Author Topic: default.vcl discussion  (Read 4415 times)

0 Members and 1 Guest are viewing this topic.

Offline
**
default.vcl discussion
« on: December 31, 2015, 06:29:10 PM »
Like lot's of folks I installed varnish and saw no speed increase on my site. I switched from disk cache to malloc and still, little improvement with a huge miss rate. I started looking into vanish settings and here's what I cam up with. A little background: I'm running wordpress, joomla, and static web pages. I didn't really write this, I just took the most useful stuff from a number of sources (some of their comments I left intact, some are my own). Right now my hit rate is over 50%, the sites behave like they should (including admin access), apache logging works, everything seems good! Feedback is appreciated. If you're doing something different that's working awesome I'd love to check it out.

Code: [Select]
backend default { .host = "X.X.X.X"; .port = "8181";}
include "/etc/varnish/backends.vcl";
#set IP for apache logging
sub vcl_recv { include "/etc/varnish/sites.vcl";
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
# Setup grace mode.
  # Allow Varnish to serve up stale (kept around) content if the backend is
  #responding slowly or is down.
  # We accept serving 6h old object (plus its ttl)
  if (! req.backend.healthy) {
   set req.grace = 6h;
  } else {
   set req.grace = 15s;
  }
 
  # If our backend is down, unset all cookies and serve pages from cache.
  if (!req.backend.healthy) {
    unset req.http.Cookie;
  }

# Drop any cookies sent to Wordpress.
if(
        req.url ~ "^/administrator" ||
        req.url ~ "^/component/banners" ||
        req.url ~ "^/component/users" ||
        req.url ~ "^/wp-admin" ||
        req.url ~ "^/wp-login.php" ||
        req.url ~ "^/any-other-url-path"
    ) {
        return (pass);
    } else {
unset req.http.cookie;
}
# As mentioned before, remove all cookies for static files, images etc
  # Varnish will always cache the following file types and serve them (during TTL).
  # Note that Drupal .htaccess sets max-age=1209600 (2 weeks) for static files.
  if (req.url ~ "(?i)\.(bmp|png|gif|jpeg|jpg|doc|pdf|txt|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
    // Remove the query string from static files
    set req.url = regsub(req.url, "\?.*$", "");
 
    unset req.http.Cookie;
 
    # Remove extra headers
    # We remove Vary and user-agent headers that any backend app may set
    # If we don't do this, Varnish will cache a separate copy of the resource
    # for every different user-agent
    unset req.http.User-Agent;
    unset req.http.Vary;
 
    return (lookup);
  }


}
#####
#If something gets super popular, super cache it
sub vcl_hit {
        if (obj.hits == 500) {
                set obj.ttl = 3h;
        } elsif (obj.hits == 10000) {
                set obj.ttl = 2d;
        } elsif (obj.hits == 1000000) {
                set obj.ttl = 4w;
        }
}
#####
#shutdown backend connections so unprivileged users donít get privileged content
sub vcl_pass { 
    set bereq.http.connection = "close";
    if (req.http.X-Forwarded-For) {
        set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
    }
    else {
        set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
    }
}
#####
#shutdown backend connections so unprivileged users donít get privileged content
sub vcl_pipe { 
    set bereq.http.connection = "close";
    if (req.http.X-Forwarded-For) {
        set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
    }
    else {
        set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
    }
}

#####
sub vcl_fetch {
# Don't allow static files to set cookies. Cache static content for a long time
  if (req.url ~ "(?i)\.(bmp|png|gif|jpeg|jpg|doc|pdf|txt|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
    unset beresp.http.set-cookie;
    # default in Drupal, you may comment out to apply for other cms as well
    set beresp.ttl = 2w;
  }
#Cache stuff you shouldnít for a min, just bout everything else a day
if (beresp.ttl < 24h) {
            if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") {
                set beresp.ttl = 60s;
            }
            else {
                set beresp.ttl = 24h;
}
}
 if (beresp.status == 301) {
    set beresp.ttl = 1h;
    return(deliver);
  }
  # Allow items to be stale if backend goes down. This means we keep around all objects for 6 hours beyond their TTL which is 2 minutes
  # So after 6h + 2 minutes each object is definitely removed from cache
  set beresp.grace = 6h;
 
  # If you need to explicitly set default TTL, do it below.
  # Otherwise, Varnish will set the default TTL by looking-up
  # the Cache-Control headers returned by the backend
  # set beresp.ttl = 6h;

  # if you have misbehaving sites (i.e Drupal6 or cookie-setters)
  # and you have forced Varnish to cache them in vcl_recv,
  # here you can instruct Varnish about their ttl, and
  # force Varnish to strip any cookies send from backend
  #if (req.http.host ~ "(?i)^(www.)?yourURL.com") {
  # unset beresp.http.set-cookie;
  # set beresp.http.Cache-Control = "public,max-age=602";
  # set beresp.ttl = 120s;
  #}

}

Offline
***
Re: default.vcl discussion
« Reply #1 on: January 01, 2016, 05:20:59 PM »
change your apache server to port 8181 and let varnish have port 80 so it receives the http traffic fist and then talks to apache

Offline
**
Re: default.vcl discussion
« Reply #2 on: January 01, 2016, 11:17:27 PM »
That's how it is setup. The backend should be your apache server. This is how I configured varnish to do things cache static content, pass logins, and stuff like that.

Offline
***
Re: default.vcl discussion
« Reply #3 on: January 02, 2016, 10:28:39 AM »
Ok and do you see any of the varnish headers in the " inspect element" or developer tools "network" section ? I'm assuming you used the one click thingy under select web server in CWP to to set the config ?

Offline
**
Re: default.vcl discussion
« Reply #4 on: January 02, 2016, 03:42:48 PM »
Everything is working well, just when I started I had a huge miss rate and almost no improvement on server speed. This is what I did to improve that. I'm sharing a solution to the "one click thingy" not really helping much. It works and installs varnish but with a minimal config that passes most things and makes your apache logs useless. Then when you start doing things to cache more like stripping cookies and user agents and setting longer cache times it can break things, especially logins. The above config tries to balance caching with usability (esp not interfering with joomla and WordPress logins/administration). I'm  new to varnish and while I think I understand what's going on I wouldn't mind a sanity check on that. In the process, if we can help people who do the "one click thingy" only to be disappointed by providing a fairly straight forward cut and paste solution that has been vetted by the CWP community, all the more better! I spent some time assembling this config and I think it might help others. This config can also probably be improved and suggestions to that effect are welcomed.

In a more perfect world there could be a step in the varnish install of CWP that has check boxes with options and notes like "Strip user Agent (increases hit rate)", "Pass WordPress login/Admin traffic (recommended if running wordpress)", "Cache from Memory or Disk", "Cache Static Content longer" etc. A varnish config generator like the mod_sec only a bit more interactive would be damn skippy!  :D
« Last Edit: January 02, 2016, 03:49:49 PM by Darkroom »