varnish related

varnish vcl

There's a neat feature of varnish that helps weed out expensive (backend) requests.

Within a vcl you can hook into the cache 'miss' target:

sub vcl_miss {
    if (req.http.CIDR_RS_CC ~ "^(CC|CODE)$" ) {
        return(synth(491, "Access denied"));
    }
}

Replace CC with a lits of | separated country codes.

What this does is effectively abort the request if the cache content doesn't exist already, you'd use this potentially in a case where a bot in another country is hitting the backend a little too greedily.

Want to check the conf?

varnish logs

What if you want to see error responses from the backend?

varnishncsa -q 'RespStatus >= 500 or BerespStatus >= 500'

What if you want to let the logs scroll by similar to other web logs?

varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-ent}i" "%{Varnish:hitmiss}x"'

I like to add a hit/miss statement at the end so we can see how effective the cache is. This can be graphed.

adjusting the http host header

Some requets might have to go a different server with a different host header. This can't be done in the vcl_recv, but has to be done in vcl_backend_fetch:

sub vcl_recv {
    if (req.url ~ "(list|of|urls)" ) {
        set req.backend_hint = different_server_pool.backend();
    }
}

sub vcl_backend_fetch {
    if (bereq.url ~ "(list|of|urls)" ) {
        set bereq.http.host = "different-server-host.com";
    }
}

dropping cached content

Content can be dropped when it matches an expression, such as all content matching .:

varnishadm 'ban req.url ~ .'

Or if you want to clear a host:

varnishadm 'ban req.http.host (www.)example.com'

test a varnish conf before reloading

Like with other daemons, if the conf is invalid, the daemon will shutdown.

/usr/sbin/varnishd -C -f /etc/varnish/default.vcl

If there are faults it will complain with "VCL compilation failed", so lets grep for that.

/usr/sbin/varnishd -C -f /etc/varnish/default.vcl 2>&1 \
| grep -c 'VCL compilation failed' | grep ^0 \
&& systemctl reload varnish

post filtering with varnish

Well, back against the wall, filtering out URLs is dead simple, but what about data that is sent in POST when the backend can't stop it?

apt-get install varnish varnish-modules

import bodyaccess;
import std;

sub vcl_recv {
    if (req.method == "POST") {
        std.cache_req_body(8KB);
        if (bodyaccess.len_req_body() > 0 && bodyaccess.rematch_req_body("(base64_decode|eval\()") != 0) {
            return(synth(403));
        }
    }
}

Remember, this is a regex comparison, if it matches it will evaluate to 1, no match is 0 and errors are -1.

Then reload it like above. In the above, a request now that has '$(curl)' within it will be responded to with a 403 page. It isn't friendly but might help you out.

varnish throttle

If you need to rate limit a characteristic in varnish you can use the throttle vmod. The is_denied method takes a key, which you can use to store request access timestamps. In this case we're using the client-ip, but you could use the user agent itself. If there are more than 5 requests from the IP in 5s, block for 30s:

import vsthrottle;

sub vcl_recv {
    if (req.http.user-agent ~ "curl" ) {
        if (vsthrottle.is_denied(client.identity, 5, 5s, 30s)) {
            return (synth(499, "too many requests"));
        }
    }
}

varnish utm strip

set req.url = regsuball(req.url, "(utm_source|utm_medium|utm_campaign|utm_term|utm_id|fbclid)=[^&]+", "");
set req.url = regsuball(req.url, "&&+", "&");

wp login post spam

You'll need the redis module, a redis server

apt-get install varnish-redis valkey-server

This include will put a 15-minute block on repeated wp-login POSTs, save and include early in default.vcl:

sub vcl_init {
    new db = redis.db(location = "127.0.0.1:6379");
}

sub vcl_recv {
    # Intercept login attempts to WordPress admin
    if (req.method == "POST" && req.url ~ "wp-login\.php|^/wp-admin/") {
        # Check current failure count for this IP
        db.command("GET");
        db.push("wp_fail:" + req.http.X-Real-IP);
        db.execute();

        # If the failure key exists and is greater than or equal to 3, block it
        if (!db.reply_is_error() && std.integer(db.get_string_reply(), 0) >= 3) {
            return (synth(429, "Too Many Login Attempts"));
        }
    }
}

sub vcl_backend_response {
    # Detect failed POST requests to the login page
    if (bereq.method == "POST" && bereq.url ~ "wp-login\.php") {

        # WordPress login failures typically omit the auth cookie (wordpress_logged_in_)
        if (!beresp.http.Set-Cookie ~ "wordpress_logged_in_") {

            # Atomically increment the counter for this IP
            db.command("INCR");
            db.push("wp_fail:" + bereq.http.X-Real-IP);
            db.execute();

            # Set a 60-second expiration on the key if it was just created
            if (!db.reply_is_error() && db.get_integer_reply() == 1) {
                db.command("EXPIRE");
                db.push("wp_fail:" + bereq.http.X-Real-IP);
                db.push("900");
                db.execute();
            }
        }
    }
}