You are viewing a single comment's thread.

view the rest of the comments →

[–] PuttItOut [M] 1 points 145 points (+146|-1) ago 

Yes, something is wrong.

Our thumbnail proxy server appears to have been flooded with inbound traffic starting about ~12 hours ago.

The delay with posting is because the proxy server request is timing out.

Trying to get a quick fix up now but don't have much time.

[–] TheAntiZealot 0 points 25 points (+25|-0) ago 

Thanks for the prompt confirmation!

[–] Nalbarcam 0 points 23 points (+23|-0) ago 

it's more important that you're aware and communicated :)

site still operates well enough for now

[–] sakuramboo 2 points 13 points (+15|-2) ago  (edited ago)

#!/bin/bash
EMAIL=puttitout@someemail.com
PROXY=0.0.0.0 # Put ip address there

ping -c 1 -t 5 $PROXY > /dev/null
if [ $? != 0 ]; then
     echo "Subject: Proxy server issue" | sendmail -v $EMAIL
fi

It's rudimentary, but quick host monitor. Put it in cron and run every hour.

0 */1 * * * /opt/monitor_proxy_script.sh

[–] PuttItOut 1 points 41 points (+42|-1) ago 

Now I'm getting DDOS spammed at my private email address. :)

[–] love_rape 0 points 8 points (+8|-0) ago 

Not such a useful suggesting; a proxy server will be making HTTP requests, whether you can ping it is largely irrelevant.

Besides if the server really is struggling you'll want more useful metrics; you want to have a HTTP-health end-point and you want to poll it, handling timeouts, backoff, and flap-detection.

The worst thing you need is an email every minute saying "Yup, still fucked". Instead you want an email "Response time degraded 5%", "Response time degraded 10%", "Timeout", then every 30 minutes "Yup, still fucked". Or better yet "99% percentile returning to normal".

Sure you can code up a simple script invoking curl, but it's not enough for real monitoring. (For example you want to do tests "externally", running curl against localhost is 99% of the time a mistake; doesn't catch DNS errors, doesn't handle DoS attacks, doesn't test the same experience a user would have.)

[–] Caesarkid1 0 points 6 points (+6|-0) ago 

Hey look at that an actual suggestion!

[–] ptr55 0 points 1 points (+1|-0) ago 

You so fancy.

ping -c 1 -t 5 $PROXY > /dev/null  &&  echo "Subject: Proxy server issue" | sendmail -v $EMAIL 

ttl of 5, that proxy server better not have many hops.

[–] thelma 0 points 4 points (+4|-0) ago 

I put the blame squarely where it belongs: on the wife