We just were made aware of a security issue on Voat concerning the saving of anonymous content. As a security measure I took the website down immediately in order to patch it. Patch is now in place and the security issue has been resolved.
I have a bottleneck detection routine running currently and when it thinks we are getting backed up, it will flip on the maintenance page, take a debug dump of the necessary processes, wait until the existing queue is reduced to expected levels, and then reenable the UI/API traffic. Based on what I've noticed this process never runs over 30 seconds. If you experience this, just wait for one goat minute, then click back in. Told you it would be fun!
I will be pushing up new versions of the website as we build them that attempt to address the issues we are experiencing. When a new version is pushed, the site will go offline temporarily and you will see ugly errors and/or slow or hanging responses. Isn't this fun? I'm all smiles over here. /s
The thought crossed my mind about how I'd explain the issue Voat is experiencing and I came up with "Everything is fine until it isn't." I laughed.
On a technical note, Voat is experiencing a thread starvation issue (randomly, not based load, likely old code somewhere). When this happens, queues build up, the process doesn't recover, and requires manual intervention. Also known as "baby sitting."
I'm going to keep my eye on things as best I can. Until I get it fixed, it's going to be fun.
Other than this rather annoying situation, the update went smoother than expected, so at least I have that going for me.
With software you often just trade one problem for another. Old code wouldn't use enough resources, this code uses too much. It is funny.
Making a few changes, let's see if we can correct this issue.
Just noticed very high memory consumption. Had to restart the service. Obviously we have an issue somewhere.
I've noticed we have some hanging (slow responses) in a few areas: Messages inbox, comment replies, loading more comments. Seems to be intermittent but we still have some work to do on this version.
Main site has now been updated. Had to purge the entire cache so will take a while to rebuild it. I will continue monitoring the site and hopefully things continue working as they should.
Just a quick note that we will be incrementally updating Voat over the next few days (don't forget to know your password). We are doing this piecemeal, one area at a time because of the scope of this update. If we don’t experience any major issues we will continue moving forward.
Voat may very well experience hiccups or downtime during this transition.
Wish us luck and see you Goats on the other side!
Thank you to every Goat who has helped test on the preview site. Couldn't do it without you guys. Much love.