I finally tackled a problem I've been having at work. It took me three days of Googling, tweaking settings, and restarting stuff. And today, after eight and a half hours of head-scratching, it all came together and worked as intended.
Red Hat Enterprise Linux 7 ships with both systemd-journald and Rsyslog. This is mostly for backward compatibility and the fact that journald can't yet connect to a remote logging server. Thankfully, Rsyslog has multiple ways to pull log data from journald, and RHEL7 ships with them. I can have journald keep a minimal journal, which will clear on reboot, and Rsyslog will continuously read from it to log in plaintext on the filesystem.
What RHEL7 doesn't ship with is a decent config for Rsyslog. Either that or someone else botched mine up. Worst part is it was syntactically correct. "rsyslogd -N6" does a syntax check of the config. It was not correct though, in very subtle ways. Like a random hyphen in front of a path for... reasons?
Why was I bothering with logging in the first place? I had a brilliant idea to solve another logging problem while drastically improving our security posture. We run an Oracle Database. STIGs dictate we need database auditing cranked up to 11, audit records have to stick around for at least a year, they need to be offloaded to another server, and we need the capability to reduce them.
Turns out you can configure Oracle to log audit records to syslog, which has reduction (filtering) capabilities and can connect to a remote syslog server to offload a copy. Space might be an issue though. Last time someone tried to apply all of the auditing STIGs, the database filled up in a day. Well, that won't be a problem now. I can setup logrotate to compress the log at whatever rate I need it to.
So the stack goes like this:
Oracle -> syslog (kernel) -> journald -> Rsyslog -> remote syslog server
And I made it work today! Next week I will be turning up the auditing little by little to see how quickly the log grows. If it's too much, that's where the filtering will have to come into play. Anyway, thanks for reading! This win put me in a much better mood today.