Recently, I have been dealing with an issue with rsyslog server, where it logs error messages with TLS connections. I posted a question on ServerFault but have not received any response for 10 days.
It is hard to debug TLS connection issue as application data is encrypted in the traffic. I tried tcpdump to capture some packets anyways.
Looking through the captured packets, it seems that the syslog client, in this case an EPIC server, may have dropped the TLS connection without properly closing it.
Checking this theory with ChatGPT - here is what may have been happening:
A party to a TCP connection should send a close_notify inside the TLS stream to properly close the connection;
TLS library receives data until end-of-stream (read returns 0 or error) OR a close_notify is processed.
If a close_notify was received before end-of-stream, the TLS library can report a clean shutdown; otherwise, it may report an error or truncated data.
If a peer closes its TCP write side without sending close_notify, your TLS library will report an unexpected EOF, truncated data, or similar error.
That seems to match the errors emitted by rsyslogd. For now, we're just going to chalk it up to that unless any further issues pop up. If I have time, I might dig into the rsyslog source code on GitHub.
