go-syslog

2018-03-28T00:00:00+00:00

This is the official continuation of influxdata/go-syslog.

To wrap up, this package provides:

an RFC5424-compliant parser and builder</a></li>
an RFC3164-compliant parser</a> - ie., BSD-syslog messages</li>
an auto-detect parser</a> that determines RFC 3164 vs RFC 5424 format per-message</li>
an RFC3195 parser</a> for syslog over BEEP</a> (RAW and COOKED profiles)</li>
a parser that works on streams for syslog with octet counting</a> framing technique, see octetcounting</a></li>
a parser that works on streams for syslog with non-transparent</a> framing technique, see nontransparent</a></li> </ul>
This library provides the pieces to parse Syslog messages transported following various RFCs.
For example:

TLS with octet count (RFC5425</a>)</li>
TCP with non-transparent framing or with octet count (RFC 6587</a>)</li>
UDP carrying one message per packet (RFC5426</a>)</li> </ul>
Installation §</a> </h2>
Requires Go 1.22 or later.
go get github.com/leodido/go-syslog/v4</code></pre>Docs§</a> </h2> The docs</a> directory contains .dot</code> files representing the finite-state machines (FSMs) implementing the syslog parsers and transports. Usage§</a> </h2> Suppose you want to parse a given sequence of bytes as a RFC5424 message. Notice that the same interface applies for RFC3164. But you can always take a look at the examples file</a>. i := []byte(`<165>4 2018-10-11T22:14:15.003Z mymach.it e - 1 [ex@32473 iut="3"] An application event log entry...`) p := rfc5424.NewParser() m, e := p.Parse(i)</code></pre> This results in m</code> being equal to: // (*rfc5424.SyslogMessage)({ // Base: (syslog.Base) { // Facility: (*uint8)(20), // Severity: (*uint8)(5), // Priority: (*uint8)(165), // Timestamp: (*time.Time)(2018-10-11 22:14:15.003 +0000 UTC), // Hostname: (*string)((len=9) "mymach.it"), // Appname: (*string)((len=1) "e"), // ProcID: (*string)(<nil>), // MsgID: (*string)((len=1) "1"), // Message: (*string)((len=33) "An application event log entry...") // }, // Version: (uint16) 4, // StructuredData: (*map[string]map[string]string)((len=1) { // (string) (len=8) "ex@32473": (map[string]string) (len=1) { // (string) (len=3) "iut": (string) (len=1) "3" // } // }) // })</code></pre> And e</code> being equal to nil</code> since the i</code> byte slice contains a perfectly valid RFC5424 message. Best effort mode§</a> </h3> RFC5424 parser has the ability to perform partial matches (until it can). With this mode enabled, when the parsing process errors out it returns the message collected until that position, and the error that caused the parser to stop. Notice that in this modality the output is returned iff it represents a minimally valid message - ie., a message containing almost a priority field in [1,191]</code> within angular brackets, followed by a version in ]0,999]</code> (in the case of RFC5424). Let's look at an example. i := []byte("<1>1 A - - - - - -") p := rfc5424.NewParser(rfc5424.WithBestEffort()) m, e := p.Parse(i)</code></pre> This results in m</code> being equal to the following SyslogMessage</code> instance. // (*rfc5424.SyslogMessage)({ // Base: (syslog.Base) { // Facility: (*uint8)(0), // Severity: (*uint8)(1), // Priority: (*uint8)(1), // Timestamp: (*time.Time)(<nil>), // Hostname: (*string)(<nil>), // Appname: (*string)(<nil>), // ProcID: (*string)(<nil>), // MsgID: (*string)(<nil>), // Message: (*string)(<nil>) // }, // Version: (uint16) 1, // StructuredData: (*map[string]map[string]string)(<nil>) // })</code></pre> And, at the same time, in e</code> reporting the error that actually stopped the parser. // expecting a RFC3339MICRO timestamp or a nil value [col 5]</code></pre> Both m</code> and e</code> have a value since at the column the parser stopped it already was able to construct a minimally valid RFC5424 SyslogMessage</code>. Builder§</a> </h3> This library also provides a builder to construct valid syslog messages. Notice that its API ignores input values that does not match the grammar. Let's have a look to an example. msg := &rfc5424.SyslogMessage{} msg.SetTimestamp("not a RFC3339MICRO timestamp") msg.Valid() // Not yet a valid message (try msg.Valid()) msg.SetPriority(191) msg.SetVersion(1) msg.Valid() // Now it is minimally valid</code></pre> Printing msg</code> you will verify it contains a nil</code> timestamp (since an invalid one has been given). // (*rfc5424.SyslogMessage)({ // Base: (syslog.Base) { // Facility: (*uint8)(23), // Severity: (*uint8)(7), // Priority: (*uint8)(191), // Timestamp: (*time.Time)(<nil>), // Hostname: (*string)(<nil>), // Appname: (*string)(<nil>), // ProcID: (*string)(<nil>), // MsgID: (*string)(<nil>), // Message: (*string)(<nil>) // }, // Version: (uint16) 1, // StructuredData: (*map[string]map[string]string)(<nil>) // })</code></pre> Finally you can serialize the message into a string. str, _ := msg.String() // <191>1 - - - - - -</code></pre>Auto-detect§</a> </h3> Suppose you don't know whether incoming messages are RFC 5424 or RFC 3164. The auto</a> package figures it out per-message. m := auto.NewMachine() msg, err := m.Parse(input) fmt.Println(auto.DetectFormat(msg)) // "rfc5424" or "rfc3164"</code></pre> You can pass format-specific options to each inner parser: m := auto.NewMachine( auto.WithRFC3164Options(rfc3164.WithYear(rfc3164.Year{YYYY: 2025})), auto.WithRFC5424Options(rfc5424.WithCompliantMsg()), )</code></pre> It also works with the stream parsers via NewParserAuto</code> - see octet counting</a> and non-transparent</a> below. Performance-wise, auto-detect peeks at a few bytes after the PRI to pick the format - no allocations, no copying. It's as fast as calling the right parser yourself. Message transfer§</a> </h2> Excluding encapsulating one message for packet in packet protocols there are two ways to transfer syslog messages over streams. The older - ie., the non-transparent framing - and the newer one - ie., the octet counting framing - which is reliable and has not been seen to cause problems noted with the non-transparent one. This library provide stream parsers for both. Octet counting§</a> </h3> In short, RFC5425</a> and RFC6587</a>, aside from the protocol considerations, describe a transparent framing technique for Syslog messages that uses the octect counting technique - ie., the message length of the incoming message. Each Syslog message is sent with a prefix representing the number of bytes it is made of. The octecounting package</a> parses messages stream following such rule. To quickly understand how to use it please have a look at the example file</a>. If you have mixed RFC 5424 and RFC 3164 messages in the same stream, use NewParserAuto</code>: p := octetcounting.NewParserAuto( syslog.WithListener(func(r *syslog.Result) { fmt.Println(auto.DetectFormat(r.Message)) }), syslog.WithBestEffort(), ) p.Parse(reader)</code></pre>Non transparent§</a> </h3> The RFC6587</a> also describes the non-transparent framing transport of syslog messages. In such case the messages are separated by a trailer, usually a line feed. The nontransparent package</a> parses message stream following such technique</a>. To quickly understand how to use it please have a look at the example file</a>. Same as octet counting, use NewParserAuto</code> for mixed-format streams: p := nontransparent.NewParserAuto( syslog.WithListener(func(r *syslog.Result) { fmt.Println(auto.DetectFormat(r.Message)) }), syslog.WithBestEffort(), ) p.Parse(reader)</code></pre> Things we do not support: trailers other than LF</code> or NUL</code></li> trailers which length is greater than 1 byte</li> trailer change on a frame-by-frame basis</li> </ul> RFC 3195 (BEEP)§</a> </h3> RFC 3195</a> defines syslog transport over the BEEP</a> protocol. It specifies two profiles: RAW (§4.2): syslog messages are carried as CRLF-terminated payloads in BEEP ANS frames. The inner messages can be either RFC 5424 or RFC 3164 format.</li> COOKED (§4.3): syslog data is encoded as XML <entry></code> elements with attributes for facility, severity, timestamp, tag, and device identity.</li> </ul> The rfc3195 package</a> provides parsers for both profiles. It implements BEEP frame scanning (MSG, RPY, ERR, ANS, NUL, SEQ frames) and extracts syslog messages from the frame payloads. This is a parsing-only implementation - it does not handle BEEP session management, channel negotiation, or TLS. Feed it a stream of BEEP frames and it will emit parsed syslog messages. To quickly understand how to use it please have a look at the example file</a>. Performances§</a> </h2> To run the benchmark execute the following command. make bench</code></pre> On my machine1</a> these are the results obtained paring RFC5424 syslog messages with best effort mode on. [no]_empty_input__________________________________-10 32072733 185.3 ns/op 272 B/op 4 allocs/op [no]_multiple_syslog_messages_on_multiple_lines___-10 27058381 219.8 ns/op 267 B/op 7 allocs/op [no]_impossible_timestamp_________________________-10 8732960 683.8 ns/op 555 B/op 12 allocs/op [no]_malformed_structured_data____________________-10 17997814 335.6 ns/op 499 B/op 8 allocs/op [no]_with_duplicated_structured_data_id___________-10 9254920 645.7 ns/op 672 B/op 15 allocs/op [ok]_minimal______________________________________-10 48347473 123.2 ns/op 227 B/op 5 allocs/op [ok]_average_message______________________________-10 6058492 986.8 ns/op 1344 B/op 20 allocs/op [ok]_complicated_message__________________________-10 7052536 843.2 ns/op 1232 B/op 23 allocs/op [ok]_very_long_message____________________________-10 2644068 2279.0 ns/op 2272 B/op 21 allocs/op [ok]_all_max_length_and_complete__________________-10 3611186 1675.0 ns/op 1848 B/op 27 allocs/op [ok]_all_max_length_except_structured_data_and_mes-10 5729514 1059.0 ns/op 851 B/op 12 allocs/op [ok]_minimal_with_message_containing_newline______-10 43165338 142.9 ns/op 230 B/op 6 allocs/op [ok]_w/o_procid,_w/o_structured_data,_with_message-10 14832892 397.8 ns/op 308 B/op 9 allocs/op [ok]_minimal_with_UTF-8_message___________________-10 20229313 306.2 ns/op 339 B/op 6 allocs/op [ok]_minimal_with_UTF-8_message_starting_with_BOM_-10 19721539 306.7 ns/op 355 B/op 6 allocs/op [ok]_with_structured_data_id,_w/o_structured_data_-10 13860580 435.7 ns/op 538 B/op 10 allocs/op [ok]_with_multiple_structured_data________________-10 8368731 721.9 ns/op 1173 B/op 15 allocs/op [ok]_with_escaped_backslash_within_structured_data-10 9730569 632.6 ns/op 864 B/op 16 allocs/op [ok]_with_UTF-8_structured_data_param_value,_with_-10 8864156 660.6 ns/op 858 B/op 15 allocs/op</code></pre> As you can see it takes: ~125ns to parse the smallest legal message </li> less than 1µs to parse an average legal message </li> ~2µs to parse a very long legal message </li> </ul> Other RFC5424 implementations, like this one</a> in Rust, spend 8µs to parse an average legal message. [1]</a>: Apple M1 Pro</li> </ul>

leodido.dev - Syslog

go-syslog

Docs §</a> </h2>
The docs</a> directory contains `.dot</code> files representing the finite-state machines (FSMs) implementing the syslog parsers and transports.</p>`