S, M, and L Logstash Architectures: Reaching for the Sky

LogStash can scale. From all-in-one boxes (S) to architectures that involve routing log-lines to separate parsing clusters managed by diverse departments (L), LogStash can do it. If you have the foundations of LogStash down, we can talk about scaling it up. From architectures with syslog as the collector and LogStash purely as a parser, to architectures where LogStash is acting as both collector and parser, you will run into scaling issues as you get bigger. We will go over scaled up and out architectures, and equip you with the knowledge of what XL might look like for you. Also, scale means more than events per second. Scale can also mean maintaining multiple years of certain types of logs. As you scale through time, you will face upgrade problems. Are you still on LogStash 1.5 because 2.x requires ElasticSearch 2.x? Or LogStash 2.4 because 5.x requires ElasticSearch 5.x? We will go over techniques to upgrade your deep history and get your architecture closer to ‘latest’.