DevOps Classroom Series – 24/Apr/2021

Logstash processing pipeline has 3 stages
- inputs
- filters
- outputs
The basic architecture is as shown below
The logstash pipeline is stored in a configuration file that ends with .conf and is as shown below

input
{
    #input plugins
}
filter
{
    # filter plugins
}
output
{
    # output plugins
}

For input plugins Refer Here and for filter plugins Refer Here and for output plugins Refer Here
Lets start creating logstash configurations
Scenario1: : Read input from standard input and write the output the standard ouput
Refer Here for the configuration file created
Run the logstash from commandline
Scenario 2: Read input from standard input and convert the message into uppercase and write the output the standard output
Refer Here for the conf file
Run the logstash from command line
Scenario3: Read the input from standard input and split the message whenever there is | character, give the first item (message[0]) a new field name firstname and second field as lastname.
Refer Here for the configuration file
Listing Plugins
we can also group the plugins by either input, output, filter and codec
Scenario4: Try to read the movies from movies.csv and display the contents
- Refer Here for the configuration file used
Now we can conclude given the message format, we can do some formatting/filter using logstash and send it to elastic search
When applications run they generate logs
We need to extract meanigful information from these logs to gain insights.
Basic Log Pipeline
Extracting meaningful information from logs is challenging and we need to look into some additional filters offered by logstash
- grok filter Refer Here