🏠   Back Home

Configuring Splunk Cloud Universal Forwarder

Splunk is a platform for aggregating, indexing, searching & analyzing logs and other machine generated data. At my day job, we use it extensively to aggregate application logs hosted on different servers. Using Splunk queries, we could query those log data to debug any production issues as well as to build different charts and dashboards for both engineering and business reporting.

Even though I was fairly comfortable with writing Splunk queries to build dashboards utilizing the log data, I’ve never actually configured servers to forward logs to Splunk. I wanted to try and understand what goes into configuring the Splunk universal forwarder. This article is simply a documentation of the steps I took to configure Splunk universal forwarder to forward application logs from a EC2 server and a dockerized app deployed on ECS Fargate.

Different Ways of Forwarding Logs To Splunk

Splunk forwarders send data from data sources to Splunk cloud for indexing which makes it easier for searching, querying and building dashboards. To push logs to Splunk cloud, we’d need to use one of Splunk forwarders. Splunk provides different type of forwarders, universal forwarder, heavy forwarder and light forwarder.

While Splunk provides something called HTTP Event Collector allowing us to push data from servers to Splunk cloud over HTTP/HTTPs, I opted to use the universal forwarder. Since HTTP Event Collector relies on HTTP connections, it might impact the application performance.

Unlike HTTP Event Collector, Universal Forwarder are lightweight agents which are installed as a package on the host machine which periodically monitors the log file in the background and pushes them to the Splunk Cloud.

Configuring Splunk Universal Forwarder on EC2

You could use following steps to configure the Splunk Universal Forwarder on a EC2 machine or any other VPS servers.

Steps:

  1. SSH into the EC2 server

  2. Download the splunkforwarder package for EC2 OS from Splunk Downloads web page e.g. for Debian based ubuntu wget -O splunkforwarder-9.0.1-82c987350fde-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/9.0.1/linux/splunkforwarder-9.0.1-82c987350fde-linux-2.6-amd64.deb"

  3. Install the package using sudo dpkg -i splunk-forwarder.deb This typically install the splunkforwarder on /opt/splunkforwarder

  4. Start Splunk by accepting license sudo /opt/splunkforwarder/bin/splunk start --acept-license. You’ll be asked for a local username and password , Keep note of it, as we’ll need it later.

  5. Restart Splunk sudo /opt/splunkforwarder/bin/splunk restart

  6. Download the Splunk forwarder credentials file from the Splunk universal forwarder documentation link https://yourdomain.splunkcloud.com/en-US/app/splunkclouduf/setupuf

  7. Copy Splunk credentials file (downloaded in step6) from your local machine to the EC2 server using Secure Copy Protocol scp , Use following command on your local machine.scp /path/to/file/splunkclouduf.spl ubuntu@IP:/server/pathIf you have pem file, use scp -i <pem file> file_to_copy_local_path ec2-user@ip:location_of_server

  8. Install Splunk credentials using sudo /opt/splunkforwarder/bin/splunk install app ./splunkclouduf.spl . Use username and password setup in step 4

  9. Add your Splunk Cloud forward server  sudo /opt/splunkforwarder/bin/splunk add forward-server yourdomain.splunkcloud.com:9997

  10. Monitor log files on certain index sudo /opt/splunkforwarder/bin/splunk add monitor ./my-app/log/production.log -index my-app-prod-indexHere, my-app-prod-index is an index which is manually created in Splunk. The default index is main

Splunk universal forwarder on Docker

For dockerized applications, Splunk provides a docker image for universal forwarder. We could potentially run this as a sidecar container alongside the app container or probably use docker-compose and add it as a dependency to our application service.

However, for simplicity, I opted not to use the docker image provided by Splunk. I manually download the Splunk package and run Splunk commands when building the Docker image and when running the container.

We can download & install the Splunk universal forwarder package, expose the required ports, accept Splunk license and setup the Splunk admin user when creating a docker image.

We’ll have to configure forward server, add monitors and start Splunk forwarder when we run the docker container. We can use docker entrypoint to run Splunk commands when starting the container.

# Dockerfile

# Splunk Universal Forwarder
RUN wget https://download.splunk.com/products/universalforwarder/releases/9.0.0.1/linux/splunkforwarder-9.0.0.1-9e907cedecb1-linux-2.6-amd64.deb \
    && apt-get install -f ./splunkforwarder-9.0.0.1-9e907cedecb1-linux-2.6-amd64.deb \
    && rm -f splunkforwarder-9.0.0.1-9e907cedecb1-linux-2.6-amd64.deb

# PORTS for Splunk Universal Forwarder
EXPOSE 9997
EXPOSE 8000

# Accept Splunk license and setup admin user
RUN deploy/splunk/auth.sh

ENTRYPOINT ["./deploy/entrypoint.sh"]

In the above Dockerfile, we’ve downloaded and installed the Splunk universal forwarder package. Additionally, we are running commands in /splunk/auth.sh. This fill will run commands to accept Splunk’s license and setup admin user for running Splunk commands on the container. Since Splunk will ask to manually type the username and password, we are using the expect package to automate the process. These credentials can be anything, but we need to keep note of it as we require it in next stages.

/splunk/auth.sh

#!/usr/bin/expect

set timeout -1

spawn /opt/splunkforwarder/bin/splunk start --accept-license
expect "Please enter an administrator username: "
send -- "user\r"
expect "Please enter a new password: "
send -- "password\r"
expect "Please confirm new password: "
send -- "password\r"
expect eof

The docker entrypoint.sh will run commands to start Splunk, add a forward server and monitor logs. Note that splunkclouduf.spl is a file downloaded from https://yourdomain.splunkcloud.com/en-US/app/splunkclouduf/setupuf page and the user name and password should be the same used in the previous step in auth.sh

#!/bin/bash

/opt/splunkforwarder/bin/splunk start

/opt/splunkforwarder/bin/splunk install app /path/to/app/splunkclouduf.spl -auth user:password

/opt/splunkforwarder/bin/splunk add forward-server yourdomain.splunkcloud.com:9997

/opt/splunkforwarder/bin/splunk add  monitor /path/to/app/log/ -index app-prod-index

/opt/splunkforwarder/bin/splunk restart

Basically we are following the same commands that we used to configure the Splunk universal forwarder in EC2, however we are separating and defining them in Dockerfile and [entrypoint.sh](http://entrypoint.sh) instead on manually SSH-ing into the server and configuring them.

Useful Splunk Cli Commands

  1. Check Splunk Status sudo /opt/splunkforwarder/bin/splunk status
  2. List Forward Servers sudo /opt/splunkforwarder/bin/splunk list forward-server
  3. List Monitors sudo /opt/splunkforwarder/bin/splunk list monitor
  4. Restart Splunk sudo /opt/splunkforwarder/bin/splunk restart
🏠   Back Home