Install and Enable Splunk Add-On for Unix and Linux on a Splunk Forwarder

We assume that you have a splunk enteprise server installed and the Splunk Add-On for Unix addon downloaded and installed on the server side.

We now go ahead and install the same on an ubuntu 18.4.0 forwarder.

Upload the same package you used on your server for the installation onto the splunk forwarder. At the time of writing this file is splunk-add-on-for-unix-and-linux_602.tgz

Untar the file to a location of your choice:

tar -xvzf splunk-add-on-for-unix-and-linux_602.tgz

Copy the Splunk_TA_nix directory and its contents across to the splunk addons directory:

cp -R /app/images/splunk_linux/Splunk_TA_nix /opt/splunkforwarder/etc/apps

The default configuration file for the Splunk Add-On for Unix addon has all stanzas disabled. Edit the /opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf configuration file and change the disabled = 1 sections to disabled = 0 at the stanzas you would like to get covered. We disabled the ps and top sections at the test environment as they were generating way too much traffic. We used the following inputs.conf:

Copyright (C) 2019 Splunk Inc. All Rights Reserved.
 [script://./bin/vmstat.sh]
 interval = 60
 sourcetype = vmstat
 source = vmstat
 disabled = 0
 [script://./bin/iostat.sh]
 interval = 60
 sourcetype = iostat
 source = iostat
 disabled = 0
 [script://./bin/nfsiostat.sh]
 interval = 60
 sourcetype = nfsiostat
 source = nfsiostat
 disabled = 0
 [script://./bin/ps.sh]
 interval = 30
 sourcetype = ps
 source = ps
 disabled = 1
 [script://./bin/top.sh]
 interval = 60
 sourcetype = top
 source = top
 disabled = 1
 [script://./bin/netstat.sh]
 interval = 60
 sourcetype = netstat
 source = netstat
 disabled = 0
 [script://./bin/bandwidth.sh]
 interval = 60
 sourcetype = bandwidth
 source = bandwidth
 disabled = 0
 [script://./bin/protocol.sh]
 interval = 60
 sourcetype = protocol
 source = protocol
 disabled = 0
 [script://./bin/openPorts.sh]
 interval = 300
 sourcetype = openPorts
 source = openPorts
 disabled = 0
 [script://./bin/time.sh]
 interval = 21600
 sourcetype = time
 source = time
 disabled = 0
 [script://./bin/lsof.sh]
 interval = 600
 sourcetype = lsof
 source = lsof
 disabled = 0
 [script://./bin/df.sh]
 interval = 300
 sourcetype = df
 source = df
 disabled = 0
 Shows current user sessions
 [script://./bin/who.sh]
 sourcetype = who
 source = who
 interval = 150
 disabled = 0
 Lists users who could login (i.e., they are assigned a login shell)
 [script://./bin/usersWithLoginPrivs.sh]
 sourcetype = usersWithLoginPrivs
 source = usersWithLoginPrivs
 interval = 3600
 disabled = 0
 Shows last login time for users who have ever logged in
 [script://./bin/lastlog.sh]
 sourcetype = lastlog
 source = lastlog
 interval = 300
 disabled = 0
 Shows stats per link-level Etherner interface (simply, NIC)
 [script://./bin/interfaces.sh]
 sourcetype = interfaces
 source = interfaces
 interval = 60
 disabled = 0
 Shows stats per CPU (useful for SMP machines)
 [script://./bin/cpu.sh]
 sourcetype = cpu
 source = cpu
 interval = 30
 disabled = 0
 This script reads the auditd logs translated with ausearch
 [script://./bin/rlog.sh]
 sourcetype = auditd
 source = auditd
 interval = 60
 disabled = 0
 Run package management tool collect installed packages
 [script://./bin/package.sh]
 sourcetype = package
 source = package
 interval = 3600
 disabled = 0
 [script://./bin/hardware.sh]
 sourcetype = hardware
 source = hardware
 interval = 36000
 disabled = 0
 [monitor:///Library/Logs]
 disabled = 1
 [monitor:///var/log]
 whitelist=(.log|log$|messages|secure|auth|mesg$|cron$|acpid$|.out)
 blacklist=(lastlog|anaconda.syslog)
 disabled = 1
 [monitor:///var/adm]
 whitelist=(.log|log$|messages)
 disabled = 0
 [monitor:///etc]
 whitelist=(.conf|.cfg|config$|.ini|.init|.cf|.cnf|shrc$|^ifcfg|.profile|.rc|.rules|.tab|tab$|.login|policy$)
 disabled = 1
 bash history
 [monitor:///root/.bash_history]
 disabled = true
 sourcetype = bash_history
 [monitor:///home/*/.bash_history]
 disabled = true
 sourcetype = bash_history
 Added for ES support
 Note that because the UNIX app uses a single script to retrieve information
 from multiple OS flavors, and is intended to run on Universal Forwarders,
 it is not possible to differentiate between OS flavors by assigning
 different sourcetypes for each OS flavor (e.g. Linux:SSHDConfig), as was
 the practice in the older deployment-apps included with ES. Instead,
 sourcetypes are prefixed with the generic "Unix".
 May require Splunk forwarder to run as root on some platforms.
 [script://./bin/openPortsEnhanced.sh]
 disabled = true
 interval = 3600
 source = Unix:ListeningPorts
 sourcetype = Unix:ListeningPorts
 [script://./bin/passwd.sh]
 disabled = true
 interval = 3600
 source = Unix:UserAccounts
 sourcetype = Unix:UserAccounts
 Only applicable to Linux
 [script://./bin/selinuxChecker.sh]
 disabled = true
 interval = 3600
 source = Linux:SELinuxConfig
 sourcetype = Linux:SELinuxConfig
 Currently only supports SunOS, Linux, OSX.
 May require Splunk forwarder to run as root on some platforms.
 [script://./bin/service.sh]
 disabled = true
 interval = 3600
 source = Unix:Service
 sourcetype = Unix:Service
 Currently only supports SunOS, Linux, OSX.
 May require Splunk forwarder to run as root on some platforms.
 [script://./bin/sshdChecker.sh]
 disabled = true
 interval = 3600
 source = Unix:SSHDConfig
 sourcetype = Unix:SSHDConfig
 Currently only supports Linux, OSX.
 May require Splunk forwarder to run as root on some platforms.
 [script://./bin/update.sh]
 disabled = true
 interval = 86400
 source = Unix:Update
 sourcetype = Unix:Update
 [script://./bin/uptime.sh]
 disabled = true
 interval = 86400
 source = Unix:Uptime
 sourcetype = Unix:Uptime
 [script://./bin/version.sh]
 disabled = true
 interval = 86400
 source = Unix:Version
 sourcetype = Unix:Version
 This script may need to be modified to point to the VSFTPD configuration file.
 [script://./bin/vsftpdChecker.sh]
 disabled = true
 interval = 86400
 source = Unix:VSFTPDConfig
 sourcetype = Unix:VSFTPDConfig

The last step is to restart the splunk forwarder:

/opt/splunkforwarder/bin/splunk restart

Now verify if the changes took place by running:

/opt/splunkforwarder/bin/splunk cmd btool inputs list

You should see all the Linux OS related monitoring options listed. Just like this:

[SSL]
 _rcvbuf = 1572864
 allowSslRenegotiation = true
 cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
 ecdhCurves = prime256v1, secp384r1, secp521r1
 host = zds.ztacs.com
 index = default
 sslQuietShutdown = false
 sslVersions = tls1.2
 [batch:///opt/splunkforwarder/var/run/splunk/search_telemetry/*search_telemetry.json]
 _rcvbuf = 1572864
 crcSalt = 

 host = zds.ztacs.com
 index = _introspection
 log_on_completion = 0
 move_policy = sinkhole
 sourcetype = search_telemetry
 [batch:///opt/splunkforwarder/var/spool/splunk]
 _rcvbuf = 1572864
 crcSalt = 

 host = zds.ztacs.com
 index = default
 move_policy = sinkhole
 [batch:///opt/splunkforwarder/var/spool/splunk/…stash_new]
 _rcvbuf = 1572864
 crcSalt = 

 host = zds.ztacs.com
 index = default
 move_policy = sinkhole
 queue = stashparsing
 sourcetype = stash_new
 [blacklist:/opt/splunkforwarder/etc/auth]
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = default
 [blacklist:/opt/splunkforwarder/etc/passwd]
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = default
 [fschange:/opt/splunkforwarder/etc]
 _rcvbuf = 1572864
 delayInMills = 100
 filesPerDelay = 10
 followLinks = false
 fullEvent = false
 hashMaxSize = -1
 host = zds.ztacs.com
 index = default
 pollPeriod = 600
 recurse = true
 sendEventMaxSize = -1
 signedaudit = true
 [http]
 _rcvbuf = 1572864
 allowSslCompression = true
 allowSslRenegotiation = true
 dedicatedIoThreads = 2
 disabled = 1
 enableSSL = 1
 host = zds.ztacs.com
 index = default
 maxSockets = 0
 maxThreads = 0
 port = 8088
 sslVersions = *,-ssl2
 useDeploymentServer = 0
 [monitor:///Library/Logs]
 _rcvbuf = 1572864
 disabled = 1
 host = zds.ztacs.com
 index = default
 [monitor:///etc]
 _rcvbuf = 1572864
 disabled = 1
 host = zds.ztacs.com
 index = default
 whitelist = (.conf|.cfg|config$|.ini|.init|.cf|.cnf|shrc$|^ifcfg|.profile|.rc|.rules|.tab|tab$|.login|policy$)
 [monitor:///home/*/.bash_history]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 sourcetype = bash_history
 [monitor:///opt/splunkforwarder/etc/splunk.version]
 _TCP_ROUTING = *
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = _internal
 sourcetype = splunk_version
 [monitor:///opt/splunkforwarder/var/log/splunk]
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = _internal
 [monitor:///opt/splunkforwarder/var/log/splunk/license_usage_summary.log]
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = _telemetry
 [monitor:///opt/splunkforwarder/var/log/splunk/metrics.log]
 _TCP_ROUTING = *
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = _internal
 [monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log]
 _TCP_ROUTING = *
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = _internal
 [monitor:///opt/splunkforwarder/var/log/watchdog/watchdog.log*]
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = _internal
 [monitor:///root/.bash_history]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 sourcetype = bash_history
 [monitor:///var/adm]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 whitelist = (.log|log$|messages)
 [monitor:///var/log]
 _rcvbuf = 1572864
 blacklist = (lastlog|anaconda.syslog)
 disabled = 1
 host = zds.ztacs.com
 index = default
 whitelist = (.log|log$|messages|secure|auth|mesg$|cron$|acpid$|.out)
 [monitor:///var/log/apache2/zds_access.log]
 _rcvbuf = 1572864
 disabled = false
 host = zds.ztacs.com
 index = default
 sourcetype = access_log
 [monitor:///var/log/syslog]
 _rcvbuf = 1572864
 disabled = false
 host = zds.ztacs.com
 index = remotelogs
 sourcetype = linux_logs
 [script]
 _rcvbuf = 1572864
 host = zds.ztacs.com
 index = default
 interval = 60.0
 start_by_shell = true
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/bandwidth.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = bandwidth
 sourcetype = bandwidth
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/cpu.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 30
 source = cpu
 sourcetype = cpu
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/df.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 300
 source = df
 sourcetype = df
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/hardware.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 36000
 source = hardware
 sourcetype = hardware
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/interfaces.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = interfaces
 sourcetype = interfaces
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/iostat.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = iostat
 sourcetype = iostat
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 300
 source = lastlog
 sourcetype = lastlog
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 600
 source = lsof
 sourcetype = lsof
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/netstat.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = netstat
 sourcetype = netstat
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/nfsiostat.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = nfsiostat
 sourcetype = nfsiostat
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/openPorts.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 300
 source = openPorts
 sourcetype = openPorts
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/openPortsEnhanced.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 3600
 source = Unix:ListeningPorts
 sourcetype = Unix:ListeningPorts
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/package.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 3600
 source = package
 sourcetype = package
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/passwd.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 3600
 source = Unix:UserAccounts
 sourcetype = Unix:UserAccounts
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/protocol.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = protocol
 sourcetype = protocol
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/ps.sh]
 _rcvbuf = 1572864
 disabled = 1
 host = zds.ztacs.com
 index = default
 interval = 30
 source = ps
 sourcetype = ps
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = auditd
 sourcetype = auditd
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/selinuxChecker.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 3600
 source = Linux:SELinuxConfig
 sourcetype = Linux:SELinuxConfig
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/service.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 3600
 source = Unix:Service
 sourcetype = Unix:Service
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/sshdChecker.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 3600
 source = Unix:SSHDConfig
 sourcetype = Unix:SSHDConfig
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/time.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 21600
 source = time
 sourcetype = time
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/top.sh]
 _rcvbuf = 1572864
 disabled = 1
 host = zds.ztacs.com
 index = default
 interval = 60
 source = top
 sourcetype = top
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/update.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 86400
 source = Unix:Update
 sourcetype = Unix:Update
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/uptime.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 86400
 source = Unix:Uptime
 sourcetype = Unix:Uptime
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/usersWithLoginPrivs.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 3600
 source = usersWithLoginPrivs
 sourcetype = usersWithLoginPrivs
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/version.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 86400
 source = Unix:Version
 sourcetype = Unix:Version
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 60
 source = vmstat
 sourcetype = vmstat
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vsftpdChecker.sh]
 _rcvbuf = 1572864
 disabled = true
 host = zds.ztacs.com
 index = default
 interval = 86400
 source = Unix:VSFTPDConfig
 sourcetype = Unix:VSFTPDConfig
 [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/who.sh]
 _rcvbuf = 1572864
 disabled = 0
 host = zds.ztacs.com
 index = default
 interval = 150
 source = who
 sourcetype = who
 [splunktcp]
 _rcvbuf = 1572864
 acceptFrom = *
 connection_host = ip
 host = zds.ztacs.com
 index = default
 route = has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue
 [tcp]
 _rcvbuf = 1572864
 acceptFrom = *
 connection_host = dns
 host = zds.ztacs.com
 index = default
 [udp]
 _rcvbuf = 1572864
 connection_host = ip
 host = zds.ztacs.com
 index = default

Once you open the splunk console and go to Search and Reporting, filter for the hostname of your forwarder then click on sourcetype on the left hand side. You should see data already flowing across just like this:

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.