Splunk Cheat Sheet (Linux)

1. set root's password: 

sudo su
passwd root
Enter new UNIX password: <new_root_password>
Retype new UNIX password: <new_root_password>
passwd: password updated successfully
# su -

2. Remove any existing Splunk directories & create user etc:

# rm -rf /opt/splunkforwarder
# userdel -r splunk # this will remove as above if user splunk's home directory
# groupadd siem
# useradd -g siem -s /bin/bash -d /home/siem -m siem # vi ~/.profile

# chage -I -1 -m -0 -M -99999 -E -1 siem

If above fails because of multiple passwd fails: # pam_tally --reset
check with #chage -l siem

# uname -a # check OS version
# dpkg -i splunk-4.3.1...........intel.deb
# chown -R siem:siem /opt/splunk
# su - siem
: $SPLUNK_HOME/bin/splunk start --accept-license
: $SPLUNK_HOME/bin/splunk edit user admin -password newpassword -role admin -auth admin:changeme

3. vi ~/.profile (as follows) (OR .bash_profile)

# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022


# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
    . "$HOME/.bashrc"
    fi
fi



SPLUNK_HOME=/opt/splunk
export SPLUNK_HOME

SPLUNK_BIN=/opt/splunk/bin
export SPLUNK_BIN

SPLUNK_DB=/IndexedData/HotWarm
export SPLUNK_DB


# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi


PATH=$PATH:$SPLUNK_BIN
export PATH

invoke with: 
# source ~/.profile OR source ~/.bash_profile
# env

4. root# mkdir /IndexedData/...../IndexedData/HotWarm

Then # chown -R siem:siem /IndexedData

5. Reset Splunk Admin password

mv $SPLUNK_HOME/etc/passwd $SPLUNK_HOME/etc/passwd.bak
splunk restart and Admin password reset to changeme.

6. Read the Splunk bucket timestamps

e.g. /IndexedData/HotWarm/_internaldb/db/db_1354400065_1354385978_1
to convert epoch: date -ud @1354385978
Sat Dec  1 18:19:38 UTC 2012

7. Splunk btool - check all inputs.conf components

siem@geoff-arc:/opt/splunk/bin$ splunk cmd btool inputs list --debug
siem@geoff-arc:/opt/splunk/bin$ splunk cmd btool inputs list --debug | grep "\["




8. $SPLUNK_HOME/etc/log.cfg - splunkd.log contents
e.g. deploymentclient = DEBUG (change from WARN for more)
  tcpoutputprocessing = DEBUG (change from WARN for more)

9. More $SPLUNK_BIN commands

splunk enable app SplunkForwarder
splunk disable app SplunkLightForwarder
splunk add forward-server 192.168.1.10:9997
splunk remove forward-server
splunk list monitor (need to be splunk Admin to see logs monitored)
splunk set deploy-poll 192.168.1.11:8089 -auth admin:password
splunk enable listen port -auth username:password
splunk enable boot-start -user siem ( as root)
splunk enable deploy-client -auth admin:password
splunk list deploy-clients (on deployment server - asks for admin PW)
splunk list licenser-messages
splunk list licenser-slaves
splunk reload deploy-server (after changing deployment server app)
splunk status
splunk disable webserver
splunk enable webserver 

10. Misc 

For any changes to $SPLUNK_HOME/etc/system/local files e.g. serverclass.conf to take effect, splunk must be restarted (as siem)

If splunk install files are installed from inside /opt/splunk, the resulting 'splunk' directory structure needs to be moved up a level with # mv splunk /* . Then delete install .gz and splunk directory.

index = main username = * clienti p= * | stats dc(clientip) as ipaddresses values(clientip) by username | where ipaddresses>1

index = main username = geoff  clientip = * | lookup geoip clientip

/bin/cp /opt/splunk/etc/system/local/*.conf /home/siem/splunk

11. License Usage

Search as Admin for daily License volumes in GB over last fortnight (evaluated at mid day):

index=_internal source=*license_usage.log type=RolloverSummary earliest=-14d | eval GB = b/1024/1024/1024 | eval _time = _time -43200 | timechart span=1d sum(GB) AS "Total GB Used"

12. SSH No Login

Add the following to /etc/ssh/sshd_config:
#Block local functional account from ssh
DenyUsers siem

service sshd resart # Linux
svcadm resart network/ssh # Solaris

13 Error Messages

Checking prerequisites...
    Checking mgmt port [8089]: already bound
ERROR: The mgmt port [8089] is already bound.  Splunk needs to use this port.
Would you like to change ports? [y/n]: n




root@master-indexer:/opt# lsof -i TCP:8089
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
splunkd 1157 root    4u  IPv4   5618      0t0  TCP *:8089 (LISTEN)

root@master-indexer:/opt# kill -9 1157

Then splunk restart.

14. Useful Linux Commands

ps -ef | grep splunk or | grep 8000 # will show siem users processes
ps -ef | grep -i syslog # syslog process running
netstat -an | grep 514 # is port 514 open?

OR to display all established, recently terminated, and listening TCP and UDP network connections along with the program name related to each socket:

netstat -anp | grep -e tcp -e udp 

 
free -m # memory available
df # free hard disc totals
df - H # check mount points
du IndexedData #directory size
fuser . # shows all processes using file/dir
find . -name outputs.conf # find all instances of outputs.conf from current dir down
find . type f -exec grep -l "192.168.2.10" {} \; # all instances of ip from here down
cat > /etc/init.d/splunk # create file, ctrl d to save and exit
cat >> audit.log # append a test line to audit log, ctrl d to exit
telnet 192.168.2.11 8089 # check route

OR
nmap -Pn -n -sS -p 19000-19003  -T 5 86.162.8.18

sed -i s/INFO/WARN/g $SPLUNK_HOME/etc/log.cfg #Stop splunk sending INFO logging
awk < /etc/passwd -F: '{print $6}' #use : delimiter & list home dir of users
cat /etc/group # show users in (wheel) group