Where do my Splunk settings go?

In our previous article about apps, we s either aw that an app is a collection of config files with either general settings of your deployment or data related settings (for example extractions for Palo Alto firewall logs).

To understand and to be able troubleshoot this process there are 2 things that are vital to understand:

1. Where does which setting take effect?
2. What is the precedence between the different apps?

Where do I configure which setting?

To answer the first question we need to go back to a webpage that Splunk no longer exposes to their users but that served for me as a cheat sheet during my career as a Splunk architect.

So again, no credit or copyright for me on this information. I just spread the document, as I think that the settings mentioned are really important and because I think it can really help Splunk beginners or even mediocre engineers to grasp the concept of ‘what goes where’ in a Splunk deployment.

What about precedence of different configuration files?

To anser this question we need to set the scene about context. Context is the reach of a configuration setting. For example indexing is independent of an app so it will be executed in the global context. Some other settings are used in app/user context. These settings will only work within the app or user context.

Global context precedence

In this global context this is the order of precendence:

• System local directory — highest priority
• App local directories
• App default directories
• System default directory — lowest priority

Note that for cluster peers (indexers part of a cluster) the precedence is a little different.

The app in a index cluster are pushed from the cluster manager into the peer-apps directory.