Logging as much information as possible is something critical for modern applications. You know that. More recently, adding intelligence to generated data towards to identify patterns, prevent future issues, and such, has also become critical.

Adding to that, considering the fact that most companies are in the process of moving applications, databases and its underlying infrastructures into the public cloud (Azure, AWS, GCP and others) it is almost certain that some critical portion of the logs collected on-premise throughout the years will also need to move upwards, what does suggest an unprecedented increase around data ingestion.

Not to mention the fact that, streaming data (logs included) out from applications sitting on different locations in (near) real-time to a centralized “lake” is something almost certain. On top of that, it is important to mention that logging data is usually unstructured and extremely verbose, what makes it pretty hard to read and understand.

What above paragraphs are trying to highlight is simple: the demand for an efficient, performant and cost-effective solution to support modern demands related to logging data processing and management (search and indexing included) is high. The good news is: nowadays, there are already very good solutions for that purpose out there, and Splunk is one of them.

Few words about Splunk

If this is the first time you’re hearing about Splunk, this session will give you a high-level view about what it is. As homework and further reading though, I strongly recommend you to go to Splunk Documentation to learn more about the solution itself.

Splunk is a software platform widely used for monitoring, searching, analyzing and visualizing the machine-generated data in real time. It performs capturing, indexing, and correlating the real time data in a searchable container and produces graphs, alerts, identify patterns, dashboards and visualizations. Splunk provides easy to access data over the whole organization for easy diagnostics and solutions to various business problems.

Splunk is a big platform composed by dozens of modules. Companies can select one, multiple or package everything together in one single suite. You can take a look at their current offers here. For the scope of this article, we are going to explore our own instance of Splunk Enterprise for containers. It is free for 60 days, and afterwards is automatically converted into the free version (Splunk Light) with the loss of some functionalities (if not licensed before that time comes, of course).

Architecture considerations

While working with a single instance of Splunk Enterprise over containers in this article (demo purposes only), it is important to notice that Splunk was designed to act as a distributed search/indexing platform. It means that it can handle whatever amount of data by leveraging the strategy of “dividing to conquer”.

From a ten-thousand feet view, Splunk’s distributed search architecture allows you to scale up to deal with large amounts of data, and better handle access control and geo-dispersed data, which makes it a perfect solution to run in the public cloud.

In a distributed search scenario (which is not our case here), a “search head” sends search requests to a group of “indexers”, also called search peers. The indexers perform the search locally and return results to the search head, which merges the results and returns them to the user. The process is depicted in Figure 1.

Figure 1. Splunk distributed search/indexing model
Source: Splunk’s Documentation

Figure 2 presents a comprehensive view about Splunk’s building blocks. As you can see, every instance under a given Splunk deployment is composed by a number of components, that come together to provide a rich search experience. Search and Indexers are only two of them.

Figure 2. Splunk’s building blocks
Souce: Splunk’s Documentation

In our case, because we’re running a single instance of Splunk Enterprise in a container, everything is packed up into a single Docker image. A distributed scenario for containers is also a possibility on top of clusters orchestrated by technologies like Kubernetes, Docker Swarm, and others.

For details about Splunk’s architecture, I strongly recommend looking into platform’s official documentation. It is really good!

Discussing options for containers on Azure

The idea of writing this article came out after working with a customer that required us a Proof of Concept (POC) for Splunk for Containers on Azure. They would like to not only see it being spun up, but also, how performant it would be when it comes to Splunk doing its things on top of Microsoft’s PaaS services for containers. Challenge accepted!

If you are familiar with Azure platform, you might know that when it comes to Containers as a Service (often times referred as CaaS out there) there are few options you can choose from. Depending on the scenario, one can suit better than the other. Figure 3 presents the main options for that purpose in Microsoft’s cloud (let’s start by looking at the Figure 3 below).

Figure 3. Services available on Azure that can host solutions on top of containers

This is all about options. Lets take a quick look into each option above to elucidate the rational behind our decision of leveraging App Service for this POC rather than whatever other service.

Azure Container Instance (ACI)

ACI is designed to run Docker containers on-demand in a managed, serverless Azure environment. Azure Container Instances is a solution for any scenario that can operate in isolated containers, without orchestration. Run event-driven applications, quickly deploy from your container development pipelines, and run data processing and build jobs.

It would serve the purpose of the POC (once we were looking into have a single instance of Splunk up and running) wasn’t the ephemerality of the storage volume (natural for serverless approaches) tied to it. However, as you might imagine, Splunk does require persistent volumes to be able to properly operate. We could attach a persistent volume to the instance, but we decided no taking down that path for technical reasons.

Azure Service Fabric

Azure Service Fabric is a distributed systems platform that makes it easy to package, deploy, and manage scalable and reliable microservices and containers. It could be a contender if we were working towards to build a distributed cluster for our Splunk environment.

Again, because we want a single instance of Splunk to proof a concept, that approach would certainly add unnecessary complexity and elevated cost. It is a great option for sure, just not ideal for what we’ve been looking at as solution for this particular case.

Azure Kubernetes Service (AKS)

Azure Kubernetes Service (AKS) simplifies deploying a managed Kubernetes cluster in Azure by offloading the operational overhead to Azure. As a hosted Kubernetes service, Azure handles critical tasks, like health monitoring and maintenance.

Another viable solution, however, does bring the same concerns of unnecessary complexity and cost to a scenario that requires a very simple approach. Again, if we were hosting a distributed version of Splunk on top of containers, AKS would definitely be the best choice. But, that’s not the case here so we’re bypassing this one.

Azure App Services for Containers

App Services for Container is a managed platform service on Azure that allows you to host web applications also on top of containers in a scalable way. However, differently from ACI that implements a serverless approach for containers, App Services does rely on a previously created web servers farm, known in the platform as App Service Plan.

Because App Service Plan is a farm of actual servers behind the scenes, it does deliver stateful storage by design, which is a requirement for Splunk. Also, if needed, you can attach new volumes to extend its storage capabilities, what is perfect for scenarios like the one we’re looking to accomplish here.

Also, with App Services for Containers you can natively pull off images from both Azure Container Registry (Azure’s offer for enterprise registry in the cloud) and Docker Hub, which is the place where our Splunk Enterprise Trial image lives.

With that said, it seems to be clear that App Services for Containers is the logical choice for us. It is cheap, scalable, also allows us deploying a single container instance over a scalable web server farm with stateful storage and easily integrates with DevOps tools like GitHub Actions, Jenkins and Azure DevOps. So, we’re going to go for it.

Deploying Splunk Enterprise in a App Service for Containers

From now on, I’m going to outline the steps I took to get Splunk up and running in Azure App Services. I’m going to use Azure CLI for that purpose, but you could easily get the same result by performing the same steps through the Azure Portal. Figure 4 presents the workflow we’re going to go through to get it done.

Figure 4. Steps to get Splunk deployed in App Services for Containers

From now on, to follow along with the examples, you will need to have Azure CLI. I’m assuming you already have it either installed and configured in your local environment or through a cloud shell option. The content available through this link will give you all the information you’ll need to get it done either way.

Step 1. Authentication in Azure

First, we need to request authorization to Azure Resource Manager (ARM) to perform operations against a given subscription in the cloud. This is what I’m doing by running the command line below.

az login --query '[].{Name:name, SubscriptionId:id, TenantId:tenantId}' -o table

This will raise up either a new tab in your browser asking you for the account information that holds the subscriptions you have access to, or (if this is coming from a bash environment) will give you a code to be used via device login authentication method. Either way, provide that information to get yourself authenticated.

Once credentials were provided and validated, Azure CLI is going to show up a list of all the subscriptions under the tenants you have access to (see Figure 5) in table format.

Figure 5. Subscriptions I have access to being listed

Great! We’re in. Take a note of the Subscription Id you’re planning to use for the deployment. You’ll need that information to successfully complete Step 2.

Step 2. Selecting the target subscription

Next, we need to tell Azure Resource Manager what is the subscription we’re going to deploy App Service against. For that purpose, we’re going to run the following command line.

az account set -s "your subscription id here"

Step 3. Creating a new Resource Group

Our next step now is to create a new Resource Group. This is going to be the logical container (nothing to do with docker just yet) in charge of bearing the App Service Plan and Web App later on. That’s what the command line below does for us.

az group create -l eastus -n Demo-SplunkContainer

The result of that run should be a Json object with the details about the resource newly created. Something like what is depicted by Figure 6.

Figure 6. Resource Group “Demo-SplunkContainer” successfully created

Step 4. Creating a new Linux-based App Service Plan

Next, we need to create the App Service Plan that will host our Splunk Enterprise instance. For this, just run the following command line according to your operational environment.

For bash:

az appservice plan create \
	-n splunk-farm \
	-g Demo-SplunkContainer \
	--is-linux \
	-l eastus \
	--sku P1V2

For PowerShell:

az appservice plan create `
	-n splunk-farm `
	-g Demo-SplunkContainer `
	--is-linux `
	-l eastus `
	--sku P1V2

After a couple of seconds, you should be able to see another Json object that describe the details related to the plan newly created, as shown by Figure 7.

Figure 7. “splunk-farm” demo successfully created

According the documentation, Docker-based Splunk Enterprise does require a minimum of 2 processors with 4 GB of RAM. Make sure to select a SKU that meet those requirements.

Step 5. Creating a new Web App for Containers

Next, we need to create the infrastructure that is going to host Splunk itself. That infrastructure that will run on top of the App Service Plan we just created is the Web App for Containers we’ve been discussing for a little while. The command line below performs that creation for us. Note that now we have a parameter -i that describes the image we’re pulling off from Docker Hub.

For Bash:

az webapp create \
	-n splunk-instance \
	-g Demo-SplunkContainer \
	-p splunk-farm \
	-i splunk/splunk:latest

For PowerShell:

az webapp create `
	-n splunk-instance `
	-g Demo-SplunkContainer `
	-p splunk-farm `
	-i splunk/splunk:latest

If everything went well, again, you should be seeing a Json object describing all the aspects related to the Web App newly created, as shown by Figure 8.

Figure 8. Web App for container successfully deployed

Next logical step then is to go ahead and browser the application under splunk-instance.azurewebsites.net, right? Well, you can do so, but you’ll face an error. This is because, we have to set up some environment variables required by Splunk’s image and we haven’t done so. It means that, before sending requests to the Splunk instance, we have to set up that requirement first and this is what I’m doing by running the command lines below.

As detailed in this article, Splunk Enterprise does require two initial configurations via environment variables:

1. SPLUNK_START_ARGS='--accept-license'
2. SPLUNK_PASSWORD='<password>'

Additionally, we have to set up the configuration related to the registry from where the Splunk image is coming from (in our case, Docker Hub). This is what I’m tackling with the command lines below.

For Bash:

az webapp config container set \
	az webapp config container set \
	--docker-registry-server-url https://index.docker.io/v1 \
	-n splunk-instance \
	-g Demo-SplunkContainer \
	--enable-app-service-storage false
az webapp config appsettings set \
	-g Demo-SplunkContainer \
	-n splunk-instance \
	--settings SPLUNK_PASSWORD=your-password
az webapp config appsettings set \
	-g Demo-SplunkContainer \
	-n splunk-instance \
	--settings SPLUNK_START_ARGS=--accept-license

For PowerShell:

az webapp config container set `
	--docker-registry-server-url https://index.docker.io/v1 `
	-n splunk-instance `
	-g Demo-SplunkContainer `
	--enable-app-service-storage false
az webapp config appsettings set `
	-g Demo-SplunkContainer `
	-n splunk-instance `
	--settings SPLUNK_PASSWORD=your-password
az webapp config appsettings set `
	-g Demo-SplunkContainer `
	-n splunk-instance `
	--settings SPLUNK_START_ARGS=--accept-license

Done. Container deployed. Next up, getting access to Splunk Enterprise web console at https://splunk-instance.azurewebsites.net.

Step 6. Accessing Splunk Enterprise web console

Everything looks good so, we reached the point of making real tests. The very first test consists of making sure we can access the web console of Splunk Enterprise by leveraging the root username (admin) e password (the we’ve defined when we created a new Web App for Container).

Figure 9 below presents the success of my first try getting access to Splunk’s web console.

Figure 9. Accessing Splunk Enterprise for the first time

Step 7. Upload test data and perform search

Great! We’re ready then to perform our last test, which consists of uploading a considerable amount of data into our instance of Splunk, performing a search over that data and then, exporting a portion of the search result as a Json file to be consumed by another data ingestion tool (let’s say, Synapse, Databricks, so on so forth).

For that test, I have downloaded an amount of log data gathered from a set of VMs I have sitting on Azure. I’ve generated some different CSV files from there and bundled them as a unique ZIP file to be imported here. So, that’s the data I’m going to use to perform the test.

To do so, from the initial screen on Splunk’s web console, I gave a click on “Add data”, as highlighted by Figure 10.

Figure 10. Accessing the session “Add data”

Next, I’ve browsed through the “Upload files from my computer” option, as depicted by Figure 11.

Figure 11. Heading to the option of uploading files from computer

That click led me to the data import wizard, provided by the tool. What I did then was the following sequence of steps:

  1. Selected the zip file that contain all the exported CSV files I mentioned early on.
  2. In the session “Input Settings”, informed Splunk to use the number 1 as host indexing element.
  3. Then, reviewed the import process. All looking good.
  4. Finally, submitted the data, which resulted on the success screen presented by Figure 12.
Figure 12. Data import successfully performed

Now, in order to perform a search, I gave a click on the button highlighted in green (Start Searching) on Figure 12. That action automatically headed me to the “Search” area within the tool. From there, the universe expands.

To demonstrate the efficiency of the search functionality, I have performed a pretty basic search asking Splunk to return all my logging information under all the files within the zip file I’ve uploaded previously. It didn’t take three seconds to give me all back, as you can see through Figure 13.

Figure 13. Searching for logs in Splunk Enterprise

Done! Concept proved and now you know how to work with Splunk Enterprise over App Services for Containers on Azure.

Hope it helps. Enjoy!


Leave a Reply

Avatar placeholder

Your email address will not be published.