As a professional services partner, Aplura excels in delivering customer-centric solutions that empower organizations to leverage Cribl’s data engine for IT and security. Our focus on collaboration, continuous innovation, and measurable results has positioned us as a trusted advisor in key customer accounts, showcasing our ability to drive adoption of Cribl’s latest solutions. This recognition underscores our dedication to empowering enterprises with flexibility, choice, and control when it comes to their data. We’re honored to contribute to the growth of Cribl’s ecosystem and look forward to continuing to deliver exceptional value to our partners and customers.

Thanks, Cribl!

See the list of winners here.

TL;DR:

• Old versions of Splunk (<=6.4) had a limitation of 255 bytes per credential.
• New versions of Splunk (>=6.5) [since 2016] do NOT have a limit of characters per credential.
• Splunk’s Add-on Builder still implements the old limitation within its configuration.
• Update your credentials using a script to allow for a single stanza compatible with Secure Password Stores!

History

Way back when, in the years of Splunk Enterprise 4.3 (2012), a new feature was introduced called “Storage Passwords”. This feature replaced the old endpoint /admin/passwords and allowed for more flexibility in storing encrypted secrets within Splunk. The “new” endpoint is /storage/passwords, and has been in use for many iterations of the product. However, when implemented initially, there was a limit of 255 bytes per credential. This was presumably part of limitations in the code during that time (it was the wild west of course!). It should be noted during my research, I could not find that limitation documented anywhere.

So what?

Because more recently, that limitation was removed. Being an older creature within the Splunk realm, I maintain backups of older copies of Splunk for research purposes. So, I loaded up multiple versions of Splunk to see when that limitation was removed. The winning version was Splunk 6.5, released in 2016. How did I test this? Very simply, I generated a file with 43,700 bytes of Lorem ipsum and used the curl command to upload to the storage passwords endpoint. I used a script to generate varying lengths of “credentials” and uploaded each difference size to the credential store. Once uploaded, I ran an SPL command (found below in References) that gave me the Splunk version, and the results of the credential store. I repeated for versions 5.0, 6.0, 6.1, 6.2, 6.3, 6.4, and 6.5.

As seen here, the results of the experiment for Splunk 6.4 show that no matter the length of the “credential”, the value was truncated to 255 characters.

Once I tested Splunk 6.5, I found that indeed the limit was removed and I can store full-fledged strings within the credential store. Once again, I failed to find any documentation on the change from 255 characters to basically unlimited.

And So?

Now we come to the crux of the issue. Splunk’s Add-on Builder is used to create various Add-ons and Apps for Splunk. This add-on has a release listed at Feb. 21, 2017 for version 2.1.0, listing compatibility for Splunk 6.4 AND 6.5, which bridges the versions with the limitation removal. The add-on utilizes (currently at least, it is unknown if earlier versions used the same library) the library solnlib to allow for various Splunk integrations. It is in this library that (even as of today [v7.0.0]) the encrypted credential store code splits each credential up into stanzas of 255 characters. With AoB, each credential is split, and then encrypted individually within the passwords.conf file, so it is very difficult to update the credentials via configuration file using actual automation techniques. Therefore a credential with length 1600 would be split into 6 stanzas, with a final stanza to determine “the end of the credential”. The splitting of credentials is causing issues with automated credential rotation and integrations with newer credential management systems due to method in which the credentials are handled. This splitting of credentials is no longer required, and the AoB (and by extension solnlib) should be updated to use a single stanza per credential.

Now What?

Therefore, a script must be used to transform the old-style stanzas into a newer single credential per stanza, and disregard the splitting of the credentials. To help remediate the issue, the python found here can be utilized to combine the multiple stanzas into a single credential that can used with AoB developed credential management. It accepts new passwords over standard in from either a file or from a piped command, as well as inline with the command as the final argument.

This script should be run anytime there are multiple stanzas for a single credential in a passwords.conf file, and need to be a singular stanza to handle rotations/updates from a CMS.

With the warnings out of the way, let’s discuss how this can be used. This script requires the use of the Splunk provided python for specific libraries to find the splunk.secret for encryption and decryption.

On Linux based systems, this is the command to use Splunk’s python.

/opt/splunk/bin/splunk cmd python3

Let’s assume we will be convert credentials for the app IA-generic-app.

cp convert-aob-cred.py /opt/splunk/etc/apps/IA-generic-app/bin
cd /opt/splunk/etc/apps/IA-generic-app/bin

For the actual execution, you will need the stanza name for the credential in question. That can be found by reviewing the passwords.conf file and extracting the information from the stanza. It is also an option to grab it from the command line (Linux based), with varying results depending on the OS in use.

splunk cmd btool passwords list | grep REST_CREDENTIAL__ | awk -F "_#" '{print $2}' | awk -F "\`\`" '{print $1}' | sort | uniq

In our case, the stanza credential name will be IA-generic-app#configs/conf-app_inputs:Sample, which was pulled from a stanza

[credential:__REST_CREDENTIAL__#IA-generic-app#configs/conf-app_inputs:Sample``splunk_cred_sep``1:]

Once the credential name stub is found, execute the conversion on the passwords.conf file.

/opt/splunk/bin/splunk cmd python3 ./convert-aob-cred.py ../local/passwords.conf IA-generic-app#configs/conf-app_inputs:Sample

Once it runs, the stanzas will be updated within the passwords.conf file, to have a single stanza for the credential, and a second for the end delimiter to provide compatibility with AoB inputs. It is also possible to update the password from the command line.

Consume StdOut from a command

echo "new_password" | /opt/splunk/bin/splunk cmd python3 convert-aob-cred.py <password_file> <input_name> -

Consume contents of a file

/opt/splunk/bin/splunk cmd python3 convert-aob-cred.py <password_file> <input_name> - < <filename>

Just set via string

/opt/splunk/bin/splunk cmd python3 convert-aob-cred.py <password_file> <input_name> "new_password"

And that is it! The resulting updates should still be compatible with existing inputs, and are encrypted on disk. Happy Converting!

References

• Splunk 4.3 was released on January 10, 2012.
• Splunk 4.3 API Documentation
• Splunk Enterprise 6.5 was released on September 27, 2016.
• Add on Builder 2.1.0 was released Feb. 21, 2017.

SPL to verify credential lengths

|rest splunk_server=local /servicesNS/-/-/storage/passwords  
| eval length=len(clear_password) 
| eval clear_password = substr(clear_password,0,50)."..." 
| table title length clear_password 
| append [ rest /services/server/info
    | table version
    | rename version as length 
    | eval title="Splunk Version"
    ] 
| stats values(*) as * by title 
| sort - title 
| table title length clear_password

Please join us in congratulating this year’s honorees:

• Keara Spoor
• David Shpritz
• Kyle Smith

You can read the full announcement here: Congratulations to the 2025–2026 SplunkTrust!

We’d also like to extend our congratulations to all of the 2025–2026 SplunkTrust inductees. Each of these individuals demonstrates a remarkable passion for the Splunk community and serves as an outstanding ambassador.

And if you’re headed to .conf in Boston, be sure to join us on Monday, September 8th for the SplunkTrust induction ceremony. We’ll be there in the crowd, cheering our colleagues on!

Overview

We recently worked with a client that wanted to replace their Splunk Heavy Forwarder with a Cribl Stream worker. One of the challenges was that we needed to replace functionality of the Tenable Add-On for Splunk that pulls scan, asset and plugin data from the Tenable SC appliance. This document outlines the steps needed to setup that integration to act as a one-for-one replacement of the Splunk app inputs.

Configuring Line Breakers

The first step to setup the collectors is to create custom line breakers for the vulnerabilities, assets and plugins. When making requests to Tenable, the API server responds with a large JSON object with the data embedded in the JSON. The line breakers are slightly different for each type of requested data. These will instruct Cribl how to parse the data out of the JSON and assign appropriate timestamps to the logs.

1. In the Cribl Worker Group pages, go to Processing -> Knowledge.
2. Select Event Breaker Rules from the menu on the left.
3. Click Add Ruleset for each of the types below. Assign the ID to be the name of each group title below. Then create a single rule in each, using the settings provided below.

Tenable JSON Basic Ruleset

• Name = Parse Results
• Filter condition = true
• Event Breaker Settings – Enabled = On
• Event Breaker type = JSON Array
• Array field = response.results
• JSON extract fields = On
• Timestamp field = lastSeen
• Max event bytes = 5000000
• Timestamp anchor = ^
• Timestamp format = Manual format - %s

Tenable JSON Asset Ruleset

• Name = Parse Results
• Filter condition = true
• Event Breaker Settings – Enabled = On
• Event Breaker type = JSON Array
• Array field = response.results
• JSON extract fields = On
• Timestamp field = (leave empty)
• Max event bytes = 5000
• Timestamp anchor = ^
• Timestamp format = Current time

Tenable JSON Plugins Ruleset

• Name = Parse Results
• Filter condition = true
• Event Breaker Settings – Enabled = On
• Event Breaker type = JSON Array
• Array field = response
• JSON extract fields = On
• Timestamp field = modifiedTime
• Max event bytes = 300000
• Timestamp anchor = ^
• Timestamp format = Manual format - %s

Storing the Tenable API Host in Variables

As we create the Tenable ingests, we will reference the Tenable API endpoint in URLs to connect to. We will also add a host field to every event tagging it with the API host name. Given the repetitive nature of this data, we setup a variable in Cribl to track this and then reference that variable name in all the configurations. This also makes it easier to change later if the API server moves.

1. From the Cribl Worker Group configuration pages, go to Processing -> Knowledge.
2. Click on the Variables menu on the left side of the page.
3. Click the Add Variable button.
4. Set the following values and click Save.
   • Name = tenableScHost
   • Description = Hostname for the Tenable SC server – used in collector source definitions.
   • Type = string
   • Value = (Set the value to be the IP or DNS name of the Tenable API server to connect to.)

Storing the Tenable Credentials as a Cribl Secret

We need to securely store the access and secret key needed to authentication to the Tenable API server. These will be stored as secrets in Cribl.

1. From the Cribl Worker Group configuration pages, go to Group Settings.
2. Click on the Security menu on the left side of the page.
3. Click on the Secrets menu.
4. Click the Add Secret button.
5. Set the following values and click Save.
   • Secret name = Tenable_SC_Access_Key
   • Secret type = API key and secret key
   • Description = Access Tenable SC appliance
   • API key = (paste API key value)
   • Secret key = (paste secret key value)

Configuring REST Collectors

With the line breakers, variables and credentials defined, we then create data sources to pull each of the data sets from Tenable. In the Data Sources page for a Cribl Worker Group, you will look for the Collectors REST type source and create new sources for each of the ingests below.

tenable_sc_vuln

Extracts the open and reopened vulnerabilities

• Collect URL = `https://${C.vars.tenableScHost}/rest/analysis`
• Collect method = POST with Body
• Collect POST body (NOTE: The filter against the lastSeen field is configured to use a state variable to track the last ingested date. This will get defined during the scheduling of this collector. If the state has not been previously set, it will default to the date 7 days prior to the current date.):

{
    "type": "vuln",
    "sourceType": "cumulative",
    "sortField": "lastSeen",
    "sortDir": "asc",
    "query": {
        "tool": "vulndetails",
        "type": "vuln",
        "filters": [
            {"filterName": "lastSeen", "operator": "=", "value": `${(state.since + 1) || (Math.floor((Date.now() - (7 * 24 * 60 * 60 * 1000))/1000))}-${Math.floor((Date.now())/1000)}`},
            {"filterName": "wasVuln", "operator": "=", "value": "excludeWas"}
        ],
        "startOffset": (+__e['response.endOffset'] || 0),
        "endOffset": ((+__e['response.endOffset'] || 0) + 200)
    }
}

• Collect headers
  • Name: x-apikey, Value: `accesskey=${C.Secret('Tenable_SC_Access_Key', 'keypair').* apiKey}; secretkey=${C.Secret('Tenable_SC_Access_Key', 'keypair').* secretKey};`
• Pagination = Response Body Attribute
• Response attributes:
  • response.endOffset
  • response.totalRecords
• Page limit = 0
• Last-page expression = (+__e['response.endOffset']) >= (+__e* ['response.totalRecords'])
• Event Breaker rulesets – Select the Tenable JSON Basic Ruleset
• Fields:
  • Field Name: host, Value: C.vars.tenableScHost

tenable_sc_vuln_patched

Retrieves the vulnerabilities that have been patched.

• Collect URL = `https://${C.vars.tenableScHost}/rest/analysis`
• Collect method = POST with Body
• Collect POST body (NOTE: The filter against the lastMitigated field is configured to use a state variable to track the last ingested date. This will get defined during the scheduling of this collector. If the state has not been previously set, it will default to the date 7 days prior to the current date.):

{
    "type": "vuln",
    "sourceType": "patched",
    "sortField": "lastSeen",
    "sortDir": "asc",
    "query": {
        "tool": "vulndetails",
        "type": "vuln",
        "filters": [
            {"filterName": "lastMitigated", "operator": "=", "value": `${(state.since + 1) || (Math.floor((Date.now() - (7 * 24 * 60 * 60 * 1000))/1000))}-${Math.floor((Date.now())/1000)}`},
            {"filterName": "wasVuln", "operator": "=", "value": "excludeWas"}
        ],
        "startOffset": (+__e['response.endOffset'] || 0),
        "endOffset": ((+__e['response.endOffset'] || 0) + 200)
    }
}

• Collect headers
  • Name: x-apikey, Value: `accesskey=${C.Secret('Tenable_SC_Access_Key', 'keypair').apiKey}; secretkey=$* {C.Secret('Tenable_SC_Access_Key', 'keypair').secretKey};`
• Pagination = Response Body Attribute
• Response attributes:
  • response.endOffset
  • response.totalRecords
• Page limit = 0
• Last-page expression = (+__e['response.endOffset']) >= (+__e['response.totalRecords'])
• Event Breaker rulesets – Select the Tenable JSON Basic Ruleset
• Fields:
  • Field Name: host, Value: C.vars.tenableScHost
  • Field Name: state, Value: 'fixed'

tenable_sc_asset

Retrieves the aggregated asset data tracked by Tenable.

• Collect URL = `https://${C.vars.tenableScHost}/rest/analysis`
• Collect method = POST with Body
• Collect POST body (NOTE: The filter against the lastSeen field is configured to use a state variable to track the last ingested date. This will get defined during the scheduling of this collector. If the state has not been previously set, it will default to the date 7 days prior to the current date.):

{
    "type": "vuln",
    "sourceType": "cumulative",
    "sortField": "lastSeen",
    "sortDir": "asc",
    "query": {
        "tool": "sumip",
        "type": "vuln",
        "filters": [
            {"filterName": "lastSeen", "operator": "=", "value": `${Math.floor(state.since || ((Date.now() - (7 * 24 * 60 * 60 * 1000))/1000))}-${Math.floor((Date.now())/1000)}`},
            {"filterName": "wasVuln", "operator": "=", "value": "excludeWas"}
        ],
        "startOffset": (+__e['response.endOffset'] || 0),
        "endOffset": ((+__e['response.endOffset'] || 0) + 200)
    }
}

• Collect headers
  • Name: x-apikey, Value: `accesskey=${C.Secret('Tenable_SC_Access_Key', 'keypair').apiKey}; secretkey=$* {C.Secret('Tenable_SC_Access_Key', 'keypair').secretKey};`
• Pagination = Response Body Attribute
• Response attributes:
  • response.endOffset
  • response.totalRecords
• Page limit = 0
• Last-page expression = (+__e['response.endOffset']) >= (+__e['response.totalRecords'])
• Event Breaker rulesets – Select the Tenable JSON Asset Ruleset
• Fields:
  • Field Name: host, Value: C.vars.tenableScHost

tenable_sc_plugin

Retrieves details about the plugin definitions as they are updated.

• Collect URL = `https://${C.vars.tenableScHost}/rest/plugin`
• Collect method = GET
• Collect parameters:
  • Name: startOffset, Value: 0
  • Name: endOffset, Value: 10000
  • Name: fields, Value: 'id,name,description,family,type,copyright,version,sourceFile,dependencies, requiredPorts,requiredUDPPorts,cpe,srcPort,dstPort,protocol,riskFactor,solution,seeAlso, synopsis,checkType,exploitEase,exploitAvailable,exploitFrameworks,cvssVector,cvssVectorBF, baseScore,temporalScore,stigSeverity,pluginPubDate,pluginModDate,patchPubDate,patchModDate, vulnPubDate,modifiedTime,md5,xrefs,vprScore,vprContext'
  • Name: since, Value: `${state.since || 0}`
  • Name: sortField, Value: 'modifiedTime'
  • Name: sortDirection, Value: 'ASC'
• Collect headers
  • Name: x-apikey, Value: `accesskey=${C.Secret('Tenable_SC_Access_Key', 'keypair').apiKey}; secretkey=${C.Secret* ('Tenable_SC_Access_Key', 'keypair').secretKey};`
• Pagination = None
• Event Breaker rulesets – Select the Tenable JSON Plugins Ruleset
• Fields:
  • Field Name: host, Value: C.vars.tenableScHost

Scheduling Collectors

With the four collectors defined, we need to setup schedules to periodically make the REST calls to Tenable and pull the data. The schedules also define the logic for setting up state variables. These variables track dates from previous runs that are incremented so that Cribl doesn’t duplicate data on future runs.

From your list of REST collector sources, you will click on the Schedule button to define each of the schedules below. NOTE: The cron schedules are arbitrary and can be adjusted to meet your needs. They are designed to not run on the quarter hours which are popular times to run jobs:

tenable_sc_vuln

• Enabled = On
• Cron schedule = 25 * * * *
• Concurrent run limit = 1
• Skippable = On
• Mode = Full Run
• Time range = Relative
• Earliest = (leave blank)
• Latest = (leave blank)
• State Tracking – Enabled = On
• State update expression = __timestampExtracted !== false && {since: (state.since || 0) > _time ? state.since : _time}
• State merge expression = (prevState.since || 0) > newState.since ? prevState : newState

tenable_sc_vuln_patched

• Enabled = On
• Cron schedule = 25 * * * *
• Concurrent run limit = 1
• Skippable = On
• Mode = Full Run
• Time range = Relative
• Earliest = (leave blank)
• Latest = (leave blank)
• State Tracking – Enabled = On
• State update expression = __timestampExtracted !== false && {since: (state.since || 0) > _time ? state.since : _time}
• State merge expression = (prevState.since || 0) > newState.since ? prevState : newState

tenable_sc_asset

• Enabled = On
• Cron schedule = 47 * * * *
• Concurrent run limit = 1
• Skippable = On
• Mode = Full Run
• Time range = Relative
• Earliest = (leave blank)
• Latest = (leave blank)
• State Tracking – Enabled = On
• State update expression = __timestampExtracted !== false && {since: (state.since || 0) > _time ? state.since : _time}
• State merge expression = (prevState.since || 0) > newState.since ? prevState : newState

tenable_sc_plugin

• Enabled = On
• Cron schedule = 39 */4 * * *
• Concurrent run limit = 1
• Skippable = On
• Mode = Full Run
• Time range = Relative
• Earliest = (leave blank)
• Latest = (leave blank)
• State Tracking – Enabled = On
• State update expression = __timestampExtracted !== false && {since: (state.since || 0) > _time ? state.since : _time}
• State merge expression = (prevState.since || 0) > newState.since ? prevState : newState

Processing Tenable Data in Pipelines

Finally, we need to do some processing of this data as it passes through Cribl. The Splunk TA for Tenable has some logic to manipulate the data before storing it in Splunk. The following pipelines are designed to replicate those changes. Create a new pipeline for each section and then create the functions per the instructions given.

tenable_sc_vuln

The first pipeline is for processing logs from the tenable_sc_vuln and tenable_sc_vuln_patched collectors.

Eval Function – The first function in the pipeline does a number of manipulations of fields on the event. This includes additional parsing, hard coding information and changing values to a human readable string.

Code Function – The Splunk TA generates a field called sc_uniqueness. This is based on the fields that Tenable provided in the uniqueness field, which we split into an array and stored in _uniqueness. This code function walks through each of those field names and attempts to pull the value from the event. It then creates an underscore separated value with those field values.

__e['sc_uniqueness'] = (__e['_uniqueness'] || [])
.map(key => key=='repositoryID' ? (__e['repository']['id'] || '') : (__e[key] || ''))
.join('_');

Serialize Function – This function takes all the data changes and rewrites the raw event as a new JSON representation of the data.

• Type = JSON Object
• Fields to serialize:
  • !_*
  • !host
  • !source
  • !sourcetype
  • !index
  • !cribl*
  • *
• Destination field = _raw

Eval Function – The final eval function sets all the meta data that we need for Splunk and removes all other fields.

• Keep fields:
  • _raw
  • _time
  • host
  • index
  • source
  • sourcetype
• Remove fields:
  • *

tenable_sc_asset

The next pipeline is for processing logs from the tenable_sc_asset collector.

Eval Function – The first function in the pipeline does a minimal amount of field manipulation of the data.

Code Function – This pipeline uses the same code function as above to create the sc_uniqueness field.

Serialize Function – This pipeline uses the same Serialize function as above to rewrite the _raw event.

Eval Function – Finally, we set the meta for Splunk.

• Keep fields:
  • _raw
  • _time
  • host
  • index
  • source
  • sourcetype
• Remove fields:
  • *

tenable_sc_plugin

The final pipeline is for processing logs from the tenable_sc_plugin collector.

Eval Function – The first function in the pipeline does a minimal amount of field manipulation of the data.

Code Function – This pipeline uses the same code function as above to create the sc_uniqueness field.

Serialize Function – This pipeline uses the same Serialize function as above to rewrite the _raw event.

Eval Function – Finally, we set the meta for Splunk.

• Keep fields:
  • _raw
  • _time
  • host
  • index
  • source
  • sourcetype
• Remove fields:
  • *

Create Your Routes

With all of the infrastructure above setup, you can finish your implementation by creating routes that connect your new collectors to Splunk, using the pipelines defined above.

Tenable API References

• Tenable Security Center API: Analysis - https://docs.tenable.com/security-center/api/Analysis.htm
• Tenable Security Center API: Plugin - https://docs.tenable.com/security-center/api/Plugin.htm
• Retrieve Vulnerability Data for a Specific Time Range - https://docs.tenable.com/security-center/best-practices/api/Content/RetrieveVulnerabilityDataForSpecificTimeRange.htm
• Retrieve Asset Data from Tenable Security Center - https://docs.tenable.com/security-center/best-practices/api/Content/RetrieveAssetDataFromSC.htm

Topic of the month is .conf19! We will review what’s new and recently announced and also get an insight from employees on their first professional technology conference!

Speakers:

• Sulove Khanal (Aplura, LLC) PDF
• Steven Bochniewicz PDF

Our team members are:

• Nancy Kafer
• David Shpritz
• Kyle Smith

We would also like to congratulate and thank the other SplunkTrustees. SplunkTrust members represent the most helpful members of the Splunk community. That might mean posts to Answers, contributing to Splunkbase, or helping folks out on the Splunk User Groups Slack Team.

SplunkTrustees are what make the Splunk Community so great, and we are proud to have three of them in the Aplura family.

That’s also why we are so proud that during .conf19, the Splunk Community presented our Director of Services, Dave Shpritz with a special award.

Dave helps to administer and moderate the Splunk User Groups Slack team, which at times can be a thankless and time consuming task. Well, the SplunkTrust and the community gave him a big “Thank You” in the form of a tall trophy. With flames.

We’ve always known that Dave puts in extra effort for our consultants, the community, and even Splunk itself. We are proud that now he has a trophy to prove it.

Thanks!