Graylog Extractors for pfSense 2.2 filter logs

I’m trying out Graylog for log collection, aggregation, and analysis. It’s free and pretty damn easy to deploy, available in OVA format.

The first thing I noticed is there seemed to be no extractors for pfSense 2.2’s new log format. Extractors allow you to parse a syslog message and place certain values into ‘fields’ for analysis or use in graphs.

Here’s one I prepared relatively quickly. You can import by:

1. Click System -> Inputs in the Graylog UI
2. Click ‘Manage extractors’ next to the relevant input
3. Click ‘Import extractors’ in the ‘Actions’ menu at the top right of the page
4. Paste the below script into the window and then click ‘Add extractors to input’

The extractors will parse the following fields out of the pfSense 2.2 filterlog messages:

• Rule number into pfsense_filter_rulenum
• Direction into pfsense_filter_direction
• Ingress interface into pfsense_filter_ingress
• Action into pfsense_filter_action
• Protocol into pfsense_filter_proto
• Source IP into pfsense_filter_sourceip
• Source Port into pfsense_filter_sourceport
• Destination IP into pfsense_filter_destip
• Destination Port into pfsense_filter_destport

Right now they only interpret IPv4 logs, IPv6 log entries don’t get parsed (thanks to the condition regex) as they are formatted differently.

The script is

{
	"extractors": [
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 17,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_proto",
	"title": "pfSense - Protocol"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 19,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_sourceip",
	"title": "pfSense - Source IP"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 7,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_action",
	"title": "pfSense - Action"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 20,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_destip",
	"title": "pfSense - Destination IP"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 21,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_sourceport",
	"title": "pfSense - Source Port"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 22,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_destport",
	"title": "pfSense - Destination Port"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 8,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_direction",
	"title": "pfSense - Direction"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"index": 5,
	"split_by": ","
	},
	"extractor_type": "split_and_index",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_ingress",
	"title": "pfSense - Ingress Interface"
	},
	{
	"condition_type": "regex",
	"condition_value": "^filterlog:.*,(in|out),4,.*",
	"converters": [],
	"cursor_strategy": "copy",
	"extractor_config": {
	"regex": "^filterlog: ([1-9]+),.*$",
	"replacement": "$1",
	"replace_all": false
	},
	"extractor_type": "regex_replace",
	"order": 0,
	"source_field": "message",
	"target_field": "pfsense_filter_rulenum",
	"title": "pfSense - Rule Number"
	}
	],
	"version": "1.3.0 (04201bb)"
	}

Thanks for

Nathan Manzi