So I’ve been meaning to do a post about this for a while. Minemeld is a cool open source project from Palo Alto Networks that allows you to take threat feeds such as IP and URL lists, that contain indicators of compromise and transform them into a single list for use with your favourite Next Gen Firewall. Minemeld is essentially a multiplexer for threat feeds. Take multiple threat feeds, transform them, set confidence and output into a single consumable feed.
For example you could transform lists from public sources such as Spamhaus and Abuse.ch and transform them into one list that can be used by your firewall to block those URLs. You can also take sources and transform them with Minemeld for consumption with security operations tools such as Splunk. This tutorial will centre around setting up a URL feed for consumption with the External Dynamic List feature on a Palo Alto firewall.
Setting up Minemeld
The first part of the setup requires you to have an Ubuntu 18.04 (you can use Redhat and CentOS but that is out of scope for this) VM ready to go. 2vCPU, 4GB memory, 80GB disk is enough for this lab.
1. We are going to do the Ansible playbook deployment for Ubuntu 18.04. You will be asked to supply a password during the install. This will be your admin account password for the Minemeld application.
Make sure your Ubuntu install is up to date then run the following:
sudo apt update
sudo apt upgrade
sudo apt install -y gcc git python-minimal python2.7-dev libffi-dev libssl-dev make
wget https://bootstrap.pypa.io/get-pip.py
sudo -H python get-pip.py
sudo -H pip install ansible
git clone https://github.com/PaloAltoNetworks/minemeld-ansible.git
cd minemeld-ansible
ansible-playbook -K -i 127.0.0.1, local.yml
sudo usermod -a -G minemeld
2. To change the default NGINX web server certifcates (if you have your own PKI etc) replace the following and restart NGINX:
sudo cp ~/cert.pem /etc/nginx/minemeld.cer
sudo cp ~/cert.key /etc/nginx/minemeld.pem
sudo service nginx restart
3. Login to your Minemeld instance with https://servername. Default username is admin and the password is what you set in step 1 during the Ansible install.
Configuring Minemeld
We are going to configure Minemeld to process a URL text feed from Abuse.ch
1. On the Minemeld dashboard click on config and the elipse button. Search for the prototype itcertpa.URLS and click on it.
2. We are using this prototype as a template, click on new in the top right. Name the prototype rwtracker and configure it as follows.
age_out:
default: null
interval: 600
sudden_death: true
attributes:
confidence: 80
direction: inbound
share_level: green
type: URL
ignore_regex: ^#
indicator:
regex: ^(http[s]*:\/\/)(.*)
transform: \2
source_name: itcertpa.URLS
url: https://urlhaus.abuse.ch/downloads/text_online/
This prototype will download the URLHaus feed from Abuse.ch. The regex will strip the http/s from the URL feed for consumption with a PA Firewall. Click OK to save.
3. Now its time to create the miner node. To add a miner click on the eye button on the bottom left corner of config then the + icon. The add node screen will appear. Set the name to rwMiner and base it off prototype minemeldlocal.rwtracker. Click OK to save.
4. Lets now add the processor prototype. The processor will remove any duplicate entries from the feeds. Click on config and then click on the elipse. Search for ‘stdlib.aggregatorURL’. Click on it. Then click on New. Name the prototype rwtrackerprocessor. Type a description “URL processor for urlhaus.abuse.ch”.
Configure it as follows:
infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
conditions:
- type == 'URL'
name: accept URL
Click OK to save.
5. Now add the processor node. Click config, the eye button and then +. Name the node rwtrackerprocessor. Select the prototype as minemeldlocal.rwtrackerprocessor. Select the input as rwMiner.
6. Its now time to create the output prototype. Click Config and the elipse and search for “stdlib.feedGreenWithValue”. Click on it.
Click new and name the prototype rwtrackerOuput. Configure it like so:
infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
7. Add the output node. Click Config -> eye icon > plus icon. Name the node rwtrackeroutput, select the prototype minemeldlocal.rwtrackeroutput. Select the input rwprocessor and click ok.
8. You have configured all three nodes: rwMiner, rwtrackerprocessor and rwtrackerOutput. On the config screen click Commit to apply the configuration.
You can navigate to the nodes screen and the see the indicators column is now processing entries from the urlhause feed in Minemeld.
The rwtrackOutput feed can also be browsed to via the url https://sitname.domain/feeds/rwtrackerOutput.
Configure Authentication for a Minemeld Feed
1. SSH to your Ubuntu VM running minemeld and run the following command:
sudo -u minemeld sh -c 'echo "FEEDS_AUTH_ENABLED: True" > /opt/minemeld/local/config/api/30-feeds-auth.yml'
2. Login to Minemeld Web UI and browse to Admin > Feed Users. Add a username (edluser) and password and select create a tag for the user. The tag we will use is called “ransomware”.
3. Click on Nodes and rwtrackerOutput. Click tags on this node and add ransomware to it. Now the edluser you created in step two is authenticated to access the feed URL for rwtrackerOutput.
Updating Minemeld
To update Minemeld with the Ansible deployment, simply run the command again from CLI:
cd ~/minemeld-ansible/
ansible-playbook -K -i 127.0.0.1, local.yml
You can check the version your running by clicking on Minemeld logo on the top left of the Web UI:
Click here for part two which I will detail how to add this Minemeld feed to a External Dynamic List on a PA Firewall. I cover using the EDL feature with URL filtering and a security policy.
Comments