Splunk collector IP address: 11.12.13.14
OS: Ubuntu 23.10 (GNU/Linux 6.5.0-25-generic x86_64)
Installing and configuring Cowrie
Installing cowrie dependencies:
# apt-get install git python3-virtualenv libssl-dev libffi-dev build-essential libpython3-dev python3-minimal authbind virtualenv -y
Create non-root user:
# adduser --disabled-password cowrie
Install python-env package
# apt install python3.11-venv
Login to the new non-root user:
# su - cowrie
From this point on your file system location should be: /home/cowrie/cowrie
Pull the git repository code for cowrie:
# git clone http://github.com/cowrie/cowrie
Then move to the downloaded source code directory:
# cd cowrie
Setup environment:
- setup
# python3 -m venv cowrie-env
- activate
# source cowrie-env/bin/activate
- Install python package installer and upgrade it
(cowrie-env) $ python -m pip install --upgrade pip
(cowrie-env) $ python -m pip install --upgrade -r requirements.txt
Cowrie is ssh and telnet monitor but telnet would need to be enabled through the configuration file separately:
Open /etc/cowrie.cfg
and enter the following lines:
[telnet]
enabled = true
One of the things you might want to do as a “must” is to set a password that attackers will use to enter the system thinking it’s the legitimate password and they successfully cracked it. In reality, they will get access to the virtual environment not being able to do any real harm.
Cowrie has a configuration file at /home/cowrie/cowrie/etc/userdb.example
which you can use as an example of how to set up a production configuration file userdb.txt. The syntax within the userdb.txt file is:
user:x:password
“X” is not used for anything, Cowrie probably left this slot fo future features, so for now we are just ignoring it and leaving “x” as in the example.
Create a file with root:x:pass
content:
# echo “root:x:pass” > /home/cowrie/cowrie/etc/userdb.txt
NOTE: Cowrie will let the attacker log in only with username root and password pass! Check out userdb.example
for more understanding of different formats.
Moving on with a focus on SSH monitoring. Let’s activate the Cowrie service:
# bin/cowrie start
Log files location:
# ls var/log/cowrie/
cowrie.json cowrie.log
cowrie.log
holds a more summarized format with timestamps for particular events:
2024-03-19T16:28:02.817428Z [cowrie.ssh.factory.CowrieSSHFactory] New connection: 103.212.211.151:60528 (146.190.55.100:2222) [session: a7bcf90c3823]
2024-03-19T16:28:04.689284Z [cowrie.ssh.factory.CowrieSSHFactory] New connection: 168.90.91.124:33870 (146.190.55.100:2222) [session: 724a359c778c]
2024-03-19T16:28:05.881523Z [cowrie.ssh.factory.CowrieSSHFactory] New connection: 103.212.211.151:60542 (146.190.55.100:2222) [session: f5a57dffd6db]
2024-03-19T16:28:06.849347Z [HoneyPotSSHTransport,129,168.90.91.124] Saved redir contents with SHA-256 a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2 to var/lib/cowrie/downloads/a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2
cowrie.json
is a more verbose log format that we normally want to feed into collection platforms like Splunk:
{"eventid":"cowrie.session.file_download","duplicate":true,"outfile":"var/lib/cowrie/downloads/a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2","shasum":"a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2","destfile":"/root/.ssh/authorized_keys","message":"Saved redir contents with SHA-256 a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2 to var/lib/cowrie/downloads/a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2","sensor":"ivan-testing","timestamp":"2024-03-19T16:28:37.898126Z","src_ip":"43.153.211.210","session":"43682aac7f7e"}
{"eventid":"cowrie.session.connect","src_ip":"43.153.211.210","src_port":35410,"dst_ip":"146.190.55.100","dst_port":2222,"session":"6a941fc47642","protocol":"ssh","message":"New connection: 43.153.211.210:35410 (146.190.55.100:2222) [session: 6a941fc47642]","sensor":"ivan-testing","timestamp":"2024-03-19T16:28:38.078918Z"}
{"eventid":"cowrie.session.connect","src_ip":"43.153.211.210","src_port":36214,"dst_ip":"146.190.55.100","dst_port":2222,"session":"04490ba7e7f6","protocol":"ssh","message":"New connection: 43.153.211.210:36214 (146.190.55.100:2222) [session: 04490ba7e7f6]","sensor":"ivan-testing","timestamp":"2024-03-19T16:28:40.441290Z"}
{"eventid":"cowrie.session.connect","src_ip":"103.175.221.6","src_port":59650,"dst_ip":"146.190.55.100","dst_port":2222,"session":"9458e4c08bae","protocol":"ssh","message":"New connection: 103.175.221.6:59650 (146.190.55.100:2222) [session: 9458e4c08bae]","sensor":"ivan-testing","timestamp":"2024-03-19T16:28:55.170367Z"}
{"eventid":"cowrie.session.file_download","duplicate":true,"outfile":"var/lib/cowrie/downloads/a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2","shasum":"a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2","destfile":"/root/.ssh/authorized_keys","message":"Saved redir contents with SHA-256 a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2 to var/lib/cowrie/downloads/a8460f446be540410004b1a8db4083773fa46f7fe76fa84219c93daa1669f8f2","sensor":"ivan-testing","timestamp":"2024-03-19T16:28:57.481658Z","src_ip":"103.175.221.6","session":"9458e4c08bae"}
{"eventid":"cowrie.session.connect","src_ip":"103.175.221.6","src_port":59658,"dst_ip":"146.190.55.100","dst_port":2222,"session":"ebf20b898351","protocol":"ssh","message":"New connection: 103.175.221.6:59658 (146.190.55.100:2222) [session: ebf20b898351]","sensor":"ivan-testing","timestamp":"2024-03-19T16:28:57.666094Z"}
Install and configure Splunk universal forwarder
Before installing Splunk Universal Forwarder let’s open necessary ports on the Cowrie box (if not open already):
# iptables -A INPUT -p tcp --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp --dport 80 -j ACCEPT
# iptables -A INPUT -p tcp --dport 443 -j ACCEPT
# iptables -A INPUT -p tcp --dport 9997 -j ACCEPT
# iptables -A INPUT -p tcp --dport 8088 -j ACCEPT
# iptables -A INPUT -p tcp --dport 8090 -j ACCEPT
Save above rules:
# iptables-save > /etc/network/iptables.rules
Create non-root user for Splunk universal forwarder:
# useradd -m splunk
Create home directory for new non-root user:
# mkdir /opt/splunkforwarder/
Set proper permissions:
# chown -R splunk:splunk /opt/splunkforwarder
Download Splunk package:
# wget -O splunkforwarder-9.2.0.1-d8ae995bf219-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/9.2.0.1/linux/splunkforwarder-9.2.0.1-d8ae995bf219-linux-2.6-amd64.deb"
Note: always check the latest version so you don’t download the deprecated version.
Install downlaoded package:
# dpkg -i splunkforwarder-9.2.0.1-d8ae995bf219-linux-2.6-amd64.de
Install license:
# sudo /opt/splunkforwarder/bin/splunk start --accept-license
Move to splunkforwarder home directory:
# cd /opt/splunkforwarder/bin
Make sure that splunk fowarder starts every time our machine is restarted:
- Stop the service if already running:
# /opt/splunkforwarder/bin/splunk stop
- Enable start on boot:
./splunk enable boot-start -user splunk
- Start the service again:
# /opt/splunkforwarder/bin/splunk start
NOTE: if you used root user throughout this part of installation of splunkforwarder, as I have, you may need to rerun the persmissions fix:
# chown -R splunk:splunk /opt/splunkforwarder/
Define the collector’s destination:
./splunk add forward-server 11.12.13.14:9997
The script will ask for an administrator’s credentials, enter the username of the user we created to administer the Splunk forwarder service (splunk), and password per your security policy if you have any, otherwise just make sure it’s a strong password.
NOTE: It is NOT asking for a login to the Splunk collecting data!
Finally, define the
./splunk add monitor -auth splunk:<your password> /home/cowrie/cowrie/var/log/cowrie/cowrie.json
NOTE: I used the cowrie.json log file to stream to collector Splunk, if you have different plans adjust the command accordingly.
Analyze data in Splunk collector
- Login to your Splunk instance
- Navigate to the Search section and locate Data Summary button
- Click on it and select the hostname of the server you previously installed the forwarder on
- For future reference, you can use this search phrase in the search field for the same effect:
host="ivan-testing"
For testing purposes I have logged into the SSH server we have Cowrie on which Splunk immediately recorded:
Then I executed pwd
command which also got immediately recorded:
Creating alert
With monitoring setup and data stream properly landing into Splunk index we can set the alert in case there is an important event. With SSH being monitored we can name a few important events that will be good alert conditions and with that in mind we’ll show how two of those events we can map and then send to alerts tracker.
Login success events
First, since we are monitoring the honeypot, we want to know when someone gets in the system so we can monitor their actions. Therefore, we’ll use the following seach parameter to know when attacker gets in:
hostname="ivan-testing" eventid="cowrie.login.success"
- On the top right corner choose
Save as -> Alert
- The dialog will open for more details.
Note that we used simple alerting method: Add to Triggered Alerts for the simplicity of the tutorial. In prodiction, we’d go with more robust solutons:
- After we save it the confirmation will appear:
To test the alert let’s go and try to login to our honeypot, make sure to enter the proper login credentials and observe the Triggered Alerts page:
Forward SSH port to Cowrie listener
The idea is to separate real ssh traffic from attackers’ traffic. To do this, we need to move real ssh port to custom number (so we don’t get locked out of the box) and forward traffic coming to port 22 to port 2222 (cowrie listener):
Change default SSH port
Open file /etc/ssh/sshd_config
:
# /etc/ssh/sshd_config
and locate
# Port 22
Remove the comment (“#”) and change the number to something within the custom range of ports (ex: 5000). Save the changes and restart the SSH service:
# systemctl restart ssh
Forward SSH traffic from port 22 to 2222
Using iptables we’ll capture traffic on the port 22 (remember that real SSH port is now on custom port; ex: 5000):
# iptables -t nat -A PREROUTING -p tcp --dport 22 -j REDIRECT --to-port 2222
At this point, we are pretty much done with all the necessary steps:
- iptables is forwarding traffic from port 22 to port 2222
- we have cowrie monitoring the port 2222
- it logs the behavior and tracks the malicious user upon the potential successful entry
- logs are streamed by Splunk forwarder to Splunk collection instance
- regular users are going to custom port 5000