I thought my website was hacked. Here is how I hardened my Linux servers security with Lynis Enterprise

Published on 24 October 2020 by Simon Fearby

I thought my website was hacked. Here is how I hardened my Linux servers security with Lynis Enterprise

I have waited a year before posting this, and I have tried my best to hide the bank's identity as I never got a good explanation back from them about they the were whitelisting my website.

I was casually reading Twitter one evening and found references to an awesome service (https://publicwww.com/) that allows you to find string references in CSS, JS, CSP etc files on websites.

I searched https://publicwww.com/ for "". I was expecting to only see only resources that were loading from my site.

I was shocked to see a bank in Asia was whistling my website and my websites CDN (hosted via ewww.io) in it's Content Security Policy.

I was not hosting content for a bank and they should not be whitelisting my site?

Were they hacked? Was I hacked and delivering malware to their customers? Setting up a Content Security Policy (CSP) is not a trivial thing to do and I would suggest you check out https://report-uri.com/products/content_security_policy (by Scott Helme) for more information on setting up a good Content Security Policy (CSP).

I have written a few blog posts on creating Content Security Policies, and maybe they did copy my starter Content Security Policy and added it to their site?

I do have a lot of blog readers from their country.

I went to https://www.securityheaders.com and scanned their site and yes they have whitelisted my website and CDN. This was being sent in a header from their server to any connecting client.

I quickly double-checked the banks Content Security Policy (CSP) with https://cspvalidator.org/ and they too confirmed the bank was telling their customers that my website was ok to load files from.

I would not be worried if a florist's website had white-listed my website but a bank that has 250 physical branches, 2,500 employees in a country that has 29 million people.

Below is the banks Content Security Policy.

I thought I had been hacked into so I downloaded my Nginx log files (with MobaXTerm,) and scanned them for hits to my site from their website.

After I scanned the logs I could see that I had zero traffic from their website

I sent a direct message to Scott Helme on Twitter (CSP Guru) and he replied with advice on the CSP.

Blocking Traffic

As a precaution, I edited my /etc/nginx/sites-available/default file and added this to block all traffic from their site.

if ($http_referer ~* "##########\.com") {
  return 404;
}
      

I tested and reloaded my Nginx config and restarted my web server

nginx -t
nginx -s reload
/etc/init.d/nginx restart
      

I also emailed my website CDN's admin at https://ewww.io/ and asked them to block traffic from the bank as a precaution. They responded quickly as said this was done and they enabled extra logging in case more information was needed data.

If you need a good and fast WordPress Content Delivery Network (CDN) check out https://ewww.io/. They are awesome. Read my old review of ewww.io here.

I contacted the Bank

I searched the bank's website for a way to contact them, their website was slow, their contact page was limited, they have a chat feature but I needed to log in with FaceBook (I don't use FaceBook)

I viewed their contact us web page and they had zero dedicated security contacts listed. The CIO was only contactable via phone only.

They did not have a security.txt file on their website.

TIP: If you run a website, please consider creating a security.txt file, information here.

I then viewed their contact us page and emailed everyone I could.

I asked if they could..

• Check their logs for malicious files loaded from my site
• Please remove the references to my website and CDN from their CSP.
• Hinted they may want to review your CI/CD logs to see why this happened

My Server Hardening (to date)

My website was already hardened but was my site compromised?

Hardening actions to date..

• Using a VPS firewall, Linux firewall 2x software firewalls
• I have used the free Lynis Scan
• Whitelisting access to port 22 to limited IP's
• Using hardware 2FA keys on SSH and WordPress Logins
• Using the WordFence Security Plugin
• Locked down unwanted ports.
• I had a strong HTTPS certificate and website configuration (test here)
• I have set up appropriate security headers (test here). I did need to re-setup a Content Security Policy (keep reading)
• Performed many actions (some blogged a while ago) here: /article/securing-ubuntu-cloud/
• etc

I had used the free version of Lynis before but now is the time to use the Lynis Enterprise.

A free version of Lynis can be installed from Github here: https://github.com/CISOfy/lynis/

What is Lynis Enterprise?

Lynis Enterprise software is commercial auditing, system hardening, compliance testing tool for AIX, FreeBSD, HP-UX, Linux, macOS, NetBSD, OpenBSD, Solaris etc. The Enterprise version is a paid version (with web portal). Lynis Enterprise has more features over the free version.

Snip from here: "Lynis is a battle-tested security tool for systems running Linux, macOS, or Unix-based operating system. It performs an extensive health scan of your systems to support system hardening and compliance testing. The project is open-source software with the GPL license and available since 2007."

Visit the Lynis Enterprise site here: https://cisofy.com/solutions/#lynis-enterprise.

I created a Lynis Enterprise Trial

I have used the free version of Lynis in the past (read here), but the Enterprise version offers a lot of extra features (read here).

View the main Lynis Enterprise site here and the pricing page here

View a tour of features here: https://cisofy.com/lynis-enterprise/

Create a Cisofy Trial Account

You can request a trial of Lynis Enterprise here: https://cisofy.com/demo/

After the trial account was set up I logged in here. Upon login, I was prompted to add a system to my account (also my licence key was visible)

Install Lynis (Clone GIT Repo/latest features)

I am given 3 options to install Lynis from the add system page here.

• Add the software repository and install the client (The suggested and easiest way to install Lynis and keep it up-to-date).
• Clone the repository from Github (The latest development version, containing the most recent changes)
• Manually install or activate an already installed Lynis.

I will clone a fresh install from Github as I prefer seeing the latest issues, latest changes from GitHub notifications. I like getting notifications about security.

I logged into my server via SSH and ran the following command(s).

sudo apt-get instal git
mkdir /thefolder
cd /thefolder
git clone https://github.com/CISOfy/lynis

Cloning into 'lynis'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10054 (delta 0), reused 1 (delta 0), pack-reused 10047
Receiving objects: 100% (10054/10054), 4.91 MiB | 26.60 MiB/s, done.
Resolving deltas: 100% (7387/7387), done.
      

I logged into https://portal.cisofy.com/ and clicked 'Add' system to find my API key

I noted my licence key.

I then changed to my Lynis folder

cd lynis
    

I then created a "custom.prf" file

touch custom.prf
    

I ran this command to activate my licence (I have replaced my licence with ########'s).

./lynis configure settings license-key=########-####-####-####-############:upload-server=portal.cisofy.com
    

View the documentation here.

Output:

Configuring setting 'license-key'
Setting changed
Configuring setting 'upload-server'
Setting changed
      

I performed my first scan and uploaded the report.

TIP: Make sure you have curl installed

./lynis audit system --upload
    

After the scan is complete, make sure you see the following.

Data upload status (portal.cisofy.com) [ OK ]
    

Data upload status (portal.cisofy.com) [ OK ]

I logged into https://portal.cisofy.com/enterprise/systems/ and I could view my systems report.

You can read the basic Lynis documentation here: https://cisofy.com/documentation/lynis/

Manual Lynis Scans

I can run a manual scan at any time

cd /thefolder/lynis/
sudo ./lynis audit system --upload
      

To view results I can login to https://portal.cisofy.com/

Automated Lynis Scans

I have created a bash script that updates Lynis (basically running 'sudo /usr/bin/git pull origin master' in the lynis folder)

#!/bin/bash

sendemail -f from@yourdomain.com -t recipient@yourdomain.com -u "CRON: Updating Lynis (yourserver.com) START" -m "/folder/runlynis.sh" -s smtp.gmail.com:587 -o tls=yes -xu recipient@yourdomain.com -xp ***my*google*gsuite*email*app*password***

echo "Changing Directory to /folder/lynis"
cd /folder/lynis

echo "Updating Lynis"
sudo /usr/bin/git pull origin master

sendemail -f from@yourdomain.com -t recipient@yourdomain.com -u "CRON: Updated Lynis (yourserver.com) END" -m "/folder/runlynis.sh" -s smtp.gmail.com:587 -o tls=yes -xu recipient@yourdomain.com -xp ***my*google*gsuite*email*app*password***
This is my bash script that runs Lynis scans and emails the report

#!/bin/bash

sendemail -f from@yourdomain.com -t recipient@yourdomain.com -u "CRON: Run Lynis (yourserver.com) START" -m "/folder/runlynis.sh" -s smtp.gmail.com:587 -o tls=yes -xu recipient@yourdomain.com -xp ***my*google*gsuite*email*app*password***

echo "Running Lynis Scan"
cd /utils/lynis/
sudo /utils/lynis/lynis audit system --upload > /folder/lynis/lynis.txt

sendemail -f from@yourdomain.com -t recipient@yourdomain.com -u "CRON: Run Lynis (yourserver.com) END" -m "/folder/runlynis.sh" -s smtp.gmail.com:587 -o tls=yes -xu recipient@yourdomain.com -xp ***my*google*gsuite*email*app*password***  -a /folder/lynis/lynis.txt
I set up two cron jobs to update Lynis (from Git) and to scan with Lynis every day.

#Lynis Update 11:55PM
55 21 * * * /bin/bash /folder/runlynis.sh && curl -fsS --retry 3 https://hc-ping.com/########-####-####-####-############ > /dev/null

#Lynis Scan 2AM
0 2 * * * /bin/bash /folder/runlynis.sh && curl -fsS --retry 3 https://hc-ping.com/########-####-####-####-############ > /dev/null
      

This is my bash script that runs Lynis scans and emails the report

I set up two cron jobs to update Lynis (from Git) and to scan with Lynis every day.

Thanks to sendemail I get daily emails

I have set up cronjob motoring and emails at the start and end of the bash scripts.

The attachment is not a pretty text file report but a least I can see the output of the scan (without logging into the portal).

Maybe I add the following file also

Lynis Enterprise (portal.cisofy.com)

Best of all Lynis Enterprise comes with a great online dashboard available at
https://portal.cisofy.com/enterprise/dashboard/.

Dashboard (portal.cisofy.com)

Clicking the 'Dashboard' button in the toolbar at the top of the portal reveals a summary of your added systems, alerts, compliance, system integrity, Events and statistics.

The dashboard has three levels

• Business (less information)
• Operational
• Technical (more information)

Read about the differences here.

Each dashboard has a limited number of elements, but the technical dashboard has all the elements.

Technical Dashboard

From here you can click and open server scan results (see below)

Server Details

If you click on a server name you can see detailed information. I created 2 test servers (I am using the awesome UpCloud host)

A second menu appears when you click on a server

Test Server 01: Ubuntu 18.04 default Scan Results (66/100)

Test Server 02: Debian 9.9 default Scan Results (65/100)

It is interesting to see Debian is 1 point below Ubuntu.

The server page will give a basic summary and highlights like the current and previous hardening score, open ports, firewall status, installed packages, users.

When I click the server name to load the report I can click to see 'Warnings' or 'Suggestions' to resolve

Suggested System Hardening Actions

I had 47 system hardening recommendations on one system

Some of the security hardening actions included the following.

e.g

Hardening Suggestion (Ignore or Solve)

If you click 'Solve' Cisofy will provide a link to detailed information to help you solve issues.

I will not list every suggested problem and fix but here are some fixes below.

ACCT-9630 Audit daemon is enabled with an empty ruleset. Disable the daemon or define rules (fixed)

TIP: If you don't have auditd installed run this command below to install it

sudo apt-get install auditd
/etc/init.d/auditd start
/etc/init.d/auditd status
      

I added the following to '/etc/audit/rules.d/audit.rules' (thanks to the solution recommendations on the Cisofy portal.

# This is an example configuration suitable for most systems
# Before running with this configuration:
# - Remove or comment items which are not applicable
# - Check paths of binaries and files

###################
# Remove any existing rules
###################

-D

###################
# Buffer Size
###################
# Might need to be increased, depending on the load of your system.
-b 8192

###################
# Failure Mode
###################
# 0=Silent
# 1=printk, print failure message
# 2=panic, halt system
-f 1

###################
# Audit the audit logs.
###################
-w /var/log/audit/ -k auditlog

###################
## Auditd configuration
###################
## Modifications to audit configuration that occur while the audit (check your paths)
-w /etc/audit/ -p wa -k auditconfig
-w /etc/libaudit.conf -p wa -k auditconfig
-w /etc/audisp/ -p wa -k audispconfig

###################
# Monitor for use of audit management tools
###################
# Check your paths
-w /sbin/auditctl -p x -k audittools
-w /sbin/auditd -p x -k audittools

###################
# Special files
###################
-a exit,always -F arch=b32 -S mknod -S mknodat -k specialfiles
-a exit,always -F arch=b64 -S mknod -S mknodat -k specialfiles

###################
# Mount operations
###################
-a exit,always -F arch=b32 -S mount -S umount -S umount2 -k mount
-a exit,always -F arch=b64 -S mount -S umount2 -k mount

###################
# Changes to the time
###################
-a exit,always -F arch=b32 -S adjtimex -S settimeofday -S stime -S clock_settime -k time
-a exit,always -F arch=b64 -S adjtimex -S settimeofday -S clock_settime -k time
-w /etc/localtime -p wa -k localtime

###################
# Use of stunnel
###################
-w /usr/sbin/stunnel -p x -k stunnel

###################
# Schedule jobs
###################
-w /etc/cron.allow -p wa -k cron
-w /etc/cron.deny -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /etc/cron.daily/ -p wa -k cron
-w /etc/cron.hourly/ -p wa -k cron
-w /etc/cron.monthly/ -p wa -k cron
-w /etc/cron.weekly/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/crontabs/ -k cron

## user, group, password databases
-w /etc/group -p wa -k etcgroup
-w /etc/passwd -p wa -k etcpasswd
-w /etc/gshadow -k etcgroup
-w /etc/shadow -k etcpasswd
-w /etc/security/opasswd -k opasswd

###################
# Monitor usage of passwd command
###################
-w /usr/bin/passwd -p x -k passwd_modification

###################
# Monitor user/group tools
###################
-w /usr/sbin/groupadd -p x -k group_modification
-w /usr/sbin/groupmod -p x -k group_modification
-w /usr/sbin/addgroup -p x -k group_modification
-w /usr/sbin/useradd -p x -k user_modification
-w /usr/sbin/usermod -p x -k user_modification
-w /usr/sbin/adduser -p x -k user_modification

###################
# Login configuration and stored info
###################
-w /etc/login.defs -p wa -k login
-w /etc/securetty -p wa -k login
-w /var/log/faillog -p wa -k login
-w /var/log/lastlog -p wa -k login
-w /var/log/tallylog -p wa -k login

###################
# Network configuration
###################
-w /etc/hosts -p wa -k hosts
-w /etc/network/ -p wa -k network

###################
## system startup scripts
###################
-w /etc/inittab -p wa -k init
-w /etc/init.d/ -p wa -k init
-w /etc/init/ -p wa -k init

###################
# Library search paths
###################
-w /etc/ld.so.conf -p wa -k libpath

###################
# Kernel parameters and modules
###################
-w /etc/sysctl.conf -p wa -k sysctl
-w /etc/modprobe.conf -p wa -k modprobe
###################

###################
# PAM configuration
###################
-w /etc/pam.d/ -p wa -k pam
-w /etc/security/limits.conf -p wa -k pam
-w /etc/security/pam_env.conf -p wa -k pam
-w /etc/security/namespace.conf -p wa -k pam
-w /etc/security/namespace.init -p wa -k pam

###################
# Puppet (SSL)
###################
#-w /etc/puppet/ssl -p wa -k puppet_ssl

###################
# Postfix configuration
###################
#-w /etc/aliases -p wa -k mail
#-w /etc/postfix/ -p wa -k mail
###################

###################
# SSH configuration
###################
-w /etc/ssh/sshd_config -k sshd

###################
# Hostname
###################
-a exit,always -F arch=b32 -S sethostname -k hostname
-a exit,always -F arch=b64 -S sethostname -k hostname

###################
# Changes to issue
###################
-w /etc/issue -p wa -k etcissue
-w /etc/issue.net -p wa -k etcissue

###################
# Log all commands executed by root
###################
-a exit,always -F arch=b64 -F euid=0 -S execve -k rootcmd
-a exit,always -F arch=b32 -F euid=0 -S execve -k rootcmd

###################
## Capture all failures to access on critical elements
###################
-a exit,always -F arch=b64 -S open -F dir=/etc -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/bin -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/home -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/sbin -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/srv -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/usr/bin -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/usr/local/bin -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/usr/sbin -F success=0 -k unauthedfileacess
-a exit,always -F arch=b64 -S open -F dir=/var -F success=0 -k unauthedfileacess

###################
## su/sudo
###################
-w /bin/su -p x -k priv_esc
-w /usr/bin/sudo -p x -k priv_esc
-w /etc/sudoers -p rw -k priv_esc

###################
# Poweroff/reboot tools
###################
-w /sbin/halt -p x -k power
-w /sbin/poweroff -p x -k power
-w /sbin/reboot -p x -k power
-w /sbin/shutdown -p x -k power

###################
# Make the configuration immutable
###################
-e 2

# EOF
      

I reloaded my audit daemon config

auditctl -R /etc/audit/rules.d/audit.rules
      

Further configuration can be added (read this), read the auditd man page here or read logs you can use the 'auditsearch' tool (read the Ubuntu Man Page here)

Here is a great guide on viewing audit events.

Because we have this rule ( '-w /etc/passwd -p wa -k etcpasswd ) to monitor the passwords file, If I read the contents of \etc\passwd it will show up in the audit logs.

We can verify the access of this file by running this command

ausearch -f /etc/passwd
      

Output

      ausearch -f /etc/passwd
----
time->Mon Jun 10 16:58:13 2019
type=PROCTITLE msg=audit(##########.897:3639): proctitle=##########################
type=PATH msg=audit(##########.897:3639): item=1 name="/etc/passwd" inode=1303 dev=fc:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(##########.897:3639): item=0 name="/etc/" inode=12 dev=fc:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(##########.897:3639): cwd="/root"
type=SYSCALL msg=audit(##########.897:3639): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=556241ea9650 a2=441 a3=1b6 items=2 ppid=1571 pid=1572 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=446 comm="nano" exe="/bin/nano" key="etcpasswd"
      

I might write a list of handy ausearech commands and blog about this in the future

SSH Permissions (fixed)

To fix the ssh permissions warning I ran the command to show the issue on my server

./lynis show details FILE-7524
2019-05-25 23:00:04 Performing test ID FILE-7524 (Perform file permissions check)
2019-05-25 23:00:04 Test: Checking file permissions
2019-05-25 23:00:04 Using profile /utils/lynis/default.prf for baseline.
2019-05-25 23:00:04 Checking /etc/lilo.conf
2019-05-25 23:00:04   Expected permissions:
2019-05-25 23:00:04   Actual permissions:
2019-05-25 23:00:04   Result: FILE_NOT_FOUND
2019-05-25 23:00:04 Checking /root/.ssh
2019-05-25 23:00:04   Expected permissions: rwx------
2019-05-25 23:00:04   Actual permissions: rwxr-xr-x
2019-05-25 23:00:04   Result: BAD
2019-05-25 23:00:04 Warning: Incorrect permissions for file /root/.ssh [test:FILE-7524] [details:-] [solution:-]
2019-05-25 23:00:04 Using profile /utils/lynis/custom.prf for baseline.
2019-05-25 23:00:04 Checking permissions of /utils/lynis/include/tests_homedirs
2019-05-25 23:00:04 File permissions are OK
2019-05-25 23:00:04 ===---------------------------------------------------------------===
      

I tightened permissions on the /root/.ssh folder with this command

chmod 700 /root/.ssh
      

Configure minimum/maximum password age in /etc/login.defs (fixed)

I set a maximum and minimum password age in '/etc/login.defs'

Defaults

PASS_MAX_DAYS   99999
PASS_MIN_DAYS   0
PASS_WARN_AGE   7
      

Add a legal banner to /etc/issue, to warn unauthorized users (fixed)

I edited '/etc/issue' on Ubuntu and Linux

Ubuntu 18.04 default

Ubuntu 18.04.2 LTS \n \l
      

Debian Default

Debian GNU/Linux 9 \n \l
      

Cisofy said this "Define a banner text to inform both authorized and unauthorized users about the machine and service they are about to access. The purpose is to share your policy before an access attempt is being made. Users should know that there privacy might be invaded, due to monitoring of the system and its resources, to protect the integrity of the system. Also unauthorized users should be deterred from trying to access it in the first place."

Done

Default umask in /etc/login.defs could be more strict like 027 (fixed)

Related files..

• /etc/profile
• /etc/login.defs
• /etc/passwd

I edited '/etc/login.defs' and set

I ran

umask 027 /etc/profile
umask 027 /etc/login.defs
umask 027 /etc/passwd
      

Check iptables rules to see which rules are currently not used (fixed)

I ran the following command to review my firewall settings

iptables --list --numeric --verbose
      

TIP: Scan for open ports with 'nmap'

Watch this handy video if you are not sure how to use nmap

Install nmap

sudo apt-get install nmap
      

I do set firewall rules in ufw (guide here) and ufw is a front end for iptables.

Scan for open ports with nmap

nmap -v -sT localhost

Starting Nmap 7.60 ( https://nmap.org ) at 2019-06-12 22:09 AEST
Initiating Connect Scan at 22:09
Scanning localhost (127.0.0.1) [1000 ports]
Discovered open port 443/tcp on 127.0.0.1
Discovered open port 22/tcp on 127.0.0.1
Discovered open port 8080/tcp on 127.0.0.1
Discovered open port 25/tcp on 127.0.0.1
Discovered open port 80/tcp on 127.0.0.1
Completed Connect Scan at 22:09, 0.02s elapsed (1000 total ports)
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00012s latency).
Not shown: 994 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
443/tcp  open  https
8080/tcp open  http-proxy
      

Everything looked good.

Harden compilers like restricting access to root user only (fixed)

Cicofy said

Compilers turn source code into binary executable code. For a production system a compiler is usually not needed, unless package upgrades are performed by means of their source code (like FreeBSD ports collection). If a compiler is found, execution should be limited to authorized users only (e.g. root user).

To solve this finding, remove any unneeded compiler or change the file permissions. Usually chmod 700 or chmod 750 will be enough to prevent normal users from using a compiler. Related compilers are as, cc, ld, gcc, go etc. To determine what files are affected, check the Lynis log file, then chmod these files.
      

I ran

Chmod 700 /usr/bin/as
chmod 700 /usr/bin/gcc
      

Turn off PHP information exposure (fixed)

Cisofy siad

Disable the display of version information by setting the expose_php option to 'Off' in php.ini. As several instances of PHP might be installed, ensure that all related php.ini files have this setting turned off, otherwise this control will show up again.
      

This was already turned off but a unused php.ini may have been detected.

I searched for all php.ini files

find / -name php.ini
      

Output

/etc/php/7.3/apache2/php.ini
/etc/php/7.3/fpm/php.ini
/etc/php/7.3/cli/php.ini
      

yep, the cli version of php.ini had the following

I set this to Off

Purge old/removed packages (59 found) with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. (fixed)

Cisofy said

While not directly a security concern, unpurged packages are not installed but still have remains left on the system (e.g. configuration files). In case software is reinstalled, an old configuration might be applied. Proper cleanups are therefore advised.

To remove the unneeded packages, select the ones marked with the 'rc' status. This means the package is removed, but the configuration files are still there.

I ran the following recommended command

dpkg -l | grep "^rc" | cut -d " " -f 3 | xargs dpkg --purge
      

Done

Install debsums utility for the verification of packages with known good database. (fixed)Cisofy said

Install the debsums utility to do more in-depth auditing of your packages.

I ran the following suggested command

apt-get remove --purge python-pip
      

I googled and found this handy page

I scanned packages and asked 'debsums'' to only show errors with this command

The only error was..

I did not need pip so I removed it

Install a PAM module for password strength testing like pam_cracklib or pam_passwdqc (fixed)

I ignore this as I do not allow logins via password and only I have an account (it's not a multi user system).

I white list logins to IP's.

I only allow ssh access with a private key and long passphrase.

I have 2FA OTP enabled at logins.

I have cloudflare over my domain.

I setup fail2ban to auto block logins using this guide

Reboot (fixed)

I restated the server

shutdown -r now
      

Done

Check available certificates for expiration (fixed)

I tested my SSL certificate with https://dev.ssllabs.com

Add legal banner to /etc/issue.net, to warn unauthorized users (fixed)

Cisofy said...

Define a banner text to inform both authorized and unauthorized users about the machine and service they are about to access. The purpose is to share your policy before an access attempt is being made. Users should know that there privacy might be invaded, due to monitoring of the system and its resources, to protect the integrity of the system. Also unauthorized users should be deterred from trying to access it in the first place.

Do not reveal sensitive information, like the specific goal of the machine, or what can be found on it. Consult with your legal department, to determine appropriate text.

I edited the file '/etc/issue.net' and added a default pre login message (same as '/etc/issue').

Install Apache mod_evasive to guard webserver against DoS/brute force attempts (ignored)

I ignored this message and I don't use the Apache (I use the Nginx web server). I have added Apache to be blocked from installing.

I clicked Ignore in the Cisofy portal.

Install Apache modsecurity to guard webserver against web application attacks (ignored)

I clicked Ignore for this one too

Check your Nginx access log for proper functioning (reviewed)

Cisofy said...

Disabled logging:
Check in the Lynis log for entries which are disabled, or in the nginx configuration (access_log off).

Missing logging:
Check for missing log files. They are references in the configuration of nginx, but not on disk. The Lynis log will reveal to what specific files this applies.

I checked my Nginx config ('/etc/nginx/nginx.conf') for all log references and ensured the logs were writing to disk (OK).

I checked my '/etc/nginx/sites-available/default' config and I did have 2 settings of 'access_log off ' (this was added during the setup for two sub reporting subfolders for the Nixstats agent.

I restarted Nginx

nginx -t
nginx -s reload
/etc/init.d/nginx restart
      

Check what deleted files are still in use and why. (fixed)

Cisofy said..

Why it matters
Deleted files may sometimes be in use by applications. Normally this should not happen, as an application should delete a file and release the file handle. This test might discover malicious software, trying to hide its presence on the system. Investigate the related files by determining which application keeps it open and the related reason.

Details
The following details have been found as part of the scan.

/lib/systemd/systemd-logind(systemd-l)
/tmp/ib1ekCtf(mysqld)
/tmp/ibhuK1At(mysqld)
/tmp/ibmTO5F5(mysqld)
/tmp/ibR0dkxD(mysqld)
/tmp/ibvf69KH(mysqld)
/tmp/.ZendSem.gq3mnz(php-fpm7.)
/usr/bin/python3.6(networkd-)
/usr/bin/python3.6(unattende)
/var/log/mysql/error.log.1(mysqld)

I ran the following command to show deleted files in use

lsof | grep deleted
      

I noticed on my database server a php-fpm service was using files. I don't have a webserver enabled on this server, so I uninstalled the web-based services.

I have separate web and database servers.

sudo apt-get remove apache*
sudo apt-get remove -y --purge nginx*
sudo apt-get remove -y --purge php7*
sudo apt autoremove
      

Check DNS configuration for the dns domain name (fixed)

Cisofy said..

Some software can work incorrectly when the system can't resolve itself. 
Add the IP name and fully qualified domain name (FQDN) to /etc/hosts. Usually this is done with an entry of 127.0.0.1, or 127.0.1.1 (to leave the localhost entry alone). 

I edited my '/etc/hosts' file

I added a domain name to the end of the localhost entry and added a new line with my server(s) IP and domain name

Disable the 'VRFY' command (fixed)

I was advised to run this command

postconf -e disable_vrfy_command=yes
      

(Debian) Enable sysstat to collect accounting (no results) (fixed)

Cisofy said..

The sysstat is collection of utilities to provide system information insights. While one should aim for the least amount of packages, the sysstat utilities can be a good addition to help recording system details. They can provide insights for performance monitoring, or guide in discovering unexpected events (like a spam run). If you already use extensive system monitoring, you can safely ignore this control.

I ran the suggested commands

apt-get install sysstat
sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
      

More info on sysstat here.

Consider running ARP monitoring software (arpwatch,arpon) (fixed)

Cisofy said

Networks are very dynamic, often with devices come and go as they please. For sensitive machines and network zones, you might want to know what happens on the network itself. An utility like arpwatch can help tracking changes, like new devices showing up, or others leaving the network.

I read this page to setup and configure arpwatch

sudo apt-get install arpwatch
/etc/init.d/arpwatch start
      

I will add more on how to use arpwatch soon

Disable drivers like USB storage when not used, to prevent unauthorized storage or data theft (fixed)

Cosofy siad..

Disable drivers like USB storage when not used. This helps preventing unauthorized storage, data copies, or data theft.

I ran the suggested fix

echo "# Block USB storage" >> /etc/modprobe.d/disable-usb-storage.conf
echo "install usb-storage /bin/false" >> /etc/modprobe.d/disable-usb-storage.conf
      

Determine if automation tools are present for system management (ignored)

I ignored this one

One or more sysctl values differ from the scan profile and could be tweaked

Cisofy said..

By means of sysctl values we can adjust kernel related parameters. Many of them are related to hardening of the network stack, how the kernel deals with processes or files. This control is a generic test with several sysctl variables (configured by the scan profile).

I was advised to adjust these settings

• net.ipv4.conf.all.send_redirects=0
• net.ipv4.conf.default.accept_source_route=0
• kernel.sysrq=0
• net.ipv4.conf.all.log_martians=1
• net.ipv4.conf.default.log_martians=1
• kernel.core_uses_pid=1
• kernel.kptr_restrict=2
• fs.suid_dumpable=0
• kernel.dmesg_restrict=1

I edited '/etc/sysctl.conf' and made the advised changes along with these (I Googled each item first)

Install a file integrity tool to monitor changes to critical and sensitive files (fixed)

Cisofy said..

To monitor for unauthorized changes, a file integrity tool can help with the detection of such event. Each time the contents or the properties of a file change, it will have a different checksum. With regular checks of the related integrity database, discovering changes becomes easy. Install a tool like AIDE, Samhain or Tripwire to monitor important system and data files. Additionally configure the tool to alert system or security personnel on events.

It also gave a solution

I installed 'aide' (read the guide here).

TIP: Long story but the steps above were not exactly correct. Thanks to this post for I was able to set up aide. without seeing this error.

Couldn't open file /var/lib/aide/please-dont-call-aide-without-parameters/aide.db.new for writing

This is how I installed aide

apt-get install aide
apt-get install aide-common
      

I initialised aide.

aideinit
      

This was the important part (I was stuck for hours on this one)

aide.wrapper --check
      

I can run the following to see what files have changed.

I could see many files have changed since the initial scan (e.g mysql, log files nano search history).

Nice

Now lets schedule daily checks and create a cron job.

cat /folder/runaide.sh
#!/bin/bash

sendemail -f from@yourdomain.com -t recipient@yourdomain.com -u "CRON: AIDE Run (yourserver.com) START" -m "/folder/runaide.sh" -s smtp.gmail.com:587 -o tls=yes -xu recipient@yourdomain.com -xp ***my*google*gsuite*email*app*password***

MYDATE=`date +%Y-%m-%d`
MYFILENAME="Aide-"$MYDATE.txt
/bin/echo "Aide check !! `date`" > /tmp/$MYFILENAME
/usr/bin/aide.wrapper --check > /tmp/myAide.txt
/bin/cat /tmp/myAide.txt|/bin/grep -v failed >> /tmp/$MYFILENAME
/bin/echo "**************************************" >> /tmp/$MYFILENAME
/usr/bin/tail -100 /tmp/myAide.txt >> /tmp/$MYFILENAME
/bin/echo "****************DONE******************" >> /tmp/$MYFILENAME

#/usr/bin/mail -s"$MYFILENAME `date`" recipient@yourdomain.com < /tmp/$MYFILENAME

sendemail -f from@yourdomain.com -t recipient@yourdomain.com -u "CRON: AIDE Run (yourserver.com) END" -m "/folder/runaide.sh" -s smtp.gmail.com:587 -o tls=yes -xu recipient@yourdomain.com -xp ***my*google*gsuite*email*app*password*** -a /tmp/$MYFILENAME -a /tmp/myAide.txt

      

Above thanks to this post

I setup a cron job to run this daily

ACCT-9622 - Enable process accounting. (fixed)

Solution:

Install "acct" process and login accounting.

sudo apt-get install acct
      

Start the "acct" service

etc/init.d/acct start
touch /var/log/pacct
chown root /var/log/pacct
chmod 0644 /var/log/pacct
accton /var/log/pacct 
      

Check the status

/etc/init.d/acct status
* acct.service - LSB: process and login accounting
   Loaded: loaded (/etc/init.d/acct; generated)
   Active: active (exited) since Sun 2019-05-26 19:42:15 AEST; 4min 42s ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 0 (limit: 4660)
   CGroup: /system.slice/acct.service

May 26 19:42:15 servername systemd[1]: Starting LSB: process and login accounting...
May 26 19:42:15 servername acct[27419]: Turning on process accounting, file set to '/var/log/account/pacct'.
May 26 19:42:15 servername systemd[1]: Started LSB: process and login accounting.
      

Run CISOfy recommended commands

touch /var/log/pacct
chown root /var/log/pacct
chmod 0644 /var/log/pacct
accton /var/log/pacct 
      

Manual Scan of Lynis

I re-ran an audit of the system (and uploaded the report to the portal) so I can see how I am progressing.

lynis audit system --upload
      

I then checked the error status and the warnings were resolved.

Progress?

I rechecked my servers and all warnings are solved, now I just need to work on information level issues

Cisofy Portal Overview

Quick breakdown of the Cisofy Portal

Overview Tab (portal.cisofy.com)

The Overview lab displays any messages, change log, API information, add a new system link, settings etc.

Dashboard Tab (portal.cisofy.com)

The dashboard tab will display compliant systems any outdated systems, alerts and events.

TIP: If you have a system that reports "Outdated" run the following command.

lynis audit system --upload
      

Systems Tab (portal.cisofy.com)

The systems tab shows all systems, OS version, warnings, information counts, the date the system's client last uploaded a report and the client version.

If you are making many changes and manual Lynis scans keep an eye on your upload credits, You can see by the above and below image, I have lowered my suggested actions to harden my servers (red text).

Clicking a host name reveals a summary of the system.

Remaining information level issues are listed.

I can click Solve and see more information about the issue to resolve.

TIP: I thought it would be a good idea to copy this list to a spreadsheet for detailed tracking.

I had another issue appear a few days later.

Compliance Tab (portal.cisofy.com)

A lot of information is listed here.

Best practice guides are available

I could go on an on but https://cisofy.com/ is awesome.

TIP: Manually updating Lynis

./lynis --version
2.7.4
      

from the command line I can view the Linus version with this command

To update the Lynis git repository from the Lynis folder run this command

git pull
Already up to date.
      

Automatically updating and running Lynis scans

I added the following commands to my crontab to update then scan and report Lynis results to the portal.

TIP: Use https://crontab.guru/ to choose the right time to run commands (I chose 5 mins past 1 AM every day to update and 5 mins past 2 AM to run a scan.

#Lynis Update
5 1 * * * root -s /bin/bash -c 'cd /utils/lynis && /usr/bin/git pull origin master'

#Lynis Scan
5 2 * * * root -s /bin/bash -c '/utils/lynis/lynis audit system --upload'
      

Troubleshooting

fyi: Lynis Log file location: /var/log/lynis.log

Cisofy Enterprise Conclusion

Pros:

• I can learn so much about securing Linux just from the Cisofy Fix recommendations.
• I have secured my server beyond what I thought possible.
• Very active development on Github: https://github.com/CISOfy/lynis/
• Cisofy has a very good inteface and updates often.
• New security issues are synced down and included in new scans (if you update)

Cons:

• I am unable to pay for this for my servers here in Australia (European legal issues).
• Needs Hardware 2FA

Tips

Make sure you have curl installed to allow reports to upload. I had this error on Debian 9.4.

View the latest repository version information here.

I added my Lynis folder to the Linux $PATH variable

export PATH=$PATH:/folder/lynis
    

Fatal: can't find curl binary. Please install the related package or put the binary in the PATH. Quitting..

Lynis Enterprise API

View the Lynis Enterprise API documentation here

Lynis Enterprise Support

Support can be found here, email support support@cisofy.com.

Getting started guide is found here.

Bonus: Setting Up Content Security Policy and reporting violations to https://report-uri.com/

I have a few older posts on Content Security Policies (CSP) but they are a bit dated.

• 2016 -Beyond SSL with Content Security Policy, Public Key Pinning etc
• 2018 - Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx

Wikipedia Definition of a Content Security Policy

Content Security Policy (CSP) is a computer security standard introduced to prevent cross-site scripting (XSS), clickjacking and other code injection attacks resulting from execution of malicious content in the trusted web page context.[1] It is a Candidate Recommendation of the W3C working group on Web Application Security,[2] widely supported by modern web browsers.[3] CSP provides a standard method for website owners to declare approved origins of content that browsers should be allowed to load on that website—covered types are JavaScript, CSS, HTML frames, web workers, fonts, images, embeddable objects such as Java applets, ActiveX, audio and video files, and other HTML5 features.

If you want to learn about to setup CSP's head over to https://report-uri.com/products/content_security_policy or https://report-uri.com/home/tools and read more.

I did have Content Security Policies (CSP) set up a few years back, but I had issues with broken resources. A lack of time on my behalf to investigate the issues forced me to disable the Content Security Policy (CSP). I should have changed the "Content-Security-Policy" header to "Content-Security-Policy-Report-Only."

I will re-add the Content Security Policy (CSP) to my site but this time I will not disable it and will report to https://report-uri.com/, and if need be I will change the header from "content-security-policy" to "content-security-policy-report-only". That way a broken policy won't take down my site in future.

If you want to set up a Content Security Policy header and with good reporting of any violations of your CSP policy simply head over to https://report-uri.com/ and create a new account.

Read the official Report URI help documents here: https://docs.report-uri.com/.

Create a Content Security Policy

The hardest part of creating a Content Security Policy is knowing what to add where.

You could generate your own Content Security Policy by heading here (https://report-uri.com/home/generate) but that will take a while.

TIP: Don't make your policy live straight away by using the "Content-Security-Policy" header, instead use the "Content-Security-Policy-Report-Only" header.

To create a content Security Policy faster I would recommend you to use this Firefox plugin to generate a starter Content Security Policy.

Install this plugin to Firefox, enable it and click the Plugins icon and ensure "Record this site..." is ticked.

Then simply browse to your site (browse as many pages as possible) and a Content Security Policy will be generated based on the content on the page(s) loaded.

TIP: Always review the generated CSP, it allows everything needed to display your site.

This is the policy that was generated for me in 5 minutes browsing 20 pages.

default-src 'none'; connect-src 'self' https://onesignal.com/api/v1/apps/772f27ad-0d58-494f-9f06-e89f72fd650b/icon https://onesignal.com/api/v1/notifications https://onesignal.com/api/v1/players/67a2f360-687f-4513-83e8-f477da085b26 https://onesignal.com/api/v1/players/67a2f360-687f-4513-83e8-f477da085b26/on_session https://yoast.com/feed/widget/; font-src 'self' data: https://fearby-com.exactdn.com https://fonts.gstatic.com; form-action 'self' https://fearby.com https://syndication.twitter.com https://www.paypal.com; frame-src 'self' https://en-au.wordpress.org https://fearby.com https://googleads.g.doubleclick.net https://onesignal.com https://platform.twitter.com https://syndication.twitter.com https://www.youtube.com; img-src 'self' data: https://a.impactradius-go.com https://abs.twimg.com https://fearby-com.exactdn.com https://healthchecks.io https://pagead2.googlesyndication.com https://pbs.twimg.com https://platform.twitter.com https://secure.gravatar.com https://syndication.twitter.com https://ton.twimg.com https://www.paypalobjects.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' https://adservice.google.com.au/adsid/integrator.js https://adservice.google.com/adsid/integrator.js https://cdn.onesignal.com/sdks/OneSignalPageSDKES6.js https://cdn.onesignal.com/sdks/OneSignalSDK.js https://cdn.syndication.twimg.com/tweets.json https://fearby-com.exactdn.com/wp-content/cache/fvm/1553589606/out/footer-45a3439e.min.js https://fearby-com.exactdn.com/wp-content/cache/fvm/1553589606/out/footer-e6604f67.min.js https://fearby-com.exactdn.com/wp-content/cache/fvm/1553589606/out/footer-f4213fd6.min.js https://fearby-com.exactdn.com/wp-content/cache/fvm/1553589606/out/header-1583146a.min.js https://fearby-com.exactdn.com/wp-content/cache/fvm/1553589606/out/header-823c0a0e.min.js https://fearby-com.exactdn.com/wp-content/piwik.js https://onesignal.com/api/v1/sync/772f27ad-0d58-494f-9f06-e89f72fd650b/web https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js https://pagead2.googlesyndication.com/pagead/js/r20190610/r20190131/show_ads_impl.js https://pagead2.googlesyndication.com/pub-config/r20160913/ca-pub-9241521190070921.js https://platform.twitter.com/js/moment~timeline~tweet.a20574004ea824b1c047f200045ffa1e.js https://platform.twitter.com/js/tweet.73b7ab8a56ad3263cad8d36ba66467fc.js https://platform.twitter.com/widgets.js https://s.ytimg.com/yts/jsbin/www-widgetapi-vfll-F3yY/www-widgetapi.js https://www.googletagservices.com/activeview/js/current/osd.js https://www.youtube.com/iframe_api; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com/ https://onesignal.com/sdks/ https://platform.twitter.com/css/ https://ton.twimg.com/tfw/css/; worker-src 'self' 
      

I can truncate starter Content Security Polity and remove some elements. Remove duplicated entries to separate files on a remote server add a wildcard (if I trust the server).

I truncated the policy with the help of the sublime text editor and Report URI CSP Generator.

I added this to the file '/etc/nginx/sites-available/default'

I added the following to the file '/etc/nginx/sites-available/default' (inside the server node).

add_header "Content-Security-Policy-Report-Only" "default-src 'self' https://fearby.com/; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://adservice.google.com.au https://adservice.google.com https://cdn.onesignal.com https://cdn.syndication.twimg.com https://fearby-com.exactdn.com https://onesignal.com https://pagead2.googlesyndication.com https://platform.twitter.com https://s.ytimg.com https://www.googletagservices.com https://www.youtube.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com https://onesignal.com https://platform.twitter.com https://ton.twimg.com; img-src 'self' data: https://a.impactradius-go.com https://abs.twimg.com https://fearby-com.exactdn.com https://healthchecks.io https://pagead2.googlesyndication.com https://pbs.twimg.com https://platform.twitter.com https://secure.gravatar.com https://syndication.twitter.com https://ton.twimg.com https://www.paypalobjects.com; font-src 'self' data: https://fearby-com.exactdn.com https://fonts.gstatic.com; connect-src 'self' https://onesignal.com https://yoast.com; object-src https://fearby.com/; frame-src 'self' https://en-au.wordpress.org https://fearby.com https://googleads.g.doubleclick.net https://onesignal.com https://platform.twitter.com https://syndication.twitter.com https://www.youtube.com; worker-src 'self'; form-action 'self' https://fearby.com https://syndication
      

Any issues with the Content Security policy will be reported to my web browsers development console and to https://report-uri.com/.

My Chrome development console reports an issue with a graphic not loading from Namecheap.

The event was also reported to the Report URI server.

Don't forget to check the reports often. When you have no more issues left you can make the Policy live by renaming the "Content-Security-Policy-Report-Only" header to "Content-Security-Policy".

FYI: I had directive reports of 'script-src-elem' and it looks like they are new directives added to Chrome 75.

Don't forget to visit the Report URI setup page and get a URL for where live reports get sent to.

If you go to the Generate CSP page and import your website's policy you can quickly add new exclusions to your policy

After a few months of testing and tweaking the policy, I can make it live ('Content-Security-Policy').

Lynis Enterprise

I have learned so much by using Lynis Enterprise from https://cisofy.com/

I am subscribed to issues notifications at https://github.com/CISOfy/lynis/issues/ and observe about 20 notifications a day in this GitHub community. Maybe one day I will contribute to this project?

Finally, Did the Bank reply?

Yes but it was not very informative.

I tried to inquire how this happened and each time the answer was vague.

It looks like the bank used my blog post to create their CSP.

Oh well at least I have a secured my servers.

Thanks for reading.

Version:

v1.1 - Changed the URL, Removed Ads and added a Lynis Enterprise Conclusion

v1.01 - Fixed the URL

v1.0 - Initial Version