IoT, Code, Security and Server Stuff

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

Terminal

How to send email via G Suite from Ubuntu in the cloud

by

Here is how I send emails from the command line in Ubuntu servers in the cloud via G SUote connect emails

Jan 2018 Update

Post on adding a second domain to G Suite

Post on adding email aliases to G Suite

Main

If you use ufw for your Ubuntu firewall then allow port 587 out traffic (read more in securing Ubuntu in the cloud here).

Ensure your port is open on IPV4 and IPV6.

If you have a GUI managed firewall with your server host then configure it to allow port 587 (out).

Read more on useful terminal commands here or setting up a Digital Ocean Ubuntu server here for as low as $5 a month here, Vultr server for as low as $2.5 here ($10 free credit). Read more about setting up an AWS Ubuntu server here.

Install sendmail and other pre requisites

Now you are ready to send an email with 1 line in the terminal (use your Gmail email unless you have diverted your email to G Suite (then use that email, guide here). Create a G Suite account here.

FYI: I have setup my email for my domain to redirect via G Suite (see my guide here and older guide here)

Send an email from the command line

This is not a drop-in replacement for Outlook or Thunderbird email clients but it is perfect for command-line alerts to con-jobs or start-up notifications.

Sending an email with an attachment

Coming soon: A guide on backing up Ubuntu with Rsync etc.

More

Post on adding a second domain to G Suite

Post on adding email aliases to G Suite

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

v1.2 added other links

v1.1 added links to two G Suite guides.

Useful Linux Terminal Commands

by

Below are Ubuntu Linux commands I use often to setup, debug maintain servers.

Read this guide for Useful OSX Commands (for setting up Apache, PHP, MySQL, Adminer etc on OSX)

I recently moved my domain from a C-Panel hosted domain (and Email to Google G Suite (my guide here)) to a self-managed Digital Ocean domain (my LetsEncrypt Guide here, my Digital Ocean guide here, my AWS setup guide here, my Vultr setup guide here) and needed to transfer my WordPress site. Setup your own Digital Ocean Ubuntu server form $5  a month (get the first 2 months free by clicking here) or setup your own Vultr Ubuntu server for as low as $2.5/month by clicking here.

How to Reboot (from time to time when prompted with *** system restart required ***” messages appear).

How to set up a  bash file (*.sh) as Executable.

The file will now be executable.

Viewing your crontab (Windows Task Scheduler equiv)

Ping a port

Rename a folder

Set the owner of a folder

Check the rsync port

Run a file every 1 minute

Show server name

How to verify patch status for Meltdown and Spectre

Read my guide here to install the patch here.

Verify Spectre and Meltdown patch status

or

Restart network

More here.

Show Operating System Name

hostnamectl ! grep "Operating System"
 Operating System: Ubuntu 10.04.3 LTS

Show installed Packages

Show all packages with “PHP” in the name.

Send Messages Other Logged In Users (CLI)

Show all user logged in

Sent a message to usr1

Sent a message to the root console

Messages will appear at the bottom of the user’s console.

Processor

List processes in a tree view (… = removed)

View major processes by usage/memory

Read more on ps here.

How big is a folder

Change FIle Create/Modify time’s

Change File Creation Date

Change file modify/accessed time

Tree

Tree needs to be installed first

Show an ASCII representation of a folder structure

Show files in a  structure

Show directories

List all files and folders in a structure

Backup a  www folder

Common (Digital Ocean) Debugging commands

Networking

Display all TCP connections

Display all UDP connections

List all Listening Connections

Show all Network stats

Show all UDP Network stats

Show all TCP Network stats

Show network packets

Displaying raw info

Show open ports

Upload a file to a remote server over ssh

Zip files

Install zip

Zip a  whole directory (with high compression)

Zip a  whole directory (excluding *.tmp, *temp, *.bak and *.zip file types)

Zip Help

Backup NGINX

Unzip

Installing Unzip

Unzip a  file

unzip filename.zip

Updates

Setup Unattended Ubuntu Security updates

At login, you should receive

Show Configured NGINX server names

Services

Reload daemon services

Verify the MongoDB service file exists

View the status of services

Local Dump

Show All Defined Ports

Show defined rsync ports

Show listening ports (active connections)

Show service status

View your bash history for a past command (e.g “openssl”).

grep "openssl" ~/.bash_history
openssl req -new -newkey rsa:4096 -nodes -keyout fearby.key -out fearby.csr

View last 1 line of a file.

Ping a server

How to check system uptime (and load average)

Uptime in friendly format

The load averages at the end are the last 1, 5 min and 15 minutes.

The w command is handy for showing uptime information as-well as logged in users (The ‘w -i’  parameter -i is handy for seeing what IP people are logged in from).

How to download a file

How to edit NGINX configuration

sudo nano /etc/nginx/nginx.conf

and

How to find a file

also

locate php.ini

Find contents in files (recursive)

Find files by name and run a command on each

Show differences in files

Show contents of file (e.g urls.txt)

cat urls.txt
http://www.server1.com
http://www.server2.com
http://www.server3.com
http://www.server4.com
http://www.server5.com

Download all files mentioned in a text file.

View all packages with updates

Output:

or (on ubuntu 16.04)

Updates

Always backup your server’s configuration before updating.

Backup MySQL

Crontab

Add this to crontab -e to backup at 1 am every day

/scripts/shrinkmysql.sh  script to shrink SQL files

I had to add this to crontab

Cron job to shrink SQL dumps at 2am every day

Write to a single log file from the crontab at 2 am every day.

todo

Query a package (e.g. siege package)

How to set up a free SSL certificate (see my guide here).

How to set your timezone.

How to restart PHP

How to show the time on the server

sudo hwclock --show

Reload and restart the NGINX configuration and web server.

JSON viewing program

Installing

Using

or

output from json tool

Below are utilities I use a lot.

ncdu file size utility

Installing

Using

pydf disk checking utility

Installing

Using

output from pydf tool

ntp timezone service

Installing

Using

Displaying startup processes

Installing

Using

htop process manager.

Installing

Using

output from htop tool

Network Benchmarking (between two servers)

I use iperf to measure total bandwidth between two servers. You will need to allow port 5001 (TCP IPV4 and TCP IPV6 in and out) in any local firewalls and hosts GUIs.

Allow port 5001 on an ufw firewall (IN and OUT)

Deny IP

Allow port 22 access to known IP

Deny Outgoing Port

Allow out on port to known IP

More securing Ubuntu in the cloud here.

Install iperf on the target and source Ubuntu server.

Run this on the listening server.

Run this on benchmarking server (and add the IP of the listening server).

iperf -c 123.123.123.123

Results

Screen dump of ipref -c ip
iperf benchmarking output

Testing concurrent connections to a web server with siege.

Install siege

Benchmarking a HTTP server

./siege -t1m c10 'https://fearby.com'

#10 Concurent users

Benchmarking HTTPS sites

Install siege 4.0.2 ( steps here ).

Verify siege 4.0.2 is installed by running

Now can benchmark https sites

View incoming connections on the target server

I always increase my history size and tell it not to store duplicates.

Viewing your typed terminal history

Increasing your history size

How to Update Ubuntu

or

Edit SSH authorized keys

Search file and show lines where text matches

View packages with updates

View Boot Text

Automatic Monitoring (ever 1 second)

Active network connections

Network Packets

Free memory

Memory breakdown

or

Monitor NGINX memory

more soon…

Read this guide for Useful OSX Commands (for setting up Apache, PHP, MySQL, Adminer etc on OSX)

50 most useful Linux commands (view here).

View other Linux command informing sites here, here and here.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

v1.8 Show users and senda messages to other users

v.71 Show Operating System Name

v1.7 added Meltdown and Spectre patch information

v1.6 Removed -x fom zip directory

V1.5 Restart Network

v1.4 added Useful OSX Commands

Securing Ubuntu in the cloud

by

It is easy to deploy servers to the cloud within a few minutes, you can have a cloud-based server that you (or others can use). ubuntu has a great guide on setting up basic security issues but what do you need to do.

If you do not secure your server expects it to be hacked into. Below are tips on securing your cloud server.

First, read more on scanning your server with Lynis security scan.

Always use up to date software

Always use update software, malicious users can detect what software you use with sites like shodan.io (or use port scan tools) and then look for weaknesses from well-published lists (e.g WordPress, Windows, MySQL, node, LifeRay, Oracle etc). People can even use Google to search for login pages or sites with passwords in HTML (yes that simple).  Once a system is identified by a malicious user they can send automated bots to break into your site (trying millions of passwords a day) or use tools to bypass existing defences (Security researcher Troy Hunt found out it’s child’s play).

Portscan sites like https://mxtoolbox.com/SuperTool.aspx?action=scan are good for knowing what you have exposed.

You can also use local programs like nmap to view open ports

Instal nmap

Find open ports

Limit ssh connections

Read more here.

Use ufw to set limits on login attempts

Only allow known IP’s access to your valuable ports

sudo ufw allow from 123.123.123.123/32 to any port 22

Delete unwanted firewall rules

Only allow known IP’s to certain ports

Also, set outgoing traffic to known active servers and ports

Don’t use weak/common Diffie-Hellman key for SSL certificates, more information here.

More info on generating SSL certs here and setting here and setting up Public Key Pinning here.

Intrusion Prevention Software

Do run fail2ban: Guide here https://www.linode.com/docs/security/using-fail2ban-for-security

I use iThemes Security to secure my WordPress and block repeat failed logins from certain IP addresses.

iThemes Security can even lock down your WordPress.

You can set iThemes to auto lock out users on x failed logins

Remember to use allowed whitelists though (it is so easy to lock yourself out of servers).

Passwords

Do have strong passwords and change the root password provided by the hosts. https://howsecureismypassword.net/ is a good site to see how strong your password is from brute force password attempts. https://www.grc.com/passwords.htm is a good site to obtain a strong password.  Do follow Troy Hunt’s blog and twitter account to keep up to date with security issues.

Configure a Firewall Basics

You should install a firewall on your Ubuntu and configure it and also configure a firewall with your hosts (e.g AWS, Vultr, Digital Ocean).

Configure a Firewall on AWS

My AWS server setup guide here. AWS allow you to configure the firewall here in the Amazon Console.

TypeProtocolPort RangeSourceComment
HTTPTCP800.0.0.0/0Opens a web server port for later
All ICMPALLN/A0.0.0.0/0Allows you to ping
All trafficALLAll0.0.0.0/0Not advisable long term but OK for testing today.
SSHTCP220.0.0.0/0Not advisable, try and limit this to known IP’s only.
HTTPSTCP4430.0.0.0/0Opens a secure web server port for later

Configure a Firewall on Digital Ocean

Configuring a firewall on Digital Ocean (create a $5/m server here).  You can configure your Digital Ocean droplet firewall by clicking Droplet, Networking then Manage Firewall after logging into Digital Ocean.

Configure a Firewall on Vultr

Configuring a firewall on Vultr (create a $2.5/m server here).

Don’t forget to set IP rules for IPV4 and IPV6, Only set the post you need to allow and ensure applications have strong passwords.

Ubuntu has a firewall built in (documentation).

Enable the firewall

Adding common ports

sudo ufw allow ssh/tcp
sudo ufw logging on
sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 53
sudo ufw allow 443
sudo ufw allow 873
sudo ufw enable
sudo ufw status
sudo ufw allow http
sudo ufw allow https

Add a whitelist for your IP (use http://icanhazip.com/ to get your IP) to ensure you won’t get kicked out of your server.

sudo ufw allow from 123.123.123.123/24 to any port 22

More help here.  Here is a  good guide on ufw commands. Info on port numbers here.

If you don’t have a  Digital Ocean server for $5 a month click here and if a $2.5 a month Vultr server here.

Backups

rsync is a good way to copy files to another server or use Bacula

Basics

Initial server setup guide (Digital Ocean).

Sudo (admin user)

Read this guide on the Linux sudo command (the equivalent if run as administrator on Windows).

Users

List users on an Ubuntu OS (or compgen -u)

Common output

Add User

e.g

Add user to a group

Show users in a group

This will show users in a group

Remove a user

Rename user

Change user password

Groups

Show all groups

compgen -ug

Common output

compgen -g
root
daemon
bin
sys
adm
tty
disk
lp
mail
proxy
sudo
www-data
backup
irc
etc

You can create your own groups but first, you must be aware of group ids

cat /etc/group

Then you can see your systems groups and ids.

Create a group

Permissions

Read this https://help.ubuntu.com/community/FilePermissions

How to list users on Ubuntu.

Read more on setting permissions here.

Chmod help can be found here.

Install Fail2Ban

I used this guide on installing Fail2Ban.

Check Fail2Ban often and add blocks to the firewall of known bad IPs

Best practices

Ubuntu has a guide on basic security setup here.

Startup Processes

It is a good idea to review startup processes from time to time.

Accounts

  • Read up on the concept of least privilege access for apps and services here.
  • Read up on chmod permissions.

Updates

Do update your operating system often.

Minimal software

Only install what software you need

Exploits and Keeping up to date

Do keep up to date with exploits and vulnerabilities

Secure your applications

  • NodeJS: Enable logging in applications you install or develop.

Ban repeat Login attempts with FailBan

Fail2Ban config

Hosts File Hardening

Add

Add a whitelist with your ip on /etc/fail2ban/jail.conf (see this)

Restart the service

sudo service fail2ban restart
sudo service fail2ban status

Intrusion detection (logging) systems

Tripwire will not block or prevent intrusions but it will log and give you a heads up with risks and things of concern

Install Tripwire.

Running Tripwire

This will scan your system for issues of note

My Output.

sudo nano /var/log/tiger/security.report.username.170809-18:42

Security scripts *** 3.2.3, 2008.09.10.09.30 ***
Wed Aug  9 18:42:24 AEST 2017
20:42> Beginning security report for username (x86_64 Linux 4.4.0-89-generic).

# Performing check of passwd files...
# Checking entries from /etc/passwd.
--WARN-- [pass014w] Login (bob) is disabled, but has a valid shell.
--WARN-- [pass014w] Login (root) is disabled, but has a valid shell.
--WARN-- [pass015w] Login ID sync does not have a valid shell (/bin/sync).
--WARN-- [pass012w] Home directory /nonexistent exists multiple times (3) in
         /etc/passwd.
--WARN-- [pass012w] Home directory /run/systemd exists multiple times (2) in
         /etc/passwd.
--WARN-- [pass006w] Integrity of password files questionable (/usr/sbin/pwck
         -r).

# Performing check of group files...

# Performing check of user accounts...
# Checking accounts from /etc/passwd.
--WARN-- [acc021w] Login ID dnsmasq appears to be a dormant account.
--WARN-- [acc022w] Login ID nobody home directory (/nonexistent) is not
         accessible.

# Performing check of /etc/hosts.equiv and .rhosts files...

# Checking accounts from /etc/passwd...

# Performing check of .netrc files...

# Checking accounts from /etc/passwd...

# Performing common access checks for root (in /etc/default/login, /securetty, and /etc/ttytab...
--WARN-- [root001w] Remote root login allowed in /etc/ssh/sshd_config

# Performing check of PATH components...
--WARN-- [path009w] /etc/profile does not export an initial setting for PATH.
# Only checking user 'root'

# Performing check of anonymous FTP...

# Performing checks of mail aliases...
# Checking aliases from /etc/aliases.

# Performing check of `cron' entries...
--WARN-- [cron005w] Use of cron is not restricted

# Performing check of 'services' ...
# Checking services from /etc/services.
--WARN-- [inet003w] The port for service ssmtp is also assigned to service
         urd.
--WARN-- [inet003w] The port for service pipe-server is also assigned to
         service search.

# Performing NFS exports check...

# Performing check of system file permissions...
--ALERT-- [perm023a] /bin/su is setuid to `root'.
--ALERT-- [perm023a] /usr/bin/at is setuid to `daemon'.
--ALERT-- [perm024a] /usr/bin/at is setgid to `daemon'.
--WARN-- [perm001w] The owner of /usr/bin/at should be root (owned by daemon).
--WARN-- [perm002w] The group owner of /usr/bin/at should be root.
--ALERT-- [perm023a] /usr/bin/passwd is setuid to `root'.
--ALERT-- [perm024a] /usr/bin/wall is setgid to `tty'.

# Checking for known intrusion signs...
# Testing for promiscuous interfaces with /bin/ip
# Testing for backdoors in inetd.conf

# Performing check of files in system mail spool...

# Performing check for rookits...
# Running chkrootkit (/usr/sbin/chkrootkit) to perform further checks...
--WARN-- [rootkit004w] Chkrootkit has detected a possible rootkit installation
Possible Linux/Ebury - Operation Windigo installetd

# Performing system specific checks...
# Performing checks for Linux/4...

# Checking boot loader file permissions...
--WARN-- [boot02] The configuration file /boot/grub/menu.lst has group
         permissions. Should be 0600
--FAIL-- [boot02] The configuration file /boot/grub/menu.lst has world
         permissions. Should be 0600
--WARN-- [boot06] The Grub bootloader does not have a password configured.

# Checking for vulnerabilities in inittab configuration...

# Checking for correct umask settings for init scripts...
--WARN-- [misc021w] There are no umask entries in /etc/init.d/rcS

# Checking Logins not used on the system ...

# Checking network configuration
--FAIL-- [lin013f] The system is not protected against Syn flooding attacks
--WARN-- [lin017w] The system is not configured to log suspicious (martian)
         packets

# Verifying system specific password checks...

# Checking OS release...
--WARN-- [osv004w] Unreleased Debian GNU/Linux version `stretch/sid'

# Checking installed packages vs Debian Security Advisories...

# Checking md5sums of installed files

# Checking installed files against packages...
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/udev/hwdb.bin' does not belong to any package.

# Performing check of root directory...

# Checking device permissions...
--WARN-- [dev003w] The directory /dev/block resides in a device directory.
--WARN-- [dev003w] The directory /dev/char resides in a device directory.
--WARN-- [dev003w] The directory /dev/cpu resides in a device directory.
--FAIL-- [dev002f] /dev/fuse has world permissions
--WARN-- [dev003w] The directory /dev/hugepages resides in a device directory.
--FAIL-- [dev002f] /dev/kmsg has world permissions
--WARN-- [dev003w] The directory /dev/lightnvm resides in a device directory.
--WARN-- [dev003w] The directory /dev/mqueue resides in a device directory.
--FAIL-- [dev002f] /dev/rfkill has world permissions
--WARN-- [dev003w] The directory /dev/vfio resides in a device directory.

# Checking for existence of log files...
--FAIL-- [logf005f] Log file /var/log/btmp permission should be 660
--FAIL-- [logf007f] Log file /var/log/messages does not exist

# Checking for correct umask settings for user login shells...
--WARN-- [misc021w] There is no umask definition for the dash shell
--WARN-- [misc021w] There is no umask definition for the bash shell

# Checking symbolic links...

# Performing check of embedded pathnames...
20:47> Security report completed for username.

More on Tripwire here.

Hardening PHP

Hardening PHP config (and backing the PHP config it up), first create an info.php file in your website root folder with this info

Now look for what PHP file is loadingPHP Config

Back that your PHP config file

TIP: Delete the file with phpinfo() in it as it is a security risk to leave it there.

TIP: Read the OWASP cheat sheet on using PHP securely here and securing php.ini here.

Some common security changes

Don’t forget to review logs, more config changes here.

Antivirus

Yes, it is a good idea to run antivirus in Ubuntu, here is a good list of antivirus software

I am installing ClamAV as it can be installed on the command line and is open source.

ClamAV help here.

Scan a folder

Setup auto-update antivirus definitions

I set auto updates 24 times a day (every hour) via daemon updates.

tip: Download manual antivirus update definitions. If you only have a 512MB server your update may fail and you may want to stop fresh claim/php/nginx and mysql before you update to ensure the antivirus definitions update. You can move this to a con job and set this to update at set times over daemon to ensure updates happen.

Manual scan with a bash script

Create a bash script

Bash script contents to update antivirus definitions.

Edit the crontab to run the script every hour

Uninstalling Clam AV

You may need to uninstall Clamav if you don’t have a lot of memory or find updates are too big.

Setup Unattended Ubuntu Security updates

At login, you should receive

Other

  • Read this awesome guide.
  • install Fail2Ban
  • Do check your log files if you suspect suspicious activity.

Check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

v1.92 added hardening a linux server link