• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Create a VM ($25 Credit)
  • Buy a Domain
  • 1 Month free Back Blaze Backup
  • Other Deals
    • Domain Email
    • Nixstats Server Monitoring
    • ewww.io Auto WordPress Image Resizing and Acceleration
  • About
  • Links

IoT, Code, Security, Server Stuff etc

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

  • Cloud
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to buy a new domain and SSL cert from NameCheap, a Server from Digital Ocean and configure it.
    • Setting up a Vultr VM and configuring it
    • All Cloud Articles
  • Dev
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to setup pooled MySQL connections in Node JS that don’t disconnect
    • NodeJS code to handle App logins via API (using MySQL connection pools (1000 connections) and query parameters)
    • Infographic: So you have an idea for an app
    • All Development Articles
  • MySQL
    • Using the free Adminer GUI for MySQL on your website
    • All MySQL Articles
  • Perf
    • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap
    • All Performance Articles
  • Sec
    • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
    • Using OWASP ZAP GUI to scan your Applications for security issues
    • Setting up the Debian Kali Linux distro to perform penetration testing of your systems
    • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
    • PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API
    • Setting strong SSL cryptographic protocols and ciphers on Ubuntu and NGINX
    • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
    • All Security Articles
  • Server
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All Server Articles
  • Ubuntu
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Useful Linux Terminal Commands
    • All Ubuntu Articles
  • VM
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All VM Articles
  • WordPress
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
    • How to backup WordPress on a host that has CPanel
    • Moving WordPress to a new self managed server away from CPanel
    • Moving a CPanel domain with email to a self managed VPS and Gmail
    • All WordPress Articles
  • All

Linux

MobaXterm tabbed SSH client (etc) for Windows

July 7, 2020 by Simon

After posting my last blog post (Goodbye Dropbox, One Drive, iCloud and Hello Nextcloud private cloud on UpCloud) I received comments from readers as to why I used Putty/WinSCP and not MobaXterm.  To be honest I had no idea MobaXterm existed.

I did a quick Google and checked out the pros and cons of Putty/WinSCP v MobaXTerm.  

MobaXterm Highlights

  • MobaXterm is a lot faster at uploading and downloading files over SSH connections
  • MobaXterm can connect to practically anything (SSH, Telnet, RDP, RSH, Xdmcp, VNC, FTP, SFTP, Serial, Shell etc)
  • Multiple terminals can either be in tabs or split horizontally or vertically.
  • Edit remote files via SSH and SFTP – Pro feature
  • Supports macro controls (multi commands in single screen) – Pro feature
  • MobaXterm has many plugins
  • MobaXterm can be run from a portable USB drive

Nice

MobaXterm Website

I visited https://mobaxterm.mobatek.net/

https://mobaxterm.mobatek.net/ website screenshot

I had an option to download the Free Version or to purchase a Professional Version.

Free v Pro version comparison

I will be reviewing the Pro version (you will see why later). I am reviewing Version 20.2 (Build 249). You can download the free version at https://mobaxterm.mobatek.net/download.html.

If you purchase the Pro version (with 12  months of updates) for $69 USD. Delivery is done by email within 24 hours after payment is received.

Download Home or Pro Version

Installing MobaXterm

I downloaded the 1.5MB Installer (the link was emailed to me after purchase)

installer exe

I opened the MobaXterm installer and entered my Pro serial number (emailed to me) then I clicked Next

Enter serial number

Customise Options (during the install)

I clicked Customise MobaXTerm Professional settings button

Advanced options button

The customizer options are impressive (Import and export settings, enable/disable features, generate a portable package etc)

Customizer options

I had an option to customise the banner or any server I connect to but I left the banner as the default banner.

Banner editing

I reviewed all options for the application. This is more in depth than Putty for sure.

Enable or disable just about everything.

I reviewed my default SSH settings.  SSH Keep alive will be turned on for sure.

SSH Settings Screen

I had an option to add plugins to MobXTerm

Dialog to save files to another folder.

I had the option to generate a portable of full installer 

generate Installer

Now I can resume the setup of MobaXterm, I clicked Next

Installer Click Next

I accepted the licence agreement and clicked Next

Licence Agreement

I installed MobaXterm to my C:\ Drive

Installer Choose destination folder.

I clicked Install

Installer Install

MobaXterm was now installed

Installer: Installed Screenshot

MobaXterm

I opened MobaXterm

Apologies in advance I setup Dark Mode in MobaXterm as soon as I opened the Configuration. All screenshots will be shown below in Dark Mode.

There are 12 colour themes to choose from.

I set the Dark theme.

Main Interface

The MobaXterm interface is more feature packed than Putty.

Menu Bar, Toolbar, Tabs and options are available

Windows prompted me to allowed access to MobaXTerm in my Windows firewall.

Time to create a connection to a server.

Adding a SSH Connection to a Server

I created a new SSH server connection to one of my existing servers.

  • I specified the IP address, username and protocol.
  • I set “SCP (enhanced speed)“
add server with options

I specified the path to my private key (for the server)

I specified my servers private key

I reviewed other terminal settings.

Other settings dialog

I changed the font to Courier New, Size 12 (I am sad). I also increased the initial size of the terminal (width and height)

Set terminal font and size

I reviewed network settings (all good)

network settings

I added a shortcut to the server onto my desktop

Save shortcut to the desktop

When I opened the connection to the server I was prompted to enter the Passphrase I set on my Public Key (nice).

Then I was prompted to put in my 2FA One Time Password (OTP) in the main terminal windows and a second time in a popup form (for the connection that builds the folder structure on the left).  This is how I configured my website. To get the OTP I need to insert my YubiKey and one the YubiCo Authenticator app and enter my keys password.

Some people say that having a passphrase on a public/private key and 2FA Authentication is a bit overkill but that is not everything I do.

enter passphrase

I need to enter two OTP’s to connect to my site.

If you are fast you can use the same OTP in both windows (but they expire in 30 seconds)

enter OTP

Nice I was able to connect to my Server just like Putty.  The cool thing is I can see files and folders without opening WinSCP.

folder list and a terminal side by side

This is way more convenient than using two programs. 

I enabled the Remote Monitoring status bar in MobaXterm. Now I can see system resources on my server without running CLI commands.

Upload Speed to Singapore from Australia

I uploaded a 1GB file with to the same server from WinSCP and MobaXterm to simulate a restore.

winscp v mobaxterm at uploading

I was gobsmacked, WinSCP could only upload a file at 342KB/s, MobaXterm could upload at 4.42MB/s.

I was uploading a 1GB file from my house to a server in Singapore (within 5 minutes of each other). I did set “SCP (enhanced speed)” when Installing MobaXterm.

MobaXTerm is 12x faster than WinSCP at uploading for me. This is well worth upgrading to a Professional licence.

Download Speed (from USA/East Coast) to Australia

I ran a practical download test (this time from Chicago). In the right-hand side of MobaXterm I zipped up my entire website folder (1.66GB)

zip -r /temp/www-22-Jun-2020.zip /www-folder

Then in the left-hand side of MobaXterm I downloaded the file, this is as simple as right-click then download.

Right click download.

I was presented with a directory tree specify where to save the file to.

Specify download location

I noticed the file was downloading at 16Mbps then 18Mbps then 22Mbps within seconds.

The download speed settled down to 26Mbps after a few minutes.

This is amazing for a download from the other side of the world (200ms ping and on a VPN, 19 traceroute server hops away).

download speed

I downloaded the same file from the same server with WinSCP to compare and MobaXTerm was 10x faster at downloading. WinSCP could only manage 2.57/MB/s

winscp download speed

Plugins

I will look into plugins soon: https://mobaxterm.mobatek.net/plugins.html

Local Terminal

I can also start a local terminal (documentation here)

open terminal

I can run Linux commands on my Windows machine via Cygwin

“ls -al” on Windows anyone

ls -al screenshot

I noticed MobaXterm had found my WSL 2 Installation of Debian on my Windows 10 v2020.

I can now use MobaXterm to open my WSL Linux installations.

update wls linux

MobaXterm can easily connect to my local and online Linux servers with ease.

overview of the mobaxterm ui

Multiple Exec

I used the Multiple Exec (similar to the multiple cursors in Sublime Text Editor) to control 2 serves at once and run identical commands.

Multiple Exec sending keystrokes to multiple servers.

Import Putty Sessions

I did not notice until later but you can import existing Putty connections.

MobaXterm can import Putty sessions

Generate Public/Private Keys

Also, you can generate SSH Public and Private Keys from the Tools menu or the left-hand side (tab).

generate ssh key

Pros

  • File/Folder and Terminal window side by site
  • 13x faster file uploads over SSH
  • 10x faster downloads over SSH
  • Integration with WSL Linux Images
  • Import Putty Settings
  • Local Terminals/Cygwin

Cons

  • It would be nice to only have to enter my 2FA OTP once when connecting to one of my serves.
  • The user interface is full-on, I am still learning it

Conclusion

Honestly having faster uploads and downloads over SSH is a dream come true. I live in Australia and we have terrible latency and I have better things to do than to watch slow uploads/downloads.

I look forward to investigating MobaXterm plugins in the near future as I feel I I have just scratched the surface.

Links

MobaXterm Documentation: https://mobaxterm.mobatek.net/documentation.html

 

v.1.1 Plugin info fix

Filed Under: SSH, Uncategorized Tagged With: Linux, MobaXterm, ssh, terminal

Setting up a Raspberry PI as a DNS Sinkhole to block ads and Trackers

April 15, 2020 by Simon

What is PiHole (Version 5)?

fyi: I updated this post 3 months after I created it as I killed my 32GB Raspberry Pi Micro SD card when I pulled the power (before a storm) without shutting down the Raspberry PIU first. Always shutdown the pi before removing the power. I have a 16GB Micro SD card that I will use instead.

I am following my guide to re setup up my PiHole.

Snip from WikiPedia:  “Pi-hole is a Linux network-level advertisement and Internet tracker blocking application which acts as a DNS sinkhole (and optionally a DHCP server), intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations. Pi-hole has the ability to block traditional website advertisements as well as advertisements in unconventional places, such as smart TVs and mobile operating system advertisements.”

What is a Raspberry PI?

A Raspberry PI is an inexpensive (5V Volt, 2 Amp) ARM based computer that can run off the power from a USB cable.

Raspberry PI models abvailable

Here is a photo of my Raspberry Pi 3B+  with an Adafruit LCD Screen

My Raspberry Pi has the following specifications 

  • 4 x 1.4GHz 64-bit (quad-core processor)
  • 1GB LPDDR2 SDRAM
  • Dual-band wireless LAN
  • Bluetooth 4.2/BLE,
  • Faster Ethernet
  • Extended 40-pin GPIO header
  • Full-size HDMI 4 USB 2.0 ports
  • 5V/2.5A DC power input

My screen has the following specifications (purchased from Pakronics)

  • 3.5″ display with 480×320 16-bit colour pixels
  • Resistive touch overlay

I plugged in a full sized USB Keyboard, Mouse and HDMI cable.

SD Card Choice

Read my guide to download and write an Raspberry Pi Operating System to an SD card.

I would not put a cheap/slow MicroSD card in the Raspberry PI, aim for at least a UHS (1) or UHS (3) speed SD card for the best bang for buck.

SD card speeds

fyi: I bought a new 32GB Samsung UHS 1 Ultra Micro SD card and it died after 12 hours of use. I replaced it with another 32GB No name brand CLASS 10 SD Card I had laying around.

(after I killed my 32GB Micro SD card I have chosen a 16GB Micro SD card as it is all I have spare)

Dead SD

Raspberry Setup

I download and saved the Raspian (Full) Operating System to a SD Card and inserted it into my Raspberry PI 3B+ (view the guide here on preparing an Operating System on a SD card).

I used the American 110-240V AC to 5.25V  2500ma DC power supply (with a US to AUS adaptor) that came with the Adafruit Screen.  It had a Micro USB connection on one end.

5.25V DC POwer Supply

It did not work though (I just had a flashing red light on the Raspberry Pi).

I had an Australian 240V AC to 5V 2500ma DC power supply to Micro USB.  from a previous project and it worked (the Raspberry Pi Started up).

5V 2500 mA power pack

I also have a number of Moki brand 240V to USB (1A and 2.4A) adapters. 

I will use the 2.4mA  plug. I know my Adafruit screen uses 100mA so this will do.

MOKI 240V 1A and 2.4A USB plugs

I plugged the HDMI cable into my Monitor and set up the HDMI as a Picture in Picture output so I can see my Main 4K screen (Display Port) and the Raspberry Pi HDMI input at the same time.

My First Raspberry Pi Boot

Mmmm my 4K screen with a 1080P HDMI picture in picture image (from the Raspberry Pi).

4k screen with a PiP HDMI input

The Raspberry Pi  booted fast and a welcome screen appeared

Apologies in advance, photos below are bad (I don’t have a HDMI capture card).

I clicked Next to setup the Raspberry PI

Welcome to the raspberry PI

I set my timezone and language

Set Timezone screenshot

I set a password

Set Password Screenshot

I skipped connecting to WiFi (I want pure Ethernet)

Join WiFi Screen

I was prompted to update the software (I clicked Next)

Update complete

Setup is complete

Setup is complete

I rebooted the Raspberry Pi

Second Boot

I changed further configuration by clicking the Raspberry Pi start button then Preferences then Raspberry PI Configuration

Screenshot of the Raspberry Pi menu showing Raspberry Pi Configuration

I changed the hostname to “raspberrypihole”, set Boot to CLI , Login as “pi“, and set Wait for network.

Update: After my Samsung SD card died I re setup my PI with a no name brand SD card and entered the name “raspberrypihole“

Set system options screenshot

Under display I reviewed the display options

Set video resolution options

I enabled SSH, SPI and I2C.

Enable SSH, SPI and I2C features screenshot.

I increased the GPU memory to 132GB

Allocate mempory screenshot

Time for a Reboot

Reboot warning.

SSH Access

I do not want to leave a keyboard, mouse and screen connected once I finish setting it up so I setup a SSH connection to the Raspberry Pi.

TIP: Putty is a free program for SSH connections.

I SSH’ed (more information on SSH below) to the Raspberry Pi and ran these commands to update it’s software and firmware.

sudo apt-get update  && sudo apt-get upgrade

Output

[email protected]:~ $ sudo apt-get update  && sudo apt-get upgrade
Hit:1 http://archive.raspberrypi.org/debian buster InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

The program “htop” is good for viewing system resources.

htop screenshot

Now it’s time to look at the Adafruit screen and case.

I put the Raspberry PI in the Adafruit Case

I purchased this kit for the Raspberry Pi, the LCD screen just connects to the Raspberry Pi GPIO pins.  It has a Adafruit LCD screen and a case for my Raspberry Pi 3B+

Photo of a Raspberry Pi 3B+, LCD screen and case

The screen just connects onto the GPIO pins

LCD Screens just connects to the Pi

The LCD screen allows you to use pins below the screen.

Male pins beside the female GPIO pins

GPIO pins documentation from https://www.raspberrypi.org/documentation/usage/gpio/

GPIO pin documentation from https://www.raspberrypi.org/documentation/usage/gpio/

A nice stack 🙂

Photo showing the LCD screen connected to the Pi

The case clips are hard to clip over the Raspberry Pi (Don’t force it or you might break for Pi)

Photo showing a case clip over the raspberry pi board

The case clip near the GPIO pins is on

Photo showing the case clip near the GPIO pins.

The clip near the power plug was off because the Raspberry Pi was not positioned correctly

Photo of the Raspberry PI off center of the lugs

After 30 minutes I carefully put the Raspberry Pi and LCD screen into the Case.

Photo showing the LCD screen in the case.

Side of the case with USB and Ethernet and USB porws exposed.

Photo of the Case exposing the USB and Ethernet port

HDMI, Power and Audio plugs are visible and lined up 🙂

Photo showing HDMI, Power and Audio plugs

The screen is visible through the case

Photo showing the case and LCD screen

The screen dips down on one side, I might have to prop it up (hot glue gun) a bit inside later

Photo showing the LCD screen dips to one side

SSH Connections to the Raspberry PI

I created an SSH connection to my Raspberry PI with MobaXterm (review here) and connected to it.

MobaXTerm connected to the pI

I ran the “ifconfig” command to get a list of all network interfaces.

I ran “ifconfig” to list all network interfaces.

MpobaXTerm ifconfig

I ran these commands to update my Raspberry PI Software

  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo apt full-upgrade
  • sudo apt -y dist-upgrade

I updated the Pi Firmware too  (this is dangerous, only update if you have issues).

  • sudo rpi-update
Firmware update

I rebooted and connected to the Raspberry Pi and ran this command to get the Ethernet and wireless mac address.

The first interface is my Ethernet adopter the second if the WiFi adaptor.

ifconfig |grep ether
ether b8:27:eb:d9:00:86 txqueuelen 1000
ether b8:27:eb:8c:55:d3 txqueuelen 1000

The first Mac address is my Ethernet address on The Raspberry PI and the second is WiFi.

I logged into my router (Telstra DJA0230) and clicked Advanced then Local Network.  I could see my DHCP range was from 192.168.0.2 to 192.168.0.254, I shortened this to 192.168.0.2 to 192.168.0.200 (so I can set a static IP Address for the Raspberry PI) then I set a Static IP address for the Raspberry pi to 192.168.0.201.

I rebooted the Raspberry PI and checked the IP address 

I logged into my Router (at https://192.168.0.1)

Screenhshot of my routers DHCP range

When my Samsung SD card died I had to re-setup a new SD card but the IP address came across as the mac address stayed the same (as it was the same hardware), I did, however, change the name of the Static IP hostname in my home router to match the new name “raspberrypihole” (not “pihole”)

I set a static IP for this Ethernet address and defined 192.168.0.201 as the IP address.

Setting Up PiHole on the RaspBerry PI

I SSH’ed to my Raspberry Pi (with the new IP address) and ran this command

Now its time to install Pi Hole onto My Raspberry Pi

wget -O basic-install.sh https://install.pi-hole.net
sudo bash basic-install.sh
PiHole INstall

I was presented with “This installer will transform your device into a network-wide ad blocker! “

Install PiHole?

I was presented with “This installer will transform your device into a network-wide ad blocker! “

PiHole is free, but powered by your donations (consider donating)

Donate Plea

I donated. Thanks PiHole Team.

My doantion screenshot.

This will pay for itself in no time.

Donation receipt.

Static IP address is required.

Static IP Warning

I chose to just have PiHole work on Ethernet (and not Wifi)

Interface Select

I was prompted to set my upstream DNS provider.

Upstream DNS Provider

I selected all default blacklist lists.

Third Party Lists

I allowed PiHole to use IPv4 and IPv6.

TCP Support

My IP and Gateway was displayed on the screen.

IP and Gateway Info

Final warning about setting a static IP address.

Final Static IP Warning

The PiHole IPv6 address is show

Ready

Install a admin interface (Yes)

Web Admin on

Install lighthttpd (Yes)

thttpd

I chose to log all DNS queries.

Log HTTP Queries

I oped to allow the viewing of all logged data. This is less secure but I can reduce this later.

Debugging

PiHole is now setting up

PiHole Installing

Installation took about 10 minutes

Installing

A PiHole admin URL and Password was displayed (write this down)

PiHole Setup

I loaded the PiHole initial admin screen (http://192.168.0.201/admin/) and it was a bit empty.

PiHole Interface

I logged into my PiHole (at http://192.1768.0.201/admin/) with the password provided during setup.

Blank PiHole

The Raspberry Pi Pi Hole service was up and waiting for connections

I have Zero traffic going through the PiHole.

Before I add computers on my network to the PiHole I had better uninstall the nextdns.io (my blog post about NextDNS.io here) as the Pi will now be the main DNS blocking Sinkhole in our house.

Uninstall NextDNS.io

On my Windows 10 PC I added the DNS server for the PiHole in IPV4 and IPV6.

I obtained the PiHole IPV4 and IPV6 addresses (1) PiHole Admin, 2) Pi Hole Settings Page, 3) IP Address)

Pi Hole Settings Screen

PiHole IPV4 and IPV6 addresses.

PiHole IP Settings

I added the Pi Holes IPV4 IP address to my Windows 10 IP Settings.

I added the PIHole DNS to the IPV4 and IPV6 on my Windows 10 Ethernet adaptor

I added the Pi Holes IPV6 IP address to my Windows 10 IP Settings.

Setting IPV6 DNS Server

After 20 hours or rining computers through the Pi-Hole Admin interface I loaded the PiHole Admin Interface (at http://192.168.0.201/admin/index.php) was reporting stats.

I can view stats for Protocol and answered queries

Dashboard

I can also see stats for permitted and blocked domains

Top Allowed and blocked traffic

Default Block Lists

I can also see the source blocked domains

Ad Lists

Add 3rd party block lists

I added these block lists to my PiHole list of sites to block (Thanks Jol)

https://v.firebog.net/hosts/Easyprivacy.txt
https://v.firebog.net/hosts/Prigent-Ads.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
https://hostfiles.frogeye.fr/multiparty-trackers-hosts.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt
https://v.firebog.net/hosts/Airelle-trc.txt
https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
https://gist.githubusercontent.com/anudeepND/adac7982307fec6ee23605e281a57f1a/raw/5b8582b906a9497624c3f3187a49ebc23a9cf2fb/Test.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
https://hosts-file.net/ad_servers.txt
https://hosts-file.net/emd.txt
https://hosts-file.net/exp.txt
https://hosts-file.net/grm.txt
https://hosts-file.net/psh.txt
https://isc.sans.edu/feeds/suspiciousdomains_Medium.txt
https://mirror1.malwaredomains.com/files/justdomains
https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt
http://someonewhocares.org/hosts/hosts
https://phishing.army/download/phishing_army_blocklist_extended.txt
https://ransomwaretracker.abuse.ch/downloads/CW_C2_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/LY_C2_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/TC_C2_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/TL_C2_DOMBL.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/win10/spy.txt
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://raw.githubusercontent.com/ZeroDot1/CoinBlockerLists/master/hosts
https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://v.firebog.net/hosts/AdguardDNS.txt
https://v.firebog.net/hosts/Airelle-hrsk.txt
https://v.firebog.net/hosts/Easylist.txt
https://v.firebog.net/hosts/Easyprivacy.txt
https://v.firebog.net/hosts/Prigent-Ads.txt
https://v.firebog.net/hosts/Prigent-Malware.txt
https://v.firebog.net/hosts/Prigent-Phishing.txt
https://v.firebog.net/hosts/Shalla-mal.txt
https://v.firebog.net/hosts/static/SamsungSmart.txt
https://v.firebog.net/hosts/static/w3kbl.txt
https://www.malwaredomainlist.com/hostslist/hosts.txt
https://www.squidblacklist.org/downloads/dg-malicious.acl
http://sysctl.org/cameleon/hosts
https://zerodot1.gitlab.io/CoinBlockerLists/hosts
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
http://www.joewein.net/dl/bl/dom-bl.txt
http://www.networksec.org/grabbho/block.txt

I updated all block lists at http://192.168.0.201/admin/gravity.php

Update Gravity Success.

666,862 domains on my block list 🙂

I now have the Pi Hole blocking 666,862 domains, this number will increase as sites are added to the remote lists, nice.

Dashboard Stats

LCD Screen Setup

I followed thin guide to setup the screen.

I can this code from the pi (logged in as root)

cd ~
wget https://raw.githubusercontent.com/adafruit/Raspberry-Pi-Installer-Scripts/master/adafruit-pitft.sh
chmod +x adafruit-pitft.sh
sudo ./adafruit-pitft.sh

I was prompted to choose a screen

Select configuration:
1. PiTFT 2.4", 2.8" or 3.2" resistive (240x320)
2. PiTFT 2.2" no touch (240x320)
3. PiTFT 2.8" capacitive touch (240x320)
4. PiTFT 3.5" resistive touch (320x480)
5. PiTFT Mini 1.3" or 1.54" display (240x240)
6. MiniPiTFT 1.14" display (240x135) - WARNING! CUTTING EDGE! WILL UPGRADE YOUR KERNEL TO LATEST
7. Quit without installing

SELECT 1-7:

I entered “3” for PiTFT 2.8″ capacitive touch (240×320)

I then was prompted to set the rotation of the screen

Select rotation:
1. 90 degrees (landscape)
2. 180 degrees (portait)
3. 270 degrees (landscape)
4. 0 degrees (portait)

SELECT 1-4: 

I entered “3” for 270 degrees (landscape).

I was prompted to allow the console to appear on the screen

Would you like the console to appear on the PiTFT display? [y/n]
y

Install Summary

Install Summary

I rebooted

Reboot [y/n]
y

I edited /boot/config.txt and changed these values

framebuffer_width=320
framebuffer_height=240

Installing PADD to display PiHole stats on the LCD

I followed this guide to install PADD (the software that displays the PiHole stats on the LCD screen)

cd ~
wget -N https://github.com/jpmck/PADD/files/4320681/padd.txt
mv padd.txt paddsimon.sh
chmod +x paddsimon.sh

Making PADD starts at boot

Edit this file

sudo nano  ~/.bashrc

and add the following to the end of the file

# Run PADD
# If we're on the PiTFT screen (ssh is xterm)
if [ "$TERM" == "linux" ] ; then
  while :
  do
    /root/paddsimon.sh
    sleep 0.2
  done
fi

I rebooted the PI.

sudo showdown -r now

How to Update the PiHole from the CLI

I ran the following command to update the PiHole block lists

pihole -g

Output…

  [i] Pi-hole blocking is enabled
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: mirror1.malwaredomains.com (justdomains)
  [✓] Status: No changes detected

  [i] Target: sysctl.org (hosts)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_tracking.txt)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_ad.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (ad_servers.txt)
  [✓] Status: No changes detected

  [i] Target: raw.githubusercontent.com (ytadblock.txt)
  [✓] Status: Retrieval successful

  [i] Target: v.firebog.net (Easyprivacy.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Prigent-Ads.txt)
  [✓] Status: No changes detected

  [i] Target: gitlab.com (notrack-blocklist.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (spy.txt)
  [✓] Status: Retrieval successful

  [i] Target: www.github.developerdan.com (ads-and-tracking-extended.txt)
  [✓] Status: Retrieval successful

  [i] Target: hostfiles.frogeye.fr (firstparty-trackers-hosts.txt)
  [✓] Status: Retrieval successful

  [i] Target: hostfiles.frogeye.fr (multiparty-trackers-hosts.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (android-tracking.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (SmartTV.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (AmazonFireTV.txt)
  [✓] Status: Retrieval successful

  [i] Target: v.firebog.net (Airelle-trc.txt)
  [✓] Status: No changes detected

  [i] Target: bitbucket.org (Mandiant_APT1_Report_Appendix_D.txt)
  [✓] Status: No changes detected

  [i] Target: gist.githubusercontent.com (Test.txt)
  [✓] Status: Retrieval successful

  [i] Target: gitlab.com (notrack-malware.txt)
  [✓] Status: Retrieval successful

  [i] Target: hosts-file.net (emd.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (exp.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (grm.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (psh.txt)
  [✓] Status: No changes detected

  [i] Target: isc.sans.edu (suspiciousdomains_Medium.txt)
  [✓] Status: Retrieval successful

  [i] Target: mirror.cedia.org.ec (immortal_domains.txt)
  [✓] Status: No changes detected

  [i] Target: someonewhocares.org (hosts)
  [✓] Status: No changes detected

  [i] Target: phishing.army (phishing_army_blocklist_extended.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (CW_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (LY_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (RW_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (TC_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (TL_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (spy.txt)
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: reddestdream.github.io (minimalhosts)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_malvertising.txt)
  [✓] Status: Retrieval successful

  [i] Target: v.firebog.net (AdguardDNS.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Airelle-hrsk.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Easylist.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Prigent-Malware.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Prigent-Phishing.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Shalla-mal.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (SamsungSmart.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (w3kbl.txt)
  [✓] Status: No changes detected

  [i] Target: www.malwaredomainlist.com (hosts.txt)
  [✓] Status: No changes detected

  [i] Target: www.squidblacklist.org (dg-malicious.acl)
  [✗] Status: Connection Timed Out (Cloudflare)
  [✗] List download failed: no cached list available

  [i] Target: zerodot1.gitlab.io (hosts)
  [✓] Status: No changes detected

  [i] Target: zeustracker.abuse.ch (blocklist.php?download=domainblocklist)
  [✓] Status: Retrieval successful

  [i] Target: www.joewein.net (dom-bl.txt)
  [✓] Status: Retrieval successful

  [i] Target: www.networksec.org (block.txt)
  [✓] Status: Retrieval successful
  [i] Received empty file: using previously cached list

  [✓] Consolidating blocklists
  [✓] Extracting domains from blocklists
  [i] Number of domains being pulled in by gravity: 1178534
  [✓] Removing duplicate domains
  [i] Number of unique domains trapped in the Event Horizon: 954486
  [i] Number of whitelisted domains: 2
  [i] Number of blacklisted domains: 0
  [i] Number of regex filters: 0
  [✓] Parsing domains into hosts format
  [✓] Cleaning up stray matter

  [✓] Force-reloading DNS service
  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

I can view all possible command line options by running 

pihole /?

Output..

Usage: pihole [options]
Example: 'pihole -w -h'
Add '-h' after specific commands for more information on usage

Whitelist/Blacklist Options:
  -w, whitelist       Whitelist domain(s)
  -b, blacklist       Blacklist domain(s)
  --wild, wildcard     Wildcard blacklist domain(s)
  --regex, regex       Regex blacklist domains(s)
                        Add '-h' for more info on whitelist/blacklist usage

Debugging Options:
  -d, debug           Start a debugging session
                        Add '-a' to enable automated debugging
  -f, flush           Flush the Pi-hole log
  -r, reconfigure     Reconfigure or Repair Pi-hole subsystems
  -t, tail            View the live output of the Pi-hole log

Options:
  -a, admin           Web interface options
                        Add '-h' for more info on Web Interface usage
  -c, chronometer     Calculates stats and displays to an LCD
                        Add '-h' for more info on chronometer usage
  -g, updateGravity   Update the list of ad-serving domains
  -h, --help, help    Show this help dialog
  -l, logging         Specify whether the Pi-hole log should be used
                        Add '-h' for more info on logging usage
  -q, query           Query the adlists for a specified domain
                        Add '-h' for more info on query usage
  -up, updatePihole   Update Pi-hole subsystems
                        Add '--check-only' to exit script before update is perfo                                                     rmed.
  -v, version         Show installed versions of Pi-hole, Web Interface & FTL
                        Add '-h' for more info on version usage
  uninstall           Uninstall Pi-hole from your system
  status              Display the running status of Pi-hole subsystems
  enable              Enable Pi-hole subsystems
  disable             Disable Pi-hole subsystems
                        Add '-h' for more info on disable usage
  restartdns          Restart Pi-hole subsystems
  checkout            Switch Pi-hole subsystems to a different Github branch
                        Add '-h' for more info on checkout usage

After 1 Week

After 1 week stats were rolling into the PIHole.

40% of all traffic was being blocked.

PiHole stats screen

I could see blocked and allowed domain calls

Top permitted and blocked domains

I can white list domains if they are blocked.

I white listed fearby.com and events.gfe.nvidia.com

Done

This is what it looks like done

Done

Nice

Did it block Ad’s

Mostly Yes. Not all advertisements are blocked but most are.

Some YouTube Advertisements seem to get through but I am seeing far less Advertisements in web pages

Using Python to use buttons on the PiTFT Plus 320×240 TFT Touchscreen with a PiHole

Read this guide to make the buttons word: Using Python to use buttons on the PiTFT Plus 320×240 TFT Touchscreen with a PiHole

Troubleshooting

If you receive an update about updating languages on your first boot while updating you can manually update all software by running this after you first reboot in a Terminal window.

sudo apt-get update && apt-get full-upgrade

If your Micro SD cad is filling up you can run to free some space

sudo apt clean

I needed to white list “events.gfe.nvidia.com” to allow my video card drivers to upodate.

Cooling

The Raspberry Pi is running cool at 47c (even though it is in a tight space).

47c image

I might add a heat pipe to it and have an external fan.  I will thermal epoxy the hat pipe to the Pi CPU and run it outside to a external heat sink and fan.

eBay purchase for a fan, thermal epoxy and heatpipes.

I have many spare heat sinks laying around.

copper and aluminium heatsinks.

I will update when the part’s arrive.

Update: I did not end up adding extra cooling, there was no need in summer.

Rotating the screen

I did exit my /boot/config.txt to rotate my LCD Screen orientation 

Buttons

Read this guide to see how I setup a Python script to make my buttons work.

Do edit your /boot.config.txt to configure your screen rotation (if need be) and to check if the LCD screen is setup (by Adafruit)

framebuffer_width=240
framebuffer_height=320

Backup and Restore PiHole Settings

I used the PiHole Backup feature (at http://192.168.0.201/admin/settings.php?tab=teleporter) to backup all of my PiHole Settings to a zip file.

Handy Links

Handy Guide: https://learn.adafruit.com/pi-hole-ad-pitft-tft-detection-display/pitft-configuration to configure the LCD Screen

Schematics of the screen: https://learn.adafruit.com/assets/25555

Donate to PiHole: https://pi-hole.net/donate/

Raspberry Pi GPIO Pins: https://www.raspberrypi.org/documentation/usage/gpio/

 

 

v 2.3 Updating to PiHole 5.1.2

Filed Under: Uncategorized Tagged With: a, acts, advertisement, and, application, as, blocking, DNS, How, I, internet, is, Linux, network-level, one, Pi-hole, set, sinkhole. See, tracker, which

Monitoring cronjobs on Linux

June 9, 2019 by Simon

I recently created a number of cron jobs to backup databases, backup files offsite, virus scan and perform a number of tasks (at various times) I put a call out on Twitter as to the best program to monitor these jobs.

@nixcraft What’s a good tool for monitoring cron job events and output? #Thanks

— Simon Fearby (Aussie DevSecOps) (@FearbySoftware) June 8, 2019

George Liu from https://community.centminmod.com/ kindly replied with this advice.

one i have used before https://t.co/SbsMfYFUg9

— George Liu (@centminmod) June 9, 2019

I visited https://healthchecks.io/. A quick Google reveals this software is open source too, nice.

Screenshot of https://healthchecks.io/

Create a healthchecks.io account

I created an account at https://healthchecks.io/ by visiting https://healthchecks.io/accounts/login/ and clicking the signup link then entering my email address.

I click the link that was sent to my email account to log in.

I then logged in and reviewed my account settings at https://healthchecks.io/accounts/profile and set a password (I then click the link in my email to set a password).

After my password was setup I visited https://healthchecks.io/

Pricing

The service is free to use up to 20 checks, then it’s $20/mo for 100 cron job checks. I am using the free version.

View the pricing page here.

Creating Monitors (Checks)

I logged into https://healthchecks.io and clicked the project icon (the one with my email)

Screenshot of the dashboard at This post will.

I deleted the sample item then clicked ‘Add Check‘

Add Check Button

I was prompted to give a name to the job and configure it.

If you click the ‘Name’ (unnamed) item a form will appear.

Add check job to https://healthchecks.io screenshot (naming it)

TIP: Name the jobs something memorable or similar to the job in “crontab”

https://healthchecks.io form asking for a job name, tag and description

I completed the form and clicked ‘Save‘

I click the gear icon for the job to configure it

https://healthchecks.io - new job settings screenshot

After clicking the Settings icon I could see the full details of the job

  • Name
  • Description
  • How to ping it (HTTP or email)
  • Current Status (Pause or Ping Now)
  • Timezone (Important, make sure this is the same timezone as your server)
  • Expected Schedule
  • Notification Method
  • Log
https://healthchecks.io screenshot of  a jobs settings

I could see the custom URL I needed to request in my cronjob to make the job pass the success test (a random GUI will be created for each job)

https://hc-ping.com/######-####-####-####-############

I added the following to the end of my desired cron job entry (I have removed my GUID for security reasons and replaced it with #’s)

&& curl -fsS --retry 3 https://hc-ping.com/########-####-####-####-######## > /dev/null

The complete cron entry looks like this (not sure what “12 12 * * *” means, check here)

12 12 * * *  /bib/bash /scripts/thescript.sh && curl -fsS --retry 3 https://hc-ping.com/########-####-####-####-######## > /dev/null

I can also configure the job to expect when to receive an indication when the job has been run.

Click Change Schedule to set a schedule

You can set a schedule with sliders..

Screenshot of when https://healthchecks.io expects a job to run

Or you can set a schedule by pasting in the cron job format and see when the job will be expected to run in the future.

Visit https://crontab.guru/ to learn how to set crontab schedules.

Crontab schedule for 12 12 * * *

After you save the job settings you will be returned to the main page.

Screenshot pf the portal once logged into https://healthchecks.io/

At any time you can reopen the job’s to adjust settings.

You can even see details about the requested clients are accessing the URLs, nice.

Screenshot of past jobs run

I created 17 jobs on 2x servers

Screenshot of 17 jobs added to https://healthchecks.io/

I setup 2 jobs that will fail just to see what failed jobs look like.

Badges

You can also use badges to quickly display the status of cron jobs in other apps or webpages (by embedding an image)

Badges available at https://healthchecks.io/projects/

I created a quick HTML (PHP) file that linked to the badge images. I have swapped out my badge images for security but paste in yours (from above)

HTML

Cron Jobs

Cron Jobs

  • Server 1: f
  • Server 2: d
  • Health Checks: h

    This is what the simple html file looks like in a browser.

    View of the html file showing cron status

    Reports

    I logged into the portal and it did not take long for me to start seeing errors.

    I had an invalid cron job entry (the wrong user was assigned and the job was failing).

    I had an error all this time and did not know it.

    Screenshot of a report of a bad cron job entry.

    My dummy job that was setup to fail did fail.

    My job to purposely fail did fail.

    An email alert was sent to my defined email address, nice.

    Email is failing report

    I could also see the email with the failing event in the log (and an initial test call made with a browser)

    Failing log table

    I was able to pause the reporting of this job in future.

    Conclusion

    I really like https://healthchecks.io/ and will continue using it.

    Sending Emails

    I have added a send email command to the end of my cron job called bach script(s) to be alerted of the jobs running.

    sendemail -f [email protected] -t [email protected] -u "CRON: xxx Run (server.com)" -m "/scripts/script.sh" -s smtp.gmail.com:587 -o tls=yes -xu [email protected] -xp YourEmailPasswordGoesHere -a /folder/file-to-attach.log

    Read my post here How to send email via G Suite from Ubuntu in the cloud

    If you have a G Suite account with 2FA enabled you will need to create a App Specific Password for this to work in your G Suite admin panel.

    Links

    Related links can be found here

    • Official Docs
    • API Reference
    • Third Party Resources
    • https://twitter.com/healthchecks_io

    Version:

    1.2 Added send email Info

    1.1 Initial Post

Filed Under: Audit, Automation, Cron Tagged With: Cron, Jobs, Linux, monitoring, on

Backing up files to a Backblaze B2 Cloud Bucket with Duplicati

June 6, 2019 by Simon

This guide will show how you can automatically backup files to a Backblaze B2 clouds bucket with the Duplicati software on Windows and Linux.

I have blogged about how you can use the Backblaze personal backup program ( here, Use my link and get your first month of Backblaze backups free). The personal application from Backblaze is cool but what if you wanted to store 10GB for free (the first 10GB is free) or setup different buckets for different computers or applications.

Welcome to Backblaze B2 Cloud Storage.

In Australia B2 cloud costs (+GST).

  • First 10GB is free.
  • $0.005c per GB stored per month.
  • To download, 1x GB costs $0.01c.

Check our my other related posts

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways

This works alongside the private Backblaze backup mentioned here.

Screenshot: Duplicati Downalods page https://www.duplicati.com/download

Backblaze B2 Pricing

How does Backblaze B2 Compare that to other cloud data storage provides.

Screenshot of https://www.backblaze.com/b2/cloud-storage.html

Comparison chart.

ProviderStorage ($/GB/Month)Download ($/GB)
Backblaze$0.005$0.01
Amazon S3$0.021$0.05+
Microsoft Azure$0.018+$0.05+
Google cloud$0.020$0.08

Vendor cost breakdown (source).

Read here to view the official page on pricing.

Snip:

Storage:

The first 10 GB of total storage is free, and is $0.005 per GB per month for additional storage beyond 10 GB.   

Download:

The first 1 GB of downloads per day are free.  The price to download stored data costs $0.01 per GB beyond the free daily 1 GB. 

Transactions:

A 'transaction' refers to any time an api call is made with your account, such as listing the contents of your bucket or downloading a file.   B2 will charge for some transactions however, for the average user, these charges are largely negligible unless you are making an extremely high volume of api calls.

Class A transactions are free.

The first 2,500 Class B transactions are free each day. Additional Class B API calls are charged at $0.004 per 10,000 calls.  

The first 2,500 Class C transactions are free each day. Additional Class C API calls are charged is $0.004 per 1,000 calls.

Vendor Relability

https://downdetector.com is  a great site for viewing is  a vendor is down or has issues, lets compare, Backblaze with Amazon, Azure and Google Cloud

  • Downdetector.com: Backblaze
  • Downdetector.com: Azure
  • Downdetector.com: Amazon
  • Downdetector.com: Google Cloud

Backblaze have has far fewer downtime issues during the great outage of May/June 2019 compared to its competitors. What I like is Backblaze tell us how they do it and tell us what hard drives are good or bad.

Creating a Backblaze B2 Account

Login/SignUp to backblaze at https://secure.backblaze.com/user_signin.htm

You can have a sneak peek at the personal Backblaze backup signup steps in this guide (why not take up a free 1 month back free by clicking this link when you create an account).

From your account page click billing and add your account details

Screenshot of https://secure.backblaze.com/billing.htm?billing_page=b2

Read the B2 Strater Guide: https://www.backblaze.com/b2/docs/

Also review your review the following under ‘My Settings‘

  • Phone Numer (mobile) for usage alerts.
  • Two Factor Authorization for extra security (don’t forget to çopy your backup codes)

Setup Billing Alerts (just in case)

In the Backblaze portal, under ‘Caps and Alerts‘ you should set..

  • Alerts to your mobile phone (for Daily storage caps,bandwidth caps and class B/C caps)
  • Under ‘Daily Storage Caps’ click ‘edit’ and set monetary alerts (you can increase alert limits later)
Added $5 and $10 daily storage alerts

Click ‘Edit Caps’ and set daily limits on all categories

Edit Caps button

I set alert limits of 10c (I can increase this later)

Daily 10c alerts should see is costs creep above $3 a month.

Screenshot of cap alerts, I set 10c warnings on everyitng

Create a cloud bucket to store files

Go to https://secure.backblaze.com/b2_buckets.htm and click ‘Create Bucket’

Screenshot of https://secure.backblaze.com/b2_buckets.htm (create bucket button)

You can create upto 100 buckets per account.

I created a descriptive bucket name that reminded me what server, drive and folder this was used to back up.

"#######-drive-backup-folder" (####### was the name of my computer)
Screenshot of bucket being created an named.

I created a number of buckets (all private).

The Backblaze portal lists all of my new storage buckets.

Screenshot of my 4 new storage buckets

Bucket Storage (Lifestyle Settings)

You can click the ‘Lifecycle Settings‘ button and set the desired buckets file retention policy (overwrite or keep all versions of a file etc).

Read the official Lifestyle settings faq here.

The default is to keep all versions of a file. Choose a setting that is required for each bucket(for me ‘Keep only the latest version of the file’ is best and lowers potential cost blowouts. Having all versions of a file may be handy for a Documents folder.

Screenshot of  Lifestyle settings at https://secure.backblaze.com/b2_buckets.htm

You could opt to keep all versions of a file (forever) but that would be expensive.

You tell Backblaze to delete older versions of a file after xx days (this is a good option)

You can also setup custom lifecycle rules to hide a file with a certain prefix after xx days then delete after xx days.

Create a Master Application Key for all buckets

You will need to create a Master Application Key that will grant you full access (Permissions: listKeys, writeKeys, deleteKeys, listBuckets, writeBuckets, deleteBuckets, listFiles, readFiles, shareFiles, writeFiles, deleteFiles ) to each bucket.

Snip from: https://secure.backblaze.com/app_keys.htm

Application keys are used as a pair: Key ID and Application Key. This allows B2 to communicate securely with different devices or apps. Once you generate your Master Application Key, this key has full capabilities. Create your own Application Keys to limit features like read/write. Learn more.

Do write down this “KeyID” and “applicationKey”

Screenshot of https://secure.backblaze.com/app_keys.htm (generated "KeyID" and "applicationKey")

Don’t lose the Master Application Key.

Create an App Key for each bucket

A master key is like an administrator account, its great for complete access but you should create a separate key per bucket for better security.

Back under the App Keys you can click “Add a New Application Key” for each bucket.

Screenshot of Add a new application key button

Read the official guide on Application Keys here

Screenshot of add application key (add name, option to choose a bucket and assign read/write acess

Do Set..

  • “Name of Key” (similar to your bucket (e.g “my-bucket-name-key”))
  • “Allow access to Bucket(s)” (choose a bucket to grant access to).
  • “Type of Access” (Read and Write, Read Only, Write Only)

Optional..

  • Set a “File name prefix” (snip from here)
"The file name prefix of files the download authorization token will allow b2_download_file_by_name to access. For example, if you have a private bucket named "photos" and generate a download authorization token for the fileNamePrefix "pets/" you will be able to use the download authorization token to access: https://f345.backblazeb2.com/file/photos/pets/kitten.jpg but not: https://f345.backblazeb2.com/file/photos/vacation.jpg."
  • Set a “Duration” (snip from here)
"The number of seconds before the authorization token will expire. The minimum value is 1 second. The maximum value is 604800 which is one week in seconds."

I now has 1x Master Application key and 4x Application Keys for my 4x Buckets

Screenshot of https://secure.backblaze.com/app_keys.htm (1 master key and 4x app keys)

TIP: Make sure you save the keys, if you did not grab the keys delete the keys and buckets (if there is not data and try again). If you don;t have the keys you are locked out of the buckets

OK, Let’s fill those buckets with data with Duplicati

Go to https://www.duplicati.com/ and familiarise yourself with the site

Setup Duplicati on Windows (10)

Go to https://www.duplicati.com/download and download the latest Windows client. At the time of writing Duplicati 2.0.4.5 beta was the latest version. Thew instal was 1MB (the download serve was not that fast)

Warning: Beta software may have bugs, feel free to check out the Github page for Issues etc.

Duplicati 2.0.4.5 download button

Open the install and Click ‘Next‘

Duplicati install wizard start, click Next

Agree to the ‘Licence Agreement‘ by clicking ‘Next‘

Duplicati licence agreement screenshot

Select Duplicati components to install and click ‘Next‘

Install an components screenshot.

Tick ‘Launch Duplicati‘ an click ‘Finish‘

Duplicati has finished installing screenshot.

Using Duplicati Software

Upon first open you will be prompted to set a password (do this even if you are the only user of thew computer)

First run screenshot

Set a strong and unique password

Screenshot of setting a duplicati password

Click ‘Ok‘

Ok Button

When the page reloads it will report it is not logged in.

Screenshot of user is not logged in error.

Enter your password

Enter password screenshot

The main screen of Duplicati reports that there are no scheduled set.

Screenshot no scheduled tasks screenshot.

OK, lets create a backup schedule.

Creating a Backup on Windows

If your screen is small you will need to click the menu at the top right and click ‘Add Backup’, if your screen is large there will be an ‘Add Backup’ menu on the left hand side of the screen.

Duplicati menu screenshot

Select ‘Configure a new backup‘ and click ‘Next‘

Configure a new backup screenshot

You will be able to name and describe the backup schedule name and be able to set up an encryption password.

Then click ‘Next‘

Screenshot of create backup schedule with name, description and encryption password

Under ‘Storage Type‘ choose ‘B2 Cloud Storage‘

When you choosing ‘B2 Cloud Storage‘ you will be directed to the screen below where you stecify..

  • B2 Cloud bucket name
  • Path in the bucket to store the files (leave blank)
  • Enter your Backblaze Account ID (e.g. ##########################)
  • Enter the Buckey ApplicationKey (e.g. ###############################)

Click ‘Next‘

Screenshot of enter B2 credentials

I entered my credentials

I entered my B2 bucket credentials

Click ‘Test Connection‘, if you entered everything OK you should see ‘Connection Worked‘

Connection worked Screenshot

Click ‘Next‘

Next button screenshot

Now you can select the folders to backup on your local computer

Duplicati, select folders to backup.

Click ‘Next‘

Next button

Now you can choose a schedule to backup

Screenshot choose time and days to run the backup.

Click ‘Next‘

Click Next button screenshot

Now you can set the remote backup size and backup retention mode.

Read the official guide on setting the volume size here

  • Increase the ‘Remote Volume Size’ if you are worried about consuming your daily API quota in Backblaze (this may happen if your upload speed is fast and your backup is very large).
  • Lower the ‘Remote Volume Size’ size if backups are small or your upload speed is low.
Screenshot of remote volume size and backup retention

Choose a desirable backup retention mode. Duplicati only offers full backups (not incremental or differential backups). This will be an issue if your backups are large.

  • Keep all backups (“Nothing will be deleted. The backup size will grow with each change.“)
  • Delete backups that are older than (“If at least one newer backup is found, all backups older than this date are deleted.“)
  • Keep a specific number of backups (“Once there are more backups than the specified number, the oldest backups are deleted.“)
  • Smart Backup Retention (“Over time backups will be deleted automatically. There will remain one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup.“)
  • Custom backup retention (“Enter a retention strategy manually. Placeholders are D/W/Y for days/weeks/years and U for unlimited. The syntax is: 7D:1D,4W:1W,36M:1M. This example keeps one backup for each of the next 7 days, one for each of the next 4 weeks, and one for each of the next 36 months. This can also be written as 1W:1D,1M:1W,3Y:1M.“)

Each storage option has pro’s and cons with potential cost implications. I would not set 300GB to “Keep all backups” and backup daily. That would be expensive.

Screenshot of the 5 backup retention modes

Click ‘Save‘

Save button screenshot

TIP: Don”t forget to save your encryption key if you set one

Screenshot of a duplicati warning saying have we saved our encryption ley

Now you will see the backup schedule on your Duplicati screen.

You can run, edit or delete the backup status here.

Screenshot of the backup schedule on the home screen.

I clicked ‘Run‘ and the backup was under way.

A progress bar updates as the backup happens.

Screenshot of backup underway

A slight delay happens at the end of the backup.

Screenshot of backup waiting to finish screen.

I logged into the Backblaze portal to see how many files and usage credits were used.

Note it seems?

Screenshot of backblaze portal

When the files are backed up a verification processes happens.

screenshot of verify files screen

Now the backup has completed.

Screenshot of the backup completed screen

I waited 2 minutes and finally Backblaze updated it’s status, Duplicati used 3 files and 87MB

Screenshot of B2 Cloud uploads stats

I logged into the view the Duplicati files that were uploaded and I could see the three files were there and they were encrypted.

Screenshot of B2 cloud encrypted Duplicati files.

Awesome

Restoring a Backup on Windows

Having a backup is useless you can restore the files (especially when they are encrypted).

Lets restore a few files and see if it works.

I opened Duplicati and clicked ‘Restore‘

Screenshot: Duplicati restore menu item

I clicked the bottom option that listed my bucket name/backup name and clicked ‘Next‘

Screenshot restore from the backup

Before I ran the restore I deleted some location files that were just backed up.

Screenshot of me deleting 2 files.

Duplicati now asked me to choose files to restore.

Screenshot restore files treeview with selections

I chose the original location to restore to and clicked ‘Restore‘

Screenshot restore to.

Duplicati took a few minutes to restore the files.

Screenshot: Duplicati downloading files

The restore was a success.

Screenshot restore success

Nice

Backblaze Costs?

Beware, make sure you calculate the costs to upload and download files.

Do set alarms on cap’s limits and review them often.

Screenshot of data usage and caps.

Second Backup Schedule on Windows

I quickly set a second backup schedule of a different folder to a different bucket.

This backup was 300GB so I will only backup once a month

Schedule once a month backup schedule.

The backup is listed on the front screen of Duplicati.

Do create multiple buckets and set multiple backup schedules to break up the backups into smaller chunks.

Screenshot of multiple backup schedules.

Setup Duplicati on Linux (Debian or Ubuntu)

Lets setup B2 Cloud Storage backup on Linux.

I visited the Duplicati forums here. Duplicati informed me that the Linux client needs mono installed

Duplicati depends on other software. For Windows, Microsoft .NET Framework 4.5 or higher needs to be installed. Linux and Mac OS X require Mono to be installed.

Mono is a Cross Platform open source .NET that runs on Linux (read here)

On Ubuntu 1.08 I installed mono with these commands

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update
sudo apt install mono-devel gtk-sharp2
sudo apt-get install libmono-2.0-1

On Debian 9.9 I installed Mono with these commands (from here)

sudo apt install apt-transport-https dirmngr gnupg ca-certificates
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update
sudo apt install mono-devel gtk-sharp2
sudo apt-get install libmono-2.0-1

I rebooted the serves and the CPU and memory went crazy after I installed Mono. My Nixstats Service reports were going off telling me my servers CPU was busy.

If you don’s have reports on Linux server CPU/MEM/Disk reports coming to you via Email and Telegram then read this.

Screensot of Nixstats Reports

I felt sorry for my web and database server server(s) so I rebooted them, wow, My website came back up in about 10 seconds thanks to UpClouds fast Max IOPS disks.

Installing Duplicati on Linux

I visited the download Duplicati page here and copied the Ubuntu and Debian download URL.

At the time of writing it was..

https://updates.duplicati.com/beta/duplicati_2.0.4.5-1_all.deb

I downloaded the deb package

cd /utils/duplicati
wget https://updates.duplicati.com/beta/duplicati_2.0.4.5-1_all.deb

I then ran this command to install the package

sudo dpkg -i duplicati_2.0.4.5-1_all.deb

I received this error even thpogh libmono-2.0-1 was installed

dpkg: dependency problems prevent configuration of duplicati:
duplicati depends on libmono-2.0-1; however:
Package libmono-2.0-1 is not installed.

This forum post helped me, I ran

sudo apt install -f

Output

sudo apt install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4
  libindicator7 libmono-2.0-1
Suggested packages:
  indicator-application
The following NEW packages will be installed:
  libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4
  libindicator7 libmono-2.0-1
0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.
1 not fully installed or removed.
Need to get 143 kB of archives.
After this operation, 541 kB of additional disk space will be used.
Do you want to continue? [Y/n]

And I ran the following

sudo apt dist-upgrade --fix-missing

I now ran the following to install the duplicati package.

sudo dpkg -i duplicati_2.0.4.5-1_all.deb

It worked on Ubuntu 18.08 bt not Debian 9.9

Debian produced the following error

sudo dpkg -i duplicati_2.0.4.5-1_all.deb                (Reading database ... 70082 files and directories currently installed.)
Preparing to unpack duplicati_2.0.4.5-1_all.deb ...
Unpacking duplicati (2.0.4.5-1) over (2.0.4.5-1) ...
dpkg: dependency problems prevent configuration of duplicati:
 duplicati depends on libappindicator0.1-cil | libappindicator3-0.1-cil; however:
  Package libappindicator0.1-cil is not installed.
  Package libappindicator3-0.1-cil is not installed.

dpkg: error processing package duplicati (--install):
 dependency problems - leaving unconfigured
Processing triggers for mime-support (3.60) ...
Errors were encountered while processing:
 duplicati

On Debian I installed the stretch version of the package

sudo apt-get install libappindicator1

Then in Debian I fixed broken packages

sudo apt --fix-broken install

This I was able to install Duplicati on Debian

sudo dpkg -i duplicati_2.0.4.5-1_all.deb

Running Duplicati CLI

I can now Run Duplicati CLI (it works)

duplicati-cli

See duplicati.commandline.exe help <topic> for more information.
  General: example, changelog
  Commands: backup, find, restore, delete, compact, test, compare, purge, vacuum
  Repair: repair, affected, list-broken-files, purge-broken-files
  Debug: debug, logging, create-report, test-filters, system-info, send-mail
  Targets: aftp, amzcd, azure, b2, box, cloudfiles, dropbox, ftp, file,
  googledrive, gcs, hubic, jottacloud, mega, msgroup, onedrive, onedrivev2,
  sharepoint, openstack, rclone, s3, ssh, od4b, mssp, sia, tahoe, webdav
  Modules: aes, gpg, zip, 7z, console-password-input, mssql-options,
  hyperv-options, http-options, sendhttp, sendmail, runscript, sendxmpp,
  check-mono-ssl
  Formats: date, time, size, encryption, compression
  Advanced: mail, advanced, returncodes, filter, filter-groups, <option>

http://www.duplicati.com/              Version:  - 2.0.4.5_beta_2018-11-28

TIP: Don’t run “duplicati” that’s the desktop verison.

Creating a Duplicati Backup on Linux

I read this page and this page to get an understanding of how to back up from the command line. I was not sure if I needed to run a service or run the CLI from a bach script so I asked on the forums here.

Within minutes I had this reply

kenkendk
1h
Hi @feartec, welcome to the forum!

The CLI and Web-UI are not the same, and operate independently.

If you use the CLI, you need to use cron or something similar to run the backups periodically. You can use duplicati-cli help to get started. You do not need the server running for this.

For the web-UI we do not currently have any tools for manipulating the backups other than the main WebUI itself. You need to somehow forward/tunnel access to the server, and then access the WebUI from a local browser.

Backing up a folder with Duplicati to a Backblaze B2 Bucket

I created this folder structure

/test/
/test/testfile.txt < hello
/test/subfolder/
/test/subfolder/file.txt < world

I ran this command to back up the folder structure above to backblaze (official guide). See advanced switches.

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=**********

TIP: read more about “dblock” size here

Output

Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  4 files need to be examined (8 bytes)
Checking remote backup ...
  Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
  Downloading file (1021 bytes) ...
  Downloading file (1.06 KB) ...
  Downloading file (2.00 KB) ...
  0 files need to be examined (0 bytes)
  Duration of backup: 00:00:06
  Remote files: 3
  Remote size: 4.05 KB
  Total remote quota: 0 bytes
  Available remote quota: 0 bytes
  Files added: 0
  Files deleted: 0
  Files changed: 0
  Data uploaded: 0 bytes
  Data downloaded: 4.05 KB
Backup completed successfully!

I logged into Backblaze to check the backups

Duplicati backup success

I created a second test file and added 2 test files

mkdir /test2
cd /test2
wget http://ipv4.download.thinkbroadband.com/10MB.zip
wget http://ipv4.download.thinkbroadband.com/5MB.zip

ls -al
total 15368
...
-rw-r--r--  1 user user 10485760 Jun  3  2008 10MB.zip
-rw-r--r--  1 user user 5242880 Jun  3  2008 5MB.zip

I ran the backup with the second folder in the command line

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=********** --retention-policy="1W:1D,4W:1W,12M:1M"

Output

Backup started at 06/07/2019 23:05:58
Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  7 files need to be examined (15.00 MB)
  Uploading file (15.03 MB) ...
  Uploading file (11.43 KB) ...
  Uploading file (1.29 KB) ...
Checking remote backup ...
  Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
  Downloading file (1.29 KB) ...
  2 files need to be examined (8 bytes)
  Downloading file (11.43 KB) ...
  Downloading file (15.03 MB) ...
  0 files need to be examined (0 bytes)
  Duration of backup: 00:00:10
  Remote files: 6
  Remote size: 15.05 MB
  Total remote quota: 0 bytes
  Available remote quota: 0 bytes
  Files added: 2
  Files deleted: 0
  Files changed: 0
  Data uploaded: 15.05 MB
  Data downloaded: 15.05 MB

Success

Restoring a Duplicati Backup on Linux

O’k lets delete the local “/test2/” folder then restore it

rm -R /test2

Now lets restore it (official guide)

/usr/bin/duplicati-cli restore "b2://bucket-name-goes-here/test" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --passphrase=************* --overwrite=true --restore-path="/test2"

Output

Restore started at 06/07/2019 23:21:51
Checking remote backup ...
  Listing remote folder ...
Checking existing target files ...
  2 files need to be restored (15.00 MB)
Scanning local files for needed data ...
  Downloading file (15.03 MB) ...
  0 files need to be restored (0 bytes)
Verifying restored files ...
Restored 0 (0 bytes) files to /test2
Duration of restore: 00:00:05

Did it work

cd /test2
ls -al
total 15368
...
-rw-r--r--  1 use user 10485760 Jun  3  2008 10MB.zip
-rw-r--r--  1 user user 5242880 Jun  3  2008 5MB.zip

Yes it worked

Backup 2 folders an emailing the results

I use Google GSuite for email and have an app password setup just for duplicati. Here is the official guide for sending an email after a backup operation

I downloaded a 100MB file

cd /test2
wget http://ipv4.download.thinkbroadband.com/100MB.zip

I ran this command to backup an emailthe progress

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=***************** [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"

Output

Backup started at 06/08/2019 00:02:28
Checking remote backup …
Listing remote folder …
Scanning local files …
8 files need to be examined (115.00 MB)
7 files need to be examined (15.00 MB)
Uploading file (49.99 MB) …
Uploading file (18.01 KB) …
Uploading file (49.99 MB) …
Uploading file (17.97 KB) …
Uploading file (233.33 KB) …
Uploading file (33.17 KB) …
Uploading file (1.42 KB) …
Compacting remote backup …
Checking remote backup …
Listing remote folder …
Verifying remote backup …
Remote backup verification completed
Downloading file (1.42 KB) …
4 files need to be examined (15.00 MB)
Downloading file (18.01 KB) …
Downloading file (49.99 MB) …
0 files need to be examined (0 bytes)
Duration of backup: 00:00:15
Remote files: 13
Remote size: 115.33 MB
Total remote quota: 0 bytes
Available remote quota: 0 bytes
Files added: 1
Files deleted: 0
Files changed: 0
Data uploaded: 100.28 MB
Data downloaded: 50.01 MB
Backup completed successfully!

Email

Screenshot of and email from Duplicati software

Backup Types and Versions

Read up here on backup retention policies here

TIP: Use the desktop Duplicati software to generate a backup then use the export to command line option to get the command line parameters to use in the command line version.

Screenshot of the desktop version of Duplicati and the export to command line link

After you click Commandline you can view each separate command liene option and parameter.

This was handy for finding this backup retention parameter

--retention-policy="1W:1D,4W:1W,12M:1M

“This will give me “one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup. “

Setup Auto Backup Cron Scripts

Login to your linux machines and edit a bash file (e.g in “/scripts/backupnow.sh” folder).

Make the script executable

chmod +x /scripts/backupnow.sh

Add the following

#!/bin/bash

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=123 [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"

Now you can add the script to your crontab by running the following.

crontab -e

Make sure you have the following in your crontab

SHELL=/bin/bash

Add a cron entry to run the bash script

#Backup Database
0 22 * * * /bin/bash /scripts/backupnow.sh

Use https://crontab.guru/ to check the time you want to run thew script

https://crontab.guru/ screenshot

Links

  • B2 Starter Guides
  • Duplicati Manual

 

 

 

Version

v1.1 Fixed typo (”downalod” thanks to @NicolasWolf for reporting.)

v1.0 Initial Post (added retention polity and Linux crontabs

v0.95 Backup, Restore and Send Email

v0.91 Added Duplicati service info

v0.9 Working Copy

Filed Under: Uncategorized Tagged With: B2, backblaze, Backup, cli, cloud, duplicati, Linux, windows

How to backup and restore a MySQL database on Windows and Linux

April 21, 2019 by Simon

Why backup and restore

This is a quick guide demonstrating how you can backup and restore a MySQL database on Windows and Linux using Adminer.

You may need to know how to backup a restore a database for a number of reasons..

e.g

  • Send the database to someone to debug or give feedback while learning.
  • Move the database from a local machine to the cloud
  • Move the database from cloud vendor A to cloud vendor B
  • etc.

Having a backup of the VM is good but having a backup of the database too is better. I use UpCloud for hosting my VM’s and setting backups is easy. But I cannot download those backups.

UpCloud Backup Screen

Murphy’s Law

“If anything can go wrong, it will”

The most important reason for taking a backup and knowing how to restore it is for disaster recovery reasons.

Backup (the easiest way) with Adminer

Adminer is a free PHP based IDE for MySQL and other databases. Simply install Adminer and save the file on your local computer or remote web server directory.

FYI: The Adminer author Jakub Vrana has a patron page, I am a patron of this awesome software.

Snip from Adminers website. “Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch andMongoDB.”

adminer.php file icon screenshot

TIP: The file would be publicly accessible to anyone so don’t save it to a common area, obfuscate the file, protect it of delete the file when you are done using it.

Once Adminer is installed load it in a web browser, login with your MySQL credentials. Once you login you will see all databases and an Import and Export menu.

Adminer main screen, all databases and import and export menu.

tbtest is a simple database with one table and 4 fields (ID, Key, Value and Modified)

.Click Export to open the export screen.

Export screen showing a list of databases and export options

Click Export, a SQL file will be generated (this is the export of the database).

Here is a save of the file:
https://fearby.com/wp-content/uploads/export.txt

Exported view of https://dev.mysql.com/doc/workbench/en/wb-admin-export-import-management.html

Its that simple.

If I add a binary blob file to the table and upload a PNG file lets see how the export looks.

Screenshot o the new table with a blog field in Adminer UI

Let export the database again in Adminer and check out the output. I used Sublime Text editor to view the export file.

New Export shows the binary file in the Backup SQL file

Restore (the easiest way) with Adminer

OK lets delete the tbtest database and then restore it with Adminer. I used Adminer to delete (DROP) the database.

Database dropped with Adminer

Database “dbtest” deleted.

Now lets create a blank database to restore to (same name).

Create database screen.

Database created.

dbtest created.

Now lets import the database backup using Adminer.

Click Import, select the backup file and un-tick Stop on errors.

Import screenshot, dxtest selectded, Restore file selected, stop on errors disabled

TIP: The 2MB next the the choose file button is defined by your web server and PHP configuration. If you are trying to import a larger database (e.g 80MB) first increase the limits in your web server and PHP (via php.ini).

The Import (restore should take seconds)

Import Success

The database was imported from a backup, all tables and records imported just fine.

The database was imported from a backup

Bonus methods.

On Ubuntu use this guide to backup from the command line. If you use the Oracle MySQL Workbench read this.

I hope this helps someone.

Filed Under: Adminer, Backup, Database, MySQL, Restore Tagged With: and, Backup, How, Linux, MySQL, on, restore, to, windows

Setting up the Debian Kali Linux distro to perform penetration testing of your systems

March 7, 2018 by Simon

This post will show you how to setup the Kali Linux distro to perform penetration testing of your systems

I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Securing your systems is very important (don’t stop) and keep learning (securing ubuntu in the cloud, securing checklist, run a Lynis system audit etc)

snip from: https://www.kali.org/about-us/

“Kali Linux is an open source project that is maintained and funded by Offensive Security, a provider of world-class information security training and penetration testing services. In addition to Kali Linux, Offensive Security also maintains the Exploit Database and the free online course, Metasploit Unleashed.”

Download Kali

I downloaded the torrent version (as the HTTP version kept stopping (even on 50/20 NBN).

Download Kali

After the download finished I checked the SHA sum to verify it’a integrity

cd /Users/username/Downloads/kali-linux-2018.1-amd64
shasum -a 256 ./kali-linux-2018.1-amd64.iso 
ed88466834ceeba65f426235ec191fb3580f71d50364ac5131daec1bf976b317  ./kali-linux-2018.1-amd64.iso

A least it matched the known (or hacked) hash here.

Installing Parallels in a VM on OSX

I use Parallels 11 on OSX to set up a VM os Demina Kali, you can use VirtualBox, VMWare etc.

VM Setup in Parallels

Hardware: 2x CPU, 2048MB Ram, 32MB Graphics, 64GB Disk.

I selected Graphical Install (English, Australia, American English, host: kali, network: hyrule, New South Wales, Partition: Guided – entire disk, Default, Default, Default, Continue, Yes, Network Mirror: Yes, No Proxy, Installed GRUB bootloader on VM HD.

Post Install

Install Parallel Tools

Official Guide: https://kb.parallels.com/en/123968

I opened the VM then selected the Actions then Install Parallels Tools, this mounted /media/cdrom/, I copied all contents to /temp/

As recommended by the Parallels instal bash script I updated headers.

apt install linux-headers-4.14.0-kali1-amd64

Then the following from https://kb.parallels.com/en/123968

apt-get clean
apt-get update
apt-get upgrade -y
apt-get dist-upgrade -y
apt-get install dkms kpartx printer-driver-postscript-hp

Parallels will not install, I think I need to upgrade to parallel 12 or 12 as the printer driver detection is not detecting (even though it is installed).

Installing Google Chrome

I used the video below

I have to run chrome with

/usr/bin/gogole-chrome-stable %U --no-sandbox --user-data=dir &

It works.

Chrome

Running your first remote vulnerability scan in Kali

I found this video useful in helping me scan and check my systems for exploits

Simple exploit search in Armitage (metasploit)

Armitage Scan

A quick scan of my server revealed three ports open and (22, 80 and 443). Port 80 redirects to 443 and port 22 is firewalled.  I have WordPress and exploits I rued failed to work thanks to patching (always stay ahead of patching and updating of software and the OS.

k006-ports

Without knowing what I was doing I was able to check my WordPress against known exploits. 

If you open the Check Exploits menu at the end of the Attacks menu you can do a bulk exploit check.

kali_bulk

WP Scan

Kali also comes with a WordPress scanner

wpscan --url https://fearby.com

This will try and output everything from your web server and WordPress plugins.

/xmlrpc.php was found and I was advised to deny access to that file in NGINX. xmlrpc.php is ok but can be used in denial of service attacks.

location = /xmlrpc.php {
	deny all;
	access_log off;
	log_not_found off;
}

I had a hit for a vulnerability in a Youtube Embed plugin but I had a patched version.

k007-wpscan

TIP: Check your WordPress often.

More to come (Draft Post).

  • OWASP scanner
  • WPSCAN
  • Ethical Hacker modules
  • Cybrary training
  • Sent tips to @FearbySoftware

Tips

Don’t have unwanted ports open, securely installed software, Use unattended security updates in Ubuntu, update WordPress frequently and limit plugins and also consider running more verbose audit tools like Lynis.

More Reading

Read my OWASP Zap guide on application testing and Cloudflare guide.

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.2 added More Reading links.

v1.1 Added bulk exploit check.

v1.0 Initial post

Filed Under: Exploit, Linux, Malware, Security, Server, SSH, Vulnerability Tagged With: debian, distro, Kali, Linux, of, penetration, perform, Setting, systems, testing, the, to, up, your

Useful Linux Terminal Commands

August 13, 2017 by Simon

Below are Ubuntu Linux commands I use often to setup, debug maintain servers.

Read this guide for Useful OSX Commands (for setting up Apache, PHP, MySQL, Adminer etc on OSX)

I recently moved my domain from a C-Panel hosted domain (and Email to Google G Suite (my guide here)) to a self-managed Digital Ocean domain (my LetsEncrypt Guide here, my Digital Ocean guide here, my AWS setup guide here, my Vultr setup guide here) and needed to transfer my WordPress site. Setup your own Digital Ocean Ubuntu server form $5  a month (get the first 2 months free by clicking here) or setup your own Vultr Ubuntu server for as low as $2.5/month by clicking here.

How to Reboot (from time to time when prompted with *** system restart required ***” messages appear).

sudo shutdown -r now

How to set up a  bash file (*.sh) as Executable.

chmod +X filename.sh

The file will now be executable.

Viewing your crontab (Windows Task Scheduler equiv)

crontab -e

Ping a port

nmap -p 80 google.com

Rename a folder

mv /www/oldname /www/newname

Set the owner of a folder

sudo chown -R www-data:www-data /wwwfolder/wp-content/uploads/2017/11/

Check the rsync port

nmap -p 873 theserver.com

Starting Nmap 7.01 ( https://nmap.org ) at 2017-08-19 10:34 AEST
Nmap scan report for theserver.com (xxx.xxx.xxx.xxx)
Host is up (0.00012s latency).
Other addresses for theserver.com (not scanned): xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
rDNS record for xxx.xxx.xxx.xxx: theserver.com
PORT    STATE SERVICE
873/tcp open  rsync

Run a file every 1 minute

*/1 * * * * /scripts/script1.sh

Show server name

hostname

How to verify patch status for Meltdown and Spectre

Read my guide here to install the patch here.

Verify Spectre and Meltdown patch status

dmesg | grep isolation && echo "patched :)" || echo "unpatched :("
[ 0.000000] Kernel/User page tables isolation: enabled
patched :)

or

sudo grep "cpu_insecure\|cpu_meltdown\|kaiser" /proc/cpuinfo && echo "patched :)" || echo "unpatched :("
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
patched :)

Restart network

sudo /etc/init.d/networking restart

More here.

Show Operating System Name

hostnamectl ! grep "Operating System"
 Operating System: Ubuntu 10.04.3 LTS

Show installed Packages

sudo apt-get install apt-show-versions

Show all packages with “PHP” in the name.

sudo apt-show-versions | grep php | more
 
libapache2-mod-php7.0:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
libapache2-mod-php7.0:i386 not installed
php-common:all/xenial 1:55+ubuntu16.04.1+deb.sury.org+1 uptodate
php-xdebug:amd64/xenial 2.5.5-3+ubuntu16.04.1+deb.sury.org+1 uptodate
php-xdebug:i386 not installed
php7.0:all/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-cli:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-cli:i386 not installed
php7.0-common:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-common:i386 not installed
php7.0-curl:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-curl:i386 not installed
php7.0-dev:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-dev:i386 not installed
php7.0-fpm:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-fpm:i386 not installed
php7.0-gd:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-gd:i386 not installed
php7.0-imap:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-imap:i386 not installed
php7.0-intl:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-intl:i386 not installed
php7.0-json:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-json:i386 not installed
php7.0-ldap:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-ldap:i386 not installed
php7.0-mbstring:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-mbstring:i386 not installed
php7.0-mysql:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-mysql:i386 not installed
php7.0-opcache:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-opcache:i386 not installed
php7.0-pgsql:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-pgsql:i386 not installed
php7.0-phpdbg:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-phpdbg:i386 not installed
php7.0-pspell:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-pspell:i386 not installed
php7.0-readline:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-readline:i386 not installed
php7.0-recode:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-recode:i386 not installed
php7.0-snmp:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-snmp:i386 not installed
php7.0-tidy:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-tidy:i386 not installed
php7.0-xml:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-xml:i386 not installed
php7.0-zip:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-zip:i386 not installed

Send Messages Other Logged In Users (CLI)

Show all user logged in

w
 20:11:51 up 1 day, 10:25,  2 users,  load average: 0.00, 0.04, 0.01
USER     TTY      FROM             [email protected]   IDLE   JCPU   PCPU WHAT
root     tty1                      20:03   31.00s  0.24s  0.20s -bash
user1    pts/0    123.123.123.123  20:09    0.00s  0.08s  0.01s w

Sent a message to usr1

echo "Hello User1" > /dev/pts/0

Sent a message to the root console

echo "Hello Admin" > /dev/tty1

Messages will appear at the bottom of the user’s console.

Processor

List processes in a tree view (… = removed)

ps -e --forest
  PID TTY          TIME CMD
    2 ?        00:00:00 kthreadd
    3 ?        00:00:00  \_ ksoftirqd/0
    5 ?        00:00:00  \_ kworker/0:0H
    7 ?        00:01:56  \_ rcu_sched
    8 ?        00:00:00  \_ rcu_bh
    9 ?        00:00:00  \_ migration/0
   35 ?        00:00:00  \_ vmstat
   37 ?        00:00:00  \_ ecryptfs-kthrea
   54 ?        00:00:00  \_ acpi_thermal_pm
   55 ?        00:00:00  \_ vballoon
   65 ?        00:00:00  \_ scsi_eh_0
   66 ?        00:00:00  \_ scsi_tmf_0
   67 ?        00:00:00  \_ scsi_eh_1
   68 ?        00:00:00  \_ scsi_tmf_1
   74 ?        00:00:00  \_ ipv6_addrconf
   36 ?        00:00:00  \_ kpsmoused
  456 ?        00:00:00  \_ iscsi_eh
...
    1 ?        00:00:02 init
...
  452 ?        00:00:00 upstart-file-br
  453 ?        00:00:00 dbus-daemon
...
 1489 ?        00:00:00 cron
 1514 ?        00:00:06 irqbalance
 1518 ?        00:00:00 sshd
11855 ?        00:00:00  \_ sshd
11914 pts/4    00:00:00      \_ bash
12008 pts/4    00:00:00          \_ ps
 1523 ?        00:00:09 php-fpm7.0
 1785 ?        00:00:03  \_ php-fpm7.0
 1786 ?        00:00:02  \_ php-fpm7.0
...
 1692 ?        00:01:52 mysqld
 ...
 1891 ?        00:00:53 fail2ban-server
...
 1956 ?        00:00:00 nginx
 1957 ?        00:00:02  \_ nginx
 1958 ?        00:00:03  \_ nginx
 1959 ?        00:00:01  \_ nginx
 1978 ?        00:00:24 ntpd
 2000 ?        00:00:00 systemd-logind
 2011 ?        00:03:24 redis-server
 ...

View major processes by usage/memory

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head                                    Sat Sep 16 18:51:17 2017

  PID  PPID CMD                         %MEM %CPU
 1692     1 /usr/sbin/mysqld             5.0  0.0
 1523     1 php-fpm: master process (/e  1.0  0.0
 1662     1 /usr/bin/lxd --group lxd --  0.4  0.0
 1785  1523 php-fpm: pool www            0.4  0.0
 1786  1523 php-fpm: pool www            0.3  0.0
 1891     1 /usr/bin/python3 /usr/bin/f  0.3  0.0
 1957  1956 nginx: worker process        0.2  0.0
 1958  1956 nginx: worker process        0.2  0.0
11855  1518 sshd: [email protected]/4             0.1  0.0

Read more on ps here.

How big is a folder

du -hs ./foldername
412MB    ./foldername

Change FIle Create/Modify time’s

Change File Creation Date

SetFile -d '11/25/2019 23:00:00' ./file.doc

Change file modify/accessed time

touch -mt 201911282300 ./filename.doc

Tree

Tree needs to be installed first

sudo apt-get install tree

Show an ASCII representation of a folder structure

tree

Show files in a  structure

tree -a -h -v
.
├── [4.0K]  folder
├── [3.0K]  logfile.log
└── [1.7M]  zipfile.tgz

Show directories

tree -d
.
└── [4.0K]  subfolder

List all files and folders in a structure

tree -a -f -p -h  -l -R

Backup a  www folder

cp -rTv /www/ /backup/www

Common (Digital Ocean) Debugging commands

cat /etc/network/interfaces.d/50-cloud-init.cfg
cat /etc/network/interfaces
ip addr
ip route
uname -a
iptables -nvL --line-numbers
ls -l /lib/modules
cat /etc/udev/rules.d/70-persistent-net.rules

Networking

Display all TCP connections

netstat -at

Display all UDP connections

netstat -au

List all Listening Connections

netstat -l

Show all Network stats

netstat -s

Show all UDP Network stats

netstat -st

Show all TCP Network stats

netstat -su

Show network packets

netstat -i

Displaying raw info

netstat --statistics --raw

Show open ports

netstat -a | grep "LISTEN "

Upload a file to a remote server over ssh

scp /local/folder/local-file.zip [email protected]:/remote/server/destination-folder/

Zip files

Install zip

sudo apt-get install zip

Zip a  whole directory (with high compression)

zip -r -9 /folder/zipfile.zip /directory-to-zip

Zip a  whole directory (excluding *.tmp, *temp, *.bak and *.zip file types)

zip -r -9 /folder/zipfile.zip /directory-to-zip -x "*.tmp" -x "*.temp" -x"./backup/*.bak"-x "./backup/*.zip" -x "*promo*.mp4""

Zip Help

zip
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
Zip 3.0 (July 5th 2008). Usage:
zip [-options] [-b path] [-t mmddyyyy] [-n suffixes] [zipfile list] [-xi list]
  The default action is to add or replace zipfile entries from list, which
  can include the special name - to compress standard input.
  If zipfile and list are omitted, zip compresses stdin to stdout.
  -f   freshen: only changed files  -u   update: only changed or new files
  -d   delete entries in zipfile    -m   move into zipfile (delete OS files)
  -r   recurse into directories     -j   junk (don't record) directory names
  -0   store only                   -l   convert LF to CR LF (-ll CR LF to LF)
  -1   compress faster              -9   compress better
  -q   quiet operation              -v   verbose operation/print version info
  -c   add one-line comments        -z   add zipfile comment
  [email protected]   read names from stdin        -o   make zipfile as old as latest entry
  -x   exclude the following names  -i   include only the following names
  -F   fix zipfile (-FF try harder) -D   do not add directory entries
  -A   adjust self-extracting exe   -J   junk zipfile prefix (unzipsfx)
  -T   test zipfile integrity       -X   eXclude eXtra file attributes
  -y   store symbolic links as the link instead of the referenced file
  -e   encrypt                      -n   don't compress these suffixes
  -h2  show more help

Backup NGINX

zip -r -9 /backup/nginx.zip /etc/nginx/ -x "*.tmp" -x "*.temp" -x"./backup/*.bak" -x "./backup/*.zip"

Unzip

Installing Unzip

sudo apt-get install unzip

Unzip a  file

unzip filename.zip

Updates

Setup Unattended Ubuntu Security updates

sudo apt-get install unattended-upgrades
sudo unattended-upgrades -d

At login, you should receive

0 updates are security updates.

Show Configured NGINX server names

grep "server_name" /etc/nginx/sites-available/default
server_name www.servername.com myservername.com localhost;

Services

Reload daemon services

systemctl daemon-reload

Verify the MongoDB service file exists

ls /etc/systemd/system | grep servivename

View the status of services

systemctl list-unit-files --type=service
UNIT FILE                                  STATE
accounts-daemon.service                    enabled
acpid.service                              disabled
[email protected]                    static
apt-daily-upgrade.service                  static
apt-daily.service                          static
...

Local Dump

locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US
en_US.iso88591
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
POSIX

Show All Defined Ports

cat /etc/services

Show defined rsync ports

cat /etc/services | grep rsync

Show listening ports (active connections)

netstat -plntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1707/mysqld
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      2023/redis-server 1
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1968/nginx
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      2097/pure-ftpd (SER
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1525/sshd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1968/nginx
tcp6       0      0 fe80::1:13128           :::*                    LISTEN      1708/lxd-bridge-pro
tcp6       0      0 :::80                   :::*                    LISTEN      1968/nginx
tcp6       0      0 :::21                   :::*                    LISTEN      2097/pure-ftpd (SER
tcp6       0      0 :::22                   :::*                    LISTEN      1525/sshd
udp        0      0 10.99.0.10:123          0.0.0.0:*                           1990/ntpd
udp        0      0 45.77.48.141:123        0.0.0.0:*                           1990/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           1990/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           1990/ntpd
udp6       0      0 fe80::1:123             :::*                                1990/ntpd
udp6       0      0 ::1:123                 :::*                                1990/ntpd
udp6       0      0 :::123                  :::*                                1990/ntpd

Show service status

service --status-all

View your bash history for a past command (e.g “openssl”).

grep "openssl" ~/.bash_history
openssl req -new -newkey rsa:4096 -nodes -keyout fearby.key -out fearby.csr

View last 1 line of a file.

tail -n 1 index.html
</body></html>

Ping a server

ping -c 2 fearby.com

How to check system uptime (and load average)

uptime
> 12:18:48 up 1 min,  1 user,  load average: 0.30, 0.18, 0.17

Uptime in friendly format

uptime -p
up 23 hours, 42 minutes

The load averages at the end are the last 1, 5 min and 15 minutes.

The w command is handy for showing uptime information as-well as logged in users (The ‘w -i’  parameter -i is handy for seeing what IP people are logged in from).

w -i
> 12:22:41 up 5 min,  1 user,  load average: 0.00, 0.07, 0.05
USER     TTY      FROM             [email protected]   IDLE   JCPU   PCPU WHAT
root     pts/0    123.123.123.123    12:18    1.00s  0.07s  0.00s w

How to download a file

curl -o localfile.zip http://www.serverwhereiuploadedthefile.com/remotefile.zip

How to edit NGINX configuration

sudo nano /etc/nginx/nginx.conf

and

sudo nano /etc/nginx/sites-available/default

How to find a file

find / -name filename.ext

also

locate php.ini

Find contents in files (recursive)

grep -r "ahref" *

Find files by name and run a command on each

find -iname "index.html" -exec md5sum {} \;

Show differences in files

diff index.html index2.html
< <body>Loading <a href="http://simon.fearby.com/blog/">http://simon.fearby.com/blog/</a></body></html>
---
> <body>Loading <a href="https://www.fearby.com/blog/">https://www.fearby.com/blog/</a></body></html>

Show contents of file (e.g urls.txt)

cat urls.txt
http://www.server1.com
http://www.server2.com
http://www.server3.com
http://www.server4.com
http://www.server5.com

Download all files mentioned in a text file.

cat urls.txt | xargs wget –c
.. download 4 files ...
cat urls.txt | xargs wget –c

View all packages with updates

sudo /usr/lib/update-notifier/apt-check -p

Output:

cryptsetup-bin
libdns-export162
libisccfg140
mongodb-org-mongos
linux-libc-dev
libgdk-pixbuf2.0-0
tcpdump
bind9-host
dnsutils
nodejs
libpython3.5
python3.5
python3.5-minimal
libisc160
grub-legacy-ec2
libapparmor1
libplymouth4
mongodb-org-shell
ntp
libtidy5
libapparmor-perl
libisc-export160
liblwres141
libcryptsetup4
libgdk-pixbuf2.0-common
libdns162
apache2-bin
apparmor
libisccc140
mongodb-org
libpython3.5-stdlib
libbind9-140
libpython3.5-minimal
cryptsetup
mongodb-org-server

or (on ubuntu 16.04)

apt list --upgradable

Updates

Always backup your server’s configuration before updating.

Backup MySQL

mysqldump --all-databases > /backup/dump-$( date '+%Y-%m-%d_%H-%M-%S' ).sql -u root -p

Crontab

Add this to crontab -e to backup at 1 am every day

0 1 * * * /usr/bin/mysqldump --all-databases > /backup/mysql/dump-$( date '+%Y-%m-%d_%H-%M-%S' ).sql -u root -pmysqlpassword

/scripts/shrinkmysql.sh  script to shrink SQL files

sudo nano /scripts/shrinkmysql.sh
#!/bin/bash

tar -zcf /backup/mysql-$( date '+%Y-%m-%d_%H-%M-%S' ).tgz /backup/mysql/
rm /backup/mysql/*.sql

I had to add this to crontab

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin/:/sbin:/usr/sbin:/scripts/:

Cron job to shrink SQL dumps at 2am every day

0 2 * * * /bin/bash /scripts/shrinkmysql.sh > /dev/null 2>&1

Write to a single log file from the crontab at 2 am every day.

todo

Query a package (e.g. siege package)

sudo dpkg-query -l | grep siege *

How to set up a free SSL certificate (see my guide here).

How to set your timezone.

dpkg-reconfigure tzdata

How to restart PHP

sudo systemctl restart php7.0-fpm

How to show the time on the server

sudo hwclock --show

Reload and restart the NGINX configuration and web server.

sudo nginx -t
sudo nginx -s reload
sudo /etc/init.d/nginx restart

JSON viewing program

Installing

sudo apt-get install jq

Using

wget wget https://downloads.wordpress.org/plugin/genesis-enews-extended.2.0.2.zip

or

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq .

output from json tool

Below are utilities I use a lot.

ncdu file size utility

Installing

sudo apt-get install ncdu

Using

sudo ncdu /

pydf disk checking utility

Installing

sudo apt-get install pydf

Using

pydf

output from pydf tool

ntp timezone service

Installing

sudo apt-get install ntp

Using

ntp

Displaying startup processes

Installing

sudo apt-get install rcconf

Using

sudo rcconf

htop process manager.

Installing

sudo apt-get install htop

Using

htop

output from htop tool

Network Benchmarking (between two servers)

I use iperf to measure total bandwidth between two servers. You will need to allow port 5001 (TCP IPV4 and TCP IPV6 in and out) in any local firewalls and hosts GUIs.

Allow port 5001 on an ufw firewall (IN and OUT)

sudo ufw allow 5001

#I Set port 5001 firewall on my hosts (Digital  Ocean and Vultr GUI)

Deny IP

sudo ufw deny from 123.123.123.123

Allow port 22 access to known IP

sudo ufw allow from 123.123.123.123/24  to any port 22

Deny Outgoing Port

sudo ufw deny out 22

Allow out on port to known IP

sudo ufw allow out from 123.123.123.123 to any port 22

More securing Ubuntu in the cloud here.

Install iperf on the target and source Ubuntu server.

sudo apt-get install iperf

Run this on the listening server.

iperf -s

Run this on benchmarking server (and add the IP of the listening server).

iperf -c 123.123.123.123

Results

Screen dump of ipref -c ip
iperf benchmarking output

Testing concurrent connections to a web server with siege.

Install siege

sudo apt-get install siege

Benchmarking a HTTP server

./siege -t1m c10 'https://fearby.com'

#10 Concurent users

Benchmarking HTTPS sites

Install siege 4.0.2 ( steps here ).

Verify siege 4.0.2 is installed by running

siege -v

Now can benchmark https sites

./siege -t1m c10 'https:/thedomain.com'

View incoming connections on the target server

sudo netstat -tupn

I always increase my history size and tell it not to store duplicates.

Viewing your typed terminal history

history

Increasing your history size

HISTSIZE=10000
HISTCONTROL=ignoredups

How to Update Ubuntu

sudo apt-get update
sudo apt-get dist-upgrade

or

sudo apt-get update && sudo apt-get upgrade

Edit SSH authorized keys

sudo nano ~/.ssh/authorized_keys

Search file and show lines where text matches

grep -i "href" index.html
> <body>Loading <a href="http://simon.fearby.com/blog/">http://simon.fearby.com/blog/</a></body></html>

View packages with updates

/usr/lib/update-notifier/apt-check --human-readable
35 packages can be updated.
15 updates are security updates.

View Boot Text

dmesg

Automatic Monitoring (ever 1 second)

Active network connections

watch -n 1 'netstat -at'

Network Packets

watch -n 1 'netstat -i'

Free memory

watch -n 1 'free -m'

Memory breakdown

watch -n 1 'cat /proc/meminfo'

or

watch -n 1 'vmstat -s'

Monitor NGINX memory

watch -n 1 'ps axu |grep nginx'

more soon…

Read this guide for Useful OSX Commands (for setting up Apache, PHP, MySQL, Adminer etc on OSX)

50 most useful Linux commands (view here).

View other Linux command informing sites here, here and here.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.8 Show users and senda messages to other users

v.71 Show Operating System Name

v1.7 added Meltdown and Spectre patch information

v1.6 Removed -x fom zip directory

V1.5 Restart Network

v1.4 added Useful OSX Commands

Filed Under: Terminal, Ubuntu, VM Tagged With: commands, Linux, terminal

Setting up a Raspberry Pi Zero W

June 28, 2017 by Simon

fyi: Here is my guide that I created while I set up a Raspberry Pi Zero W. My previous Raspberry PI 2 Setup Guide here and I wanted to try the single-core Raspberry Pi Zero W. Eventually, want to run my Raspberry Pi from batteries (my older guide here).

I plugged in a  2Amp 5v micro USB powder pack, micro HDMI cable, micro USB keyboard and expected to see a light but nothing.  This guide says the Raspberry Pi has no power LED. Unfortunately, I thought the power pack was faulty and I plugged it in a few times and corrupted the installed SD-card.

Failed Setup

After reading this guide I remembered I forgot to switch the monitor to the HDMI input (oops).

Yep, I realized I corrupted the OS on the SD Card.

Old Noobs Operating System

I checked the SD-Card (that I purchased from eBay) I noticed the SD-Card had an older NOOBS 2.0.0 installation in an in an invalid subfolder (that’s why it would not boot).

I plugged the Micro SD card into an SD-Card adapter and plugged it into my Mac (time to install a newer OS).

Downloading the latest Raspian Operating System

I visited https://www.raspberrypi.org/downloads/raspbian/ and download raspbian

I downloaded a 1.66GB Zip File.

The file took 30 mins to download.

I expanded the zip file to a 4.66 img file.

Formatting the SD-Card

I formatted my SD Card with the SD Card Association SD formatting tool from https://www.sdcard.org/downloads/formatter_4/ here.

Setting up the SD-Card

The Raspberry Pi Foundation has a great guide on how to install the Raspberry Pi Raspian image onto an SD-Card here.

I downloaded the free 64MB Etcher program for Mac OS here https://etcher.io/ in order to copy (flash) Raspian onto the SD-Card.

I opened Etcher and selected the Raspian Jesse image file and clicked flash.

The flash will take between 5-10 minutes.

SD Card Contents

The Raspian image is now ready for reinsertion back into the Raspberry Pi to setup

1st Raspberry Pi (Raspian) boot.

The raspberry pi does, in fact, have a  power/activity led with the latest operating system 🙂

The desktop loaded quite swiftly too. My Bay special keyboard detected just fine. You can manually connect to your wifi by entering the following command in the terminal (thanks to this guide).

Pi

sudo nano /etc/network/interfaces

Generic Config

sudo rasps-config

# > I changed the password.
# > I chnaged the hostname.
# > Configured the pi to book to console (logged in) and not the desktop
# > Configured the pi to wait for the network on boot
# > Configured the splash screen not to show at boot
# > Configured the local to be "en_AU.UTF-8 UTF-8" ("en_AU.UTF-8")
# > Configured timezone to be Australia/Sydney
# > Configured the keyboard to be a "Generic 104 Key PC" (Englist (US)
# > Configured Wifi Country to be AU
# > Configured the (Interface) Camera = ON
# > Configured the (Interface) SSH = ON
# > Configured the (Interface) SPI = ON
# > Configured the (Interface) I2C = ON
# > Configured the (Interface) Serial = ON
# > Configured the (Interface) 1-Wire = ON
# > Configured the (Interface) Remote GPIO = OFF
# > Configured the (Interface) SPI = ON
# > Configured the (Advanced) GPU Memory = 16
# Finish and Reboot

I am not using the desktop much so I am happy setting 16MB for the video card.

Get the Mac Address and set a static IP Address (to allow remote SSH management)

After a reboot, I was logged into a shell and was able to bring up the devices network details (including mac address).

ifconfig

Test the network connection

I was successful;y able to ping a remote server via wifi.

ping www.fearby.com

Pi Zero Network Update

Static IP Address

Todo: I set up my local ADSL router to give the Raspberry Pi a Static (known) IP every time it boots.

Update the Pi Software (after the network is up)

Now that I knew the network is working I was able to update the Pi software.

sudo apt-get update
sudo apt-get upgrade
sudo rpi-update
sudo reboot

Pi Zero

I re-ran the update just in case. Now my Pi is all up to date.

Installed Misc Software

sudo apt-get install image-magick
sudo apt-get install nginx
sudo /etc/init.d/nginx start
sudo apt-get install php5-fpm

More on setting up an NGINX web server here.

Set the Pi Mouse (regular sized USB devices)

I had small micro USB to larger USB adapters (from eBay)  but could only use a keyboard or mouse at any one time. I Also had a micro USB to multiple micro USB and Ethernet adapter (from eBay) (SN: YS-LAN38) but it failed to work with NOOBS 2.0.0 (maybe it will work with Raspian latest)

I wanted to test regular devices in the case of an emergency.

Pi Zer UBS Adapter

And the mouse works (and the X desktop worked with 16MB assigned)

Help

40 Useful Raspberry Pi Commands.

Todo

Add information on adding and using a 5MP Raspberry Pi camera and setting this up as a https://fearby.com/article/raspberry-pi-2b-security-webcam/secuerity webcam.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.2 DRAFT: Added more images

Filed Under: IoT, Raspberry Pi Tagged With: IoT, Linux, raspbery pi, raspian, w, zero

Primary Sidebar

Poll

What would you like to see more posts about?
Results

Support this Blog

Create your own server today (support me by using these links

Create your own server on UpCloud here ($25 free credit).

Create your own server on Vultr here.

Create your own server on Digital Ocean here ($10 free credit).

Remember you can install the Runcloud server management dashboard here if you need DevOps help.

Advertisement:

Tags

2FA (9) Advice (17) Analytics (9) App (9) Apple (10) AWS (9) Backup (21) Business (8) CDN (8) Cloud (49) Cloudflare (8) Code (8) Development (26) Digital Ocean (13) DNS (11) Domain (27) Firewall (12) Git (7) Hosting (18) HTTPS (6) IoT (9) LetsEncrypt (7) Linux (20) Marketing (11) MySQL (24) NGINX (11) NodeJS (11) OS (10) PHP (13) Scalability (12) Scalable (14) Security (44) SEO (7) Server (26) Software (7) SSH (7) ssl (17) Tech Advice (9) Ubuntu (39) Uncategorized (23) UpCloud (12) VM (44) Vultr (24) Website (14) Wordpress (25)

Disclaimer

Terms And Conditions Of Use All content provided on this "www.fearby.com" blog is for informational purposes only. Views are his own and not his employers. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. Never make changes to a live site without backing it up first.

Advertisement:

Footer

Popular

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Add Google AdWords to your WordPress blog

Security

  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • Setting up DNSSEC on a Namecheap domain hosted on UpCloud using CloudFlare
  • Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx
  • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
  • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
  • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
  • Beyond SSL with Content Security Policy, Public Key Pinning etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Run an Ubuntu VM system audit with Lynis
  • Securing Ubuntu in the cloud
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Code

  • How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains
  • Useful Java FX Code I use in a project using IntelliJ IDEA and jdk1.8.0_161.jdk
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider
  • How to setup PHP FPM on demand child workers in PHP 7.x to increase website traffic
  • Installing Android Studio 3 and creating your first Kotlin Android App
  • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
  • How to use Sublime Text editor locally to edit code files on a remote server via SSH
  • Creating your first Java FX app and using the Gluon Scene Builder in the IntelliJ IDEA IDE
  • Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

Tech

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Is OSX Mojave on a 2014 MacBook Pro slower or faster than High Sierra
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • The case of the overheating Mac Book Pro and Occam’s Razor
  • Useful Linux Terminal Commands
  • Useful OSX Terminal Commands
  • Useful Linux Terminal Commands
  • What is the difference between 2D, 3D, 360 Video, AR, AR2D, AR3D, MR, VR and HR?
  • Application scalability on a budget (my journey)
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

Wordpress

  • Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution
  • Setting web push notifications in WordPress with OneSignal
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Wordfence Security Plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
  • Moving WordPress to a new self managed server away from CPanel
  • Moving WordPress to a new self managed server away from CPanel

General

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Using the WinSCP Client on Windows to transfer files to and from a Linux server over SFTP
  • Connecting to a server via SSH with Putty
  • Setting web push notifications in WordPress with OneSignal
  • Infographic: So you have an idea for an app
  • Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl
  • Building faster web apps with google tools and exceed user expectations
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in

Some ads on this site use cookies. You can opt-out if of local analytics tracking by scrolling to the bottom of the front page or any article and clicking "You are not opted out. Click here to opt out.". Accept Reject Read More
GDPR, Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT