(laptrinhx.com – do not steal this post)
In early 2021 www.fearby.com died and it was all my fault. Here is my breakdown of the events.
On the 4th of January 2021, I woke to see my website not loading.
https://www.fearby.com had 2 servers.
- Web server (www.fearby.com)
- Database server (db.fearby.com)
Upon investigating why my website was down I found out that WordPress could not talk to the database server. I tried to log into the database server via SSH failed (no response). I tried logging into my db.fearby.com server via the root console and it too did not work.
I was locked out of my own server and memory told me it was caused by my be playing with fail2ban and other system auditing tools a few months earlier.
I tried restoring the db.fearby.com serer from backups (one at a time). I had the last 7 days as individual backups. I had no luck, all of my backups were no good (I sat on the problem too long).
In mid-2020 I locked myself out of db.fearby.com (SSH and root console) because I setup an aggressive fail2ban, AIDE intrusion detection system(s) and firewall rules. I could no longer access my db.fearby.com server via SSH or the root console. The database server was still operational and I foolishly left it running (with no access).
I did not know how to (or had enough time) reset the root password on the Debian server. I was unable to think of a fix to restore my website. I should have reset the root password, it is easy to do thanks to a post from Janne Roustemaa – How to reset root password on cloud server.
A few months ago I finally found out how to reset the root password of a Debian server.
How to Reset the Root Password on a Debian server on UpCloud
I followed Janne Roustemaa’s guide here: How to reset root password on cloud server.
I logged into the Up Cloud Hub.
I shut down db.fearby.com from the UpCloud Dashboard.
I created a backup of db.fearby.com (just in case). I upgraded my backup plan from 1 backup every 7 days to daily backups for 7 days and weekly backups for 1 month.
Deploy a temporary server to reset the root password
I deployed a new temporary (cheap) server alongside the dead server in Chicago.
Get $25 free credit on UpCloud and deploy your own server: Use this link to get $25 credit (new UpCloud users only).
I called the server “recovery.fearby.com” and set Debian 9 and the Operation System (same as the dead server).
I added the command “shutdown -h 1” to the Initialization script to ensure the server shuts down after it was deployed.
I can only add the disk to the new server if the old db.fearby.com server is shut down.
I shut down db.fearby.com and recovery.fearby.com servers.
Both servers have shut down
Detach the disk from db.fearby.com
I detach the disk from db.fearby.com server.
In the UpCloud Dashboard, I opened the db.fearby.com and clicked the resize tab
I clicked the Detach button.
I clicked Continue
Attach db.fearby.com disk to recovery.fearby.com
Now I attached this disk as a secondary disk onto the recovery.fearby.com server.
In the UpCloud hub I clicked Servers then selected the recovery.fearby.com server, then clicked the Resize Tab
I scrolled down and clicked Attach existing storage
Attach existing storage dialogue
I selected the system disk from the db.fearby.com (that I detached earlier)
I clicked Add a storage device button
Now I have attached the storage from db.fearby.com and attached it to recovery.fearby.com as a secondary disk.
Starting the recovery.fearby.com server
I started the recovery.fearby.com server by clicking Start
The server is starting
When the server started, I obtained its IP and connect to it with MobaXTerm.
Now I can access the db.fearby.com disk.
I do not want to reset the root password until I undelete the files I need.
Viewing Disks
I ran this command to verify the attached disks.
lsblkTwo disks were visible
Alternatively, I can view partitions with the following command
cat /proc/partitions
major minor #blocks name
254 0 26214400 vda
254 1 26213376 vda1
254 16 52428800 vdb
254 17 52427776 vdb1
I can see partition data with these commands
recovery.fearby.com disk: /dev/vda1
fdisk -l /dev/vda1
Disk /dev/vda1: 25 GiB, 26842497024 bytes, 52426752 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
db.fearby.com disk: /dev/vdb1
fdisk -l /dev/vdb1
Disk /dev/vdb1: 50 GiB, 53686042624 bytes, 104855552 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytesI ran this command to mount the second disk
mount /dev/vdb1 /mntI looked in the “/mnt/Backup” folder as this was where I had daily MySQL dumps saving too on the old system.
The folder was empty. It looks like the file system was corrupted.
Finding Deleted Files
I assumed the file system was corrupt, I installed Testdisk as I wanted to recover deleted SQL dump backups in the backup folder.
sudo apt-get update
sudo apt-get install testdiskI confirmed testdisk was installed
photorec --versionI ran testdisk and passed in the disk as a parameter
sudo photorec /dev/vdb1I selected the Disk /dev/vdb1 – 53GB and pressed enter
I selected ext4 partition and pressed enter (not whole disk)
I selected ext2/ext3/ext4 filesystem and pressed enter
I selected Free to only scan in unallocated space and pressed enter
When asked to choose the recovery location to restore files to I selected /recovery (on the recovery.fearby.com disk , not the db.fearby.com disk.
I just realized that the recovery.fearby.com disk is 25Gb and the db.fearby.com disk is 50GB, lets hope I do not have more than 25GB of deleted files (or I will have to delete recovery.fearby.com and deploy a 100GB system) and run undelete again.
After 20 minutes 32,809 files were recovered to /recovery/recup_dir
I pressed CTRL+C to exit photorec
I changed directory to /recovery
cd /recoveryI counted the recovered files
find . -type f | wc -l
>32810Recovered Files were placed in sub folders.
drwxr-xr-x 68 root root 4096 Jan 19 13:28 .
drwxr-xr-x 23 root root 4096 Jan 19 13:28 ..
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.1
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.10
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.11
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.12
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.13
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.14
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.15
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.16
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.17
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.18
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.19
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.2
drwxr-xr-x 2 root root 36864 Jan 19 13:22 recup_dir.20
drwxr-xr-x 2 root root 24576 Jan 19 13:22 recup_dir.21
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.22
drwxr-xr-x 2 root root 24576 Jan 19 13:22 recup_dir.23
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.24
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.25
drwxr-xr-x 2 root root 32768 Jan 19 13:22 recup_dir.26
drwxr-xr-x 2 root root 32768 Jan 19 13:22 recup_dir.27
drwxr-xr-x 2 root root 36864 Jan 19 13:22 recup_dir.28
drwxr-xr-x 2 root root 32768 Jan 19 13:22 recup_dir.29
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.3
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.30
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.31
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.32
drwxr-xr-x 2 root root 20480 Jan 19 13:22 recup_dir.33
drwxr-xr-x 2 root root 36864 Jan 19 13:22 recup_dir.34
drwxr-xr-x 2 root root 36864 Jan 19 13:22 recup_dir.35
drwxr-xr-x 2 root root 36864 Jan 19 13:22 recup_dir.36
drwxr-xr-x 2 root root 32768 Jan 19 13:22 recup_dir.37
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.38
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.39
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.4
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.40
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.41
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.42
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.43
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.44
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.45
drwxr-xr-x 2 root root 20480 Jan 19 13:23 recup_dir.46
drwxr-xr-x 2 root root 20480 Jan 19 13:24 recup_dir.47
drwxr-xr-x 2 root root 20480 Jan 19 13:24 recup_dir.48
drwxr-xr-x 2 root root 20480 Jan 19 13:24 recup_dir.49
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.5
drwxr-xr-x 2 root root 20480 Jan 19 13:24 recup_dir.50
drwxr-xr-x 2 root root 20480 Jan 19 13:24 recup_dir.51
drwxr-xr-x 2 root root 20480 Jan 19 13:25 recup_dir.52
drwxr-xr-x 2 root root 20480 Jan 19 13:25 recup_dir.53
drwxr-xr-x 2 root root 20480 Jan 19 13:25 recup_dir.54
drwxr-xr-x 2 root root 20480 Jan 19 13:25 recup_dir.55
drwxr-xr-x 2 root root 20480 Jan 19 13:26 recup_dir.56
drwxr-xr-x 2 root root 20480 Jan 19 13:26 recup_dir.57
drwxr-xr-x 2 root root 24576 Jan 19 13:26 recup_dir.58
drwxr-xr-x 2 root root 36864 Jan 19 13:26 recup_dir.59
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.6
drwxr-xr-x 2 root root 20480 Jan 19 13:26 recup_dir.60
drwxr-xr-x 2 root root 20480 Jan 19 13:26 recup_dir.61
drwxr-xr-x 2 root root 20480 Jan 19 13:27 recup_dir.62
drwxr-xr-x 2 root root 20480 Jan 19 13:27 recup_dir.63
drwxr-xr-x 2 root root 20480 Jan 19 13:27 recup_dir.64
drwxr-xr-x 2 root root 20480 Jan 19 13:28 recup_dir.65
drwxr-xr-x 2 root root 12288 Jan 19 13:28 recup_dir.66
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.7
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.8
drwxr-xr-x 2 root root 20480 Jan 19 13:21 recup_dir.9
66 folders were recoveredI calculated the folder size
sudo du -sh /recovery
13G /recoveryI immediately started downloading ALL recovered files (13GB) to my local PC with MobaXTerm
On recovery.fearby.com I searched for any trace of recovered *.sql files that I was dumping daily via cron jobs
find /recovery -name "*.sql"Many pages of recovered sql files were listed
/recovery/recup_dir.49/f36487168.sql
/recovery/recup_dir.49/f35110912.sql
/recovery/recup_dir.49/f36667392.sql
/recovery/recup_dir.49/f36995072.sql
/recovery/recup_dir.49/f35667968.sql
/recovery/recup_dir.49/f35078144.sql
/recovery/recup_dir.49/f37535744.sql
/recovery/recup_dir.49/f36913152.sql
/recovery/recup_dir.49/f33996800.sql
/recovery/recup_dir.49/f36421632.sql
/recovery/recup_dir.49/f35061760.sql
/recovery/recup_dir.49/f36143104.sql
/recovery/recup_dir.49/f36618240.sql
/recovery/recup_dir.49/f34979840.sql
/recovery/recup_dir.49/f37273600.sql
/recovery/recup_dir.49/f35995648.sql
/recovery/recup_dir.49/f36241408.sql
/recovery/recup_dir.49/f37732360.sql
/recovery/recup_dir.49/f34603008.sql
/recovery/recup_dir.49/f33980416.sql
/recovery/recup_dir.1/f0452896.sql
/recovery/recup_dir.1/f0437184.sql
/recovery/recup_dir.1/f0211232.sql
/recovery/recup_dir.17/f16547840.sql
/recovery/recup_dir.17/f16252928.sql
/recovery/recup_dir.17/f15122432.sql
/recovery/recup_dir.17/f17159720.sql
/recovery/recup_dir.17/f15089664.sql
/recovery/recup_dir.17/f15958016.sql
/recovery/recup_dir.17/f15761408.sql
/recovery/recup_dir.57/f69582848.sql
/recovery/recup_dir.57/f69533696.sql
/recovery/recup_dir.57/f69173248.sql
/recovery/recup_dir.57/f68321280.sql
/recovery/recup_dir.57/f70483968.sql
/recovery/recup_dir.57/f70746112.sql
/recovery/recup_dir.57/f68730880.sql
/recovery/recup_dir.57/f67862528.sql
/recovery/recup_dir.57/f70123520.sql
/recovery/recup_dir.57/f68337664.sql
/recovery/recup_dir.57/f70172672.sql
/recovery/recup_dir.57/f71057408.sql
/recovery/recup_dir.57/f68796416.sql
/recovery/recup_dir.57/f70533120.sql
/recovery/recup_dir.57/f69419008.sql
/recovery/recup_dir.57/f68239360.sql
/recovery/recup_dir.57/f69779456.sql
/recovery/recup_dir.57/f68255744.sql
/recovery/recup_dir.57/f67764224.sql
/recovery/recup_dir.57/f71204864.sql
/recovery/recup_dir.57/f70336512.sql
/recovery/recup_dir.57/f68501504.sql
/recovery/recup_dir.57/f67944448.sql
/recovery/recup_dir.50/f39059456.sql
/recovery/recup_dir.50/f38518784.sql
/recovery/recup_dir.50/f40206336.sql
/recovery/recup_dir.50/f40927232.sql
/recovery/recup_dir.50/f39485440.sql
/recovery/recup_dir.50/f39092224.sql
/recovery/recup_dir.50/f40861696.sql
/recovery/recup_dir.50/f39731200.sql
/recovery/recup_dir.50/f40337408.sql
/recovery/recup_dir.50/f38862848.sql
/recovery/recup_dir.50/f41664512.sql
/recovery/recup_dir.50/f41074688.sql
/recovery/recup_dir.50/f40828928.sql
/recovery/recup_dir.50/f41713664.sql
/recovery/recup_dir.50/f38092800.sql
/recovery/recup_dir.50/f39878656.sql
/recovery/recup_dir.50/f38305792.sql
/recovery/recup_dir.50/f38830080.sql
/recovery/recup_dir.50/f39534592.sql
/recovery/recup_dir.50/f39813120.sql
/recovery/recup_dir.50/f40435712.sql
/recovery/recup_dir.50/f41467904.sql
/recovery/recup_dir.50/f37901728.sql
/recovery/recup_dir.50/f38682624.sql
/recovery/recup_dir.50/f38191104.sql
/recovery/recup_dir.50/f38174720.sql
/recovery/recup_dir.50/f40878080.sql
//and many moreIt would take a few hours to download 32,000 files from the other side of the world. Next time I deploy fearby.com, I will deploy it to Sydney Australia.
It looks like MobaXTerm does not copy to the selected destination that I selected when dragging and dropping files, Being impatient I scanned my system for a file that MobaXTerm had copied. MobaXTerm saves download to “%Documents%\MobaXterm\splash\tmp\dragdrop\”.
I opened some of the recovered SQL files and it looks like all files were partial and were not the whole database backup. A compete mysql dump should be over 200 lines long. Dang.
I downloaded all files I could from the these folders
- /mnt/etc/nginx
- /mnt/var/lib/mysql
- /mnt/Scripts
Disappointed, I sat on things for a few weeks, I thought I had lost my website.
Try 2 – Reinstall a fresh db.fearby.com
After I copied files from the old db.fearby.com serve to recovery.fearby.com, I reattached the storage to the old db.fearby.com system.
In vein, I tried mending the broken MySQL service on db.fearby.com. I tried uninstalling and reinstalling MySql on db.fearby.com (each time no luck). I was having trouble with MySQL not starting.
I had too many rabbit holes (mysql errors) to list. I thought my database was corrupt.
I was kicking myself for letting my access to db.fearby.com lapse, I was kicking myself for not having more backups.
Try 3 – Reinstall MySQL
I tried deploying a new server and setting it up, maybe from the frustration I did not do it correctly. I had HTTPS issues with CloudFlare. I gave up for a few months
Try 4 – Check for Database Corruption
I downloaded DiskInternals – MySQL Recovery and scanned my database. To my amazement, it reported no database corruption.
Maybe I can recover my website?
Try 5 – Using RunCloud.io to deploy a website
Having failed with setting up a server from scratch, a friend (Hi Zach) said I should try RunCloud to deploy a server.
I tried to document everything from attaching RunCloud to UpCloud and Cloudflare’s API to deploying a server.
Long story short, RunCloud is not for me. I had too many issues with RunCloud and IPV6, CloudFlare API Integration, No SSH access to my server, no Nginx editing capabilities with RunCloud.
I deleted the RunCloud deployed server.
Try 6 – 10 minutes deploying a serer by hand
I ended up deploying a server in 10 minutes manually without taking notes.
Summary
I deployed a server (this time to Sydney).
I Installed the UFW firewall (configured and started)
sudo apt-get install ntpSet the date and time
sudo timedatectl set-timezone Australia/Sydney
Installed ntp time server
sudo apt-get install ntp
I Installed PHP 7.4 and PHP 7.4 FPM (I cheated and googled: https://www.cloudbooklet.com/install-php-7-4-on-debian-10/ )
I edited PHP Config
sudo nano /etc/php/7.4/fpm/php.ini
Installed NGINX webserver
sudo apt-get install nginxConfigured NGINX (I got an CLoudflare to NGINX SSL Certificate), I also sighned up for a Cloud flare SSL certificate
sudo nano /etc/nginx/nginx.confContents
worker_cpu_affinity auto;
worker_rlimit_nofile 100000;
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
multi_accept on;
}
http {
root /www-root;
client_max_body_size 10M;
proxy_connect_timeout 1200s;
proxy_send_timeout 1200s;
proxy_read_timeout 1200s;
fastcgi_send_timeout 1200s;
fastcgi_read_timeout 1200s;
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain application/xml;
gzip_min_length 256;
gzip_proxied no-cache no-store private expired auth;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}I edited sudo nano /etc/nginx/sites-available/default
Contents
server {
listen 80 default_server;
listen [::]:80 default_server;
# SSL configuration
#
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
ssl on;
ssl_certificate /path/to/ssl/certs/cert.pem;
ssl_certificate_key /path/to/ssl/private/key.pem;
root /path-to-www;
# Add index.php to the list if you are using PHP
index index.html index.php;
server_name fearby.com;
#Security Headers
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade";
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Frame-Options SAMEORIGIN always;
add_header Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()";
# Force HTTPS
if ($scheme != "https") {
return 301 https://$host$request_uri;
}
# DENY RULES
location ~ /\.ht {
deny all;
}
location ~ ^/\.user\.ini {
deny all;
}
location ~ (\.ini) {
return 403;
}
if ($http_referer ~* "laptrinhx.com") {
return 404;
}
if ($http_referer ~* "bdev.dev") {
return 404;
}
if ($http_referer ~* "raoxyz.com") {
return 404;
}
if ($http_referer ~* "congtyaz.com") {
return 404;
}
location / {
try_files $uri $uri/ /index.php?$args;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/run/php/php7.4-fpm.sock;
}
# DNS
resolver 1.1.1.1 1.0.0.1 valid=60s;
resolver_timeout 1m;
}I tested Nginx and PHP.
I installed MySQL (I Googled a guide)
I created a database, database user and assigned permissions for my blog.
I installed the WordPress CLI tool
I installed WordPress the using the wp-cli tool
wp core install --url=example.com --title=Example --admin_user=supervisor --admin_password=strongpassword [email protected]When I had a blank WordPress I uploaded the blog folder that I backed up before.
I ran this command to allow Nginx to read the backed up website files
sudo chown -R www-data:www-data /path-to-wwwI also uploaded the backup of my mysql database to /var/lib/mysql/oldblogdatabase
I ran this command to allow mysql to read the backed up database
sudo chown -R mysql:mysql /var/lib/mysql/oldblogdatabaseI also uploaded the following files to /var/lib/mysql
TIP: These files are very important to restore. You cannot just copy a database in a subfolder.
- ibdata1
- ib_logfile0
- ib_logfile1
- ib_logfile1
- ibtmp1
I ran this command to allow mysql ro read the ib* files
sudo chown -R mysql:mysql /var/lib/mysql/ib*I was able to load my old website
I still have some issues to solve but it is back.
Lessons Learned
- Save all passwords and have backup accounts and roll back before working backups are gone.
- Setup a Dev Test, Pre Prod Environment and do no test on production servers.
- Do not delay disaster recovery actions.
- Do not rely on automation.
- Have more than a weeks worth of backups.
Get $25 free credit on UpCloud and deploy your own server: Use this link to get $25 credit (new UpCloud users only).
Change Log
Version 1.2
