Having a self-managed Ubuntu server from Vultr for as low as $2.5 a month (or Digital Ocean for $5 a month (setup guides here or here respectively)) can certainly be cheap but you take on all the support and risk in keeping the services up.
Advertisement:
Vultr has a good monitoring information but this does not show you services performance.
Node JS monitoring service like http://docs.keymetrics.io/ is great at monitoring Node applications where webmin and PHPServerMonitor are good options for monitoring basic servers and services.
The more software you install the more complex the setup and find potential errors.
Service Interruption Notifications
How will you be notified when things are down? Having automated monitoring scripts (or Self Service Status Pages) or external monitoring services is a good idea, the last thing you wants is to see your server or service is down based on a twitter reply.
Much thanks goes to Michael Boelen on Twitter for reporting that my server was down: Follow: @mboelen
Founder of @cisofy_is, author of rkhunter and Lynis, blogger at linux-audit.com, public speaker. #linux #security
Read more Linux troubleshooting tips at https://linux-audit.com/
You can have many different types of errors and it is going to be hard to suggest where your problem is going to be.
Provider/Networking Errors
Always assume provider errors are an issue, more often than not my servers have gone offline due to provider problems outside of my control.
I have had networking errors on Vultr with the default Dynamic DHCP IP’s (prompting changes to Static IP’s). I have also had issues with Static IP’s on Vultr and had to log a Support Tickets again when my server went offline, In this case, Vultr were able to restore my server but I am unsure of what happened?
Example Problem (DHCP IP): Dynamic IP, VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).
I logged a ticket with Vultr after I rebooted the server.
Vultr support quickly identified the problem was my server was not picking up an IP address and suggested I set static IP and reboot.
Example Problem (Static IP): VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).
Again I logged a support ticket and Vultr indicated the issues was network related.
In both cases, there was nothing I could do (a reboot did not fix each error) and a support ticket had to be logged.
Broken WordPress
Example Problem (Website Down): Error 502 bad gateway
Today I had a server stay-up buy NGINX reported error 502 bad gateway. IN this case I rebooted teh server
Know your NGINX log file location
1 | cat /etc/nginx/nginx.conf |
Show the last 22 lines of the identifies NGINX log file
1 | tail -n 200 error_log /var/log/nginx/error.log |
Error Hints
PHP-FPM is reporting errors.
1 | 2017/10/22 19:45:02 [error] 12045#0: *100883 connect() to unix:/var/run/php/php7.0-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 62.210.215.115, server: www.fearby.com, request: "GET /article/how-to-setup-a-twitter-feed-api-endpoint-in-nodejs-with-nginx-ruby-t-etc/feed/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "fearby.com" |
Wordpress too is reporting errors, I could have restarted NGINX and PHP but it was just as easy to reboot the server as I forgot the right commands to restart PHP/NGINX.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | 2017/10/22 10:28:40 [error] 12045#0: *98242 FastCGI sent in stderr: "PHP message: PHP Fatal error: Maximum execution time of 30 seconds exceeded in /www/wp-content/plugins/bj-lazy-load/inc/class-bjll.php on line 329 PHP message: PHP Stack trace: PHP message: PHP 1. {main}() /www/index.php:0 PHP message: PHP 2. require() /www/index.php:17 PHP message: PHP 3. require_once() /www/wp-blog-header.php:19 PHP message: PHP 4. include() /www/wp-includes/template-loader.php:74 PHP message: PHP 5. genesis() /www/wp-content/themes/genesis/index.php:15 PHP message: PHP 6. do_action() /www/wp-content/themes/genesis/lib/framework.php:39 PHP message: PHP 7. WP_Hook->do_action() /www/wp-includes/plugin.php:453 PHP message: PHP 8. WP_Hook->apply_filters() /www/wp-includes/class-wp-hook.php:323 PHP message: PHP 9. genesis_do_loop() /www/wp-includes/class-wp-hook.php:298 PHP message: PHP 10. genesis_standard_loop() /www/wp-content/themes/genesis/lib/structure/loops.php:41 PHP message: PHP 11. do_action() /www/wp-content/themes/genesis/lib/structure/loops.php:92 PHP message: PHP 12. WP_Hook->do_action() /www/wp-includes/plugin.php:453 PHP message: PHP 13. WP_Hook->apply_filters() /www/wp-includes/class-wp-hook.php:323 PHP message: PHP 14. genesis_do_post_content() /www/wp-includes/class-wp-hook.php:298 PHP message: PHP 15. the_content() /www/wp-content/themes/genesis/lib/structure/post.php:367 PHP message: PHP 16. apply_filters() /www/wp-includes/post-template.php:240 PHP message: PHP 17. WP_Hook->apply_filters() /www/wp-includes/plugin.php:203 PHP message: PHP 18. BJLL::filter() /www/wp-includes/class-wp-hook.php:298 PHP message: PHP 19. apply_filters() /www/wp-content/plugins/bj-lazy-load/inc/class-bjll.php:169 PHP message: PHP 20. WP_Hook->apply_filters() /www/wp-includes/plugin.php:203 PHP message: PHP 21. BJLL::filter_iframes() /www/wp-includes/class-wp-hook.php:298 PHP message: PHP 22. BJLL::_get_content_haystack() /www/wp-content/plugins/bj-lazy-load/inc/class-bjll.php:255 |
I rebooted the server in 30 seconds.
How I Rebooted in Ubuntu
1 | sudo shutdown -r now |
How to prevent this error in future?
- Extend PHP file execution time (No)
- Daily reboots of the server (No)?
- Daily restarts of NGINX/PHP (Yes, Short Term)
- Check the server for errors and automatically reboot (Yes, Long Term).
How to check system uptime (verifying the server rebooted)
1 2 | uptime 20:04:02 up 14 min, 1 user, load average: 0.13, 0.09, 0.08 |
I will set a cronjob entry to perform diagnostics (via a Bash Script) and restart NGINX/PHP or the server. I might set up remote monitoring of the web server content too.
I should have checked memory/CPU usage too but forgot (sometimes it is best to do a deep investigation and get more information).
Troubleshooting Network Errors
As mentioned above the providers can have network issues (with a static or Dynamic IP) so don’t feel bad if you cannot fix every networking error.
Perform External Ping
Can you ping a remote server from your server?
1 2 3 4 5 6 | ping www.google.com PING www.google.com (172.217.25.132) 56(84) bytes of data. 64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=1 ttl=57 time=0.853 ms 64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=2 ttl=57 time=0.820 ms 64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=3 ttl=57 time=0.776 ms 64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=4 ttl=57 time=0.835 ms |
Has your domain expired?
Check the expiry of your domain here
1 2 | whois google.com | egrep -i 'Expiry Date' Registry Expiry Date: 2020-09-14T04:00:00Z |
Here is a nice post on setting up a bash script to check multiple domains expiry.
Common (Digital Ocean) Debugging commands
1 2 3 4 5 6 7 8 | cat /etc/network/interfaces.d/50-cloud-init.cfg cat /etc/network/interfaces ip addr ip route uname -a iptables -nvL --line-numbers ls -l /lib/modules cat /etc/udev/rules.d/70-persistent-net.rules |
SSL Certificate
You can check the expiry of your SSL certificate by scanning our site with https://www.ssllabs.com/ssltest/
If you use a Let’s Encrypt SSL certificate you can reissue a certificate (see my guide here)
Web Server Config
It is a good idea to make a backup of your web server (NGINX) configuration and know what each configuration value does.
View NGINX Config
1 | cat /etc/nginx/nginx.conf |
Web Console (Vultr)
Server Setup Guide
Read more here on setting up Ubuntu on Vultr
Local Network Info
1 | ifconfig |
Read more here on setting up Ubuntu Networking on Vultr
Disable Firewall
1 | sudo ufw disable |
Read more on setting up firewalls on Ubuntu here.
Logs
Common Ubuntu Log paths: https://help.ubuntu.com/community/LinuxLogFiles
If in Doubt, Reboot
1 | sudo shutdown -r now |
Log a ticket
Remember you can log a ticket with Vultr when things go pear shape.
Monitoring
Don’t forget to have a self-serve status page to alert you when things go wrong.
Vultr Status Page
View the Vultr status page here.
Your Status Page
Read my service status page guide here.
Past Data
Do document past errors and try and prevent those errors from happening again and act upon reoccurring errors (using past documentation).
Donate and make this blog better
Ask a question or recommend an article
Revision History
v1.2 added common digital ocean commands
etc