Auto-restart Raspberry Pi when its target server is not reachable

Posted by Armin on Thursday, October 29, 2015

My security camera has been running great for months. Issues have arisen only when the wireless router gets restarted or the wireless connection gets dropped for some other reason. The problem is that the Raspberry Pi doesn't automatically reconnect after a wlan connection is dropped. In that case, I usually had to unplug the RPi to restart it manually, because I couldn't reach it via ssh any longer.

When searching for a solution to my problem, I came across this post: https://www.raspberrypi.org/forums/viewtopic.php?t=16054. It provides a bash script to reset the WLAN connection after it gets dropped. I tried a version of it that I scheduled to run with cron. However, it still didn't solve the Raspberry Pi's problem with reconnecting to the "mother server" after a network issue.

The solution that has been working for a while now is to have the RPi ping its target server on a regular basis. If the ping is unsuccessful, it restarts (in the hope that this will re-establish the network connection.) The Python program I pasted below performs the pinging, logs the results to a file for later inspection, and performs a system reboot if necessary. The program runs once every 30 minutes, as defined in a crontab.

The Python code: networkchecker.py

#!/usr/bin/env python
#
# networkChecker
#
# Checks whether a server is available to the Raspberry Pi.
# If it's not available, the RasPi is told to restart
# To be run periodically (cron job)
#
# AJH 2014
 
 
import argparse
import os
from datetime import datetime
import logging
 
 
def check_for_sudo():
	"""Check whether we are dealing with the root user."""
	if os.geteuid() == 0:
		return True
	else:
		return False
 
def get_current_time():
	d = datetime.now()
	time = {}
	time["timestring_long"] = "{0}-{1:02d}-{2:02d} at {3:02d}:{4:02d}:{5:02d}".format(d.year, d.month, d.day, d.hour, d.minute, d.second)
	time["timestring_short"] = "{:02d}:{:02d}:{:02d}".format(d.hour, d.minute, d.second)
	return time
 
def setup_logging(logfilename):
	# Set up a log file
	logging.basicConfig(filename=str(logfilename), format='%(levelname)s:%(message)s', level=logging.INFO)
	ct = get_current_time()	
	logging.debug(" networkChecker -- START -- {}".format(ct['timestring_long']))
 
def ping_the_server(args):
	"""Send a ping to the server"""
	cmd = "ping -q -c {} {}".format(args.n, args.host)
	response = os.system(cmd)
	logging.debug("Sending command: {}".format(cmd))
	return response
 
def main(args):
	"""Check whether server is up"""
	# set up log file
	if not args.logfile:
		setup_logging("{}/networkchecker.log".format(os.getcwd()))
	else:
		setup_logging(args.logfile)
	is_sudo = check_for_sudo()
	if is_sudo:
	    logging.debug("Running as sudo user")
	if not is_sudo:
	    logging.debug("Not running as sudo user")
	# send a ping to the server
	response = ping_the_server(args)
	if response == 0:
		ct = get_current_time()	
		logging.info("{0} is up. ({1})".format(args.host, ct['timestring_long']))
		#everything is fine, don't do anything else
	else:
		logging.info("{0} is not reachable".format(args.host))
		# Restart Raspberry Pi when host is not reachable;
		# this should hopefully make network drive available again
		if is_sudo and args.restart:
			ct = get_current_time()
			logging.info("System shutting down now for restart at {}".format(ct['timestring_long']))
			try:
				os.system("sudo shutdown -r now") # this has to be run as sudo
			except:
				logging.info("Shutdown command unsuccessful.")
		elif args.restart and not is_sudo:
			logging.debug("Not shutting down. Sudo permissions required")
	ct = get_current_time()	
	logging.debug(" networkChecker -- STOP -- {}".format(ct['timestring_long']))
 
 
if __name__ == '__main__':
	parser = argparse.ArgumentParser(description='Network Checker')
	parser.add_argument('-s', '--server', dest='host', help="Server to ping at regular intervals")
	parser.add_argument('-n', '--number', dest='n', default=10, help="Number of pings to send")
	parser.add_argument('-r', '--restart', dest='restart', help="Restart RasPi if server is down", action="store_true")
	parser.add_argument('-l', '--log', dest='logfile', help="Log file name and path")
	args = parser.parse_args()
	main(args)

Notes: You can change the logging level in line 34 from level=logging.INFO to level=logging.DEBUG to display more detailed information during setup.

Using cron to run the command on a regular basis

After placing the script onto the Raspberry Pi, I set up a cron job to run it every 30 minutes. This has to be performed for the sudo crontab, because it involves a restart command that would otherwise fail. The command sudo crontab -e opens the sudo cron file in an editor (usually nano).

At the bottom of the file, add the line shown here if it should be run every 30 minutes:

# m h  dom mon dow   command
*/30 * * * * /home/pi/bin/networkChecker.py -s SERVER_IP_ADDRESS -n 20 -l LOG_FILE_LOCATION -r

Note that the full path to the script is given (/home/pi/bin/networkChecker.py, in my case) because path variables are set differently for the sudo user. The script might not be found if the program location is not in the sudo PATH variable. SERVER_IP_ADDRESS is the one that's going to be pinged. The "-n" parameter defines how many pings to send, and "-l" defines the log file path and name. (E.g. /var/log/networkchecker.log; make sure it's writeable for the process that executes the script.) Adding the "-r" parameter specifies that a reboot is going to be performed. Otherwise, actions will only be logged. (Useful for debugging.)

Tags:

Raspberry PI