YaDDNet Data Backup

Offsite Text file backups of all DSC messages

• Alan Spindel initially suggested an offsite backup of the DSC messages in 2019
• YaDDNet has had the facility to send copies of each new message out to specified IP addresses via UDP since that time
• The DSC messages are sent by UDP as simple plain text, for easy archiving
• The archived text file can be used if necessary to rebuild the SQL database table containing the DSC messages in case of disaster
• Since I'm now hosting YaDDNet again I decided I should run a UDP log receiver at home to improve the backup provision
• The daily archive files will be gzipped and stored on Dropbox
  • I could share the link to anyone who is interested in using the saved data

• In the PyYadd.py function a DSC message is created from the raw incoming data from YaDD (and DSC Decoder) in a standard format.
• This involves converting MMSI → Coast or Ship name, finding the [MID] and corresponding country etc.
• Prior to using the newly created message to inject new data into the SQL server PyYadd.py calls the function send_to_mirror() to send out the UDP packet to any configured IP addresses
• PyYadd.py then inserts the new data in the SQL server
• Any addressed & listening UDP clients receive the UDP packets and append them to a log file

def send_to_mirror(data):
    log_data = "[log];"+data
    port = 2505
    host = "aaa.bbb.ccc.ddd.eee"
    host2 = "fff.ggg.hhh.iii"
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.sendto(log_data, (host, port))
    sock.sendto(log_data, (host2, port))
 
    return

#!/usr/bin/env python
#
# udp_logger.py
#
 
import SocketServer
import socket
import threading
 
class ThreadedUDPRequestHandler(SocketServer.BaseRequestHandler):
 
    def handle(self):
        data = self.request[0].strip()
        socket = self.request[1]
        self.write_file(data)
 
    def write_file(self,text):
        filename = '/home/gm4slv/yaddlogs/yaddnet_udp_log.txt'
        f = open(filename, 'a+')
        log = text
        f.write(log+"\n")
        f.close()
        return
 
class ThreadedUDPServer(SocketServer.ThreadingMixIn, SocketServer.UDPServer):
    pass
 
if __name__ == "__main__":
    HOST, PORT = "", 2505
    server = ThreadedUDPServer((HOST, PORT), ThreadedUDPRequestHandler)
    ip, port = server.server_address
    server.serve_forever()
    # Start a thread with the server -- 
    # that thread will then start one
    # more thread for each request
    server_thread = threading.Thread(target=server.serve_forever)
    # Exit the server thread when the main thread terminates
    server_thread.daemon = True
    server_thread.start()
    server.shutdown()

• The UDP Log Receiver listens to UDP/2505 for incoming packets from YaDDNet. These are of the form:

[log];2024-04-19 08:43:06;NLD_MB_HF1;2187.5;SEL;002050480;COAST;205;Belgium;Ostende Radio;SAF;636092799;SHIP;636;Liberia;CISNES, D5HF4, Cargo;TEST;NOINF;--;--;REQ;OK
[log];2024-04-19 08:43:28;BDXC-Han-Remote;12577.0;SEL;002241022;COAST;224;Spain;Coruna Radio;SAF;538005991;SHIP;538;Marshall Islands;NAVIG8 VIOLETTE, V7IH3, Tanker;TEST;NOINF;--;--;REQ;OK

• each new packet is appended to the running log file /home/gm4slv/yaddlogs/yaddnet_udp_log.txt
• the file will grow indefinitely if left to itself!

• run by Crontab every day at midnight
• copies the current running log to a timestamped file
• erases the contents of the running log file
• gzips the timestamped copy and moves it to Dropbox

## CRONTAB ENTRY
 
00 00 * * * /home/gm4slv/bin/yaddlog_rotate.sh /dev/null 2>&1

#!/bin/bash
#
# yaddlog_rotate.sh
 
DROPBOX=/home/gm4slv/Dropbox/Yaddlogs/
DIR=/home/gm4slv/yaddlogs
 
INFILE=yaddnet_udp_log.txt
 
TIMESTAMP=`date +%Y%m%d_%H%M`
 
OUTFILE=$TIMESTAMP\_$INFILE
 
cp $DIR/$INFILE $DIR/$OUTFILE
cat /dev/null > $DIR/$INFILE
 
 
gzip $DIR/$OUTFILE
 
mv $DIR/$OUTFILE.gz $DROPBOX

• This creates a gzipped log file in my Dropbox of each day's new DSC messages

• crontab entry to start it as a background job at @reboot

#
# CRONTAB ENTRY
#
@reboot python /home/gm4slv/bin/udp_logger.py &

— John Pumford-Green 19/04/24 09:31 BST

Last updated : 06/03/25 06:49 GMT

YaDDNet