Differences

This shows you the differences between two versions of the page.

--- public:radio:2024:yaddnet_data_backup [19/04/24 09:37 BST] – [YaDDNet UDP Mirror] john
+++ public:radio:2024:yaddnet_data_backup [06/03/25 06:49 GMT] (current) – external edit 127.0.0.1
@@ Line 2: / Line 2: @@
-====== Yaddnet Data Backup ======
+====== YaDDNet Data Backup ======
 ** Offsite Text file backups of all DSC messages **
-===== YaDDNet UDP Mirror =====
+  * Alan Spindel initially suggested an offsite backup of the DSC messages in 2019
+  * YaDDNet has had the facility to send copies of each new message out to specified IP addresses via UDP since that time
+  * The DSC messages are sent by UDP as simple plain text, for easy archiving
+  * The archived text file can be used if necessary to rebuild the SQL database table containing the DSC messages in case of disaster
+  * Since I'm now hosting YaDDNet again I decided I should run a UDP log receiver at home to improve the backup provision
+  * The daily archive files will be ''gzipped'' and stored on Dropbox
+    * I could share the link to anyone who is interested in using the saved data
+===== YaDDNet UDP Send to Mirror =====
+  * In the ''PyYadd.py'' function a DSC message is created from the raw incoming data from YaDD (and DSC Decoder) in a standard format.
+  * This involves converting MMSI -> Coast or Ship name, finding the ''[MID]'' and corresponding country etc.
+  * Prior to using the newly created message to inject new data into the SQL server ''PyYadd.py'' calls the function ''send_to_mirror()'' to send out the UDP packet to any configured IP addresses
+  * ''PyYadd.py'' then inserts the new data in the SQL server
+  * Any addressed & listening UDP clients receive the UDP packets and append them to a log file
 <code python>
@@ Line 23: / Line 36: @@
 </code>
 ===== Backup UDP receive and logrotate =====
+==== UDP Listener ====
+<code python>
+#!/usr/bin/env python
+#
+# udp_logger.py
+#
+import SocketServer
+import socket
+import threading
+class ThreadedUDPRequestHandler(SocketServer.BaseRequestHandler):
+    def handle(self):
+        data = self.request[0].strip()
+        socket = self.request[1]
+        self.write_file(data)
+    def write_file(self,text):
+        filename = '/home/gm4slv/yaddlogs/yaddnet_udp_log.txt'
+        f = open(filename, 'a+')
+        log = text
+        f.write(log+"\n")
+        f.close()
+        return
+class ThreadedUDPServer(SocketServer.ThreadingMixIn, SocketServer.UDPServer):
+    pass
+if __name__ == "__main__":
+    HOST, PORT = "", 2505
+    server = ThreadedUDPServer((HOST, PORT), ThreadedUDPRequestHandler)
+    ip, port = server.server_address
+    server.serve_forever()
+    # Start a thread with the server --
+    # that thread will then start one
+    # more thread for each request
+    server_thread = threading.Thread(target=server.serve_forever)
+    # Exit the server thread when the main thread terminates
+    server_thread.daemon = True
+    server_thread.start()
+    server.shutdown()
+</code>
+  * The UDP Log Receiver listens to UDP/2505 for incoming packets from YaDDNet. These are of the form:
+<code>
+[log];2024-04-19 08:43:06;NLD_MB_HF1;2187.5;SEL;002050480;COAST;205;Belgium;Ostende Radio;SAF;636092799;SHIP;636;Liberia;CISNES, D5HF4, Cargo;TEST;NOINF;--;--;REQ;OK
+[log];2024-04-19 08:43:28;BDXC-Han-Remote;12577.0;SEL;002241022;COAST;224;Spain;Coruna Radio;SAF;538005991;SHIP;538;Marshall Islands;NAVIG8 VIOLETTE, V7IH3, Tanker;TEST;NOINF;--;--;REQ;OK
+</code>
+  * each new packet is appended to the running log file ''/home/gm4slv/yaddlogs/yaddnet_udp_log.txt''
+  * the file will grow indefinitely if left to itself!
+==== Logrotate ===
+  * run by Crontab every day at midnight
+  * copies the current running log to a timestamped file
+  * erases the contents of the running log file
+  * ''gzips'' the timestamped copy and moves it to //Dropbox//
+<code bash>
+## CRONTAB ENTRY
+00 * * * /home/gm4slv/bin/yaddlog_rotate.sh /dev/null 2>&1
+</code>
+<code bash>
+#!/bin/bash
+#
+# yaddlog_rotate.sh
+DROPBOX=/home/gm4slv/Dropbox/Yaddlogs/
+DIR=/home/gm4slv/yaddlogs
+INFILE=yaddnet_udp_log.txt
+TIMESTAMP=`date +%Y%m%d_%H%M`
+OUTFILE=$TIMESTAMP\_$INFILE
+cp $DIR/$INFILE $DIR/$OUTFILE
+cat /dev/null > $DIR/$INFILE
+gzip $DIR/$OUTFILE
+mv $DIR/$OUTFILE.gz $DROPBOX
+</code>
+   * This creates a ''gzipped'' log file in my Dropbox of each day's new DSC messages
+=== Starting the UDP Listener at boot-up ===
+  * crontab entry to start it as a background job at ''@reboot''
+<code bash>
+#
+# CRONTAB ENTRY
+#
+@reboot python /home/gm4slv/bin/udp_logger.py &
+</code>
 --- //John Pumford-Green 19/04/24 09:31 BST//
+Last updated : ~~LASTMOD~~
 ===== Further Information =====
+[[https://www.yaddnet.org/index.php | YaDDNet]]
-{{tag>}}
+{{tag>radio yaddnet}}