John's Vademecum

Try to learn something about everything, and everything about something -Thomas Huxley “Darwin's bulldog” (1824-1895)

User Tools

Site Tools

You are here: My digital Commonplace Book » Public Area » 03 : Radio » 2024 Radio Topics » YaDDNet Data Backup
Trace:
public:radio:2024:yaddnet_data_backup

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
public:radio:2024:yaddnet_data_backup [19/04/24 09:42 BST] – [Backup UDP receive and logrotate] johnpublic:radio:2024:yaddnet_data_backup [06/03/25 06:49 GMT] (current) – external edit 127.0.0.1
Line 2: Line 2:
  
  
-====== Yaddnet Data Backup ======+====== YaDDNet Data Backup ======
  
 ** Offsite Text file backups of all DSC messages ** ** Offsite Text file backups of all DSC messages **
  
-===== YaDDNet UDP Mirror =====+  * Alan Spindel initially suggested an offsite backup of the DSC messages in 2019 
 +  * YaDDNet has had the facility to send copies of each new message out to specified IP addresses via UDP since that time 
 +  * The DSC messages are sent by UDP as simple plain text, for easy archiving 
 +  * The archived text file can be used if necessary to rebuild the SQL database table containing the DSC messages in case of disaster 
 +  * Since I'm now hosting YaDDNet again I decided I should run a UDP log receiver at home to improve the backup provision 
 +  * The daily archive files will be ''gzipped'' and stored on Dropbox 
 +    * I could share the link to anyone who is interested in using the saved data
  
 +===== YaDDNet UDP Send to Mirror =====
 +
 +  * In the ''PyYadd.py'' function a DSC message is created from the raw incoming data from YaDD (and DSC Decoder) in a standard format.
 +  * This involves converting MMSI -> Coast or Ship name, finding the ''[MID]'' and corresponding country etc.
 +  * Prior to using the newly created message to inject new data into the SQL server ''PyYadd.py'' calls the function ''send_to_mirror()'' to send out the UDP packet to any configured IP addresses
 +  * ''PyYadd.py'' then inserts the new data in the SQL server 
 +  * Any addressed & listening UDP clients receive the UDP packets and append them to a log file
  
 <code python> <code python>
Line 23: Line 36:
 </code> </code>
 ===== Backup UDP receive and logrotate ===== ===== Backup UDP receive and logrotate =====
 +
 +
 +==== UDP Listener ====
  
 <code python> <code python>
  
 #!/usr/bin/env python #!/usr/bin/env python
 +#
 +# udp_logger.py
 +#
  
 import SocketServer import SocketServer
 import socket import socket
 import threading import threading
-import re 
  
-import time 
- 
- 
-   
-     
 class ThreadedUDPRequestHandler(SocketServer.BaseRequestHandler): class ThreadedUDPRequestHandler(SocketServer.BaseRequestHandler):
        
- 
     def handle(self):     def handle(self):
         data = self.request[0].strip()         data = self.request[0].strip()
Line 46: Line 58:
         self.write_file(data)         self.write_file(data)
                
- 
     def write_file(self,text):     def write_file(self,text):
         filename = '/home/gm4slv/yaddlogs/yaddnet_udp_log.txt'         filename = '/home/gm4slv/yaddlogs/yaddnet_udp_log.txt'
Line 73: Line 84:
  
 </code> </code>
 +
 +
 +  * The UDP Log Receiver listens to UDP/2505 for incoming packets from YaDDNet. These are of the form:
 +
 +<code>
 +
 +[log];2024-04-19 08:43:06;NLD_MB_HF1;2187.5;SEL;002050480;COAST;205;Belgium;Ostende Radio;SAF;636092799;SHIP;636;Liberia;CISNES, D5HF4, Cargo;TEST;NOINF;--;--;REQ;OK
 +[log];2024-04-19 08:43:28;BDXC-Han-Remote;12577.0;SEL;002241022;COAST;224;Spain;Coruna Radio;SAF;538005991;SHIP;538;Marshall Islands;NAVIG8 VIOLETTE, V7IH3, Tanker;TEST;NOINF;--;--;REQ;OK
 +</code>
 +
 +
 +  * each new packet is appended to the running log file ''/home/gm4slv/yaddlogs/yaddnet_udp_log.txt''
 +  * the file will grow indefinitely if left to itself!
 +
 +==== Logrotate ===
 +
 +  * run by Crontab every day at midnight
 +  * copies the current running log to a timestamped file
 +  * erases the contents of the running log file 
 +  * ''gzips'' the timestamped copy and moves it to //Dropbox//
 +
 +<code bash>
 +
 +## CRONTAB ENTRY
 +
 +00 00 * * * /home/gm4slv/bin/yaddlog_rotate.sh /dev/null 2>&1
 +</code>
 +
 +<code bash>
 +
 +#!/bin/bash
 +#
 +# yaddlog_rotate.sh
 +
 +DROPBOX=/home/gm4slv/Dropbox/Yaddlogs/
 +DIR=/home/gm4slv/yaddlogs
 +
 +INFILE=yaddnet_udp_log.txt
 +
 +TIMESTAMP=`date +%Y%m%d_%H%M`
 +
 +OUTFILE=$TIMESTAMP\_$INFILE
 +
 +cp $DIR/$INFILE $DIR/$OUTFILE
 +cat /dev/null > $DIR/$INFILE
 +
 +
 +gzip $DIR/$OUTFILE
 +
 +mv $DIR/$OUTFILE.gz $DROPBOX
 +
 +</code>
 +
 +   * This creates a ''gzipped'' log file in my Dropbox of each day's new DSC messages
 +
 +=== Starting the UDP Listener at boot-up ===
 +
 +  * crontab entry to start it as a background job at ''@reboot''
 +
 +<code bash>
 +#
 +# CRONTAB ENTRY
 +#
 +@reboot python /home/gm4slv/bin/udp_logger.py &
 +
 +</code>
 +
  
 --- //John Pumford-Green 19/04/24 09:31 BST// --- //John Pumford-Green 19/04/24 09:31 BST//
 +
 +Last updated : ~~LASTMOD~~
  
  
 ===== Further Information ===== ===== Further Information =====
  
 +[[https://www.yaddnet.org/index.php | YaDDNet]]
  
-{{tag>}}+{{tag>radio yaddnet}}
  
  
public/radio/2024/yaddnet_data_backup.1713516139.txt.gz · Last modified: 06/03/25 06:49 GMT (external edit)