Add presentation. Fix docs, docker-compose and bandswitcher

2 years ago · 75181eebf0
5 changed files with 111 additions and 32 deletions
--- a/Readme.md
+++ b/Readme.md
@ -1,20 +1,18 @@
 # what is it?

-![alt text](screenshot.png "Logo Title Text 1")
-
 * a set of tools to export your personal WSJT-X
  * live reception data into prometheus or mysql
  * historical reception data (since 2019) into mysql
 * a tool to poll pskreporter and export your personal 
-  * live transmission data to prometheus or mysql 
+  * live transmission data into prometheus or mysql 

 every time your wsjtx is/was running you are/were generating nice data points, which can be queried using grafana.
 this can be very interesting to evaluate your **personal** hf conditions from the current and the past.

 some use cases: 
 * choose dxcc, band, mode and timeframe in all combinations so see when your were successfully able to receive and transmit, to plan future activities
-* compare input of different station like for antenna comparions
-* have nice wallboard for contesting or dxpedition
+* compare data of different station for antenna comparions etc.
+* have a nice and useful wallboard for contesting or dxpedition
 * display rx/tx data on a map with custom timespan and filters
 * compare hf propagration forecasts to your personal situation
 * use grafana alerts or prometheus alertmanager to send yourself alerts when your stations receives something interesting
@ -28,7 +26,7 @@ have fun!

 ## prerequisites

-* ALL.TXT with interesting data and/or running hamradio station to provide live data
+* one or more ALL.TXT with interesting data and/or running hamradio station to provide live data
 * grafana
 * prometheus or mysql

@ -40,8 +38,6 @@ have fun!
 * **wsjtx-exporter**
  * follows live traffice in ALL.txt
  * supports prometheus and mysql
-* **alltxt2csv**
-  * reads whole ALL.txt since 2019 and creates csv-file which can be imported into mysql
 * **misc**
  * sample dashboards
  * sql table definitions
@ -58,6 +54,7 @@ pro
 * you get nicer graphs with counters and bars if you ask me
 * you can use alertmanager if you want advanced alerting
 * optimized for timeseries data
+
 con
 * not designed for keeping long timeseries (weeks or some months are fine)
 * not possible to import historical data
@ -69,10 +66,11 @@ pro:
 * you can import your ALL.txt since 2019
 * you can store a lot of data over a long timeframe
 * you can propably build more complex queries in sql
+
 con:
 * in comparison to prometheus not optimized for timeseries, but still good enough
 * gauge based graphs are not as nice as counter/bars if you ask me
-* you can use *only* graphanas internal alerting
+* you can use *only* grafanas internal alerting (or do something else)

 both allow distributed setups with multiple wsjtx instances submitting their data to a central prometheus or mysql service.
 you can as well run both in parallel and use prometheus for a live overview and mysql for historical evaluations.
@ -95,7 +93,7 @@ thats quite normal as there are propably a lot of strange/broken messages in you

 ### does it work with wsjt-x 2.3.0.RC2?

-no, there is a bug which prevents the ALL.TXT to contain them timestamp, but it should be fixed in the next release.
+no, there is a bug which prevents the ALL.TXT to contain the timestamp, but it should be fixed in the next release.

 ### what is the state of this?

@ -106,7 +104,7 @@ feedback and pull requests are highly appreciated!
 here be dragons:
 * callsign lookup performance done with regex in one goroutine is unneccessary slow for alltxt2csv
 * grids for ALL.TXT entries are always looked up by dxcc instead if being tracked in the file
-* there is a lot of space for improvements for the dashboards
+* there is a lot of room for improvements in the dashboards

 ### grafana says too many rows for mysql datasource

@ -159,5 +157,7 @@ alerting

 * WSJT-X, Grafana, Prometheus, MySQL Teams
 * Philip Gladstone for pskreporter.info
-* Todd Neal     for his nice work on https://github.com/tzneal/ham-go
-* Stefan Prodan for his nice work on https://github.com/stefanprodan/dockprom
+* Todd Neal for https://github.com/tzneal/ham-go
+* Stefan Prodan for https://github.com/stefanprodan/dockprom
+* Markus Lindenberg DO8ML for helping with a lot of stuff
+* Sebastian Schaumann DL7LE for testing and a lot of good ideas and questions
--- a/devnotes.md
+++ b/devnotes.md
@ -1,33 +1,37 @@
 # assumptions/ideas

+* parsing ALL.txt is better than udp - udp cant be rewind
 * keep client as stupid/uncomplicated as possible
+  * no decentral maintenance should be needed if something changed
  * noch parsing/resolving, just postition finding

 # before release

-* multiple repos?
-  * alltxt2http & http-exporter (inkl. shared/httpstuff und shared/wsjtx)
-  * pskreporter-exporter
-  * dashboards n docs
-
 * alltxt2csv kann weg oder?
  * resolving lokal nur bedingt sinnvoll
    * profitiert nicht von ggf. vorhandenen grids im cache
  * wenn station und identifier bekannt sind, kann das auch der admin tun
+    * und das kann dann auch der wsjtx-exporter inkl. cache lookups etc. uebernehmen 

 * think about:
  * logische trennung in zeilenverarbeitung und 'spotting' (spot in mysql&prometheus exportieren)
    * take line -> parse line -> resolve grid and lotw for line ?

+* stationpi
+  * wsjtx works
+  * alltxt2http works
+  * wireguard works
+  * audio via pulse-tcp seems to work..
+  * write doc!
+
 * http-exporter
+  * callsignlabel aus prometheus entfernen?????
  * errorhandling
    * db down
      * who keeps track of the missing lines???
    * lookup error (not an error)
-  * add instance field to mysql
-    * what happes if no instance is provided?
  * improve logging
-  * counter vs histogram in prometheus?
+  * counter vs histogram in prometheus? (fuer signal)
  * http 200 OK synchron warten lassen bis alles fertig ist?
    * scheint tendenziell eine gute idee zu sein.. wenn db problem hat, kann client aufhoeren..

@ -39,10 +43,11 @@

 * alltxt2http
  * what about readall/batchmode?
+  * errorhandling/logging
+    * see journal for obviously ending without any logs?!

 * systemd restart problematik (failed too fast oder soo ;)

-
 * Username=Station (=~ Callsign) right?!?
  * call und password in htacces und fertig!
  * auth. user kann seine instance selber waehlen
@ -62,13 +67,13 @@
 * push images to dockerhub

 * doc
-  * german docs..
  * fix single tool readmes
  * hints for using grafana
    * intervals
    * fold row which are not needed
    * use filters!
  * server und/oder skript/readme zum aufsetzen
+  * german docs..

 # later

--- a/docker-compose/docker-compose.yml
+++ b/docker-compose/docker-compose.yml
@ -48,6 +48,17 @@ services:
 #    labels:
 #      org.label-schema.group: "monitoring"

+  redis:
+    image: redis
+    container_name: cache
+    expose:
+      - 6379
+    restart: unless-stopped
+    networks:
+      - monitor-net
+    labels:
+      org.label-schema.group: "monitoring"
+
  grafana:
    image: grafana/grafana:7.3.4
    container_name: grafana
@ -145,7 +156,7 @@ services:
    image: 10.0.73.1:5000/wsjtx-exporter:latest
    container_name: wsjtx-exporter
    ports: 
-      - "4000:4000"
+      - "4001:4000"
    environment:
      - PROMETHEUS=true
      - MYSQL=true
--- a/misc/bandswitcher.sh
+++ b/misc/bandswitcher.sh
@ -1,20 +1,36 @@
 #!/bin/bash

-RIGCTL=$(which rigctl)
-PARAMETERS="-m 2"
+# stay DELAY seconds on every band
 DELAY="300"

-BANDS="14074000 21074000 28074000"
+# local rigctld, can be running dummy rig!
+PARAMETERS="-m 2"
+BANDS=(14074000 21074000 28074000)
+
+# enable gqrx intergration
+GQRX="1"
+GQRX_PARAMETERS="-m 2 -r localhost:7356"
+GQRX_BANDS=(139074000 146000740 153000740)
+
+# find rigctl binary
+RIGCTL=$(which rigctl)

 # wait for full minute
+echo "wait for full minute.."
 sleep $(((60 - $(date +%s) % 60)-1))

 # start switching
-while true
-do
-  for band in $BANDS
-  do
-    $RIGCTL $PARAMETERS F $band
+let len=${#BANDS[*]}-1
+while true ; do
+  for i in $(seq 0 $len); do
+    date
+    echo "switching local rigctl to ${BANDS[$i]}"
+    $RIGCTL $PARAMETERS F ${BANDS[$i]}
+
+    if [ "$GQRX" = "1" ]; then
+      echo "switching gqrx to ${GQRX_BANDS[$i]}"
+      $RIGCTL $GQRX_PARAMETERS F ${GQRX_BANDS[$i]}
+    fi
    sleep $DELAY
  done
 done
--- a/presentation.md
+++ b/presentation.md
@ -0,0 +1,47 @@
+# O49 Vorstellung
+
+* Funkwetterprognosen schwer verstaendlich..
+  * muss anschaulicher gehen - siehe Gridtracker
+  * WSJT-X generiert viele spannenden Daten (ab 2019)
+    * NICHT die Logdaten... sondern ALL.TXT :)
+
+* Idee - Daten in Grafanakompatible Datenquellen packen
+  * was ist Grafana?
+  * Datenmenge
+
+* Nur Daten von meinem WSJT-X sind nicht soooo interessant
+
+* PSKReporter
+  * aktuell alle 5 Minuten abolen und speichern
+    * bald mit Livedaten :)
+
+* Benefits/Features!
+  * Longtime Auswertung
+  * Wallboard
+  * DX-Jagd Vorbereitung
+  * Callsign Tracking
+  * Vergleich von Stationen
+
+* Know Issues
+  * know your data/query!
+    * live counter vs tx
+    * dl3sd 24/7 vs dl7le 6/7
+    * continental distribution
+    * map zeigt nur eine dimension (dxcc ODER band ODER receiver) -> keine ueberlagerung moeglich
+  * alles wip!
+    * viele features nicht drin
+    * Dashboards
+    * db
+    * komponenten
+  * Achtung Last!
+    * Client
+    * Server
+
+* Ausblick
+  * Pi mit VNC (+VPN ;)
+    * Red Pitaya ;)
+  * Alerting
+  * QRZ/LoTW
+
+* Bei Interesse Mail an mich
+  * Useraccount und Link zum Howto kommt