From 75181eebf0aae5fbf182d6a6d9b2882834f55d9f Mon Sep 17 00:00:00 2001 From: Sebastian Denz Date: Tue, 19 Jul 2022 00:40:40 +0200 Subject: [PATCH] Add presentation. Fix docs, docker-compose and bandswitcher --- Readme.md | 26 ++++++++--------- devnotes.md | 25 +++++++++------- docker-compose/docker-compose.yml | 13 ++++++++- misc/bandswitcher.sh | 32 +++++++++++++++------ presentation.md | 47 +++++++++++++++++++++++++++++++ 5 files changed, 111 insertions(+), 32 deletions(-) create mode 100644 presentation.md diff --git a/Readme.md b/Readme.md index 7e860bd..0228921 100644 --- a/Readme.md +++ b/Readme.md @@ -1,20 +1,18 @@ # what is it? -![alt text](screenshot.png "Logo Title Text 1") - * a set of tools to export your personal WSJT-X * live reception data into prometheus or mysql * historical reception data (since 2019) into mysql * a tool to poll pskreporter and export your personal - * live transmission data to prometheus or mysql + * live transmission data into prometheus or mysql every time your wsjtx is/was running you are/were generating nice data points, which can be queried using grafana. this can be very interesting to evaluate your **personal** hf conditions from the current and the past. some use cases: * choose dxcc, band, mode and timeframe in all combinations so see when your were successfully able to receive and transmit, to plan future activities -* compare input of different station like for antenna comparions -* have nice wallboard for contesting or dxpedition +* compare data of different station for antenna comparions etc. +* have a nice and useful wallboard for contesting or dxpedition * display rx/tx data on a map with custom timespan and filters * compare hf propagration forecasts to your personal situation * use grafana alerts or prometheus alertmanager to send yourself alerts when your stations receives something interesting @@ -28,7 +26,7 @@ have fun! ## prerequisites -* ALL.TXT with interesting data and/or running hamradio station to provide live data +* one or more ALL.TXT with interesting data and/or running hamradio station to provide live data * grafana * prometheus or mysql @@ -40,8 +38,6 @@ have fun! * **wsjtx-exporter** * follows live traffice in ALL.txt * supports prometheus and mysql -* **alltxt2csv** - * reads whole ALL.txt since 2019 and creates csv-file which can be imported into mysql * **misc** * sample dashboards * sql table definitions @@ -58,6 +54,7 @@ pro * you get nicer graphs with counters and bars if you ask me * you can use alertmanager if you want advanced alerting * optimized for timeseries data + con * not designed for keeping long timeseries (weeks or some months are fine) * not possible to import historical data @@ -69,10 +66,11 @@ pro: * you can import your ALL.txt since 2019 * you can store a lot of data over a long timeframe * you can propably build more complex queries in sql + con: * in comparison to prometheus not optimized for timeseries, but still good enough * gauge based graphs are not as nice as counter/bars if you ask me -* you can use *only* graphanas internal alerting +* you can use *only* grafanas internal alerting (or do something else) both allow distributed setups with multiple wsjtx instances submitting their data to a central prometheus or mysql service. you can as well run both in parallel and use prometheus for a live overview and mysql for historical evaluations. @@ -95,7 +93,7 @@ thats quite normal as there are propably a lot of strange/broken messages in you ### does it work with wsjt-x 2.3.0.RC2? -no, there is a bug which prevents the ALL.TXT to contain them timestamp, but it should be fixed in the next release. +no, there is a bug which prevents the ALL.TXT to contain the timestamp, but it should be fixed in the next release. ### what is the state of this? @@ -106,7 +104,7 @@ feedback and pull requests are highly appreciated! here be dragons: * callsign lookup performance done with regex in one goroutine is unneccessary slow for alltxt2csv * grids for ALL.TXT entries are always looked up by dxcc instead if being tracked in the file -* there is a lot of space for improvements for the dashboards +* there is a lot of room for improvements in the dashboards ### grafana says too many rows for mysql datasource @@ -159,5 +157,7 @@ alerting * WSJT-X, Grafana, Prometheus, MySQL Teams * Philip Gladstone for pskreporter.info -* Todd Neal for his nice work on https://github.com/tzneal/ham-go -* Stefan Prodan for his nice work on https://github.com/stefanprodan/dockprom +* Todd Neal for https://github.com/tzneal/ham-go +* Stefan Prodan for https://github.com/stefanprodan/dockprom +* Markus Lindenberg DO8ML for helping with a lot of stuff +* Sebastian Schaumann DL7LE for testing and a lot of good ideas and questions diff --git a/devnotes.md b/devnotes.md index 7650905..f359801 100644 --- a/devnotes.md +++ b/devnotes.md @@ -1,33 +1,37 @@ # assumptions/ideas +* parsing ALL.txt is better than udp - udp cant be rewind * keep client as stupid/uncomplicated as possible + * no decentral maintenance should be needed if something changed * noch parsing/resolving, just postition finding # before release -* multiple repos? - * alltxt2http & http-exporter (inkl. shared/httpstuff und shared/wsjtx) - * pskreporter-exporter - * dashboards n docs - * alltxt2csv kann weg oder? * resolving lokal nur bedingt sinnvoll * profitiert nicht von ggf. vorhandenen grids im cache * wenn station und identifier bekannt sind, kann das auch der admin tun + * und das kann dann auch der wsjtx-exporter inkl. cache lookups etc. uebernehmen * think about: * logische trennung in zeilenverarbeitung und 'spotting' (spot in mysql&prometheus exportieren) * take line -> parse line -> resolve grid and lotw for line ? +* stationpi + * wsjtx works + * alltxt2http works + * wireguard works + * audio via pulse-tcp seems to work.. + * write doc! + * http-exporter + * callsignlabel aus prometheus entfernen????? * errorhandling * db down * who keeps track of the missing lines??? * lookup error (not an error) - * add instance field to mysql - * what happes if no instance is provided? * improve logging - * counter vs histogram in prometheus? + * counter vs histogram in prometheus? (fuer signal) * http 200 OK synchron warten lassen bis alles fertig ist? * scheint tendenziell eine gute idee zu sein.. wenn db problem hat, kann client aufhoeren.. @@ -39,10 +43,11 @@ * alltxt2http * what about readall/batchmode? + * errorhandling/logging + * see journal for obviously ending without any logs?! * systemd restart problematik (failed too fast oder soo ;) - * Username=Station (=~ Callsign) right?!? * call und password in htacces und fertig! * auth. user kann seine instance selber waehlen @@ -62,13 +67,13 @@ * push images to dockerhub * doc - * german docs.. * fix single tool readmes * hints for using grafana * intervals * fold row which are not needed * use filters! * server und/oder skript/readme zum aufsetzen + * german docs.. # later diff --git a/docker-compose/docker-compose.yml b/docker-compose/docker-compose.yml index c157b92..c40be63 100644 --- a/docker-compose/docker-compose.yml +++ b/docker-compose/docker-compose.yml @@ -48,6 +48,17 @@ services: # labels: # org.label-schema.group: "monitoring" + redis: + image: redis + container_name: cache + expose: + - 6379 + restart: unless-stopped + networks: + - monitor-net + labels: + org.label-schema.group: "monitoring" + grafana: image: grafana/grafana:7.3.4 container_name: grafana @@ -145,7 +156,7 @@ services: image: 10.0.73.1:5000/wsjtx-exporter:latest container_name: wsjtx-exporter ports: - - "4000:4000" + - "4001:4000" environment: - PROMETHEUS=true - MYSQL=true diff --git a/misc/bandswitcher.sh b/misc/bandswitcher.sh index b17efd7..172cc94 100755 --- a/misc/bandswitcher.sh +++ b/misc/bandswitcher.sh @@ -1,20 +1,36 @@ #!/bin/bash -RIGCTL=$(which rigctl) -PARAMETERS="-m 2" +# stay DELAY seconds on every band DELAY="300" -BANDS="14074000 21074000 28074000" +# local rigctld, can be running dummy rig! +PARAMETERS="-m 2" +BANDS=(14074000 21074000 28074000) + +# enable gqrx intergration +GQRX="1" +GQRX_PARAMETERS="-m 2 -r localhost:7356" +GQRX_BANDS=(139074000 146000740 153000740) + +# find rigctl binary +RIGCTL=$(which rigctl) # wait for full minute +echo "wait for full minute.." sleep $(((60 - $(date +%s) % 60)-1)) # start switching -while true -do - for band in $BANDS - do - $RIGCTL $PARAMETERS F $band +let len=${#BANDS[*]}-1 +while true ; do + for i in $(seq 0 $len); do + date + echo "switching local rigctl to ${BANDS[$i]}" + $RIGCTL $PARAMETERS F ${BANDS[$i]} + + if [ "$GQRX" = "1" ]; then + echo "switching gqrx to ${GQRX_BANDS[$i]}" + $RIGCTL $GQRX_PARAMETERS F ${GQRX_BANDS[$i]} + fi sleep $DELAY done done diff --git a/presentation.md b/presentation.md new file mode 100644 index 0000000..fedae9a --- /dev/null +++ b/presentation.md @@ -0,0 +1,47 @@ +# O49 Vorstellung + +* Funkwetterprognosen schwer verstaendlich.. + * muss anschaulicher gehen - siehe Gridtracker + * WSJT-X generiert viele spannenden Daten (ab 2019) + * NICHT die Logdaten... sondern ALL.TXT :) + +* Idee - Daten in Grafanakompatible Datenquellen packen + * was ist Grafana? + * Datenmenge + +* Nur Daten von meinem WSJT-X sind nicht soooo interessant + +* PSKReporter + * aktuell alle 5 Minuten abolen und speichern + * bald mit Livedaten :) + +* Benefits/Features! + * Longtime Auswertung + * Wallboard + * DX-Jagd Vorbereitung + * Callsign Tracking + * Vergleich von Stationen + +* Know Issues + * know your data/query! + * live counter vs tx + * dl3sd 24/7 vs dl7le 6/7 + * continental distribution + * map zeigt nur eine dimension (dxcc ODER band ODER receiver) -> keine ueberlagerung moeglich + * alles wip! + * viele features nicht drin + * Dashboards + * db + * komponenten + * Achtung Last! + * Client + * Server + +* Ausblick + * Pi mit VNC (+VPN ;) + * Red Pitaya ;) + * Alerting + * QRZ/LoTW + +* Bei Interesse Mail an mich + * Useraccount und Link zum Howto kommt