diff --git a/Readme.md b/Readme.md index 19915ec..e63c3f0 100644 --- a/Readme.md +++ b/Readme.md @@ -3,10 +3,10 @@ ![alt text](screenshot.png "Logo Title Text 1") * a set of tools to export your personal WSJT-X - * live reception data into prometheus or mysql - * historical reception data (since 2019) into mysql + * live reception data into prometheus or mysql + * historical reception data (since 2019) into mysql * a tool to poll pskreporter and export your personal - * live transmission data to prometheus or mysql + * live transmission data to prometheus or mysql every time your wsjtx is/was running you are/were generating nice data points, which can be queried using grafana. this can be very interesting to evaluate your **personal** hf conditions from the current and the past. @@ -35,17 +35,17 @@ have fun! ## tooling overview * **pskreporter-exporter** - * polls pskreporter.info for your callsign - * supports prometheus and mysql + * polls pskreporter.info for your callsign + * supports prometheus and mysql * **wsjtx-exporter** - * follows live traffice in ALL.txt - * supports prometheus and mysql + * follows live traffice in ALL.txt + * supports prometheus and mysql * **alltxt2csv** - * reads whole ALL.txt since 2019 and creates csv-file which can be imported into mysql + * reads whole ALL.txt since 2019 and creates csv-file which can be imported into mysql * **misc** - * sample dashboards - * shell script to hop between bands via rigctl to allow fractional monitoring of multiple bands - * ready to launch docker-compose setup based on https://github.com/stefanprodan/dockprom + * sample dashboards + * shell script to hop between bands via rigctl to allow fractional monitoring of multiple bands + * ready to launch docker-compose setup based on https://github.com/stefanprodan/dockprom ## things you should be aware of.. @@ -53,21 +53,27 @@ have fun! show pro/con overview: -**prometheus:** -* + you get nicer graphs with counters and bars if you ask me -* + you can use alertmanager if you want advanced alerting -* + optimized for timeseries data -* - not designed for keeping long timeseries (weeks or some months are fine) -* - not possible to import historical data -* - not as flexible as mysql regarding complex queries - -**mysql**: -* + you can import your ALL.txt since 2019 -* + you can store a lot of data over a long timeframe -* + you can propably build more complex queries in sql -* - in comparison to prometheus not optimized for timeseries, but still good enough -* - gauge based graphs are not as nice as counter/bars if you ask me -* - you can use *only* graphanas internal alerting +### prometheus + +pro +* you get nicer graphs with counters and bars if you ask me +* you can use alertmanager if you want advanced alerting +* optimized for timeseries data +con +* not designed for keeping long timeseries (weeks or some months are fine) +* not possible to import historical data +* not as flexible as mysql regarding complex queries + +### mysql + +pro: +* you can import your ALL.txt since 2019 +* you can store a lot of data over a long timeframe +* you can propably build more complex queries in sql +con: +* in comparison to prometheus not optimized for timeseries, but still good enough +* gauge based graphs are not as nice as counter/bars if you ask me +* you can use *only* graphanas internal alerting both allow distributed setups with multiple wsjtx instances submitting their data to a central prometheus or mysql service. you can as well run both in parallel and use prometheus for a live overview and mysql for historical evaluations. @@ -110,7 +116,7 @@ choose a bigger interval ### how long does it take to import my data into mysql? * my ALL.TXT (new format start july 2019) contains ~ 13.7 mio lines and has ~ 850M -* converting to csv takes ~ 14min on i7-4750HQ (2015) and the result has ~ 1.2G +* converting to csv takes ~ 12min on i7-4750HQ (2015) and the result has ~ 1.2G * currently this is done using another module which uses regular expressions which is not optimial for this use case * importing the csv to mysql takes ~ 3.5min * querying the whole time (~ 1.5 years) in grafana takes some seconds diff --git a/todo.md b/todo.md index 4c8cceb..5b1e4a5 100644 --- a/todo.md +++ b/todo.md @@ -1,32 +1,32 @@ * wsjtx-exporter - * cqzone und ituzone nicht in db - * -back parameter implementieren - * remove calls metric? - * https://prometheus.io/docs/practices/naming/#labels - * cant reach database ist bei mir aufgetreten - * adressieren ;) - * systemd user unit - * windows aequivalent?? - * trace: sucessfully parsed.. loggt: fields.time - * bei kaputter Zeile: + * cqzone und ituzone nicht in db + * -back parameter implementieren + * remove calls metric? + * https://prometheus.io/docs/practices/naming/#labels + * cant reach database ist bei mir aufgetreten + * adressieren ;) + * systemd user unit + * windows aequivalent?? + * trace: sucessfully parsed.. loggt: fields.time + * bei kaputter Zeile: Nov 26 20:10:47 sebo-OptiPlex-980 wsjtx_exporter[869]: goroutine 12 [running]: Nov 26 20:10:47 sebo-OptiPlex-980 wsjtx_exporter[869]: github.com/denzs/wsjtx_dashboards/shared/wsjtx.ScanLine(0xc0000a4004, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) * alltxt2csv - * direkt importfeature wieder aufnehmen..? + * direkt importfeature wieder aufnehmen..? * database - * propably more indices + * propably more indices * create table aus binaries nehmen? - * kontrolle sollte beim db admin liegen + * kontrolle sollte beim db admin liegen * doc -** german docs.. -** server und/oder skript/readme + * german docs.. + * server und/oder skript/readme * fix dashboards - * umgang mit refresh der variablen?? + * umgang mit refresh der variablen?? * provide dashboards to grafana