wsjtx-dashboards/Readme.md

# what is it?

![alt text](screenshot.png "Logo Title Text 1")

* a set of tools to export your personal WSJT-X
  * live reception data into prometheus or mysql
  * historical reception data (since 2019) into mysql
* a tool to poll pskreporter and export your personal 
  * live transmission data to prometheus or mysql 

every time your wsjtx is/was running you are/were generating nice data points, which can be queried using grafana.
this can be very interesting to evaluate your **personal** hf conditions from the current and the past.

some use cases: 
* choose dxcc, band, mode and timeframe in all combinations so see when your were successfully able to receive and transmit, to plan future activities
* compare input of different station like for antenna comparions
* have nice wallboard for contesting or dxpedition
* display rx/tx data on a map with custom timespan and filters
* compare hf propagration forecasts to your personal situation
* use grafana alerts or prometheus alertmanager to send yourself alerts when your stations receives something interesting
* impress other people with cool dashboards ;)
* ...

sample dashboards are provided. 

have fun!
73 de DL3SD

## prerequisites

* ALL.TXT with interesting data and/or running hamradio station to provide live data
* grafana
* prometheus or mysql

## repository overview

* **pskreporter-exporter**
  * polls pskreporter.info for your callsign
  * supports prometheus and mysql
* **wsjtx-exporter**
  * follows live traffice in ALL.txt
  * supports prometheus and mysql
* **alltxt2csv**
  * reads whole ALL.txt since 2019 and creates csv-file which can be imported into mysql
* **misc**
  * sample dashboards
  * sql table definitions
  * shell script to hop between bands via rigctl to allow fractional monitoring of multiple bands
  * ready to launch docker-compose setup based on https://github.com/stefanprodan/dockprom

## things you should be aware of..

### what about prometheus and mysql?

#### prometheus

pro
* you get nicer graphs with counters and bars if you ask me
* you can use alertmanager if you want advanced alerting
* optimized for timeseries data
con
* not designed for keeping long timeseries (weeks or some months are fine)
* not possible to import historical data
* not as flexible as mysql regarding complex queries

#### mysql

pro:
* you can import your ALL.txt since 2019
* you can store a lot of data over a long timeframe
* you can propably build more complex queries in sql
con:
* in comparison to prometheus not optimized for timeseries, but still good enough
* gauge based graphs are not as nice as counter/bars if you ask me
* you can use *only* graphanas internal alerting

both allow distributed setups with multiple wsjtx instances submitting their data to a central prometheus or mysql service.
you can as well run both in parallel and use prometheus for a live overview and mysql for historical evaluations.

### pskreporter-exporter vs other access/polling of pskreporter like GridTracker

### can it read my whole ALL.txt since from the beginning?

not really. only lines in 'new' format are accepted because the old format doesnt contain the frequency and the date.

lines in incompatible format are silently ignored.

with alltxt2csv you can prepare your ALL.TXT from ca. mid 2019 for import to mysql.

for prometheus it is not possible to import historical data for obivous reasons.

### there are a lot of "cant parse callsign messages" during importing my ALL.txt

thats quite normal as there are propably a lot of strange/broken messages in your ALL.txt ;)

### does it work with wsjt-x 2.3.0.RC2?

no, there is a bug which prevents the ALL.TXT to contain them timestamp, but it should be fixed in the next release.

### what is the state of this?

this works fine for me! but it was just build from boredom during covid lockdown and it is not to be considered as perfect.
performance obviously depends on your hardware and the complexity of your queries.
feedback and pull requests are highly appreciated!

here be dragons:
* callsign lookup performance done with regex in one goroutine is unneccessary slow for alltxt2csv
* grids for ALL.TXT entries are always looked up by dxcc instead if being tracked in the file
* there is a lot of space for improvements for the dashboards

### grafana says too many rows for mysql datasource

try choosing a bigger interval

### how long does it take to import my data into mysql?

* my ALL.TXT (new format start july 2019) contains ~ 13.7 mio lines and has ~ 850M
* converting to csv takes ~ 12min on i7-4750HQ (2015) and the result has ~ 1.2G
  * currently this is done using a module which uses regular expressions which is not optimial for this use case
* importing the csv to mysql takes ~ 3.5min
* querying the whole time (~ 1.5 years) in grafana takes some seconds

### does this need a lot of ressource on my machine?

well, it depends on your queries, but propably yes! ;)

### when i count the decoded messages in wsjt-x manually it doesnt match the prometheus graphs

that is correct! due to the way prometheus get the results it doesnt know the corret timestamp of the event.
rounding/rating does the rest.

..in mysql it is correct!

## howtos

there is no special howto on integrating the components into your existing grafana/prometheus/mysql infrastructure,
as you should propably know how to deal with these kind of stuff.. ;)

### howto build binaries/containers

build binaries:
```
go get github.com/denzs/wsjtx_dashboards
```

build docker containers:
```
docker build Dockerfile.wsjtx-exporter .
docker build Dockerfile.pskreporter-exporter .
```

### to be done...

debian/win10@wsl2 + docker-compose (historical data with mysql + alltxt2csv)
raspbian + docker-compose (live data with prometheus only)
alerting

## Kudos to

* WSJT-X, Grafana, Prometheus, MySQL Teams
* Philip Gladstone for pskreporter.info
* Todd Neal     for his nice work on https://github.com/tzneal/ham-go
* Stefan Prodan for his nice work on https://github.com/stefanprodan/dockprom