The EHT has a monitoring data collection system, and two user interfaces for it: VLBI Monitor and Grafana.
This document contains instructions for viewing and creating new dashboards in Grafana.
During the observation, you can discuss vlbimonitor and Grafana in the #vlbimonitor Slack channel.
Please contact Greg, either via email or EHT Slack (@Greg Lindahl), to ask for an account. Usernames and passwords are in the same style as vlbimonitor. Having your own account means you can make new dashboards and panels.
There's also a group login for the AOC, which is read-only.
If you look in the Dashboards section (to the left), you can find a list of all of the dashboards in the system. You can also make your own! Dashboards are made up of panels, and panels have a list of time series on them, and can be applied to one of the many data sources.
All of the dashboards at the top level are looking at live vlbimon data. The dashboards in folders are either using historical data ("Historical Demo") or are prototypes made by other EHT users.
You are encouraged to create your own dashboards and panels. The best way to learn what various database queries look like is to look at existing dashboards -- the code for them is under the 3 vertical dots icon in the upper right of every panel. The dashboards in production use are set to read-only, but you can still click on "explore" see what the current settings are.
You can even change these settings, and save changed panel in your own new dashboard. Starting from an existing panel is usually the fastest method of creating a new one.
To create your new panel while exploring, use "Add to dashboard" in the upper right. Choose "New Dashboard", then "Open". You'll see your new dashboard with your new panel in it. It won't be saved until you save it, by using the floppy disk icon in the upper right. But before that, click on the gear (next to the floppy), change the title, and select the Prototypes folder.
If you end up with a dashboard named "New dashboard" not in the Prototypes folder, no worries, just ask on #vlbimon or contact Greg.
A panel has a format, usually a time series, an SQL query, and a data source.
The SQL query has to match the details of the database, which has a separate table for every vlbimonitor "parameter", containing all of the stations in the same table. The columns of this table are the station name, a time, and the value for that station at that time. As a result, the main thing you need to specify, for a normal time series showing all of the stations in a single panel, is the of the parameter. The rest is just boilerplate.
In this example, the name is if_1_systemTemp:
SELECT time, station, value from ts_param_if_1_systemTemp
WHERE time >= $__from / 1000 AND time < $__to / 1000 ORDER BY time ASC
and if you just wanted one station, KP, add station = to the WHERE clause:
SELECT time, station, value from ts_param_if_1_systemTemp
WHERE time >= $__from / 1000 AND time < $__to / 1000 AND station = 'KP'
ORDER BY time ASC
This summary shows all of the parameter names, and how many data points each station has for them:
https://grafana.ehtcc.org/static/summary2023.txt
The production dashboards all use live.db as the data source,
which is a real-time updated copy of the vlbimonitor database.
Last year's live.db is renamed to eht2023.db
There are also individual databases representing single days of
previous observations, named by the experiment name. For example,
e24j25.db is the 2024 dress rehearsal.
Grafana has a way to transform data points before showing them in a chart,
but it's difficult to use. To make life easier, vlbimon-bridge implements
set of "transformers".
As an example, sky coordinates are represented in vlbimon as a string
like 14.588+80.996. While there's no perfect way to represent a position
on the sky as a function of time, it is handy to have this string value
split into 2 additional timeseries, named foo_ra and foo_dec, with floating
point values. There is a transformer that does this for all RaDec and all Azel
in the vlbimon database.
https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/
To add a new Grafana alert "contact" that posts to a Slack channel, you need to get a webhook from here: https://api.slack.com/apps (app name: grafana-alerts). There are already channels configured in Grafana for: ehtobs_bots, ehtobs_bots_urgent, ehtobs_bots_testing, and ehtobs_bots_grafanatest.
The software that copies the vlbimonitor database into live.db, and also
does transformations, can be found at https://github.com/eventhorizontelescope/vlbimon-bridge/
I chose to install on Ubuntu from the Grafana APT repository instructions.
This means that the configuration files are in the usual system directories.
I also installed the sqlite plugin:
sudo grafana-cli plugins install frser-sqlite-datasource
An example nginx.conf file can be found in [conf/nginx.conf]. This configuration assumes that grafana is on its own subdomain (e.g. grafana.ehtcc.org.)
An example /etc/grafana/grafana.ini file can be found in [conf/grafana.ini]
To start the grafana server once, use:
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl start grafana-server
sudo /bin/systemctl status grafana-server
And to start it every boot:
sudo /bin/systemctl enable grafana-server
The grafana config looks for its databases in the /var/lib/grafana/ directory.
Note that grafana itself uses the /var/lib/grafana.db database to
store configuration information, so do not place any vlbi data in that
database.
When you add a new database, a Grafana an admin needs to add it to Connections -> Data Sources before it is visible to users as a datasource.