Monitoring

Monitoring communicates the general health and status of the FAT. These measurements are taken between all other measurements, as well as during dedicated monitoring periods. Dedicated monitoring periods include: during temperature transitions, shortly following temperature transitions, and to fill time at cold temperatures following other measurements.

Types of Monitoring

4 types of monitoring scripts exist - quick, detailed (constant HV), detailed (w/ Gain Scan), and warm reboot. Additionally, a very simple “status check” is performed, which simply checks if the mainboards are still connected & alive. Each one will be quickly discussed here.

Status Check

Status check simply creates iceboot sessions on all modules and confirms that the D-Eggs are available for further commands. Results are printed to #chiba-daq, including the temperature and HV. With a few exceptions, the HV should always be enabled to some value greater than 1000 V. If you see a value less than 1000 V and it is not immediately following a test of the cameras or the cold boot, contact an expert. This is especially important if a module shows a trend of low HV values.

Quick Monitoring

Quick monitoring is run the most frequently, needing less than 1 minute to quickly sample the modules. This is supposed to give us constant feedback regarding the health of the D-Eggs with some temporal granularity. The quick monitor measures mainly low-level quantities such as: HV readback, V-mon & I-mon rails, the pressure sensor, and calculates the dark rate using the FIR trigger. When the quick monitoring runs, the results are sent to #chiba-daq for each D-Egg with some results.

Thing to look for in the monitoring statements:

  1. Are all 16 modules printed?

  2. Are any values FAIL? - i.e. HV0 Val is FAIL

  3. Are any values below ~50 or over ~20,000 Hz?

  4. Are any values out of range? - i.e. dark rate is high

If any values are significantly out of bounds (see point 2), immediately contact an expert. Else, if any values are FAIL or the dark rate is out of range, examine the previous monitoring results. Do these results indicate a pattern for this module, or is it simply a fluctuation? If a module is consistently out of range, talk to an expert - this module may then be flagged.

Accompanying the quick monitoring, is also a monitoring analysis. This creates rolling plots, which update as new monitoring completes. It’s generally a good idea to check these plots at least once per day. They live on Grappa at /disk20/fat/software/degg_measurements/degg_measurements/analysis/monitoring/.

You should look for sudden changes in trends or large uncertainties. Make a note of which module, and post the plot to #degg_fat.

Rebooting

Warm reboots are handled automatically. These are typically set up to happen during temperature transitions. Simply watch #chiba-daq for any possible errors associated with the rebooting.

Detailed Monitoring (Constant HV)

When the temperature is constant, we do not want to modify the HV. As such, the monitoring performs a gain check (waveform), to measure the new gain. Because we want to measure the HV & gain stability over time, the HV is not modified. A new peak height is extracted and dark rate & delta-t is measured. Alternating runs of the scaler with the ADC trigger & FIR trigger are made. This measurement typically takes around 30 - 40 minutes.

No automated analysis is yet supported. For now, refer to the quick monitoring results as similar quantities are measured.

Detailed Monitoring (Gain Scan)

The extension of “Detailed Monitoring (Constant HV)”, is adding the proper gain scan to the monitoring suite. This is performed quickly using charge stamps and a new HV for 1e7 gain is extracted. As the gain changes somewhat with temperature, this script is only run when the temperature is moving. Measurements of the dark rate and delta-t are still performed.

The gain scan is a long procedure relative to the other measurements, making the script typically run for about 90 mintues. Also no automated analysis is yet supported. See quick monitoring results.

Monitoring PDFs

It is useful to generate PDFs summarise the results of the FAT. This can be achieved via the monitoring latex PDF generating script.

First, you should load the degg-fat environment, then run the command:

source /disk20/fat/software/venvs/degg-fat/bin/activate
/disk20/fat/software/degg_measurements/degg_measurements/utils/write_analysis_tex.py /disk20/fat/data/json/run/run_<RUN_NUMBER>.json

Replace the run number (without trailing zeros) with the current run you would like to be created.