Testing Schedule

The config file lives in degg_measurements/daq_scripts/configs and are .ini files.

Again, check that the run file is correct - this is a critical piece of information.

The testing schedule is constantly being iterated upon, however here is the general overview:

  • Comm. Testing

  • Gain scan

  • Cool down (-40) - monitoring running

  • Cold Boot

  • STF and Camera comm.

  • Gain scan

  • Dark rate

  • Delta-T

  • PMT Linearity

  • Double-pulse separation

  • Transit Time Spread

  • Flasher operations

  • Camera operations

  • Change temperature (-20) - monitoring running

Before finishing the FAT, verify the status of the data files and jsons!

Pre-Master Gain Scan

Currently (2022-05-26) we run a gain scan prior to enabling the master script. This serves two functions:

  1. We can confirm that the D-Egg & PMTs are behaving in a reasonable manner.

  2. The initial HV values are filled to the json files before monitoring.

More details will be on the following pages on how to run the individual DAQ and analysis scripts, but the instructions below should be enough to run those relevant for the gain scan.

Running the gain scan is straightforward. Navigate to degg_measurements/daq_scripts and run:

python3 measure_gain_online.py /home/icehap-daq/data/json/run/run_00XXX.json -j 4

where the run_00XXX.json is the current run number and -j the number of processes to spawn. This should always be 4.

The script often takes a few hours to finish - take a break, get a coffee, and come back when it’s finished…

Once it’s done run the analysis script. Navigate to degg_measurements/analysis/gain and run:

python3 analyze_gain.py /home/icehap-daq/data/json/run/run_00XX.json

and the analysis script should automatically find the most recent gain data (the one you just ran). This will take a few minutes and creates a number of plots you can check. These plots will be in figs/run_XXX/key_GainMeasurement_XX/.

The most important plots to check are called “gain_curve_SQXXXX.pdf”. Make sure there are 32 plots (1 for each PMT), and that they all look reasonable. “Reasonable” in this case means: the gain reaches 1e7 within 1200 to 1900 volts, the points appear mostly regular (some fluctuation around 1e7 is OK), and the fit is successful.

If all of these steps are successful, then we’re ready!

Executing the Master Script

python3 fat_master.py configs/fat_config.ini

To exit a screen session use the keys ‘ctrl + A’ and then ‘d’ (for detach).

Troubleshooting: If the fat_master.py is not starting, you may need to run with --force. This is to acknowledge there are uncommitted changes made in the directory. This should be avoided if possible, by committing all changes before running so we have a record of the stable version used to run the FAT.

If the script crashed in the middle of a run, use the ``–recover`` flag when running again. The schedule will resume from the last attempted task!

Before FAT Ends

Recommended time to run the run_verification.py script is before turning off the freezer to return to +20!

After the run is finished, review that all measurements have created output files and that the json files are not damaged/corrupted. An automated script to check the json files for the relevant data lives in utils and can be run by giving it the json file you want to check. Note, if you’re checking older runs, some data may have been moved to the backup on grappa already to save disk space. If you need to find old data on grappa, try checking ``disk19/users/icecube/``.

Run the check script:

python3 run_verification.py /home/icehap-daq/data/json/run/run_00XXX.json

This script will output a result for each measurement for each PMT. In the case that a file is missing or otherwise has a bad path, the script tries to provide a useful warning. Finally, a summary is printed at the end:

--------------------
Summary:
Number of Good Results        : 304
Number of Missing Results     : 0
Number of Missing Measurements: 0

Good Results mean that the json file points to a directory and it exists. Missing results mean that a measurement has been created in the json file, but the path is either invalid or ‘None’. This typically means that either the measurement crashed early or the DAQ script is actually saving the data somewhere else. If this is the case, confirm the status of the data and try to move it to the appropriate directory if possible.

Missing Measurements are more serious, as this means that during the FAT a problem occured and this measurement was skipped or otherwise failed before creating an entry. If a measurement is missing, this may be cause for a module to be re-tested.

After the Full FAT Run

Once we’re satisfied with the results of the run, open the screen session called “backup”:

screen -r backup

Inside this screen session, backup the data to Grappa. For “today” input today’s date (ex. 2022-04-26) and the run number (ex. 00129, you can find it in the config file: degg_measurements/degg_measurements/configs/fat_config.ini).

rsync -avzr /home/icehap-daq/data grappa:/disk19/users/icecube/backup_{today}_{run_number}

This will take a few hours, so you can disconnect from the screen by doing these commands (“control + A”, then “d”):

ctrl+A
d

Wait for an expert to begin the procedure for removing the D-Eggs.