Running the FAT¶

So far the FAT procedure has required a significant amount of manual intervention and testing. Hopefully this will reduce over time, but the primary automated process is the “master script”. This includes several components which work together to make the life of the shifter as simple as possible. This section will go through some of the conecpts of the master script and highlight areas your input may be required.

Before running anything for the first time, you should read this section

Quick Check list¶

Here are some things that you should check before starting your FAT shift.

Did you run setup_deggs.py?
Were all 4 DomNet instances successful?
Did you run create_configs.py?
Make sure all of the peripherals are working. (See below)
Open the FAT Logbook, login, and make a new FAT Run post.
Update the config file to use the latest run json file.
Run a gain scan at room temperature and the analysis script. (See next page)

Setup¶

In case you missed it in earlier sections: run setup_deggs.py in the screen session called “setup”. You can re-attach to the screen session with:

screen -r setup

and then simply type:

python3 setup_deggs.py

and type ‘y’ or ‘yes’ at the appropriate promts. Please check that all the DomNet instances were successful. If not run the script again.

To exit a screen session use the keys ‘ctrl + A’ and then ‘d’ (for detach).

New Run Number¶

This is given in more detail in the previous section. Go to degg_measurements/utils and simply run:

python3 create_configs.py

And provide a comment when asked. This information will be used to fill out the FAT Logbook.

Peripherals Check¶

The FAT Logbook asks you to confirm that various pieces of hardware are operational before starting a new FAT run/batch. This process is largely automated by the script:

python3 simple_check.py

which lives in the degg_measurements/utils directory. This checks the thermometer connection, humidity sensors, filter wheel, and function generator. The only other component which requires manual intervention is the laser power supply. As discussed in the previous section on Laboratory Setup, please ensure this power supply is on.

To catch any issues with the light system (laser, fibres, function generator, tabletop mainboard) before starting an FAT run, two diagnostic scripts should also be run. In both cases, make sure that setup_deggs.py and create_configs.py have been run, and that the hardware is all correctly configured.

First inside utils/ run:

python3 check_laser_freq.py

This verifies that the laser is producing a sync signal detectable by the tabletop mainboard and that the frequency being recorded is valid. Depending on the status of the function generator when this is run, it may need to retry once. This is normal. If the script fails after the retry, contact an expert.

Next in daq_scripts/ run:

python3 check_laser_visibility.py /home/icehap-daq/data/json/run/run_00XXX.json "check visibility for run XXX" -j4

This records information from the D-Eggs regarding their visibility. Most commonly, this could indicate that a fibre channel is not effectively transmitting light to a PMT. Depending on if uncommitted changes are in the directory, you may also need to run with --force.

FAT Logbook¶

You should use the Logbook throughout the FAT. If you cannot access the Logbook, simply make a new user for yourself.

The most important form is ‘Start New Run’. If you have followed all of the previous steps, these fields should be easy to fill-out. Logging information is important for future records, please use this to record any information.

Master Script¶

The master script is also run in a screen session. First check if a screen session already exists:

screen -ls

If a screen session called ‘master’ already exists, attach to it. You can re-attach to the screen session with:

screen -r master

If you are starting a new FAT Run, this session should be associated with the previous run. To keep the logging files separate, type ‘exit’ to close this session. Now we’ll make a new screen session for the new FAT Run. Remember that ‘XXX’ is the current run number (output from create_configs.py).

screen -L -Logfile fat_run_XXX.log -S master

This command creates a new screen session called ‘master’ and creates a logfile to track all information sent to the terminal. (std-out & std-err?)

See the next page for important steps to perform before running the master script!

Connectivity Tests¶

Although much of the previous sections has been about testing the connectivity of the D-Eggs, the master script also contains a “dummy test” to ensure all D-Eggs are alive and listening. The script should respond with errors in the event that D-Eggs listed in the run file are not responding. In this case, return to previous recommendations about troubleshooting connectivity.

Scheduler¶

During the FAT, we want certain tests to be executed in a specific order. Before any tests are executed, the full testing schedule is built by the scheduler class. This is a simple class which builds a list of tasks and then executes functions, which empties the queue.

In the case of crashes or tests completing unsuccessfully, the schedule also contains a method to resume from a given point. This is controlled mainly by the schedule config file. When the task is queued up it can also be passed a flag to perform a backup run after the task finishes. The backup task is located in utils/backup.py and creates a snapshot copy of the json files at this time on Grappa: /misc/disk19/users/icecube/fat_backup. By running this regularly after all measurement processes, we insure ourselves against DAQ scripts corrupting the json files. Actual backup and copying of data files is processed between runs, and will be explained later.

Additionally as part of the schedule we regularly include the status_check function to verify that D-Eggs are still responsive and ready to undergo their next measurement. This should help track if/when D-Eggs became unresponsive and prevent errors encounter during initialisation of DAQ scripts.

We can build a new schedule using the script in degg_measurements/daq_scripts/configs:

python3 create_fat_config.py this_run_file.json config.ini

where the run file is the json file created earlier, and the config.ini is an identifiable configuration file name.

Output¶

The master script also interfaces with the Slack bot to provide real-time updates of the FAT DAQ status. Upon launching the master script, the information sent to Slack looks something like this:

Executing FAT Master Script - Time is now: 2020-05-29 17:46:20.341249
-- Current Run: 00012
-- D-Egg IDs:
DVT2020-004-0917-0804.json
DVT2020-006-0903-0808.json
Schedule:
[0] wait_time
[1] readout_and_reboot
[2] stf_wrapper
[3] measure_gain
[4] measure_gain
[5] measure_scaler
[6] measure_linearity
[7] measure_long_waveform
[8] measure_baseline_flasher
[9] readout_and_reboot
[10] manual_input

and the currently running process gets returned in Slack 5:46 [3] measure_gain. If any error occurs, the trace is printed as a warning to the Slack channel and directly to the shift expert.

When the master script finishes, a summary statement is sent to Slack and the run json files are updated.

Input¶

In some cases you will be required to input a resume command to the master script. This should happen when the freezer temperature needs to be adjusted. Follow the instructions written in the laboratory section on how to adjust the freezer temperature. Then follow the instructions in the terminal to resume the DAQ.