Overview

The FAT shifter is an important part of our team, responsible for keeping the DAQ and analyses running. Since we need to test 300+ D-Eggs, it’s important that we try to keep to our schedule! This means regularly checking the status of the devices, and helping identify issues. If you’re ever unsure about something, contact an expert!

Structure

Each shift block is 1 week, composed of 2 shifters and one on-call expert. The shifters share equal responsibility for keeping the DAQ running, recording information, and contacting the on-call expert. The on-call expert is expected to handle issues if contacted by the shifters, within reason. The two shifters should organise themselves however they like, but ideally with equal coverage of the week. To avoid large delays, DAQ downtime larger than 1 hour between 09:00 - 18:00, Monday ~ Friday must be avoided. This should be less than 8 hours (as possible) after 18:00 or on weekends.

First-time shifters should be sure to read this documentation. Where possible, these people should be paired with experienced members for their first shift. Both the primary and assistant shifters should help each other out.

Talk to Colton, Max, or Ryo for pre-FAT training if you have any questions

Shifter Organization

Crashes of the DAQ can be frequent, so it is important to make sure someone is watching the system. As the shifter, you are responsible for keeping the system running. Make sure you communicate with your shift partner to find a convenient setup.

  1. If you are unavailable for any time during which you are scheduled, talk to you shifter partner and Izumi-san (ideally in-advance).

  2. If you need to re-schedule your shift, please arrange the replacement yourself and inform Izumi-san.

Example

Tanaka and Suzuki are on shift next week. They decide in advance that Tanaka will check mainly in the morning, and Suzuki in the evening between Monday and Friday. Tanaka decides he can check on Saturday, while Suzuki will check on Sunday.

Tanaka and Suzuki both coordinate regarding any errors Monday ~ Friday during working hours, and discuss things they do not understand.

After their shift they create their summary presentation for the D-Egg FAT Meeting on Monday, to let the new shifters know about any problems they encountered.

General Responsibilities

The following list includes general responsibilities for the shifter

  1. Keep the DAQ Running

  2. Watch Slack (#chiba-daq & #degg_fat) for monitoring output and for warnings

  3. Keep Analyses Running (on Grappa)

  4. Verify Analysis Results

  5. Check FATDAQ is Being Updated

  6. Summarise the weekly status at the FAT Meeting

You can use Anna’s script to pull down relevant plots for the various analyses. Anna’s script should be run on your local machine:

/disk20/fat/software/degg_measurements/degg_measurements/utils/sync_figs_for_monitoring.sh

Or pull it directly from GitHub: https://github.com/icehap/degg_measurements/blob/master/degg_measurements/utils/sync_figs_for_monitoring.sh

Also available for help is Aske’s script to tell you which measurements were run today (or any number of days) in

/disk20/fat/data/DAQ_log_check.py

This parses available log files created from analyses run on Grappa.

Weekly Meeting

The weekly meeting should be a summary of the status of the FAT from the past week. These slides should be made by the shifters who just finished their shift. This information should help the shifters starting this week.

The shift hand-over meeting is held at 11:00 AM on Mondays. A useful example of slides are Anna’s slides from: https://drive.google.com/drive/folders/1yH2DJjpNBeiMrD8bOBt-XXtupvbrl-Vd?usp=share_link

Things to include in the slides:

  1. Summarize any new entires into the logbooks (DAQ and Analysis)

  2. DAQ Uptime, and any problems?

  3. How were the various errors handled that week? Include the expert’s reponses.

  4. Were the modules rebooted?

  5. Show monitoring plots - any problems?

  6. Any STF issues?

  7. Were there any crashes? Which modules caused the issues?

  8. Were the D-Egg SW/FW updated last week?

FAT Documents

There are two locations for recording information you observe during FAT. These links are on the first introduction page of this documentation. Put DAQ errors into the LogBook, and record module errors related to monitoring or analysis results into the google form. Fill these documents out as you go along during your shift.

Notes About Shifting

The FAT is a team effort, please be responsible while you are on shift. This means restarting the DAQ and reporting errors promptly, even on weekends. Work together with the team and everyone will only have to do a little bit!