Local Setup

Error Logging

It is important to make notes of all error which occur, including errors which do not cause the FAT to entirely stop/crash. These can be temporarily logged in places such as the white boards, Slack, etc. However the errors must also be logged in a more permenant way.

This can be achieved using the FAT Logbook - use the ‘New Post’ option and provide as much detail as possible. Please also consider using the script error_logging.py located in /home/icehap-daq/dvt/. Follow the instructions to add notes about an error to the log which can be reviewed later. Add as much detail as possible, including any software errors.

Internet

This is another opportunity to confirm the computer is connected to the internet. Internet connection is critical for several functions, such as updating the shifter chat and database files. While you should have already been checking the D-Egg logbook, it can be useful to keep this page displayed throughout testing. Additionally, open Slack in the application and navigate to the #chiba-daq channel to monitor real-time updates and errors.

Cron Jobs

The previous steps required examining the thermometers and humidity sensors to verify their operations. Cron jobs are also important for performing automated backups of select data. Open the file /var/spool/mail/icehap-daq and check if there are any recent entries. Report any errors in the #chiba-daq channel and if needed contact a shifting expert.

Lastly check the crontab for any missing jobs using crontab -l. Compare to the list below:

0 */2 * * * python3 /home/icehap-daq/dvt/crontab/check_cron.py
*/10 * * * * python3 /home/icehap-daq/software/goldschmidt/goldschmidt/record_temp.py /dev/ttyUSB1 1 2 3 4 /home/icehap-daq/software/goldschmidt/goldschmidt/temp.csv
*/10 * * * * python3 /home/icehap-daq/software/USBRH_driver/humidity_readout.py
*/30 * * * * python3 /home/icehap-daq/software/check_disk/slow_disk_check.py / /home/icehap-daq/software/check_disk/disk_space.csv
0 */2 * * * python3 /home/icehap-daq/software/plotting/temperature_plot.py
0 */2 * * * python3 /home/icehap-daq/software/plotting/slow_disk_plot.py
0 */2 * * * python3 /home/icehap-daq/software/plotting/humidity_plot.py
0 */4 * * * python3 /home/icehap-daq/dvt/crontab/push_update.py
0 */23 * * * python3 /home/icehap-daq/dvt/crontab/backup_cron.py

If the cron jobs are running, they should be regularly displayed in the slack chat with a green tick mark.

Function Generator

To provide automated control over the laser system, the laser is connected to a function generator. Enabling the function generator is described during the laboratory section. Here we want to verify connectivity.

The function generator is controlled by the library called skippylab, which should already be installed on the local machine. To test if the correct skippylab libraries are visible, enter an ipython session and enter the following commands:

from skippylab.instruments.function_generators import Agilent3101CFunctionGenerator as FG3101
fg = FG3101()
fg.startup()
fg.disable()

Note: This functionality is also verified in ``utils/simple_check.py`` Instructions following in the master script introduction

Filter Wheel

In addition to the function generator controlling the laser on/off and frequency, the filter wheels give us consistent and programmable control over the intensity. This is also controlled via a python library and can be tested in an ipython session:

from filterWheel import control102C
fw0 = control102C.connect_wheel(wheelNum=0, port='/dev/ttyUSB0')
control102C.check_wheel(fw0)
fw1 = control102C.connect_wheel(wheelNum=1, port='/dev/ttyUSB2')
control102C.check_wheel(fw1)

If you’re encountering connectivity problems, the filter wheel may be on the incorrect /dev/ttyUSB*. During reboot of the PC, sometimes usb ports can get shuffled around. For example, try /dev/ttyUSB1. If you are still having problems, consider unplugging all USB connections, then inserting the filter wheel first.

Note: This functionality is also verified in ``utils/simple_check.py`` Instructions following in the master script introduction

D-Egg Measurements

Most of the local code to control the D-Eggs exists in the “degg_measurement” repository. Navigate to it from $HOME/software/degg_measurements/degg_measurements/. We will primarilly be using software from the utils/ directory to do the setup.

Setting Up D-Eggs

To begin issuing commands to the D-Eggs, two important steps are needed. These will ultimately be handled by scripts in the utils/ directory.

  1. The first is establishing a connection between the ICMs on the mainboards inside the D-Eggs and the ICMs inside the mini-fieldhubs (MFHs). To do this, we are currently using a program called DomNet, which runs similarly to the icm_probe script. How to run DomNet was briefly explained in the previous sections, however specific knowledge of running DomNet is not required to setup the D-Eggs.

  2. The second step is to enable the FPGA on the mainboard. As the FPGA must be re-programmed when powered off, this can be done manually when connected via telnet, but is more easily handled by dedicated scripts. Simply keep-in-mind that if you see an error such as FPGA NOT PROGRAMMED, this means you should try to program the FPGA again.

* In this case, “powered off” means the D-Egg mainboard. Either if the mainboard is re-booted or the power is turned-off at the mini-fieldhub.

setup_deggs

  1. Turn on the mini-fieldhubs (1st, 2nd, 3rd).

  2. Inside utils run python3 setup_deggs.py

  3. Type “yes” and then “yes” again to start up the D-Eggs. This will run DomNet for the appropriate number of connections.

  4. You should see ports 5000 through 5016 opened - make sure that you count all 16 D-Eggs and the 4 MFH ICMs. If not, re-run setup from the beginning.

  5. Flash all of the FPGAs by answering “yes” and “yes” to parallel flashing. Some D-Eggs will return a green message, indicating the flashing was successful. Others will return a yellow message stating that the FPGA was not flashed correctly. This will be manually corrected in the next step. In other cases, the prompt will say “Could not start Iceboot session”. Monitor these D-Eggs closely in the next steps.

  6. Also enable the tabletop mainboard on port 10007 (“yes” when prompted).

  7. setup_deggs.py will then attempt to get sensor data from all D-Eggs. This will only succeed if the FPGA flashing was completed without errors. Again, observe for D-Eggs which “could not start Iceboot session”.

  8. If 16 sets of sensor data are displayed, you can move to the next page. Otherwise see below for more options.

If you see an error during setup which states that the ICM number does not match what was expected, check ls /dev/ttyUSB*. In this case, if ttyUSB2 is missing, or ttyUSB8 is missing, physical action must be taken. Go to the freezer lab, turn off the mini-fieldhubs, and disconnect the white USB cable in the USB hub. Then reconnect the cable and follow the instructions above.

Flash FPGAs

This step is only necessary if you had D-Eggs which could not flash their FPGA.

setup_deggs.py tries to flash the FPGA for all modules. In some cases the D-Eggs are slower to boot or the FPGA was not flashed correctly and the module needs to be flashed manually. Inside utils is a script called flash_fpga.py which takes the port as an argument. For all D-Eggs which were not flashed correctly during setup_deggs.py, run

python3 flash_fpga.py 50XX

where 50XX is the correct port number. Do this until all D-Egg FPGAs are flashed!

Debugging Connection

If you are unable to communicate with the D-Eggs, this could indicate that the mainboard locked-up during the setup procedure or a connection is not stable. First try re-running setup_deggs.py. Then once domnet is running, you can test by using telnet as described in earlier sections: telnet localhost 50XX. If a prompt appears either like this # or like this > then you have established a connection. This # means the D-Egg needs to be “booted” (i.e. type boot). This > means the D-Egg is already booted. Test the mainboard status with any command, such as flashID.

In the case that you can connect to the mainboard via telnet, but could not setup the D-Egg, disable the MFH power and start again. If this problem persists, re-check via icm_probe. As a last resort, see the previous section about repairing the cables. Contact an expert if you are having issues.