Skip to content

Troubleshooting---ESS DAQ and EFU, KAFKA, DAQUIRI

Event Formation Unit (EFU): cmake takes long

If executing cmake .. takes very long, without any output, do not worry. This can happen. It may last 10 minutes or even longer. It can take veryyyyy long!

Error in running EFU: no config file

It is highly recommended to install the ESSDAQ software in the user home directory. Not in a specific subfolder where all VMM3a/SRS related software is placed. If the user however decides to still create a different path, the following error might occur: 

START GDGEM
Gd-GEM - applying local configuration
Ubuntu: net.core.netdev_max_backlog
Checking kernel buffer sizes
net.core.rmem_max = 12582912
net.core.rmem_max = 12582912
net.core.wmem_max = 12582912
net.core.netdev_max_backlog = 5000
HW check PASSED
Network check started
Network check PASSED
start Daquiri acquisition
START EFU
Gd-GEM - applying local configuration
Using default essdaq location
Error: No config file: /home/dateuser/essdaq/detectors/gdgem/config.ini
exiting ...

In case this error occurs, please adjust the CONFIG_FILE path in essdaq/efu/efu_start.sh:

#!/bin/bash

echo "START EFU"

source ../../config/scripts/base.sh

#
# #
#

# This is no good, cannot differentiate other processes that may have "efu" somewhere in the string
# don't start if EFU is running
#ps aux | grep -v grep | grep efu && errexit "EFU is already running on this machine"

UDPARG=""
if [[ $EFU_UDP != "" ]]; then
  UDPARG="-p $EFU_UDP"
fi

if [[ $ESSDAQROOT != "" ]]; then
  echo "Using custom essdaq location: $ESSDAQROOT"
  CONFIG_FILE=$ESSDAQROOT/essdaq/detectors/$DETECTOR/config.ini
else
  echo "Using default essdaq location"
  CONFIG_FILE=$HOME/ <b> + + + PATH TO ESSDAQ + +  </b> /essdaq/detectors/$DETECTOR/config.ini
fi

NOHWARG=""
if [[ $NOHW != "" ]]; then
  NOHWARG="--nohwcheck"
fi

test -f $CONFIG_FILE || errexit "No config file: $CONFIG_FILE"

echo "Extra EFU args: $@"

pushd ../../efu/event-formation-unit/build &> /dev/null || errexit "directory ./event-formation-unit/build does not exist"
  if [ -z "$DEBUG" ]; then
    ./bin/efu --read_config $CONFIG_FILE $UDPARG $NOHWARG -b $KAFKA_IP:9092 -g $GRAFANA_IP --log_file ../../logfile.txt $@ 2>&1 > /dev/null &
  else
    ./bin/efu --read_config $CONFIG_FILE $UDPARG $NOHWARG -b $KAFKA_IP:9092 -g $GRAFANA_IP $@
  fi
popd

Starting the EFU: ./bin/efu: No such file or directory

When starting the EFU with the ./start.sh script and an error like this occurs 

Gd-GEM - applying local configuration
Ubuntu: net.core.netdev_max_backlog
Checking kernel buffer sizes
net.core.rmem_max = 12582912
net.core.rmem_max = 12582912
net.core.wmem_max = 12582912
net.core.netdev_max_backlog = 5000
HW check PASSED
Network check started
Network check PASSED
start Daquiri acquisition
START EFU
Gd-GEM - applying local configuration
Using default essdaq location
Extra EFU args: --file /home/dateuser/VMM/essdaq/detectors/gdgem/RHUM_config.json --calibration /home/dateuser/VMM/essdaq/detectors/gdgem/RHUM_calib.json
~/VMM/essdaq/detectors/gdgem
dateuser@MiniHP-VMM:~/VMM/essdaq/detectors/gdgem$ ../../efu/efu_start.sh: line 39: ./bin/efu: No such file or directory
make sure that the EFU is actually compiled. This means, check that you followed all the instructions under point 9 of the installation procedure. You can check, if the EFU is installed if the file essdaq/efu/event-formation-unit/build/bin/efu exists.

Starting the EFU: exiting ...

When starting the EFU, an error message like this shows up: 

START GDGEM
Gd-GEM - applying local configuration
Ubuntu: net.core.netdev.max_backlog
Error: ethif [enp0s31f6] - MTU is not 9000 bytes
exiting ...
Ensure that all network interfaces are set to an MTU of 9000 (e.g. sudo ifconfig enp0s31f6 mtu 9000)

Starting the EFU: NMXConfig error

When starting the EFU and an error message like this shows up: 

dateuser@MiniHP-VMM:~/VMM/essdaq/detectors/gdgem$ ./start.sh
START GDGEM
Gd-GEM - applying local configuration
Ubuntu: net.core.netdev_max_backlog
Checking kernel buffer sizes
net.core.rmem_max = 12582912
net.core.rmem_max = 12582912
net.core.wmem_max = 12582912
net.core.netdev_max_backlog = 5000
HW check PASSED
Network check started
Network check PASSED
start Daquiri acquisition
START EFU
Gd-GEM - applying local configuration
Using default essdaq location
Extra EFU args: --file /home/dateuser/VMM/essdaq/detectors/gdgem/RHUM_config.json --calibration /home/dateuser/VMM/essdaq/detectors/gdgem/RHUM_calib.json
~/VMM/essdaq/detectors/gdgem
dateuser@MiniHP-VMM:~/VMM/essdaq/detectors/gdgem$ terminate called after throwing an instance of 'std::runtime_error'
what():  NMXConfig error - Invalid JSON file.
please check the JSON files (calibration and configuration)

Starting the EFU: max_backlog ERROR

When starting the EFU and an error message related to something like max_backlog ERROR shows up, do the following: open essdaq/detectors/gdgem/hwcheck.sh and replace
sysctl -a 2>/dev/null | grep net.core.netdev.max_backlog | grep $backlogsize || errexit "max_backlog size incorrect"
with
sysctl -a 2>/dev/null | grep net.core.netdev_max_backlog | grep $backlogsize || errexit "max_backlog size incorrect"
The change is the . to an _ after netdev.

Something similar has to be done in essdaq/config/scripts/hwcheck.sh. Replace 

if [[ -f /etc/lsb-release ]]; then
  BACKLOG=net.core.netdev.max_backlog
  echo "Ubuntu: "$BACKLOG
fi
with 
if [[ -f /etc/lsb-release ]]; then
  BACKLOG=net.core.netdev_max_backlog
  echo "Ubuntu: "$BACKLOG
fi
After performing these changes, reboot!

Starting the EFU in a second instance

When executing the start script for the EFU in a second instance, you get the following message: 

dateuser@MiniHP-VMM:~/VMM/essdaq/detectors/gdgem$ ./start.sh
START GDGEM
Gd-GEM - applying local configuration
Ubuntu: net.core.netdev_max_backlog
Checking kernel buffer sizes
net.core.rmem_max = 12582912
net.core.rmem_max = 12582912
net.core.wmem_max = 12582912
net.core.netdev_max_backlog = 5000
HW check PASSED
Network check started
Network check PASSED
start Daquiri acquisition
<DAQuiri::CommandServer> OK: opening new project and starting acquisition
START EFU
Gd-GEM - applying local configuration
Using default essdaq location
Extra EFU args: --file /home/dateuser/VMM/essdaq/detectors/gdgem/RHUM_config.json --calibration /home/dateuser/VMM/essdaq/detectors/gdgem/RHUM_calib.json --nohwcheck
~/VMM/essdaq/detectors/gdgem

dateuser@MiniHP-VMM:~/VMM/essdaq/detectors/gdgem$ terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
It is important to note that, once the start.sh script for the EFU was executed, the stop.sh script has to be executed before the start.sh is executed again.