Project

General

Profile

Bug #23477

Logfile issues

Added by Iker de Icaza Astiz about 2 months ago. Updated 22 days ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
10/26/2019
Due date:
% Done:

100%

Estimated time:
Duration:

Description

This is similar to an issue I've observed with John and Gennadiy.

Running three different components from daq33 I got the following exception message:

Timeout sending Init transition to artdaq process ptb01 at
sbn-daq01-priv:10100; try checking logfile
sbn-daq01-priv:/log/ptb01-sbn-daq01-10100/ptb01-sbn-daq01-10100-20190604104907-4484.log
for details
Traceback (most recent call last):
  File "/software/products/artdaq_daqinterface/v3_06_02/rc/control/daqinterface.py", line 1940, in do_config
    self.do_command("Init")
  File "/software/products/artdaq_daqinterface/v3_06_02/rc/control/daqinterface.py", line 1301, in do_command
    raise Exception(make_paragraph("An exception was thrown during the %s transition." % (command)))
Exception:
An exception was thrown during the Init transition.

I was running on 2019-10-25, so would have expected a log file with a corresponding name. Note that the log file there is dated 2019-06-04. This file is not accessible from daq33 but it is under daq01. No log file for this component was created at all.

Overall I don't quite follow the logic of how some log files are only accessible on sbn-daq01 and some on sbnd-daq33. ptb is a component connected to sbn-daq01 and sbnd-daq33, and can run in both. Potentially I might be being misled by log files from older versions of the code.

We are running artdaq_daqinterface v3_06_02. This might have been an issue already solved or partially solved.

Sorry that I can't pinpoint a way to reproduce this.


Related issues

Related to artdaq - Feature #23491: artdaq should create logfile subdirectories which are group writeableClosed10/29/2019

History

#1 Updated by Iker de Icaza Astiz about 2 months ago

I've investigated further.

I believe it has all to do with write permissions, it all makes sense now.

Duh!

Can the default behaviour be changed so that new folders and the files therein have group read and write permissions? or some other way which will allow all user to write into a folder?

#2 Updated by Eric Flumerfelt about 2 months ago

On artdaq-demo:bugfix/23477_qms_GroupWriteableDirectories, I have added logic to quick-mrb-start.sh where the daqlogs, daqdata, and run_records directories are created group writeable. I have also added an option to allow users to specify the location of the run_records directory.

#3 Updated by Iker de Icaza Astiz about 2 months ago

Thanks Eric.

Is artdaq-demo used inside here? Maybe I'm just misunderstanding, but I thought that the purpose of the demo was to kickstart a new project.

Currently if I start a run with a new combination of components/partition then a new folder will be created. Looking into our setup I've come across an issue were one of the nodes would not use the preferred /daq/log/ but the older one /log/, and create there folders with wrong permissions. I believe it's due to some issue in our setup, as it doesn't happen in another node.

However I can't work out why this is happening.

#4 Updated by Eric Flumerfelt about 2 months ago

The changes I made were simply caused by our discussion of this issue; they're not specifically trying to resolve your problem, just apply the lessons learned to the wider artdaq ecosystem, as the demo serves as our "best practices" example.

#5 Updated by Iker de Icaza Astiz about 2 months ago

Fair enough, I'll note the couple of commits below for quick reference.

artdaq-demo:bugfix/23477_qms_GroupWriteableDirectories
a0715b0b
db88029b

Feel free to close this issue.

#6 Updated by John Freeman about 2 months ago

The incorrect logfile reference you saw was fixed in Issue #23318, after DAQInterface v3_06_02 was released - given your write permissions issue, what you'd have seen using DAQInterface at the head of its develop branch was DAQInterface issuing an error message saying that it couldn't find the expected logfile. However, we discussed your suggestion at yesterday's artdaq meeting, and it was agreed that when DAQInterface first creates the logfile directories it should make them group-writeable, so that other users in your permissions group who try writing to those directories won't encounter the problems you've encountered in this issue. I'll implement that...

#7 Updated by John Freeman about 2 months ago

  • % Done changed from 0 to 100
  • Status changed from New to Resolved

With commit d893d50a2183175cb5fea6712c239fc9b822c510 on feature/23477_better_directory_access, I've ensured that if the log directory defined in $DAQINTERFACE_SETTINGS doesn't exist, then it's created so that both the user running DAQInterface and the group the user's a member of have write access to the log directory; the same applies to the subdirectories of the log directory in which DAQInterface places run-numbered softlinks to the logfiles produced by artdaq.

However, I noticed that the subdirectories artdaq creates and to which it writes the logfiles do not have group permissions set. I've filed Issue #23491 so that this gets changed.

#8 Updated by Eric Flumerfelt about 2 months ago

  • Related to Feature #23491: artdaq should create logfile subdirectories which are group writeable added

#9 Updated by Eric Flumerfelt 25 days ago

  • Status changed from Resolved to Reviewed
  • Co-Assignees Eric Flumerfelt, John Freeman, Ron Rechenmacher added

I've code-reviewed the DAQInterface change, it looks good.

#10 Updated by Eric Flumerfelt 22 days ago

  • Target version set to artdaq_demo v3_07_00
  • Status changed from Reviewed to Closed
  • Project changed from artdaq Utilities to artdaq Demo
  • Experiment deleted (SBND)


Also available in: Atom PDF