Project

General

Profile

Bug #23465

Nested transaction SQLite when inserting many FileCatalogMetadata entries

Added by Kyle Knoepfel 11 months ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Start date:
10/23/2019
Due date:
% Done:

100%

Estimated time:
2.00 h
Scope:
Internal
Experiment:
MicroBooNE
Duration:

Description

Herb Greenlee from MicroBooNE reports:

Hello Kyle,

Problem is that cet::sqlite::Connection::flush_no_throw opens its own
transaction:

template <std::size_t NColumns, typename Row>
int
Connection::flush_no_throw(std::vector<Row> const& buffer,
sqlite3_stmt*& insertStmt) {
// Guard against concurrent updates to the same database.
hep::concurrency::RecursiveMutexSentry sentry{*mutex_, func};
sqlite::Transaction txn{db_};

Therefore, cet::sqlite::Ntuple<>::insert can't be safely called when a
Transaction is already open.

I also dug a little into why the metadata was so large. Turns out there
are many replications of {"mc.pot": "0"} in sparse data files. I think
we'll have to do something about that ourselves (rather than waiting for a
new version of art...).

History

#1 Updated by Kyle Knoepfel 11 months ago

  • Target version set to 1.01.04
  • Status changed from Resolved to Closed

The solution to this problem was to avoid creating the nested transaction in the RootOutputFile class when writing the user-supplied FileCatalogMetadata entries to disk.

Implemented with commit art_root_io:19b8662.



Also available in: Atom PDF