Project

General

Profile

Support #18080

Slow build on mac Sierra

Added by Bruce Baller about 2 years ago. Updated about 2 years ago.

Status:
Assigned
Priority:
High
Assignee:
Christopher Green
Target version:
-
Start date:
10/31/2017
Due date:
% Done:

0%

Estimated time:
1.00 h
Spent time:
5.50 h
Scope:
Experiment:
-
SSI Package:
Co-Assignees:
Duration:

Description

Code development on mac Sierra is seriously hampered by the slow build time. The last stage of the build, Installing files (in lareco), takes over two minutes. This is summarized here:

ninja install
<wait minutes while all files are installed>
ninja install
<wait minutes while all files are installed>

The installation phase on a uboonegpvm machine only takes a few seconds.

ninja install
<wait many minutes while all files are installed>
ninja install
<wait a few seconds to check that files are Up-to-date>

The slow build can result in a loss of an hour of productivity per day when one is building many times. This problem was not obvious on mac Yosemite.

I suspect that the TimesDiffer function in cmFIleComparison.cxx may be responsible.

Associated revisions

Revision 540735b3 (diff)
Added by Christopher Green about 2 years ago

Solve slow installs on OS X, per issue #18080.

History

#1 Updated by Gianluca Petrillo about 2 years ago

The analysis that Bruce verbally reported to me boils down to two points:

  1. on OSX, CMake uses install_name_tool to install libraries at installation time (which happens long after build time); this utility rewrites rpath's and therefore changes the file on the installation side, leaving it at least a few seconds newer than the one, untouched, in the build directory
  2. CMake performs the actual installation of the file (cmFileCopier::InstallFile() in cmFileCommand.cxx) only if the destination file timestamp does not differ from the source one by more than one second (cmFileTimeComparison::FileTimesDiffer() in cmFileTimeComparison.h)

A crazy workaround: have cet_install_files touch the source file (and maybe also the local copy by cet_copy()) immediately after install(). Not clear to me which side effects would happen.

#2 Updated by Gianluca Petrillo about 2 years ago

For reference: using CMake v3_9_0 and cetbuildtools v5_14_00 from UPS.

#3 Updated by Lynn Garren about 2 years ago

  • Project changed from LArSoft to cet-is

#4 Updated by Kyle Knoepfel about 2 years ago

  • Tracker changed from Bug to Support
  • Status changed from New to Assigned
  • Assignee set to Christopher Green
  • Estimated time set to 1.00 h

First, is there a reason that you need to run the install step every time? In general, it is sufficient to just build and run the tests without using the install command.

Second, we will attempt to ascertain what causes the performance issue. We suspect it is not an mrb or cetbuildtools problem, but that it is a CMake issue.

#5 Updated by Christopher Green about 2 years ago

  • Status changed from Assigned to Feedback

As a temporary workaround, please place the following directive after every call to cet_cmake_env():

cmake_policy(SET CMP0042 NEW)
(e.g. the top-level CMakeLists.txt files for each package in your MRB set).

After the next time you do a ninja install, the system should behave as expected in the face of no updates to the library.

Please let us know if there are any other issues with the libraries as produced in this way.

Newer versions of cetbuildtools will remove the need for this workaround.

#6 Updated by Christopher Green about 2 years ago

  • Status changed from Feedback to Resolved
  • Target version set to 2.09.02
  • % Done changed from 0 to 100

Committed with cetbuildtools:540735b, and tagged for release with cetbuildtools versions v5_14_04 and v6_00_02.

#7 Updated by Lynn Garren about 2 years ago

cetbuildtools v5_14_04 will be used in the upcoming larsoft release.

#8 Updated by Christopher Green about 2 years ago

  • Status changed from Resolved to Assigned

Investigation has so far ascertained that for a given mrb build (the art suite), a handful of libraries behave properly on multiple installs; most do not.

More investigation is needed to tie down the root cause.

#9 Updated by Christopher Green about 2 years ago

  • % Done changed from 100 to 0

With further investigation, I have ascertained that this is an MacOS-specific attribute of our CMake-builds and installation of libraries. I have attempted a couple of "simple" possible ways to change the behavior, such as by defining CMAKE_INSTALL_RPATH_USE_LINK_PATH=ON, without success. What's left is an open-ended investigation of the interaction between RPATH settings and directives CMake config files that has no guarantee of a successful outcome.

I would urge you to look at whether your development workflow can be adjusted usefully to avoid needing multiple installation steps. If there is some attribute of cetbuildtools or mrb that is preventing that, please let us know so we can investigate that.

#10 Updated by Kyle Knoepfel about 2 years ago

  • Target version deleted (2.09.02)

#11 Updated by Christopher Green about 2 years ago

  • Target version set to 2.09.02

For the record, I have also removed the setting of CMAKE_SKIP_RPATH and replaced it with:

set(CMAKE_INSTALL_RPATH_USE_LINK_PATH ON)
but without good effect.

#12 Updated by Kyle Knoepfel about 2 years ago

  • Target version deleted (2.09.02)

#13 Updated by Ben Morgan about 2 years ago

I've had a quick look at this based on information from Chris Green, and think I've tracked down the cause to be CMake's handling of rpaths/install names on install. It's pretty much as reported by Gianluca/Bruce:

  1. The relevant CMake commands can bee seen the in the cmake_install.cmake file located in the binary directory where the target is installed.
  2. On both Linux/macOS, CMake will use file(INSTALL ...) to install the binary.
    1. On Linux it then uses file(CHANGE_RPATH ...) to change the build to install rpaths. This modifies the installed binary, but restores its modification time, and hence re-running install will not trigger file(INSTALL ...)
    2. On macOS it then uses execute_process(COMMAND install_name_tool ...) to change the build to install rpaths. As there's no restoration of the modification time, subsequent invocations of installation will trigger the preceding file(INSTALL ...)

The use of install_name_tool at install time can be suppressed by setting CMAKE_BUILD_WITH_INSTALL_RPATH to ON (edit: corrected the required value!). However, this almost certainly has consequences for development/testing as it may result it tests either not being able to find the needed libraries, or worse, finding the wrong ones. It's also incompatible with SIP as (I think) there's no way other than a messy mix of build/install area @DYLD_LIBRARY_PATH@s to specify which library to use.

I'd also agree with Kyle's comment that the use case for repeated invocations of install should be laid out. For development and running tests, it's usually never necessary and CMake's default RPATH behaviour tends to work well, at least in my experience.

Nevertheless, I think it'd be worth raising an issue on CMake's issue tracker to query the difference in behaviour between Linux/macOS. It should be possible to replicate it by moving the call to install_name_tool into a subcommand of file(), with a post modification update of the modification time as on Linux. However, there may be other technical reasons for doing it the existing way that I'm not aware of.

#14 Updated by Ben Morgan about 2 years ago

One other useful CMake variable to consider if frequent repeated installs are required is CMAKE_INSTALL_MESSAGE . Set this to LAZY and it'll suppress any "Up-to-date" messages, which is helpful in seeing issues in the install step.

#15 Updated by Kyle Knoepfel about 2 years ago

Ben, thank you for taking the time to look at this. We appreciate your extra set of eyes looking things over and the expertise you lend.

Also available in: Atom PDF