Support #18080
Slow build on mac Sierra
Description
Code development on mac Sierra is seriously hampered by the slow build time. The last stage of the build, Installing files (in lareco), takes over two minutes. This is summarized here:
ninja install
<wait minutes while all files are installed>
ninja install
<wait minutes while all files are installed>
The installation phase on a uboonegpvm machine only takes a few seconds.
ninja install
<wait many minutes while all files are installed>
ninja install
<wait a few seconds to check that files are Up-to-date>
The slow build can result in a loss of an hour of productivity per day when one is building many times. This problem was not obvious on mac Yosemite.
I suspect that the TimesDiffer function in cmFIleComparison.cxx may be responsible.
Associated revisions
History
#1 Updated by Gianluca Petrillo about 3 years ago
The analysis that Bruce verbally reported to me boils down to two points:
- on OSX, CMake uses
install_name_tool
to install libraries at installation time (which happens long after build time); this utility rewrites rpath's and therefore changes the file on the installation side, leaving it at least a few seconds newer than the one, untouched, in the build directory - CMake performs the actual installation of the file (
cmFileCopier::InstallFile()
incmFileCommand.cxx
) only if the destination file timestamp does not differ from the source one by more than one second (cmFileTimeComparison::FileTimesDiffer()
incmFileTimeComparison.h
)
A crazy workaround: have cet_install_files
touch
the source file (and maybe also the local copy by cet_copy()
) immediately after install()
. Not clear to me which side effects would happen.
#2 Updated by Gianluca Petrillo about 3 years ago
For reference: using CMake v3_9_0
and cetbuildtools
v5_14_00
from UPS.
#3 Updated by Lynn Garren about 3 years ago
- Project changed from LArSoft to cet-is
#4 Updated by Kyle Knoepfel about 3 years ago
- Tracker changed from Bug to Support
- Status changed from New to Assigned
- Assignee set to Christopher Green
- Estimated time set to 1.00 h
First, is there a reason that you need to run the install step every time? In general, it is sufficient to just build and run the tests without using the install command.
Second, we will attempt to ascertain what causes the performance issue. We suspect it is not an mrb
or cetbuildtools
problem, but that it is a CMake issue.
#5 Updated by Christopher Green about 3 years ago
- Status changed from Assigned to Feedback
As a temporary workaround, please place the following directive after every call to cet_cmake_env()
:
cmake_policy(SET CMP0042 NEW)(e.g. the top-level
CMakeLists.txt
files for each package in your MRB set).
After the next time you do a ninja install
, the system should behave as expected in the face of no updates to the library.
Please let us know if there are any other issues with the libraries as produced in this way.
Newer versions of cetbuildtools will remove the need for this workaround.
#6 Updated by Christopher Green about 3 years ago
- Status changed from Feedback to Resolved
- Target version set to 2.09.02
- % Done changed from 0 to 100
Committed with cetbuildtools:540735b, and tagged for release with cetbuildtools versions v5_14_04 and v6_00_02.
#7 Updated by Lynn Garren about 3 years ago
cetbuildtools v5_14_04 will be used in the upcoming larsoft release.
#8 Updated by Christopher Green about 3 years ago
- Status changed from Resolved to Assigned
Investigation has so far ascertained that for a given mrb
build (the art suite), a handful of libraries behave properly on multiple installs; most do not.
More investigation is needed to tie down the root cause.
#9 Updated by Christopher Green about 3 years ago
- % Done changed from 100 to 0
With further investigation, I have ascertained that this is an MacOS-specific attribute of our CMake-builds and installation of libraries. I have attempted a couple of "simple" possible ways to change the behavior, such as by defining CMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
, without success. What's left is an open-ended investigation of the interaction between RPATH
settings and directives CMake config files that has no guarantee of a successful outcome.
I would urge you to look at whether your development workflow can be adjusted usefully to avoid needing multiple installation steps. If there is some attribute of cetbuildtools or mrb that is preventing that, please let us know so we can investigate that.
#10 Updated by Kyle Knoepfel about 3 years ago
- Target version deleted (
2.09.02)
#11 Updated by Christopher Green about 3 years ago
- Target version set to 2.09.02
For the record, I have also removed the setting of CMAKE_SKIP_RPATH
and replaced it with:
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH ON)but without good effect.
#12 Updated by Kyle Knoepfel about 3 years ago
- Target version deleted (
2.09.02)
#13 Updated by Ben Morgan about 3 years ago
I've had a quick look at this based on information from Chris Green, and think I've tracked down the cause to be CMake's handling of rpaths/install names on install. It's pretty much as reported by Gianluca/Bruce:
- The relevant CMake commands can bee seen the in the
cmake_install.cmake
file located in the binary directory where the target is installed. - On both Linux/macOS, CMake will use
file(INSTALL ...)
to install the binary.- On Linux it then uses
file(CHANGE_RPATH ...)
to change the build to install rpaths. This modifies the installed binary, but restores its modification time, and hence re-running install will not triggerfile(INSTALL ...)
- On macOS it then uses
execute_process(COMMAND install_name_tool ...)
to change the build to install rpaths. As there's no restoration of the modification time, subsequent invocations of installation will trigger the precedingfile(INSTALL ...)
- On Linux it then uses
The use of install_name_tool
at install time can be suppressed by setting CMAKE_BUILD_WITH_INSTALL_RPATH
to ON
(edit: corrected the required value!). However, this almost certainly has consequences for development/testing as it may result it tests either not being able to find the needed libraries, or worse, finding the wrong ones. It's also incompatible with SIP as (I think) there's no way other than a messy mix of build/install area @DYLD_LIBRARY_PATH@s to specify which library to use.
I'd also agree with Kyle's comment that the use case for repeated invocations of install
should be laid out. For development and running tests, it's usually never necessary and CMake's default RPATH behaviour tends to work well, at least in my experience.
Nevertheless, I think it'd be worth raising an issue on CMake's issue tracker to query the difference in behaviour between Linux/macOS. It should be possible to replicate it by moving the call to install_name_tool
into a subcommand of file()
, with a post modification update of the modification time as on Linux. However, there may be other technical reasons for doing it the existing way that I'm not aware of.
#14 Updated by Ben Morgan about 3 years ago
One other useful CMake variable to consider if frequent repeated installs are required is CMAKE_INSTALL_MESSAGE
. Set this to LAZY
and it'll suppress any "Up-to-date" messages, which is helpful in seeing issues in the install step.
#15 Updated by Kyle Knoepfel about 3 years ago
Ben, thank you for taking the time to look at this. We appreciate your extra set of eyes looking things over and the expertise you lend.
Solve slow installs on OS X, per issue #18080.