Project

General

Profile

Task #22894

Feature #22889: Build spack_to_ups converter

Initial coding

Added by Marc Mengel 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Start date:
07/10/2019
Due date:
% Done:

60%

Estimated time:
Duration:

History

#1 Updated by Marc Mengel 2 months ago

So at the moment, pursuing this as actually two module types: ups_table and ups_version to write the version and table files respectively.

Currently on the ups_gen branch of the spack repository in this project.

#2 Updated by Marc Mengel 2 months ago

So currently, duplicated the dotkit modules files for ups_table and ups_version, and added them to the list of module types, but it blows up partway there.

So Trying to figure out how it all gets called, I injected an exception and a stack trace print into the code where it loads the jinja templates for the existing module types, and did an install, it looks like:

  File "/scratch/mengel/spack/bin/spack", line 48, in <module>
    sys.exit(spack.main.main())
  File "/home/mengel/spack/lib/spack/spack/main.py", line 696, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/mengel/spack/lib/spack/spack/main.py", line 446, in _invoke_command
    return_val = command(parser, args)
  File "/home/mengel/spack/lib/spack/spack/cmd/install.py", line 330, in install
    install_spec(args, kwargs, abstract, concrete)
  File "/home/mengel/spack/lib/spack/spack/cmd/install.py", line 205, in install_spec
    install(spec, kwargs)
  File "/home/mengel/spack/lib/spack/spack/cmd/install.py", line 194, in install
    spec.package.do_install(**kwargs)
  File "/home/mengel/spack/lib/spack/spack/package.py", line 1618, in do_install
    self, build_process, dirty=dirty, fake=fake)
  File "/home/mengel/spack/lib/spack/spack/build_environment.py", line 810, in fork
    p.start()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib64/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mengel/spack/lib/spack/spack/build_environment.py", line 765, in child_process
    return_value = function()
  File "/home/mengel/spack/lib/spack/spack/package.py", line 1578, in build_process
    spack.hooks.post_install(self.spec)
  File "/home/mengel/spack/lib/spack/spack/hooks/__init__.py", line 54, in __call__
    hook(*args, **kwargs)
  File "/home/mengel/spack/lib/spack/spack/hooks/module_file_generation.py", line 29, in <lambda>
    post_install = lambda spec: _for_each_enabled(spec, 'write')
  File "/home/mengel/spack/lib/spack/spack/hooks/module_file_generation.py", line 22, in _for_each_enabled
    getattr(generator, method_name)()
  File "/home/mengel/spack/lib/spack/spack/modules/common.py", line 726, in write
    traceback.print_stack()
  File "/scratch/mengel/spack/bin/spack", line 48, in <module>
    sys.exit(spack.main.main())
  File "/home/mengel/spack/lib/spack/spack/main.py", line 696, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/mengel/spack/lib/spack/spack/main.py", line 446, in _invoke_command
    return_val = command(parser, args)
  File "/home/mengel/spack/lib/spack/spack/cmd/install.py", line 330, in install
    install_spec(args, kwargs, abstract, concrete)
  File "/home/mengel/spack/lib/spack/spack/cmd/install.py", line 205, in install_spec
    install(spec, kwargs)
  File "/home/mengel/spack/lib/spack/spack/cmd/install.py", line 194, in install
    spec.package.do_install(**kwargs)
  File "/home/mengel/spack/lib/spack/spack/package.py", line 1618, in do_install
    self, build_process, dirty=dirty, fake=fake)
  File "/home/mengel/spack/lib/spack/spack/build_environment.py", line 810, in fork
    p.start()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib64/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mengel/spack/lib/spack/spack/build_environment.py", line 765, in child_process
    return_value = function()
  File "/home/mengel/spack/lib/spack/spack/package.py", line 1578, in build_process
    spack.hooks.post_install(self.spec)
  File "/home/mengel/spack/lib/spack/spack/hooks/__init__.py", line 54, in __call__
    hook(*args, **kwargs)
  File "/home/mengel/spack/lib/spack/spack/hooks/module_file_generation.py", line 29, in <lambda>
    post_install = lambda spec: _for_each_enabled(spec, 'write')
  File "/home/mengel/spack/lib/spack/spack/hooks/module_file_generation.py", line 22, in _for_each_enabled
    getattr(generator, method_name)()
  File "/home/mengel/spack/lib/spack/spack/modules/common.py", line 726, in write
    traceback.print_stack()

#3 Updated by Marc Mengel 2 months ago

  • Tracker changed from Feature to Task

#4 Updated by Marc Mengel about 2 months ago

  • % Done changed from 30 to 60

Okay, so now we generate table and version files, getting something resembling a UPS database.
https://cdcvs.fnal.gov/redmine/projects/spack-infrastructure/repository/spack/diff?utf8=%E2%9C%93&rev=72a1368fc866b990a1bde11c617d79897d4e252f&rev_to=030f0951d4ff888766145667a7d76dc1c21cfa88

So remaining bits to get right are:

  • generating ups flavor and qualifiers from spack info
  • finding dependencies, as they don't normally put dependencies in their module files, rather they want you to use "spack load -r" to do recursive setups.

#5 Updated by Marc Mengel about 2 months ago

Some more progress, flavor translation in version file but need the same for dependencies.
after lookining in spec._cmp_key_cache to find out specifics.

Also need suitable qualifiers in table file(?) and/or table file per instance

commit:be5ba62



Also available in: Atom PDF