UPDATE 2019-03-12: These instructions are obsolete and WILL NOT WORK. Contact FIFE support if you need a new service certificate (i.e. not a replacement for an expiring cert).

How to get a service cert to automate processes

Users are supposed to do this themselves so they're the owner of the cert and agree to the terms etc.
USDC will make production certs on request of the experiment production groups.

Generating the cert

  1. Go to
    If you don't have a certificate loaded in your browser you can just hit "Cancel" at the pop-up and fill out the user information.
    • Click on the link How can I generate a CSR? to see a reference command under "If you want to request a service certificate..." .
  2. cd to a working directory where you will create and test this key, the cert that will be issued, and the proxy generated using these. The system should be able to run vom-proxy-init and voms-proxy-info.
  3. Run the sample command, with modified DN and output file, to generate the key file to be used later with the issued cert file.
    • typically work in the ${PROJ}gpvm:/opt/${PROJ}pro directory.
      # Remember to escape any non-leading / with \/ as shown.
      umask 077
      SUBJ="/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=production\/${PROJ}" 
      openssl req -new -newkey rsa:2048 -nodes -keyout ${PROJ}pro.key -subj "${SUBJ}" 
  4. Cut and paste the encoded output from the above command line into the box. Include the BEGIN and END lines
  5. Choose "Fermilab" for the approving VO.
  6. Accept the terms
  7. Submit
  8. After the request gets approved and you'll get a web page to go to.
    • Click on Issue Certificates, to get a Download PEM option.
    • Click on Download PEM.
  9. Move this file to your working directory that already has mu2epro.key.
  10. Rename this file from its long name to something like ${PROJ}pro.cert
    mv mu2epro.cert
  11. Make sure at least the "key" file is only readable by the owner (chmod 400 <filename>) or most commands will refuse to use it
  12. Once you download the certificate, check the DN and expiration with
    openssl x509 -in mu2epro.cert -noout -subject -dates

Registering in VOMS

Before using the cert to generate a proxy you need to register it with the experiment VO.
That can be done by opening a "Scientific Computing Request" "Batch Submission" in Service Now,
with suggested short description "Please add service certificate to <EXP> VO",
and suggested long message:
"We need to add this to the <EXP> VO with a production role: Subject: <cut and paste the DN here from the output of the openssl commands above>"

USDC will will use VOMSadmin to add the cert to its owner, and give that person the needed role.
See Experiment service certs - VOMS Registration


To test the cert for grid submission you could run a command like the below substituting the appropriate experiment for mu2e in this example.

voms-proxy-init -key ./${PROJ}pro.key -cert ./${PROJ}pro.cert -valid 48:0 -voms fermilab:/fermilab/${PROJ}/Role=Production -out ${PROJ}pro.proxy

If you get an error saying "User unknown to this VO" it means that the DN of your certificate (the DN is in the openssl -subject output from above) is not listed as a member of the VO you specified in the "-voms" option to voms-proxy-init. You need to open up the Service Desk ticket noted above or if you did it wasn't entered correctly.

You can look at the generated proxy with something like

ls -l ${PROJ}pro.proxy
voms-proxy-info -all -file ./${PROJ}pro.proxy

  • This certificate does have an expiration which you can see with the "openssl .... -enddate" command above. You need to remember to request a new one before this one expires or your submissions will fail.
    • OSG will remind the requester by email about a month ahead of expiration
    • The rexbatch@fifebatch1 checkcerts weekly script sendsmail to fermigrid-help a month ahead of expiration.


Job submission is normally via jobsub_client using user kerberos authentication.

Production jobs are an exception, using the production cert for access to jobsub from a restricted shared account.
The cert and key files are distributed from fifeutilgpvm01 and installed in protected directories like /opt/mu2epro.
ECF/SSI will make the directories on request.


For general information about certificates from the Fermilab security team click HERE

Service Certificate Management

If you're interested in having SCD manage your experiment's service certificates for you, you'll need to open a Service Desk request. Do the following:

1, go to
2. Select “Scientific Computing Services” from the menu on the left sidebar.
3. Click on the "Distributed Computing" service area.
4. Select "Submit a request to service providers"
5. Select your VO/experiment
6. Fill the fields “Short description" with something like "manage service certificates for "your experiment"
7. In the details, specify each account and Role combination that you want managed. For example it might be something like

experimentpro Production
experimentcal Calibration

8. If needed add some people in the “watch list”.
9. In case of errors with the service, we email certain people from the experiment. We will assume that the person filing the ticket is the person to notify, unless we're told otherwise.
10. Finally click on “Submit” to submit the request.

Requirements for experiments to use the service

We only offer the service for service accounts that are abiding by cs-docdb 5644 and its addendums.

These requirements include, but are not limited to:

  • No more than ten people from the experiment are allowed in the .k5login file at any time. USDC and OPOS members do not count against the limit.
  • The cert and key files should never be sent over a network via any unencrypted means, especially as email attachments.
  • NOTE: experiments can request exceptions and if granted are detailed in an addendum document. At this date (2/21/2017) NOVA is the only experiment with an approved exception.
  • Common errors with Service Certificate Management service
  • When people log into shared accounts (<expt>pro, for example) and use the managed proxy for job submission or copying files, they must make sure their kerberos credentials have been forwarded to that session (run klist -c to check the default principal). Not doing so violates Fermilab security policy, and penalties may be imposed.