Fermi Redmine: System Outage w/ Lost data last week

Added by Marc Mengel 5 months ago

The system outage last Thursday was caused by a database server crash; to get things running they put up the last backup, which means Fermi Redmine is currently missing any changes from about 5:00PM Feb 5 through 12:00 noon Feb 6.

We are working on recovering the missing data;It appears to be
73 wiki pages
43 issue updates
18 attachment uploads

-- Update:

so far I've recovered the changes for issue trackers, and project wiki's 'A' through 'N'
I have another 30 wiki pages to check, and then 11 attachments to re-attach...

Facilities Integration: Meeting Summary 10/01/2019

Added by Josh Juneau 9 months ago

There was no main topic for discussion at this meeting, as this was the first meeting since the GIS Manager had left the laboratory. Therefore, there was much discussion around the GIS. Josh Kenney expressed the value of GIS in that it must be moved forward. He mentioned that the utility maps are very important, and updating this data is essential.

Kevin has been working on SURF locations, SSP Phase 4, and FIMS data.

Sabina has been working on metrics, VFA, and other related data.

Albert has been spending time repairing old links and updating the FESS website.

Josh Kenney expressed concern on who is taking over control of software such as Leica. Also, who manages and takes care of the BlueBeam software, which is used by Engineering.

Josh Juneau has been working on development of the SSP Phase 4 and Engineering Task Review applications. Josh is working on migrating the GIS environment over to the Windows 2016 based ArcGIS 10.7 infrastructure, and resolving requests from users. Also working on the FAMIS 12c Forms and Reports migration, and at this time a couple of more testers have been added (Maredith and Marcel).

Facilities Integration: Meeting Summary 08/08/2019

Added by Josh Juneau 11 months ago

Attendees: Sabina Bellino, Kevin Coppert, Josh Juneau, Josh Kenney, Karen Kosky

This week we were happy to have Josh Kenney representing the FESS Engineering department. We had a great discussion regarding FESS Engineering software, processes, and future directions. The Engineering group is currently investigating the use of Service Now for intake of Fermilab project requests, and they may be seeking the use of another project management software as well. There was also discussion around the idea of possibly seeking out an estimating software at some point in the future for the Engineering department, and having an online version of the ICE form rather than using spreadsheets. I will reach out to Michele McCusker from AD to find out if there have been any plans for an online ICE form and communicate back to the group.

Karen attended and we reviewed the Software/Request form with her. She likes it and would be happy to disseminate the information to FESS. I will work with the group to write up a short article around the group’s charge and introduce the Software/Request form. Karen plans to use that article to send the information to other department heads in FESS. We also had discussed the overall role of the group, and determined that we should not be policing software purchases or initiatives, but rather, we should be helping to make recommendations and suggestions for new software or solution intentions so that we can help guide towards better integration and less replication of functionality. Lastly, Karen would like for us to discuss and put together a short article to help others at the lab learn about how they can make use of our FESS online systems. This article may include information regarding submitting work order tickets online, searching for open tickets or active equipment, seeking out information from GIS, and so forth. We should discuss this topic in our next meeting.

Rather than having a separate meeting with FESS management and the entire FIT group, we should probably have at least one FIT group member attend the 8:30 management meeting once a month…possibly even more often. Let’s discuss this in an email or at the next meeting so that we can continue to work on informing management of the FIT group initiatives.

Other important information:

- This morning the T&M and FESS Motorpool applications were upgraded to Oracle 12 and Payara 5. Updated URLs will be posted to the website.
- The intention is to have all new T&M Requisitions go through the online requisition system in the near future.
- I would like to change the day and time of the FIT meeting to something that everyone else finds more useful…even the location of the meeting. I’ll put together an email that we can use to help determine the best day/time/location.

Facilities Integration: Meeting Summary 06/13/2019

Added by Josh Juneau about 1 year ago

In the FIT Meeting for 06/13/2019, Kevin provided a nice tour of the Self Service Property application. Prior to the FIT meeting, we met with Site Services and FESS Management as a team. There was a large discussion regarding the upcoming requirement to update RPVs on OSFs, and also how we are going to proceed with capturing condition assessments.

Facilities Integration: Meeting Summary 05/30/2019

Added by Josh Juneau about 1 year ago

It has been nearly two weeks since our last meeting, and I apologize for the late posting. Current events discussed on 05/30/2019:

- Self Service Property 3.2 work continues
- No eLog decisions have been made, as yet, from the AD with respect to providing FESS with an eLog account
- FAMIS 12c forms/reports still in testing, AppTree Connectors being installed to new servers FESS-OWLD and FESS-OWLP
- FID Validation preparation

During this meeting, Bridgett presented the eJulie application.

Thanks to everyone who was able to attend.

Facilities Integration: Meeting Summary 05/02/2019

Added by Josh Juneau about 1 year ago

During this meeting, we went around the table to converse about the latest items/topics. Items to note:

- Cedric had mentioned that they were keen on obtaining photos and Dawn said she'd be able to help out in that area.
- Database patching that will be occurring over the next several weeks.
- Self Service Property 3.2 testing and development underway
- Pat has been working lots with the latest round of AppTree updates and troubleshooting the application crashes.

Dawn presented on some slides that she had shown at the Facility Manager's meeting with respect to some of the tools available within the ArcGIS applications.

Thanks for attending.

Facilities Integration: Meeting Summary 04/18/2019

Added by Josh Juneau about 1 year ago

This was the first meeting where we went around the table to discuss what was happening in our areas of Fermi. Once we made it around the room, it became apparent that we need to keep focused on “Integration” topics, as our summaries took about 90% of the meeting time. This is a very good practice and I think we need to continue doing it each meeting so that we can all keep aware of what is occurring throughout the section. I’ll do my best to shorten my list to only those items that are integration related so that we can make it around the room and still have time for other meeting agenda items!

I provided an update on the eLog request, as it is still in the hands of AD at this time. I have been in contact with the developer and he is requesting from the AD Operations Management team that FESS is provided a separate log. I will keep this group, and the FM Group, posted with the latest details.

The Software/Solution request form is now in production. We haven’t yet publicized it, and I’ll discuss with management and keep the group updated. It is certainly apparent that we need to have fully documented requests so that we know the exact intentions of the requestor and the purpose of the requested solution. I don’t think we can capture everything with this form. For instance, Larry’s follow-up email on the eLog request provided a great deal of insight into the process. I do not believe an online form will help us to capture that detail. So with that said, I think we all have agreed that the form is a good starting point.

I briefly mentioned the “Predictive Solutions” software, as this will likely be coming to FIT for review in a future meeting. Stay tuned for more information and details on this software.

Lastly, I will follow up with management on bringing someone from FESS Engineering into the group. I think we need to have a representative from that department to keep us informed and to help collaborate on integration tactics.

Facilities Integration: Meeting Summary - 04/04/2019

Added by Josh Juneau over 1 year ago

Thanks for meeting today. This week we briefly reviewed the overall intentions of the group, reviewed an incoming Software and Solution Request, and Cedric provided us with a great presentation on the Metasys system.

Beginning with the next meeting, we will start to go around the table so that each of us can share any current initiatives that may be worth mentioning so that the group can better collaborate on solutions for the section. This is something that we haven’t really done as part of this meeting yet, and I think we are at a point where it makes sense to start.

We reviewed Larry Sliwa’s request for an ELog system. The ELog initiative is to create a solution similar to the AD electronic log ( that will allow a few operators to log in and post details regarding tasks, photos, and other information to keep everyone current on operations 24/7. There are a few pieces of software that FESS already has in-place or has the ability to use, which may be suitable for this request:

  • FAMIS and AppTree Mobile Priority 0 Work Orders
  • Survey 123
  • Look into possibly extending the AD solution to provide FESS with a separate log

After reviewing, the Facilities Integration Team collaboratively agrees that the AppTree mobile application paired with a Priority 0 work order ticket may be the solution. When a priority 0 ticket is created (using the app or the FAMIS desktop solution), it can be displayed on the web immediately using the “Zero Priority WO Search” ( We will have to add the ability to view attached pictures/documents inline without clicking on the work order, but this solution should work with minimal effort. I am planning to respond to Larry to let him know that this is the recommended approach from the team. I’ll wait until next week to respond to Larry with the recommendation so that everyone in this group has time to respond, if necessary. I will copy in the response.

Finally, Cedric provided an overview of the Metasys system. I want to thank Cedric for taking the time to do so, as it was very informative.

Facilities Integration: Meeting Summary - 03/21/2019

Added by Josh Juneau over 1 year ago

Thanks for attending the meeting yesterday. We reviewed the charter once again, and took a look at some slight modifications that have been made with respect to the intent of the group. Essentially we added one of our main purposes, as follows:

“Collaboratively focus on facilities data initiatives, seeking integration improvement opportunities”

I had the opportunity to meet with Joe, Karen, and Mark yesterday after the meeting and we discussed the charter. It was well received.

We also continued our review of the FESS Software and Applications, looking more into the FID and CAIS systems. I thought that the CAIS review was especially useful, thanks to Steve for demonstrating parts of that system. Next time we will continue these system reviews.

Lastly, I added an additional field to the “Software\Solution Request Form” entitled, “Current Procedure”. I have placed the form into the development environment and it can be viewed at the following link:


Also available in: Atom