Reduce number of queries to collector
CMS did a study highlighting how the frontend is responsible of about 100 queries per minute on the collector: https://docs.google.com/presentation/d/13C8IVjcjXacJZPPPNWtag6AQI8K34OoD1mx-Cg7ElwI/edit?usp=sharing
This has an impact on the collector performances. The estimation is a decrease in 10% of the collector duty cycle if the queries are stopped/redirected to the backup collector.
Most of the queries seems they are querying information about schedds (to follow up with a condor_q). We should cache these information since schedd info rarely change.
#1 Updated by Marco Mascheroni 3 months ago
I developed a patch to cache the schedd queries. I tested it in the CMS ITB frontend and immediately after I applied (at 16:57) the number of queries dropped from ~2000 to ~300. See the screenshot attached.
#2 Updated by Marco Mascheroni 3 months ago
- Assignee changed from Marco Mascheroni to Marco Mambelli
- Status changed from New to Feedback
FYI: The above screenshot is taken from https://monit-grafana.cern.ch/d/mX_T4JQWk/si_collectors_test?orgId=11&refresh=15m&from=now-6h&to=now&fullscreen&panelId=14
Current changes are in v36/24127.