Project

General

Profile

Feature #15600

Ideas for erlang front-end monitoring.

Added by Dennis Nicklaus almost 3 years ago. Updated 10 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
ACSys/FE Framework
Target version:
-
Start date:
02/20/2017
Due date:
% Done:

0%

Estimated time:
Duration:

Description

Things we've learned from using Wombat:

  • We need to monitor the memory usage of our process as a whole
  • * should be easily done through Acnet, but our U:EMEtnn devices aren't very optimized (acsys-fe/system_driver.erl) (0th element is total available, can't select element from SSDN, so makes alarming useless.)
  • We also need to monitor Process Queue size
  • * Not implemented now, but easily done through Acnet.
  • Collecting all info/warn/error messages from all frontends in one easily viewed place is useful.
  • * Is this something we can figure out for ourselves also?
  • It does a good job of reminding us when we get different s/w versions on different front-ends and is a good thing to track.
  • * Again, perhaps we can investigate this. Maybe a non-continuous monitor that we can run?
  • Overall CPU usage shows some interesting patterns, but I think there are cheaper tools for collecting that info, with wider implications across all clx-es, not just the front-end ones.
  • A lot of the items in Wombat metrics are non-changing for us. The I/O Metrics (e.g., TCP or UDP bytes/packets sent) could be interesting maybe, but aren't really quantitatively useful for now, more of a curiosity.
  • The erlang system_driver collects base number of processes, but we could maybe instrument more to measure Acnet activity.
  • We need nicer synoptic pages or specific app. for various diag devices we have.

History

#1 Updated by Richard Neswold over 1 year ago

  • Category set to ACSys/FE Framework

Set category field.

#2 Updated by Richard Neswold 10 months ago

As demo-ed in a recent group meeting, the Observer tool can be used to attach to and monitor an ACSys front-end running in a remote node. The front-end only needs to include the runtime_tools application in its tarball.

As an example, accessing an ACSys front-end running on CLX20 from CLX39:

clx39$ erl -sname ${USER} -setcookie fe
Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Eshell V10.1  (abort with ^G)
(neswold@clx39)1> observer:start().

This will start the Observer X-application (you need to make sure your ssh session is forwarding its X display back to your machine.) The "Nodes" menu lets you pick an alternate Erlang VM. For this example, you'd enter "fe@clx20". This tools lets you see CPU load, memory load, process trees, and other useful, internal information.

It doesn't do everything that Wombat provided, but it's still very useful, free, and available now.

Also available in: Atom PDF