Subject: [lmap] some thoughts about (additional) lmap monitoring objects

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 24 Feb 2016 12:17:16 +0100

Hi,

I have written a small monitoring interface that allows me to look at
the status of our lmapd while it is running. The tool is essentially
parsing the XML serialization of the YANG defined /lmap-state tree
(with a few exceptions). Here is how it looks:

---8<---

agent-id:     550e8400-e29b-41d4-a716-446655440000
hardware:     Linux x86_64
firmware:     #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24)
version:      lmapd version 0.3
last-started: 2016-02-23T19:31:34+01:00

SCHEDULE/ACTION S INV SUP OVR ERR SPACE LST LFS L-INVOKE   L-COMPLETE L-FAILURE
demo            R 852   0   0 851     0         09:42:35    
 mtr            R 852   0   0   0     0   0   0 09:42:35   09:41:42
 happy          E 851   0   0 851     0   1   1 09:41:42   09:41:43   09:41:43
report-primary  D   0   0   0   0 1019K
 report         E   0   0   0   0     0   0   0
report-backup   D   0   0   0   0     0                    
 report         E   0   0   0   0     0   0   0

---8<---

Explanation:

  S   = state ([R]unning, [E]nabled, [D]isabled, [S]uppressed)
  INV = number of invocations (added as discussed at the interim)
  SUP = number of invocations suppressed due to suppression
  OVR = number of invocations suppressed due to overlaps
  ERR = number of failed completions
  LST = last status code
  LFS = last failed status code
  L-INVOKE = last invocation time
  L-COMPLETE = last completion time
  L-FAILURE = last failed completion time

The demo schedule runs the two actions mtr and happy. The happy action
always fails. The mtr action feeds data to report-primary while the
hello action feeds data to both report-primary and report-backup.
Unfortunately, report-backup is disabled for whatever reason.

There are two issues that I like to bring up here.

o I think we should have status information about suppressions as
  well. Suppressions can be active or inactive. They have a start and
  an end. I think the information model should have an
  ma-status-suppression-obj and the YANG data model a corresponding
  list so that I can obtain information which suppressions are
  currently active.

o Talking about the buffers or queues that buffer results while
  results move from an action to a subsequent destination schedule has
  been controversial in the past. I still remember the Dublin interim
  meeting where we spent some cycles on this.

  Still these buffers exist and in the example above you see that data
  is piling up since the reporter is not delivering the data. So I
  implemented the SPACE column that shows how much data has been
  allocated by the schedule (usually incoming data) and the actions
  (usually data produced while running). Since this buffer space is
  somewhat important for the well being of an lmapd daemon, I would
  appreciate if we would have at least status reporting objects;
  ideally we would even have mechanism to control the usage of this
  space - but I recall that people did not like to go into that
  direction.

/js

PS: On another measurement system I am hosting, failure to deliver
    data due to a TLS cert issue did pile up more than 3GB of data.
    Luckily this runs on a host where there are still some plenty of
    GBs to allocate but thinking about CPEs, I believe it is crucial
    to be able to flush data or to control how data is aged out if
    space is getting tight or to suppress schedules in a well-defined
    manner (e.g., by defining additional events that trigger when
    available space crosses certain thresholds).

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>