Subject: [nmrg] Residual configurations

From: "Dean Bogdanovic" <[email-masked]>
Date: Wed, 07 Oct 2020 12:26:51 -0400

Hi,

During last NMRG virtual meeting on Sep 25, we touched base about 
residual configuration issues in networks. Network device configurations 
are getting bigger and often operators don’t know why certain parts of 
the config are there. One such use case that contributes to config 
growth are debugging sessions. Network operator enters the device and 
starts editing configuration. After the debug session is over, it is not 
unusual for that debug config information to stay in the config 
indefinitely.
Some operators have created central databases that contain all the 
network configuration and act as systems of record. If anything is to 
persist in the network, it has to be entered in the central database, 
but there is still an issue between on device persisting configurations 
and central configuration database.
One of the ideas was to keep the persistent configuration at minimum on 
the device and in the central database. All network services are 
generated on demand and are ephemeral.
The network topology would look something like this

![](cid:[email-masked] "Screen Shot 
2020-10-07 at 12.07.41 PM.png")

 From physical perspective, there are two networks, an optical transport 
and a packet switched network. All devices receive basic connectivity 
configuration that (could) persists on the device, you can call them 
Level 1 virtual networks (maybe glorified management networks). Those 
networks provide most basic connectivity between devices within a single 
management domain. That information could be also provided via ZTP.
On top of each network higher level networks are overlaid. All the 
network configuration for higher level networks is ephemeral. The higher 
level networks can (IMO, should be) different management domains, which 
creates clear demarcation lines in the network provisioning data

Such approach would create a (small) well defined starting configuration 
with all other services built as part of higher level networks. The 
residual config problem would be limited to each domain and with higher 
level networks being ephemeral, it should be easily reprovisioned by 
replaying the provisioning information from a central location.

As said, this an interesting problem to me, as the residual configs also 
eat data plane resources, which are quite expensive.

Dean

---

From: Alexander Clemm <[email-masked]>
Date: Wed, 7 Oct 2020 16:44:24 +0000

Hi Dean,

I agree that having a database acting as a system of record, from which device configurations can be regenerated as needed, makes a lot of sense.  Whenever the two were to get out of sync, you could also reprovision the network as needed.

I am not quite clear what you are suggesting regarding the distinction between overlay networks and substrate network and was wondering if you could clarify this.  Clearly, the same principle would apply there as well (database acts as system of record).  But why would you mandate it being it being ephemeral (versus non-ephemeral for the substrate)?  Just as a method to garbage-collect configurations made by administrators on devices directly?

(I do understand that there will be a problem if device configuration entered by an administrator and configuration kept at an external system were to diverge, in particular, when it is not clear which one is the actual system of record.  But this seems to be not an issue here – just the size of the configuration file?)

Thanks
--- Alex

From: nmrg <[email-masked]> On Behalf Of Dean Bogdanovic
Sent: Wednesday, October 7, 2020 9:27 AM
To: [email-masked]
Subject: [nmrg] Residual configurations

Hi,

During last NMRG virtual meeting on Sep 25, we touched base about residual configuration issues in networks. Network device configurations are getting bigger and often operators don’t know why certain parts of the config are there. One such use case that contributes to config growth are debugging sessions. Network operator enters the device and starts editing configuration. After the debug session is over, it is not unusual for that debug config information to stay in the config indefinitely.
Some operators have created central databases that contain all the network configuration and act as systems of record. If anything is to persist in the network, it has to be entered in the central database, but there is still an issue between on device persisting configurations and central configuration database.
One of the ideas was to keep the persistent configuration at minimum on the device and in the central database. All network services are generated on demand and are ephemeral.
The network topology would look something like this

From physical perspective, there are two networks, an optical transport and a packet switched network. All devices receive basic connectivity configuration that (could) persists on the device, you can call them Level 1 virtual networks (maybe glorified management networks). Those networks provide most basic connectivity between devices within a single management domain. That information could be also provided via ZTP.
On top of each network higher level networks are overlaid. All the network configuration for higher level networks is ephemeral. The higher level networks can (IMO, should be) different management domains, which creates clear demarcation lines in the network provisioning data

Such approach would create a (small) well defined starting configuration with all other services built as part of higher level networks. The residual config problem would be limited to each domain and with higher level networks being ephemeral, it should be easily reprovisioned by replaying the provisioning information from a central location.

As said, this an interesting problem to me, as the residual configs also eat data plane resources, which are quite expensive.

Dean

---

From: "Dean Bogdanovic" <[email-masked]>
Date: Wed, 07 Oct 2020 17:25:46 -0400

On 7 Oct 2020, at 12:44, Alexander Clemm wrote:

> Hi Dean,
>
> I agree that having a database acting as a system of record, from 
> which device configurations can be regenerated as needed, makes a lot 
> of sense.  Whenever the two were to get out of sync, you could also 
> reprovision the network as needed.
Happy we are in agreement :)

>
> I am not quite clear what you are suggesting regarding the distinction 
> between overlay networks and substrate network and was wondering if 
> you could clarify this.
Think about substrate network as providing core routing and access to 
management system. Nothing else. All other services are created as 
overlay networks.

> Clearly, the same principle would apply there as well (database acts 
> as system of record).  But why would you mandate it being it being 
> ephemeral (versus non-ephemeral for the substrate)?
To speak in specific vendor terms, logical system or logical routers are 
created for each overlay network. Those logical router/systems are 
ephemeral as are their configurations. In case something happens to the 
network, as after restart of any network service or the logical 
router/system, the restarted service/system can’t know what is the 
state of the network. So, why boot a service that might be out of sync 
with the rest of the network. Replay it from the central database, as 
all the changes should be reflected in the central database.

> Just as a method to garbage-collect configurations made by 
> administrators on devices directly?
Yes, but also think about enabling customers to change network services, 
and cleaning up after them.

>
> (I do understand that there will be a problem if device configuration 
> entered by an administrator and configuration kept at an external 
> system were to diverge, in particular, when it is not clear which one 
> is the actual system of record.  But this seems to be not an issue 
> here – just the size of the configuration file?)
IMO, the system of record should always be the central database, never 
the device. Except in the very first boot up sequence, when call home 
for basic provisioning is executed.
It is not just config file size, it is to prevent unnecessary 
provisioning of potentially stale config data.

Dean

> residual configuration issues in networks. Network device 
> configurations are getting bigger and often operators don’t know why 
> certain parts of the config are there. One such use case that 
> contributes to config growth are debugging sessions. Network operator 
> enters the device and starts editing configuration. After the debug 
> session is over, it is not unusual for that debug config information 
> to stay in the config indefinitely.
> but there is still an issue between on device persisting 
> configurations and central configuration database.
> One of the ideas was to keep the persistent configuration at minimum 
> on the device and in the central database. All network services are 
> From physical perspective, there are two networks, an optical 
> transport and a packet switched network. All devices receive basic 
> connectivity configuration that (could) persists on the device, you 
> can call them Level 1 virtual networks (maybe glorified management 
> networks). Those networks provide most basic connectivity between 
> devices within a single management domain. That information could be 
> also provided via ZTP.
> network configuration for higher level networks is ephemeral. The 
> higher level networks can (IMO, should be) different management 
> domains, which creates clear demarcation lines in the network 
> provisioning data
>
> Such approach would create a (small) well defined starting 
> configuration with all other services built as part of higher level 
> networks. The residual config problem would be limited to each domain 
> and with higher level networks being ephemeral, it should be easily 
> reprovisioned by replaying the provisioning information from a central 
> location.
>
> As said, this an interesting problem to me, as the residual configs 
> also eat data plane resources, which are quite expensive.
>
> Dean

> _______________________________________________
> nmrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/nmrg

---

From: Jeff Tantsura <[email-masked]>
Date: Thu, 8 Oct 2020 10:51:52 -0700

Hi Dean,

I see where you are coming from.
While i sympathetic to the use case, the fundamental issue here is that you are treating device configuration as a SoT.
As Alex suggested, it should not as it is ephemeral from system statte prospective, during “transformation“ of  intended state into operational (configuration/derived), clean new configuration would be created from the SSoT.

Regards,
Jeff

> On Oct 7, 2020, at 14:25, Dean Bogdanovic <[email-masked]> wrote:
> 
> 
> 
>> I am not quite clear what you are suggesting regarding the distinction between overlay networks and substrate network and was wondering if you could clarify this.
> Think about substrate network as providing core routing and access to management system. Nothing else. All other services are created as overlay networks.
> 
>> Clearly, the same principle would apply there as well (database acts as system of record).  But why would you mandate it being it being ephemeral (versus non-ephemeral for the substrate)?
> To speak in specific vendor terms, logical system or logical routers are created for each overlay network. Those logical router/systems are ephemeral as are their configurations. In case something happens to the network, as after restart of any network service or the logical router/system, the restarted service/system can’t know what is the state of the network. So, why boot a service that might be out of sync with the rest of the network. Replay it from the central database, as all the changes should be reflected in the central database.
> 
> 
>> Just as a method to garbage-collect configurations made by administrators on devices directly?
> Yes, but also think about enabling customers to change network services, and cleaning up after them.
> 
>> 
> IMO, the system of record should always be the central database, never the device. Except in the very first boot up sequence, when call home for basic provisioning is executed.
> It is not just config file size, it is to prevent unnecessary provisioning of potentially stale config data.
> 
> Dean
> 
>> 
>> Thanks
>> --- Alex
>>

---

From: LUIS MIGUEL CONTRERAS MURILLO <[email-masked]>
Date: Fri, 9 Oct 2020 10:15:38 +0000

Hi Dean,

Thanks for elaborating. After reading some thoughts come to my mind.

I think that Level 1 configuration could evolve in time, e.g. because the network infrastructure evolves along the time (e.g., new nodes, links upgrades, etc).

So it would be maybe necessary to sort out along the time what configurations can be categorized as Level 1 and what other are higher level configurations.

Also, if there are dependencies among those higher level configurations it could be convenient to keep track of those, for maybe reverting partially the high level configurations instead of totally.

In summary, this would imply to add some characterization to the configuration actions that are introduced along the time in the network (could this be made maybe but introducing new parameters the YANG models??).

Would be nice to have your feedback on this.

Best regards

Luis

De: nmrg <[email-masked]> En nombre de Dean Bogdanovic
Enviado el: miércoles, 7 de octubre de 2020 18:27
Para: [email-masked]
Asunto: [nmrg] Residual configurations

Hi,

Dean

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

---

From: "Dean Bogdanovic" <[email-masked]>
Date: Fri, 09 Oct 2020 08:22:34 -0400

On 8 Oct 2020, at 13:51, Jeff Tantsura wrote:

> While i sympathetic to the use case, the fundamental issue here is 
> that you are treating device configuration as a SoT.
Actually, I’m not treating the device as SoT (system of truth). This 
is how many network operators are treating it today.

> As Alex suggested, it should not as it is ephemeral from system state 
> prospective, during “transformation“ of  intended state into 
> operational (configuration/derived), clean new configuration would be 
> created from the SSoT.
If the operator persists the last known config state on the device, the 
device will boot from that config file. I’m stating that the device 
should receive most of the configuration post boot time from SoT (to use 
your terminology).

Regards,
Dean

>>> which device configurations can be regenerated as needed, makes a 
>>> lot of sense.  Whenever the two were to get out of sync, you could 
>>> also reprovision the network as needed.
>>> I am not quite clear what you are suggesting regarding the 
>>> distinction between overlay networks and substrate network and was 
>>> wondering if you could clarify this.
>> To speak in specific vendor terms, logical system or logical routers 
>> are created for each overlay network. Those logical router/systems 
>> are ephemeral as are their configurations. In case something happens 
>> to the network, as after restart of any network service or the 
>> logical router/system, the restarted service/system can’t know what 
>> is the state of the network. So, why boot a service that might be out 
>> of sync with the rest of the network. Replay it from the central 
>> database, as all the changes should be reflected in the central 
>> database.
>>
>>
>>> (I do understand that there will be a problem if device 
>>> configuration entered by an administrator and configuration kept at 
>>> an external system were to diverge, in particular, when it is not 
>>> clear which one is the actual system of record.  But this seems to 
>>> be not an issue here – just the size of the configuration file?)
>> IMO, the system of record should always be the central database, 
>> never the device. Except in the very first boot up sequence, when 
>> call home for basic provisioning is executed.
>>> configurations are getting bigger and often operators don’t know 
>>> why certain parts of the config are there. One such use case that 
>>> contributes to config growth are debugging sessions. Network 
>>> operator enters the device and starts editing configuration. After 
>>> the debug session is over, it is not unusual for that debug config 
>>> information to stay in the config indefinitely.
>>> network configuration and act as systems of record. If anything is 
>>> to persist in the network, it has to be entered in the central 
>>> database, but there is still an issue between on device persisting 
>>> networks. The residual config problem would be limited to each 
>>> domain and with higher level networks being ephemeral, it should be 
>>> easily reprovisioned by replaying the provisioning information from 
>>> a central location.
>>>

---

From: "Dean Bogdanovic" <[email-masked]>
Date: Fri, 16 Oct 2020 09:42:00 -0400

Luis,

On 9 Oct 2020, at 6:15, LUIS MIGUEL CONTRERAS MURILLO wrote:

> I think that Level 1 configuration could evolve in time, e.g. because 
> the network infrastructure evolves along the time (e.g., new nodes, 
> links upgrades, etc).
>
> So it would be maybe necessary to sort out along the time what 
> configurations can be categorized as Level 1 and what other are higher 
> level configurations.

Yes, that is a good suggestion and approach.

>
> Also, if there are dependencies among those higher level 
> configurations it could be convenient to keep track of those, for 
> maybe reverting partially the high level configurations instead of 
> totally.
The presumption is there are dependencies between some configuration 
levels, but it has to be sequential, level 2 can have config 
dependencies on level 1, but not level 3 to level 1.

>
> In summary, this would imply to add some characterization to the 
> configuration actions that are introduced along the time in the 
> network (could this be made maybe but introducing new parameters the 
> YANG models??).
This part I haven’t thought through yet. I have one approach in mind, 
but it is depending on the technology developed by my employer. Another 
approach could be using logical systems/routers, but those has several 
limitations. What you are suggesting is another possibility. Maybe your 
approach could be a good starting point.

Dean

> Este mensaje y sus adjuntos se dirigen exclusivamente a su 
> destinatario, puede contener información privilegiada o confidencial 
> y es para uso exclusivo de la persona o entidad de destino. Si no es 
> usted. el destinatario indicado, queda notificado de que la lectura, 
> utilización, divulgación y/o copia sin autorización puede estar 
> prohibida en virtud de la legislación vigente. Si ha recibido este 
> mensaje por error, le rogamos que nos lo comunique inmediatamente por 
> esta misma vía y proceda a su destrucción.
>
> The information contained in this transmission is privileged and 
> confidential information intended only for the use of the individual 
> or entity named above. If the reader of this message is not the 
> intended recipient, you are hereby notified that any dissemination, 
> distribution or copying of this communication is strictly prohibited. 
> If you have received this transmission in error, do not read it. 
> Please immediately reply to the sender that you have received this 
> communication in error and then delete it.
>
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu 
> destinatário, pode conter informação privilegiada ou confidencial e 
> é para uso exclusivo da pessoa ou entidade de destino. Se não é 
> vossa senhoria o destinatário indicado, fica notificado de que a 
> leitura, utilização, divulgação e/ou cópia sem autorização pode 
> estar proibida em virtude da legislação vigente. Se recebeu esta 
> mensagem por erro, rogamos-lhe que nos o comunique imediatamente por 
> esta mesma via e proceda a sua destruição