Deterministic Networking (DetNet) Controller Plane Framework
Andrew G. Malis
Independent
agmalis@gmail.com
Xuesong Geng
Huawei
gengxuesong@huawei.com
Mach (Guoyi) Chen
Huawei
mach.chen@huawei.com
Fengwei Qin
China Mobile
qinfengwei@chinamobile.com
Balazs Varga
Ericsson
balazs.a.varga@ericsson.com
29 June 2022
This document provides a framework overview for the Deterministic Networking (DetNet) controller plane. It discusses concepts and requirements for DetNet controller plane which could be basis for future solution specification.

Introduction
Deterministic Networking (DetNet) provides the capability to carry specified unicast and/or multicast data flows for real-time applications with extremely low data loss rates and bounded latency within a network domain. As defined in [RFC8655], techniques used to provide DetNet capability include reserving data plane resources for individual (or aggregated) DetNet flows in some or all of the intermediate nodes along the path of the flow, providing explicit routes for DetNet flows that do not immediately change with the network topology, and distributing data from DetNet flow packets over time and/or space to ensure delivery of each packet’s data in spite of the loss of a path.
DetNet data plane is defined in a set of documents that are anchored by the DetNet Data Plane Framework[RFC8938] (and the associated DetNet MPLS defined in [RFC8964] and DetNet IP defined in [RFC8939] and other data plane specifications defined in [RFC9023], [RFC9024], [RFC9025], [RFC9037] and [RFC9056])
While the Detnet Architecture and Data Plane documents are primarily concerned with data plane operations, they do contain some requirements for functions that would be required in order to automate DetNet service provisioning and monitoring via a DetNet controller plane. The purpose of this document is to gather these requirements into a single document and discuss how various possible DetNet controller plane architectures could be used to satisfy these requirements, while not providing the protocol details for a DetNet controller plane solution. Such controller plane protocol solutions will be the subject of subsequent documents.
Note that in the DetNet overall architecture, the controller plane includes what are more traditionally considered separate control and management planes. Traditionally, the management plane is primarily involved with fault management, configuration management and performance management(sometimes accounting management and security management is also considered in the management plane, but not in the scope of this document). , while the control plane is primarily responsible for the instantiation and maintenance of flows, MPLS label allocation and distribution, and active in-band or out-of-band signaling to support DetNet functions. In the DetNet architecture, all of this functionality is combined into a single Controller Plane. See Section 4.4.2 of [RFC8655] and the aggregation of Control and Management planes in [RFC7426] for further details.

Terminology
This document uses the terminology established in the DetNet Architecture [RFC8655], and the reader is assumed to be familiar with that document and its terminology.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174]when, and only when, they appear in all capitals, as shown here.

DetNet Controller Plane Requirements
Other DetNet documents, including [RFC8655] and [RFC8938], contain requirements for the Controller Plane. For convenience, these requirements have been compiled here. These requirements have been organized into 3 groups, including: requirements primarily applicable to control plane, requirements primarily applicable to management plane and requirements applicable to both planes.

DetNet Control Plane Requirements
The primary requirements for the DetNet Control Plane include:
- Support the dynamic creation, modification, and deletion of DetNet flows. This may include some or all of explicit path determination, link bandwidth reservations, restricting flows to specific links (e.g., IEEE 802.1 Time-Sensitive Networking (TSN) links), node buffer and other resource reservations, specification of required queuing disciplines along the path, ability to manage bidirectional flows, etc., as needed for a flow.
- Support DetNet flow aggregation and de-aggregation via the ability to dynamically create and delete flow aggregates (FAs), and be able to modify existing FAs by adding or deleting participating flows.
- Allow flow instantiation requests to originate in an end application (via an Application Programming Interface (API), via static provisioning, or via a dynamic control plane, such as a centralized SDN controller or distributed signaling protocols. See Section 3 for further discussion of these options.
- In the case of the DetNet MPLS data plane, manage DetNet Service Label (S-Label), Forwarding Label (F-Label), and Aggregation Label (A-Label) [RFC8964] allocation and distribution.
- Also in the case of the DetNet MPLS data plane, support the DetNet service sub-layer, which provides DetNet service functions such as protection and reordering through the use of packet replication, duplicate elimination, and packet ordering functions (PREOF).
- Support queue control techniques defined in Section 4.5 of [RFC8655] and [I-D.ietf-detnet-bounded-latency] that require time synchronization among network nodes.
- Advertise static and dynamic node and link resources such as capabilities and adjacencies to other network nodes (for dynamic signaling approaches) or to network controllers (for centralized approaches).
- Scale to handle the number of DetNet flows expected in a domain (which may require per-flow signaling or provisioning).
- Provision flow identification information at each of the nodes along the path. Flow identification may differ depending on the location in the network and the DetNet functionality (e.g. transit node vs. relay node).

DetNet Management Plane Requirements
The primary requirements of the DetNet Management Plane are that it must be able to:
- Monitor the performance of DetNet flows and nodes to ensure that they are meeting required objectives, both proactively and on-demand.
- Support DetNet flow continuity check and connectivity verification functions.
- Support testing and monitoring of packet replication, duplicate elimination, and packet ordering functionality in the DetNet domain.

Requirements For Both Planes
The following requirements apply to both the DetNet Controller and Management Planes:
- Operate in a converged network domain that contains both DetNet and non-DetNet flows.
- Adapt to DetNet domain topology changes such as links or nodes failures (fault recovery/restoration), additions and removals.

DetNet Control Plane Architecture
As noted in the Introduction, the DetNet control plane is responsible for the instantiation and maintenance of flows, allocation and distribution of flow related information (e.g., MPLS label), and active in-band or out-of-band information distribution to support these functions.
The following sections define three types of DetNet control plane architectures: a fully distributed control plane utilizing dynamic signaling protocols, a fully centralized SDN-like control plane, and a hybrid control plane containing both distributed protocols and centralized controlling . This document describes the various information exchanges between entities in the network for Each type of these architectures and the corresponding advantages and disadvantages.
In each of the following sections, there are examples to illustrate possible mechanisms that could be used in each type of the architectures. They are not meant to be exhaustive or to preclude any other possible mechanism that could be used in place of those used in the examples.

Distributed Control Plane and Signaling Protocols
In a fully distributed configuration model, User-to-Network Interface (UNI) information is transmitted over a DetNet UNI protocol from the user side to the network side.Then UNI and network configuration information propagate in the network via distributed control plane signaling protocols. Such a DetNet UNI protocol is not necessary in case that the End-systems are DetNet capable.
Taking an RSVP-TE MPLS network as an example, where end systems are not part of the DetNet domain:
1. Network nodes collects topology information and DetNet capabilities of the network nodes through IGP;
2. Ingress edge node receives a flow establishment request from the UNI and calculates one or more valid path(s);
3. The ingress node sends a PATH message with an explicit route through RSVP-TE [RFC3209]. After receiving the PATH message, the egress edge node sends a RESV message with the distributed label and resource reservation request.
In this example, both IGP and RSVT-TE may request extensions for DetNet.

SDN/Fully Centralized Control Plane
In the fully SDN/centralized configuration model, flow/UNI information is transmitted from a Centralized User Controller or from applications via an API/ northbound interface to a Centralized Controlle. Network node configurations for DetNet flows are performed by the controller using a protocol such as NETCONF [RFC6241]/YANG [RFC6020] or PCE-CC [RFC8283].
Take the following case as an example::
1. A Centralized Controller collects topology information and DetNet capabilities of the network nodes via NETCONF/YANG;
2. The Controller receives a flow establishment request from a UNI and calculates one or more valid path(s) through the network;
3. The Controller chooses the optimal path and configures the devices along that path for DetNet flow transmission via PCE-CC.
Protocols in the above example may require extensions for DetNet.

Hybrid Control Plane (partly centralized, partly distributed)
In the hybrid model, controller and control plane protocols work together to provide DetNet services, and there are a number of possible combinations.
In the following case, RSVP-TE and controller are used together:
1. Controller collects topology information and DetNet capabilities of the network nodes via an IGP and/or BGP-LS [RFC7752];
2. Controller receives a flow establishment request through API and calculates one or more valid path(s) through the network ;
3. Based on the calculation result, the Controller distributes flow path information to the ingress edge node and configures network nodes along the path with necessary DetNet information (e.g. for replication/duplicate elimination)
4. Using RSVP-TE, the ingress edge node sends a PATH message with an explicit route. After receiving the PATH message, the egress edge node sends a RESV message with the distributed label and resource reservation request.
There are many other variations that could be included in a hybrid control plane. The requested DetNet extensions for protocol in each possible case is for future work.

DetNet Control Plane for DetNet Mechanisms
This section discusses requested control plane features for DetNet mechanisms as defined in [RFC8655], including explicit path, resource reservation, service protection(PREOF). Different DetNet service may implement part/all of them based on the requirements.

Explicit Paths
Explicit paths are required in DetNet to provide a stable forwarding service and guarantee that DetNet service is not impacted when the network topology changes. The following features are necessary in control plane to implement explicit paths in DetNet:
- Path computation: DetNet explicit paths need to meet the SLA (Service Level Agreement) requirements of the application, which include bandwidth, maximum end- to-end delay, maximum end-to-end delay variation, maximum loss ratio, etc. In a distributed network system, IGP with CSPF (Constrained Shortest Path First) may be used to compute a set of feasible paths for a DetNet service. In a centralized network system, controller can compute paths satisfying the requirements of DetNet based on the network information collected from the DetNet domain.
- Path establishment: The computed path for the DetNet service has to be sent/configured/signaled to the network device, so the corresponding DetNet flow could pass through the network domain following the specified path.

Resource Reservation
DetNet flows are supposed to be protected from congestion, so sufficient resource reservation for DetNet service could protect service from congestion. There are multiple types of resources in the network that could be allocated to DetNet flows, e.g., packet processing resource, buffer resource, and bandwidth of the output port. The network resource requested by a specified DetNet service is determined by the SLA requirements and network capability.
- Resource Allocation: Port bandwidth is one of the basic attributes of a network device which is easy to obtain or calculate. In current traffic engineering implementations, network resource allocation is synonymous with bandwidth allocation. A DetNet flow is characterized with a traffic specification as defined in [RFC9016], including attributes such as Interval, Maximum Packets Per Interval, and Maximum Payload Size. The traffic specification describes the worst case, rather than the average case, for the traffic, to ensure that sufficient bandwidth and buffering resources are reserved to satisfy the traffic specification. However, in case of DetNet, resource allocation is more than simple bandwidth reservation. For example, allocation of buffers and required queuing disciplines during forwarding may be required as well. Furthermore, resources must be ensured to execute DetNet service sub-layer functions on the node, such as protection and reordering through the use of packet replication, duplicate elimination, and packet ordering functions (PREOF).
- Device configuration with or without flow discrimination: The resource allocation can be guaranteed by device configuration. For example, an output port bandwidth reservation can be configured as a parameter of queue management and the port scheduling algorithm. When DetNet flows are aggregated, a group of DetNet flows share the allocated resource in the network device. When the DetNet flows are treated independently, the device should maintains a mapping relationship between a DetNet flow and its corresponding resources.

PREOF Support
DetNet path redundancy is supported via packet replication, duplicate elimination, and packet ordering functions (PREOF). A DetNet flow is replicated and goes through multiple networks paths to avoid packet loss caused by device or link failures. In general, current control plane mechanisms that can be used to establish an explicit path, whether distributed or centralized, support point-to-point (P2P) and point-to-multipoint (P2MP) path establishment. PREOF requires the ability to compute and establish a set of multiple paths (e.g., multiple LSP segments in an MPLS network) from the point(s) of packet replication to the point(s) of packet merging and ordering. Mapping of DetNet (member) flows to explicit path segments has to be ensured as well. Protocol extensions will be required to support these new features. Terminology will also be required to refer to this coordinated set of path segments (such as an “LSP graph” in case of DetNet MPLS data plane).

Data Plane specific considerations

DetNet in an MPLS Domain
For the purposes of this document, “traditional MPLS” is defined as MPLS without the use of segment routing (see Section 4.4.3 for a discussion of MPLS with segment routing) or MPLS-TP [RFC5960].
In traditional MPLS domains, a dynamic control plane using distributed signaling protocols is typically used for the distribution of MPLS labels used for forwarding MPLS packets. The dynamic signaling protocols most commonly used for label distribution are LDP [RFC5036], RSVP-TE, and BGP [RFC8277] (which enables BGP/MPLS-based Layer 3 VPNs [RFC4384] and Layer 2 VPNs [RFC7432]).
Any of these protocols could be used to distribute DetNet Service Labels (S-Labels) and Aggregation Labels (A-Labels) [RFC8964]. As discussed in [RFC8938], S-Labels are similar to other MPLS service labels, such as pseudowire, L3 VPN, and L2 VPN labels, and could be distributed in a similar manner, such as through the use of targeted LDP or BGP. If these were to be used for DetNet, they would require extensions to support DetNet-specific features such as PREOF, aggregation (A-Labels), node resource allocation, and queue placement.
However, as discussed in Section 3.1, distributed signaling protocols may have difficulty meeting DetNet’s scalability requirements. MPLS also allows SDN-like centralized label management and distribution as an alternative to distributed signaling protocols, using protocols such as PCEP and OpenFlow [OPENFLOW].
PCEP, particularly when used as a part of PCE-CC, is a possible candidate protocol to use for centralized management of traditional MPLS-based DetNet domains. However, PCE path calculation algorithms would need to be extended to include the location determination for PREOF nodes in a path, and the means to signal the necessary resource reservation and PREOF function placement information to network nodes. See ((?I-D.ietf-pce-pcep-extension-for-pce-controller)) for further discussion of PCE-CC and PCEP for centralized control of an MPLS domain.

DetNet in an IP Domain
For the purposes of this document, “traditional IP” is defined as IP without the use of segment routing (see Section 4.4.3 for a discussion of IP with segment routing). In a later revision of this document, this section will discuss possible protocol extensions to existing IP routing protocols such as OSPF, IS-IS, and BGP. It should be noted that a DetNet IP data plane [RFC8939] is simpler than a DetNet MPLS data plane [RFC8964], and doesn’t support PREOF, so only one path per flow or flow aggregate is required.

DetNet in a Segment Routing Domain
Segment Routing [RFC8402] is a scalable approach to building network domains that provides explicit routing via source routing encoded in packet headers and it is combined with centralized network control to compute paths through the network. Forwarding paths are distributed with associated policy to network edge nodes for use in packet headers. As such, segment routing can be considered as a new data plane for both MPLS and IP. It reduces the amount of network signaling associated with distributed signaling protocols such as RSVP-TE, and also reduces the amount of state in core nodes compared with that required for traditional MPLS and IP routing, as the state is now in the packets rather than in the routers. This could be useful for DetNet, where a very large number of flows through a network domain are expected, which would otherwise require the instantiation of state for each flow traversing each node in the network. However, further analysis is needed on the expected gain, as DetNet flows may require various type of DetNet specific resources as well.
In a later revision of this document, this section will discuss the impact of DetNet on the Segment Routing Control and Management planes. Note that the DetNet MPLS and IP data planes described in [RFC8964] and [RFC8939] were constructed to be compatible with both types of segment routing, SR-MPLS [RFC8660] and SRv6 [I-D.ietf-6man-segment-routing-header]. However, as of this writing, traffic engineering and resource reservation for segment routing are currently unsolved problems.
Editor’s note: this section may be collapsed to previous sections and listing MPLS segment routing in the MPLS section as one of the possible explicit routing techniques for MPLS, and do the same for IP.

Management Plane Overview
The Management Plane includes the ability to statically provision network nodes and to use OAM to monitor DetNet performance and detect outages or other issues at the DetNet layer.

Provisioning
Static provisioning in a Detnet network nodes will be performed via the use of appropriate YANG models, including [I-D.ietf-detnet-yang] and [I-D.ietf-detnet-topology-yang].

DetNet Operations, Administration and Maintenance (OAM)
This document covers the general considerations for OAM.

OAM for Performance Monitoring (PM)

Active PM
Active PM is performed by injecting OAM packets into the network to estimate the performance of the network by measuring the performance of the OAM packets. Adding extra traffic can affect the delay and throughput performance of the network, and for this reason active PM is not recommended for use in operational DetNet domains. However, it is a useful test tool when commissioning a new network or during troubleshooting.

Passive PM
Passive PM monitors the actual service traffic in a network domain in order to measure its performance without having a detrimental affect on the network. As compared to Active PM, Passive PM is much preferred for use in DetNet domains.

OAM for Connectivity and Fault/Defect Management (CFM)
The detailed requirements for connectivity and fault/defect detection and management in DetNet IP domain and DetNet MPLS domain are defined in respectively in [I-D.ietf-detnet-ip-oam] and [I-D.ietf-detnet-mpls-oam].

Gap Analysis
In a later revision of this document, this section will contain a gap analysis of existing IETF control and management plane protocols not already discussed elsewhere in this document for their ability (or inability) to satisfy the requirements in Section 2, and discuss possible protocol extensions to existing protocols to fill the gaps, if any.

IANA Considerations
This document has no actions for IANA.
Note to RFC Editor: this section may be removed on publication as an RFC.

Security Considerations
Editor's note: This section needs more details.
The overall security considerations of DetNet are discussed in [RFC8655] and [I-D.ietf-detnet-security]. For DetNet networks that make use of Segment Routing (whether SR-MPLS or SRv6), the security considerations in [RFC8402] also apply.
DetNet networks that make use of a centralized controller plane may be threatened by the loss of connectivity (whether accidental or malicious) between the central controller and the network nodes, and/or the spoofing of control messages from the controller to the network nodes. This is important since such networks depend on centralized controllers to calculate flow paths and instantiate flow state in the network nodes. For networks that use both DetNet and Segment Routing with a centralized controller, this would also include the calculation of SID lists and their installation in edge/border routers.
In both cases, such threats may be mitigated through redundant controllers, the use of authentication between the controller(s) and the network nodes, and other mechanisms for protection against DOS attacks. A mechanism for supporting one or more alternative central controllers and the ability to fail over to such an alternative controller will be required.

Acknowledgments
Thanks to Jim Guichard, Donald Eastlake, and Stewart Bryant for their review comments.

Normative References
[RFC8655]

Deterministic Networking Architecture
N. Finn
P. Thubert
B. Varga
J. Farkas
October 2019
RFC 8655
DOI 10.17487/RFC8655
[RFC7426]

Software-Defined Networking (SDN): Layers and Architecture Terminology
E. Haleplidis, Ed.
K. Pentikousis, Ed.
S. Denazis
J. Hadi Salim
D. Meyer
O. Koufopavlou
January 2015
RFC 7426
DOI 10.17487/RFC7426
[RFC2119]

Key words for use in RFCs to Indicate Requirement Levels
S. Bradner
March 1997
BCP 14
RFC 2119
DOI 10.17487/RFC2119
[I-D.ietf-detnet-flow-information-model]

DetNet Flow Information Model
Balazs Varga
Janos Farkas
Rodney Cummings
Yuanlong Jiang
Don Fedyk
15 May 2020
Internet-Draft draft-ietf-detnet-flow-information-model-10
[RFC8402]

Segment Routing Architecture
C. Filsfils, Ed.
S. Previdi, Ed.
L. Ginsberg
B. Decraene
S. Litkowski
R. Shakir
July 2018
RFC 8402
DOI 10.17487/RFC8402
[I-D.ietf-detnet-security]

Deterministic Networking (DetNet) Security Considerations
Tal Mizrahi
Ethan Grossman
30 May 2020
Internet-Draft draft-ietf-detnet-security-10
[RFC8174]

Informative References
[OPENFLOW]

OpenFlow Switch Specification, Version 1.5.1 (Protocol version 0x06)
Open Networking Foundation
March 2015
ONF TS-025
[RFC3209]

RSVP-TE: Extensions to RSVP for LSP Tunnels
D. Awduche
L. Berger
D. Gan
T. Li
V. Srinivasan
G. Swallow
December 2001
RFC 3209
DOI 10.17487/RFC3209
[IEEE.802.1QBV_2015]

IEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks - Amendment 25: Enhancements for Scheduled Traffic
IEEE
18 March 2016
IEEE 802.1Qbv-2015
DOI 10.1109/IEEESTD.2016.7572858
[RFC5439]

An Analysis of Scaling Issues in MPLS-TE Core Networks
S. Yasukawa
A. Farrel
O. Komolafe
February 2009
RFC 5439
DOI 10.17487/RFC5439
[RFC6241]

Network Configuration Protocol (NETCONF)
R. Enns, Ed.
M. Bjorklund, Ed.
J. Schoenwaelder, Ed.
A. Bierman, Ed.
June 2011
RFC 6241
DOI 10.17487/RFC6241
[RFC6020]

YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)
M. Bjorklund, Ed.
October 2010
RFC 6020
DOI 10.17487/RFC6020
[RFC8283]

An Architecture for Use of PCE and the PCE Communication Protocol (PCEP) in a Network with Central Control
A. Farrel, Ed.
Q. Zhao, Ed.
Z. Li
C. Zhou
December 2017
RFC 8283
DOI 10.17487/RFC8283
[RFC7752]

North-Bound Distribution of Link-State and Traffic Engineering (TE) Information Using BGP
H. Gredler, Ed.
J. Medved
S. Previdi
A. Farrel
S. Ray
March 2016
RFC 7752
DOI 10.17487/RFC7752
[RFC5960]

MPLS Transport Profile Data Plane Architecture
D. Frost, Ed.
S. Bryant, Ed.
M. Bocci, Ed.
August 2010
RFC 5960
DOI 10.17487/RFC5960
[RFC5036]

LDP Specification
L. Andersson, Ed.
I. Minei, Ed.
B. Thomas, Ed.
October 2007
RFC 5036
DOI 10.17487/RFC5036
[RFC8277]

Using BGP to Bind MPLS Labels to Address Prefixes
E. Rosen
October 2017
RFC 8277
DOI 10.17487/RFC8277
[RFC4384]

BGP Communities for Data Collection
D. Meyer
February 2006
BCP 114
RFC 4384
DOI 10.17487/RFC4384
[RFC7432]

BGP MPLS-Based Ethernet VPN
A. Sajassi, Ed.
R. Aggarwal
N. Bitar
A. Isaac
J. Uttaro
J. Drake
W. Henderickx
February 2015
RFC 7432
DOI 10.17487/RFC7432
[RFC8660]

Segment Routing with the MPLS Data Plane
A. Bashandy, Ed.
C. Filsfils, Ed.
S. Previdi
B. Decraene
S. Litkowski
R. Shakir
December 2019
RFC 8660
DOI 10.17487/RFC8660
[I-D.ietf-6man-segment-routing-header]

IPv6 Segment Routing Header (SRH)
Clarence Filsfils
Darren Dukes
Stefano Previdi
John Leddy
Satoru Matsushima
Daniel Voyer
22 October 2019
Internet-Draft draft-ietf-6man-segment-routing-header-26
[I-D.ietf-detnet-yang]

Deterministic Networking (DetNet) Configuration YANG Model
Xuesong Geng
Mach Chen
Yeoncheol Ryoo
Zhenqiang Li
Reshad Rahman
Don Fedyk
11 June 2020
Internet-Draft draft-ietf-detnet-yang-06
[I-D.ietf-detnet-topology-yang]

Deterministic Networking (DetNet) Topology YANG Model
Xuesong Geng
Mach Chen
Zhenqiang Li
Reshad Rahman
25 January 2019
Internet-Draft draft-ietf-detnet-topology-yang-00
[RFC8938]
[RFC8964]
[RFC8939]
[RFC9023]
[RFC9024]
[RFC9025]
[RFC9037]
[RFC9056]
[I-D.ietf-detnet-bounded-latency]
[RFC9016]
[I-D.ietf-detnet-ip-oam]
[I-D.ietf-detnet-mpls-oam]