For a long time the standard point-to-point analog line to a substation was
all that was needed in the grid of the past. It worked well for many years and
by default kept data requirements simple and minimized application integration.
If a new data requirement was warranted, such as a dial-up circuit for a C&I
meter, one would just add another phone line, and communication costs increased.
Upon further advances in technology in equipment and applications, a spaghetti
factory of dedicated circuits was installed into silos of technology at the
substation.

The communication and maintenance costs for the legacy-dedicated circuits provide
little integration and no foundation for future requirements. This architecture
hampers the use of new technology which is unable to leverage the existing network
platform and becomes a budgetary hurdle for implementation of new practices.
It is also a ticking time bomb of availability as communication providers eliminate
analog circuits and continually increase the recurring costs.

The substation is rapidly becoming a distinct network of information not limited
to just operations. Applications and employees need real-time connectivity to
the data assets for constant evaluation and maintenance. Resisting change and
maintaining the status quo is quickly becoming the wrong answer for distribution
companies. A strategic investment in network architecture will enable the use
of information and provide a lasting benefit for the foreseeable future.

Considerations

Any change or investment in advancement of infrastructure must improve service
levels and network performance while minimizing the regulatory pressures on
price. Most distribution companies’ rejections to change network architecture
have been for two reasons. Cost is the first. The second has been the resistance
to change.

Not updating the network architecture may result in stable prices, but is likely
to increase service interruptions. Any change must be price sensitive and provide
real return on investment. It has to address the concerns of operations, make
use of existing capital infrastructure and create a secure environment and meet
new regulatory standards.

A major consideration for the chosen network platform is to support multiple
applications and data assets within the substation and the enterprise. The use
of new technology will assist in a positive ROI scenario, enabling better asset
management and utilization techniques. Figure 1 lists several components and
use cases possible on the network.

There is only one real choice of platform that effectively meets considerations
outlined above and that is a TCP/IP network. Implementation of a TCP/IP network
will not only be cost effective but also provide far greater functionality to
distribution companies than today’s legacy network. For instance, with the migration
of SCADA and control traffic to an IP-based solution, single use and high-cost
analog circuits can be decommissioned. The data can now be delivered over a
single digital circuit. With an IP-controlled network, engineers will now be
able to monitor and make changes to the site remotely, reducing the expense
and delay involved in making physical site visits.

An IP-based solution enables redundancies to be put in place that cannot be
achieved with today’s analog network. For sites where connectivity is imperative,
an IP solution provides the ability to dial up over a backup circuit or reroute
traffic to a secondary data center in the case of a communications failure with
the primary data center. It can also allow duplication of the information generated
so multiple data centers can simultaneously monitor the networks.

The IP solution will also allow the control of SCADA information to be encrypted
before it is sent across the network, providing an additional layer of operational
security and meeting potential future regulatory requirements. Once the IP migration
is complete, significant cost savings will be achieved with the decommissioning
of legacy infrastructure required by today’s network.

Network Considerations

There are many choices of TCP/IP platforms to evaluate for the communication
infrastructure. Most commonly used are satellite, dedicated fiber, frame relay,
cellular, radio and PSTN (public switched telephone network). Each has their
advantages and disadvantages and all can be a part of the total solution.

Satellite networks are easy to integrate and are very portable for installation
but the relative data throughput and cost are unattractive. Also, they have
a delay in transmission that not all applications can handle, especially older
equipment.

Cellular is also easy to install but has coverage problems, limiting network
availability. This would warrant it for only redundancy or noncritical data
assets. These assets would include AMR, C&I metering and remote diagnostics.
One must also be careful to secure a cellular network with proper components,
as typically they are open to the Internet. With proper security it is a credible
low cost platform for redundancy implementation. The recurring cost is associated
with bandwidth utilization and can run as low as a few dollars per month for
service. Only when a critical failure occurs in the primary network would cellular
network be utilized. The portability and ease of installation makes the total
cost of ownership of cellular the primary choice for redundancy.

Dedicated fiber fits all the requirements but is very expensive to install
and maintain once in place. Radio networks will work for most applications but
throughput is a major concern as well as interference from other sources. Only
a few types of radio networks in the non-licensed spectrum will support TCP/IP.
PSTN networks will work for low bandwidth applications – similar to cellular
only the costs are higher. It would be a good choice when cellular is not available.

Frame relay is the best choice for the primary network. It has the high availability
required for mission-critical throughput performance desired for applications.
Frame relay is a copper connection from the communication provider’s local point
of presence and is purchased in different segments as required per substation
environment. An advantage is when your bandwidth demand exceeds the current
segment throughput over the same copper connection can ratchet up as needed
from the service provider – be it a 56Kbps connection up to a full T-1.

A multi-protocol label switching (MPLS) solution as the core of the network
is recommended and connecting all of the circuits into an MPLS cloud. The MPLS
functionality rides on top of the basic TCP/IP network structure, providing
advanced features that have real relevance in the utility industry. Benefits
of an MPLS network are:

• Any-to-any connectivity;
• IP convergence;
• Redundancy and availability;
• High degree of bandwidth scalability; and
• Performance.

MPLS will provide a core network that is more resilient because, unlike the
current ATM clouds and other platforms, MPLS has the ability to route around
failures within the service provider’s network. Therefore, if the service provider
is experiencing congestion or hardware failures within its network, one will
not lose connectivity between its sites.

The MPLS network has the intelligence to find a different path between the
endpoints and will automatically reroute the traffic around the network failure.
Service level agreements are obtainable from the service providers covering
availability and performance.

MPLS provides the ability for any site connected to the cloud to communicate
directly with any other site connected to the cloud. By removing the aggregation
points within the network, any-to-any connectivity helps support future projects
such as the consolidation of servers or call centers into a centralized facility.
Any-to-any connectivity also supports the migration of analog voice to a voice
over IP solution in the future. The removal of the dependencies on aggregation
sites enables shorter circuits to be installed and the future removal of legacy
communication infrastructure. Shorter circuits are less expensive and the removal
of the legacy links will greatly reduce operating and maintenance expenses and
reduce future capital expenditures.

MPLS also allows prioritization of data and configuration within each network
endpoint. This is critical to the operation of the communication infrastructure
within the utility marketplace. The classification of each data component enables
the end user to reduce the segment size of the pipe and allocate portions of
the segment for critical applications. Examples of the classification are in
Figure 2.

Class A data is considered the highest priority for real-time control and operations.
Class B is the next level and so on. The benefit of classification is that each
class is configurable for bandwidth utilization with a standard and maximum.
This allows the higher classes to acquire bandwidth as needed. This means the
frame relay network can be focused in avalanche mode for applications critical
for operations and services. During steady-state or non-avalanche mode, a larger
percentage of the bandwidth is available for the lower classes to operate but
is limited in critical situations.

Security

New regulatory standards are requiring that all routable protocols linked to
critical assets be cyber secure, preventing unauthorized access. Also, physical
access to the critical assets in the substation is to be logged and monitored
during operation, adding another data requirement to the network architecture.

Cyber security is achieved through industry-standard techniques such as secure
shell and IPSec. This will provide both authentication and encryption of the
required network connections within the substation. The physical requirement
can be accomplished through several techniques such as cameras, RFID or bar
scanners with off-the-shelf applications. The application provides real-time
notification and logs the identification of any personnel entering each critical
asset.

Benefits

There are several tangible benefits for distribution companies that implement
a TCP/IP communication platform. Some but not all benefits can be:

Operational

• Enhanced information allows faster and cheaper recovery from faults.
• Remote asset monitoring allows faults to be anticipated and avoided.
• Real-time information provides detailed data on faults, keeping blackouts
  as short as possible.
• Higher availability of data in network performance and redundancy can increase
  service levels.
• Reduces outages and network downtime.
• Meets new regulatory requirements for cyber security of critical assets.

Financial

• Lower communication costs achievable.
• Load balancing of demand to better manage and reduce peaks.
• Custom analytics enhances asset life prediction and network planning.
• Provides low cost availability of data to back-end applications.
• Responds to opportunities derived from deregulation.
• Remote asset monitoring supports “just in time” replacement of failing assets.
• Optimizes usage of aging network infrastructure.

Conclusion

Utilities require a strategic investment in communication infrastructure to
provide a platform that enables business units to operate and grow effectively
within the enterprise. The legacy approach to build point solutions for communication
needs is a failure looking forward. An investment in a TCP/IP platform, such
as an MPLS frame relay network, is the only cost-effective approach to remain
competitive in an industry ready to enable growth.