Email a colleague    

March 2017

How Benchmarking & Streamlining NOC Operations can Lower Costs & Boost Effectiveness

How Benchmarking & Streamlining NOC Operations can Lower Costs & Boost Effectiveness

One of the hardest things in the comms business is getting efficiency out of network operations.

Given the big investments and complexity in telecom networks, operators tend to throw money, people, and tools at their network operations center (NOC) on the better-safe-than-sorry theory.

Yet operators who fail to trim the cost/performance in their NOCs are crowding out vital funds for network expansion, sales, and customer support.

Magnifying the challenge of NOC efficiency is that consistent benchmarks are hard to come by because networks vary greatly by type, equipment, scale, and configuration.

Well, despite the considerable barriers, one company, Greater Chicago-based INOC, has made a business out of benchmarking and delivering more efficient NOC services to operators large and small.

And joining us to discuss the practice of streamlining NOC operations is Hal Baylor, Head of Business Development at INOC, who I interviewed at the recent Metroconnect 2017 conference in Miami.

Dan Baker, Editor, Top Operator: Hal, I have great respect for the work INOC does: benchmarking 57+ network varieties takes tons of experience, and tailoring NOC service for clients is not a cookie cutter operation at all.

Hal Baylor: Very true, Dan.  Two clients with the exact same network equipment may want us to do very different things.

That’s why we built an operational framework: it allows us to turn things on and off depending on the client’s needs.  And our best practices are top notch because we have real-world experience delivering NOC services for over 17 years.

But if an operator knows it has network management issues, how can improvements in the NOC help?

Ironically, the answer is to first look outside the NOC because what’s generally perceived to a “problem in the NOC” is usually a much larger issue.

Operators tend to put a Band Aid on these problems, “Oh, let’s hire a new NOC director or supervisor and our problems will go away.” But hiring a new manager is usually not the cure because the NOC merely reflects problems that live in the network operation as a whole.

Often the issues you see in the NOC reveal flaws in the larger network organization’s processes, systems integration, a less-than-perfect circuit database, lack of metrics, training and/or tools.

So a large component of what makes a NOC successful is all the people outside of the NOC required to streamline processes and procedures, onboard customers, set up monitoring tools, and make sure the NOC’s infrastructure is optimized.

This is why a large percentage of our staff at INOC work in departments outside of the NOC.

So what programs do your experts work on outside the NOC?

Dan, a lot of it is driven off our operational framework — the disciplines required to be excellent in service delivery.  It includes Systems Development, Service Transition (for onboarding and change management), Advanced Engineering, Client Advocacy and Security teams.  They work on operational workflow, monitoring, integrating different systems and ticketing systems.

It also includes KPIs produced out of the ticketing system or the NMS.  A number of areas require a deep dive on a daily basis.  As mentioned, an efficient and effective NOC requires accurate reporting and analysis.  How good is it?  Are the people building the reports expert enough to handle the job?

So we have dedicated people who work on improving reporting and developing metrics.  Metrics are absolutely key to running a NOC properly.

What kinds of customers are you trying to attract?  Since you’re here at the Metroconnect conference, I assume metro fiber players are a fit for you.

Yes, fiber is one market where we have several clients, but our operational framework allows us to work in practically any technology environment.  We can easily serve enterprises, as well as service providers.

The size of the customer usually dictates the kind of work we do.

For enterprises at a Tier 1 level, for example, we can support servers, applications, and an array of enterprise odd-ball things like hosted VoIP.  At a Tier 1, we usually follow the customer’s script: we process alarms, log into equipment, dispatch staff and open carrier tickets.  So because the process is well-defined, we can support many environments at a Tier 1.

However, it’s at the tier 2 and tier 3 level where our expertise really comes into play.  That’s where our knowledge of specific networking services such as dark and lit fiber and routing and switching is required to support clients in the carrier world.

Then there are other tasks that are routine across almost any operator.  It’s things like return equipment authorizations (RMAs), opening a carrier ticket and managing a scheduled maintenance.  For INOC, we have well-regimented processes and procedures so those tasks can be trained fairly easily.  You don’t necessarily need to employ your optical engineer to deal with a circuit outage.

How does an operator know whether its NOC is performing well or not?

There’s no substitute for going through a formal NOC Assessment, which we perform as a service.  We go in and — using a highly refined methodology — interview leadership, review workflows and how each department interacts, review NOC metrics for time to ticket, mean time to repair, time in edit mode, staff loading, costs, and as well evaluate all the tools currently being used.  We then provide a detailed analysis along with recommendations.

Now this detailed NOC evaluation mirrors the way we evaluate our own NOC: its procedures and benchmarks we apply at the dozens of carriers we interact with each day.  We can tell them how many staff it should take to run their NOC, what the metrics should look like, what reports they need, and what tools they need.

Hal, you’d been doing this a long time.  How efficient are the NOCs at small to medium operators here in the US?  How bad is it out there?

It’s a mixed bag, Dan.  I’d estimate most NOCs are operating at an average of 70% efficiency and often less.

But some operators are stunningly inefficient.  We’ve seen companies that have 50+ employees in their NOC, and yet INOC could easily add them to our outsource portfolio and add a much smaller number of FTEs.  The easy answer is to add people when there is a problem.  The problem may go away, but only at the cost of adding cost and inefficiency.

This business is all about experience.  All it takes is for one experienced network engineer to arrive and you suddenly find you can do things much better — and far more cost effectively.  The same goes with a NOC.  People fail to realize how out-of-hand things can get and often don’t have the metrics to know.

Collectively our staff at INOC has suffered pain as we’ve built and maintained networks.  But creating a consistent operational framework is our way of turning that pain into knowledge, knowledge we can use to alleviate the pain — lack of efficiency and effectiveness — of our clients.

Copyright 2017 Top Operator Journal

 

About the Expert

Hal Baylor

Hal Baylor

Hal has worked in marketing, product development and business development for more than 20 years.  Prior to INOC, he was Senior Manager of Residential & Business Services at cable MSO Mediacom.  Other roles included General Manager at Trillion and Director of Internet Services at cable MSO ClearSource.

At INOC, Hal oversees business development and account management, ensuring not only growth but continuous service improvement.   Contact Hal via

Recent Articles