Monthly Archives: November 2017

How Vendor Maturity Challenges NFV Adoption

Days are changing. The physical is becoming virtual. It started in the datacenter and now it’s in the network. The surge of NFV is creating a new market freeze. Telecoms are waiting. They don’t want to buy physical, but they haven’t seen enough success in the virtual network world. A common reason for lack of success has been the vendors and devices employed. Immature is a common descriptor for the marketplace. This blog intends to educate the readers on what devices are out there. Here I catalog those VNF types I have seen. More people need to prepare themselves for the new realities of this new virtual world.   The issues of NFV adoption is one the industry must address.

Challenges in NFV Adoption

The facts on the ground are these: the maturity of vendors defines what is possible. If your vendor deems tracking next hop & neighbor topology as irrelevant, then you may not be able to perform accurate root cause analysis. If your vendor deems that MAC addresses can change every time you reboot a VNF, then your network ARP table cache will look crazy. Virtualization breaks the rules and some vendors are not ready to deal with that fallout. The greatest threat and obstacle to VNF adoption is the quality and quantity of VNFs available. This is a serious challenge. Out of all the stories I have heard and spoken – VNF vendor maturity is usually at the core of the issue.   NFV adoption will continue to be challenged until process can be developed to mitigate the issue.
Legacy VNFs for NFV Adoption

Legacy VNFs


Legacy VNFs

Let’s talk about categorizing the VNFs. We are seeing most VNFs falling into these silos. First there are legacy VNFs. They are your standard fare; just ported into a virtualized infrastructure. Virtual switches are usually imbedded into the hypervisor. Routers are common, lead by Cisco and Juniper. The mobile (3GPP) infrastructure is adopting VNFs so Ericsson, Nokia, and Cisco have offerings. Virtualized security has gotten plenty of traction. Most firewall PNFs have been ported, like Cisco, Fortinet, Palo Alto, and Checkpoint. These VNF types are PNFs emulated on a Linux OS. I compare the concept to game emulators. The “VNF” is nothing more than a Linux operating system emulating the BIOS of the pre-existing PNF. I call them “legacy” because all the integrations and interfaces are identical to their PNF brethren. What is interesting is the VNF image shipped has the Linux embedded. For any practical purpose you cannot manage the Linux operating system underneath. The VNF overrides the SSH, SNMP, telnet servers. So you cannot tell there is Linux underneath. Given that though, legacy VNFs are the easiest to manage. They are >90% like their PNF counterparts, which provides predictability.
MANO-enabled VNFs for NFV Adoption

MANO-enabled VNFs



Legacy is exactly that – old! Most product offerings focus on new services using new technologies. This is where you will experience the second type of VNF – MANO-enabled. These new VNFs support and act in concert with an orchestrated, elastic infrastructure. SD-WAN is a perfect example of new network technology with Velocloud/VMWare, Viptela/Cisco, and Versa as vendors. The trouble with these is many of the vendors don’t know the “rules”. Documentation is errant or missing. Common and core functionality support is missing – like the concept of interface monitoring. With giant gaps in functionality, especially from an assurance perspective, how can we maintain SLAs? When APIs have no documentation, you can bet that the vendor will not be able to provide best practices for KPIs. The new VNFs with new vendors have the hottest technology, but with the greatest challenges.
Custom VNFs for NFV Adoption

Custom VNFs


Custom VNFs

Rules? We don’t need rules? That is what the third VNF types is all about – custom VNFs. One of the advantages to virtualizing networks is telecom can create their own VNFs. Think this is not likely? Well one of the first customers I talked to used a customer VNF. They used a Linux OS firewalls with custom code to configure and manage them. Sprint has C3PO that will be using open source mobile core components. These VNFs enable customers increased flexibility, but they can come with a price: lack of consistency. Managing custom VNFs ends up being identical to custom applications then network elements. The good news is that you will be able to influence the changes needed. The bad news is you may be having those conversations AFTER those devices roll into production. A DevOps process can help support managing them effectively. The challenge is your assurance and delivery tools will need support agile processes. Custom VNFs are the most challenging for operations. They flip the traditional sourcing and boarding models predefined in the industry.
On-boarding Strategy for NFV Adoption

On-boarding Strategy

Onboarding VNFs

With such a diversity in VNF types, how can operations ensure proper engineering new services and offerings based up them? I call it an “on-boarding strategy”. When new VNFs are being planned or explored, operations must be involved to perform a proper assessment. This assessment must be holistic, including a multitude of different items. While many of these items are common in the PNF world, they have different levels of importance in the VNF world.  

Example Checklist

Having a mature on-boarding strategy addresses NFV adoption.   Below is my simplistic list, my recommendation is that you build you own:
  • Fault – This includes service and non-service affecting issue and log collection. Its import to understand the protocol, format, and overhead required. 
  • Performance – This includes counters, KPIs, and KQIs associated to the technology. Passive vs active collection as well as protocol/format creates challenges.
  • Inventory – This includes interfaces and “slot/card” information. Protocol/format/overhead is important as well. 
  • Configuration – This includes how to recreate the VNF. A VNF, like PNF, is a blank slate until it has a configuration. Recreating the VNF requires configuration collection, repository, and policy manager to restore. 
  • Topology – The next hops by network layer (1/2/3/4, etc) allows for accurate root cause analysis. Without accuracy, your automation loses value.
  • Automation – Can you snapshot or change configuration without service impact? Understanding how well the VNF handles change allows better understanding of the limits of your tooling.

Being Proactive

Be proactive in its use so you can identify problems before they slow down your delivery. Many vendors offer similar VNFs. Say that from a contract perspective, switching between router vendors is insignificant. But let’s say they vary from an on-boarding requirements. The more information you have, the more chance you have at avoiding challenges and delay.   The you avoid the delay, the more you NFV adoption will accelerate.
Impact on tools for NFV Adoption

Impact on Tools

 Impact on Tools

VNF maturity directly impacts your tooling. Automation becomes limited by the least common denominator of VNF capabilities. If you cannot perform accurate root cause analysis, then you cannot perform automation. If your VNF crashes when you perform a configuration pull, then automated restoration is limited. If the VNF stops logging under performance load, your operational processes are threatened. Virtualization of your network challenges legacy tools and processes like I have never seen. In my career, adopting a new tool to cover the new technology domain was enough – while increasing complexity and cost. NfV is different because it’s not a new domain, it unifies all domains.
When looking at tools, you should focus on three key areas. First is a no-brainer, scale. When you are changing the infrastructure to which all new network elements will run on – scale should be the first discussion point. Next is flexibility. When you are unifying all network domains with a common infrastructure, you need tools that cross domains . Lastly you need a tool with automation focus. Automation is how you will grow the network, but only if your tools support it. Your service delivery and assurance solutions need to embrace your new network infrastructure. You do not fight want them to fight you tooth and nail.    With the proper tools and process, your NFV adoption will accelerate and enable operations.

Lessons Learned to Enable NFV Adoption

  • Virtualization is causing market confusion and hesitancy
  •  Vendor maturity one of the largest issues
  • There are three types of VNFs
  1.  Legacy VNFs are like PNFs and you can expect similar feature/functionality, but with weird twists 
  1. MANO enabled VNFs have stable devices but inconsistency from a feature/functionality perspective 
  2. Custom VNFs that break all the rules which run more like applications then network
  • Create onboarding processes based upon your requirements. Make sure they are documented and easy to use by procurement, engineering, and vendors
  •  Make sure you have tools with a focus on flexibility, scale, and automation

My advice of NfV Adoption 

  • Leverage process -> Development and enforce a VNF onboarding strategy
  • Vendor management -> Only use VNFs from vendors you trust, that you have a strong relationship
  • Get Experience -> Model a network with the devices you want to use, then pilot that network with real traffic
Shawn Ennis

About the Author

Serial entrepreneur and operations subject matter expert who likes to help customers and partners achieve solutions that solve critical problems.   Experience in traditional telecom, ITIL enterprise, global manage service providers, and datacenter hosting providers.   Expertise in optical DWDM, MPLS networks, MEF Ethernet, COTS applications, custom applications, SDDC virtualized, and SDN/NFV virtualized infrastructure.  Based out of Dallas, Texas US area and currently working for one of his founded companies – Monolith Software.

Operations Digital Transformation Playbook

Digital Transformation is the buzz word of the day.  Whether TMF is saying it or Lightreading is reporting it, CIOs are doing it.    Here is an example roadmap for transforming operations for the business’s digital transformation.    This 4-step process leverages much of what the industry has already said. I have interweaved some color and advice .   I hope you find it useful and comment below.

Acceptance for Digital Transformation

Step 1: Acceptance

First, your organization needs to buy into the fact that something has to change.     Buy-in for digital transformation is the key to success.  While 100% agreement is not possible, getting an overwhelming majority will reduce timelines.   Forming a committee with regular cadence calls can assist on collection of use cases. As a sound board, they can be the voice of the organization. They will also provide you cover during the transformation process.
Here is some advice to help people get in the boat. Some will doubt the need for change. To those doubters, I would pose the following questions:
  • What % of the time does operations spend on firefighting?
  • What does your customers say about the quality of the services you provide?
  • How many compromises does your team make to “get the job done”?
  • Does 25% YOY staff turnover frighten you?

These questions are the canaries in the coal mine for impacting digital transformation.    If you cannot focus on improvement, growth, and resiliency – the organization is in danger   When the business is changing to a more agile footprint, operations gets left behind — or even worse, becomes the roadblock.


Selection for Digital Transformation

Step 2: Selection

Change scares people and organizations and digital transformation can get scary.   When it comes to selection, it must be a sober, deliberate decision.   RFIs are a common method for initiating change.    The trouble is the net you cast.    If you only send the RFI to your pre-existing vendors, you will get lots of the same.
I recommend that you start with a google search on “Operations transformation”, then “IT transformation”    The results should net you some NEPs/DEPs (EMC, Huawei) and Global SI players (Accenture, Deloitte).   If you are a fortune 500, they will be very kind to you and expect big money.    If you have a relationship with these guys, I recommend calling upon them and seeing the “big pitch”.    Its great for context and helps to understand the commitment involved in transformation.
The next step would be to call some analysts.   I have had great experiences with Analysys Mason, Appledore Research, and Gartner.    They can tell you what other customers have done.    Attend some webinars and trade events can help get you connected to the trend setters.    This will help you round out the group you want to invite into the RFI process.
With the RFI executed, you will want to review the material and cut down to 5 or less parties.    Make sure you have a global SI, a NEP/DEP, and some trend setters in the bunch.    Ask for presentations and documentation of best practices.   Get as much information as possible, creating quality requirements is key.
Within the transformation workgroup, create a top 10/25 list from each member of key issues.   Apply your use cases and develop a list of requirements (<100 items). Add to it a ratings system to keep it fair, to the point.   If you value verification of technical compliance (ie. support for Cisco IOS Y/N, etc), add another sheet.   Another tip; you can always demand entries to combine their offerings. This firms up and consolidates your options.   Use this list with your procurement team to create the RFP. Give at least 2 weeks to respond, and no more than 4 weeks.    Stick to your schedule and grade the entries responses.
Work within the workgroup to kill and combine entries until you get to at least 2 — the fewer the better. Based upon grading, provide a list to the down-selected parties of how they can improve their response. Giving parties the opportunity to focus and improve, will allow for better options. Schedule meetings with no more than 7 day’s notice for their presentation and response.    After all meetings are complete, revise the grading and make a selection with procurement.    Notify all parties and negotiate a contract.   I recommend all contracts as part of transformation be longer term, you will want a partner for at least 3 years.    I also recommend agreement of SLAs and penalties of failed/delayed delivery.


Execution for Digital Transformation

Step 3: Execution

After making the selection, now the hard work starts – addressing digital transformation.    Implementation should be a core concern during the selection process.   Some transformation projects are short-term (less than 6 months), longer term leverages milestones.    Phasing allows transformation projects to achieve quick wins and setup long-term success.   When building the business case, phasing allows prioritization of key objectives.   I always recommend to show significant value within a quarter, and every quarter after.   Regular improvement needs to be visible, or you will need significant executive sponsorship.    Phasing will help drive the value and keep on task.
Selection of a project mantra defines how that project will run.   Agile is very popular in IT projects.   A DevOps approach allows your transformation project to become evergreen.   For long-term projects where you need extreme flexibility, there is no better technique.    For short-term, fixed scope projects waterfall is more than satisfactory.
When executing the vision, setting phased milestones provides the director. Quarterly scheduled demonstrations keeps the faith. With consistent, planned deliveries will confirm healthy project management.   When it comes to execution focus, communication and delivery success should be the first priority.    Its always best to remember, if you have an unhealthy project — you will have poor deliverables.


RIO Renewal for Digital Transformation

Step 4: ROI & Renewal

Once the project has achieved it main objectives the question becomes “Now What?”.    In every sense of the word, there is an “end-state” with regards to transformation.   Once you get there, you will experience the fact that the goal posts moved on you.   This is another reason Agile methodologies are popular.

Meet with your workgroup and steering committee, does it make sense to continue.   One key issue my customers have seen, is that transformation can lead to change for only change’s sake.   There must be clear needs to continue.   You can always reduce team cadence and let the needs of the business denote the tempo.
In summation, executing a digital transformation is a heavy commitment.   The facts are that the change required is necessary to address industry climate.    Nobody wants to buy a T1 anymore and that is a good thing!    The good news is that meeting the needs of business is possible and profitable. Good luck transforming!

Key lessons learned for Digital Transformation:

  • Collaboration = Commitment = Success – if you communicate effectively
  • Select the best process and tools for your team.     Do not fall into the conformity for its sake.
  • Set achievable regularly delivered goals.   Show consistently increasing value to the business.
  • Focus on the present, but regularly plan for the future – and always communicate


Shawn Ennis

About the Author

Serial entrepreneur and operations subject matter expert who likes to help customers and partners achieve solutions that solve critical problems.   Experience in traditional telecom, ITIL enterprise, global manage service providers, and datacenter hosting providers.   Expertise in optical DWDM, MPLS networks, MEF Ethernet, COTS applications, custom applications, SDDC virtualized, and SDN/NFV virtualized infrastructure.  Based out of Dallas, Texas US area and currently working for one of his founded companies – Monolith Software.