NetWorker, better called "Not Workin'"
December 11, 2015
NetWorker, better called "Not Workin'"
Score 1 out of 10
Overall Satisfaction with Networker
We have been using NetWorker for the past three years for all Open Systems (Windows and Linux) backups. It has been the worst nightmare of my 40+ years in IT. We never had even one day when all of our backups completed successfully. Support was horrible. We are now in the process of completely replacing it with a different solution.
- When used with a Data Domain appliance, using either DDBOOST or a VTL, it is quick and does a wonderful job of deduplication. We have 2.3 PB stored on a 140 TB DD 4500. While this is expensive storage, the cost for 2+ PB would be even higher.
- It does a good job of brick-level backups of Exchange mailboxes, and does so in very good time. A few hours backs up our entire organizations mailbox stores in a way that provides object level restore.
- Used in combination with DPA (Data Protection Advisor), it has a very good reporting capability. DPA, however, requires more than just surface knowledge in order to get really good reports, and the DDOS changes can wreck havoc with customized reports.
- NetWorker has a number of glaring flaws. For starters, it does not have any built-in vaulting capability. I simply cannot believe that EMC thinks nobody takes tapes out of their libraries. Their response to our inquiry about it? "We can write a program for you that will cost x-thousands of dollars, or you can develop one yourself." We wrote our own customized program to vault tapes.
- NetWorker does not posses any Disaster Recovery reporting capability. Again, we had to custom code reporting for this so that tape librarians would know what tapes to recall from offsite storage for entire groups of servers. During a crisis there isn't time to be doing that on a one at a time basis for hundred or thousands of servers.
- NetWorker is extremely sensitive to DNS changes, and appears to cache DNS data in hidden locations. We have servers being reported by NetWorker as not connected when they were decommissioned years ago, removed from AD and DNS, yet we still cannot get NetWorker to stop complaining about them.
- NetWorker does not play well at all with multi-homed clients (more than one network interface). In environments where it is not conducive to backup servers of a production network, it becomes crucial to do so over a dedicated or secondary LAN. This causes huge issues with NetWorker.
- If a group contains a number of clients, and one of those gets hung up during a backup, the entire group fails. That is a very wasteful approach for both time and infrastructure resources. Instead, it should fail the one client and allow the remainder in the group to complete successfully. It should also allow the group run to be canceled and still keep the good clients backups rather than registering the entire group as failed.
- There is no way in NetWorker to identify specific file/directories that fail to backup successfully. It will report on savesets, but I need to know that file abc.dat or directory F:\Program Data\ failed and why. It does me no good to get a warning that the saveset for the F drive failed. What failed and why? It may have been a critical problem, or it may have been of no importance.
- We endured three years of NetWorker experiencing problems, enduring a grueling process of trying to get knowledgeable and rapid support -- sometimes taking days and weeks, and only after getting really pushy with support managers -- only to have the problems return over and over. For example, it has been a regular issue for the peer information to get clobbered for no apparent reason. The result is the backup fails for that client, and then I have to go in and remove the peer information on the NetWorker Server, all affected Storage Nodes, and the client. I can now run nsradmin -p nsrexec and then the print and delete statements for nsr peer information in my sleep.
- In our experience, NetWorker was extremely expensive to use. It requires very expensive proprietary hardware, like Data Domain, for deduplication. CommVault, in stark contrast, is hardware agnostic for deduplication, and will deduce across any/all hardware, even on tape.
- Extremely wasteful of personnel resources. In our experience it required a dedicated administrator, working 50-60 hours a week for the three years we used it, just to attempt to keep up with the backups. That amounted to a cost of about $270K over the three year period.
- Because it was never designed or implemented as it should have, for a three year period we were completely vulnerable to a real life disaster. If the need had arisen to recover from, say, and tornado hitting our datacenter, we would have gone out of business because the NetWorker backups were completely unreliable.
NetWorker and CommVault are both rated in the top quadrant of Gartner's assessment of backup/recovery applications. We have used both for a number of years, and CommVault was the winner without any significant opposition technically from NetWorker. We moved from CommVault to NetWorker, not because we were having any issues with CommVault, but because management in our company had a knee-jerk reaction to an highly unreasonable cost estimate from a CommVault sales person. That was followed by a strong team of sales people from EMC coming in and "making a sweet deal" to IT management. Where it comes to protecting our company's data assets, NetWorker failed miserably. CommVault, on the other hand, passed muster multiple times during 96-hour Disaster Recovery exercises. Over a nine year period of using CommVault, the number of incidents we had were rare and very well supported. The only issue we had with CommVault was the rising cost of maintenance. NetWorker, on the other hand, had issues every single day, and support was terrible.
I would not recommend NetWorker to anyone for any reasons. It is huge, cumbersome, extremely problematic and EMC's support organization with rare exceptions is the worst in the industry. We have had nothing other than serious problems with it for three years, despite consistent attempts to get it working correctly. Every time an EMC engineer came on site (always someone different) that person would ask "Why did you do it this way? That's not the best way to do it." He/she would then change things to how they thought it should be set up (in direct contrast to what the previous EMC person had done and said). In other words, there are no consistent best practices across their support organization. This caused major issues for us, and left me extremely stressed knowing that there was no way in hell we could ever recover from a true disaster.
Dell EMC Networker Feature Ratings
5 - Systems administrators are the primary users of NetWorker, with two of the five being primary and secondary SMEs who had to spend 50-60 hours a week supporting the application.
2 - One highly technical individual, acting as primary support, spending an average of 50-60 hours a week trying to keep NetWorker running, and wasting countless hours trying to get EMC support to provide true assistance. One secondary, less technical person acts as secondary more to help with the workload and to be the "go to guy" when the primary is not available.
- File level backups of both physical and virtual servers.
- Sweeping native SQL backups from CIFS and writing it to tape for offsite retention
- Providing recoverability to the end-user community for files/directories on an ad-hoc basis
- Source for the restore operation of 96-hour disaster recovery exercises performed annually
- We were never able to get it working as designed, much less to get innovative with it.
- We have no plans to continue the use of this miserable product. We are abandoning ship with it as rapidly as possible.
There are three reasons for not renewing our use of NetWorker: 1) the rising and extremely high cost of support and proprietary hardware needed for deduplication, 2) the complete unreliability of the product (we couldn't recover from a true disaster if we wanted to), and 3) the horrible support from EMC for the product
Evaluating Networker and Competitors
Yes - We replaced CommVault with Networker. The only motivating factor there was cost: CommVault support costs were rising and the EMC sales team made a "sweet deal" to IT management to sway the decision to move to NetWorker. They essentially promised NetWorker and DPA for the cost of CommVault support.
I was strongly opposed to replacing CommVault with NetWorker. The decision was completely political and based on the EMC Sales team making deals with our IT management in order to get their feet in the door.
Yes: force EMC to provide a complete design document and implementation plan prior to setting foot in the building. Both of these we missing. The implementation was a major failure because, without plan or design; it was brought in and people who had no knowledge, experience or training were told to implement it across al critical servers within 30 days
How can anyone build a house without a blueprint? NetWorker was ramrodded into place here without a design or implementation plan. The result was a setup that was doomed from the start and never worked reliable over the full three years of our contract obligation.
- Professional services company
EMC sent an engineer to our site to assist with the initial implementation for just our critical servers. We did not have a design document or an implementation plan, and neither did they. The implementation was done on-the-fly over a 30-day window. Two months later a different engineer came on site to complete the implementation, and took a completely different approach. Each and very engineer who visited us had a different take on how it should have been done. In other words, there wasn't any consistency regarding best practices.
Yes - Phase one was done in a 30-day window for only critical servers due to time constraints. We had 30 days to get it done prior to our busy season lockdown. When we came out of lockdown phase two was started to complete the rest of our systems.
Change management was a minor issue with the implementation - We did not have an implementation plan, and the issues resulting from that were compounded by inconsistent best practices viewpoints from different EMC engineers. Our company uses ITIL change management practices which understandably slowed the process down while waiting for weekly CAB meeting approvals for agent installations and configurations on production systems.
- First and foremost, there wasn't any design plan or implementation plan. Everything was done "on the fly."
- Implementation was done without anyone in our company having any knowledge, experience, or training with the product.
- Configurations were changed often during implementation, depending on what the EMC engineer here at the time though was the right way to do it. This changed from engineer to engineer, causing configurations that didn't work as well as causing re-work to accommodate the engineer's ideas about the "best way" to do it.
EMC's support of NetWorker, with rare exceptions, was a nightmare. Their "follow the Sun" approach was a good concept, but did not work well for us. We spend huge amounts of time waiting for someone to get back to us after opening support tickets. When someone would get around to responding, English was usually not their first language. Consequently there was a lot of time spent saying, "Please repeat that. I didn't understand you." Initial responses were almost alway from someone who just wanted to email a link to a document to read--usually one that had nothing to do with the issue at hand--or would promise to contact us but would not follow through for days on end. Almost every time we had a serious issue it required phone calls and emails to support managers before a technician would even be assigned to a ticket. Horrible, horrible support experiences.
Problems get solved
Not kept informed
Difficult to get immediate help
Need to explain problems multiple times
Slow Initial Response
Yes - We require 24/7/365 support. When our critical systems are down the financial impact is huge. I must say that NetWorker really dropped the ball with their support.
Yes - Very rare were the times that we received timely and lasting resolution to such issues over three years of constant problems. Most of the time it took days to get anyone with half a brain to respond. Most of the time the technician did not speak English well at all, which made for huge communications breakdowns. These same people generally seemed to be reading from scripts, asking the same questions that had been answered multiple times already. More often than not issues required companies to EMC support managers before anything of substance was done to resolve the issue at hand.
We did have once exceptional support experience. EMC sent an engineer on site for a number of weeks to help iron out some rather serious issues. He was very knowledgeable, personable, and went out of his way to ensure we understood the what and why of what he was doing, and ensured that it was all well documented.
NetWorker has the clunkiest interface and unfriendliest CLI with which I have ever had to work. I spent three years hating this application because it took ALL of my time just to keep it running. Even then, I had no confidence in our ability to recover from a disaster because of its unreliability.
Do not like to use
Difficult to use
Requires technical support
Not well integrated
Slow to learn
Feel nervous using
Lots to learn
- Nothing. Nothing at all is easy or elegant with NetWorker. It is extremely complex, finicky, cumbersome and totally unreliable.
- Everything. Configuration and maintenance are a nightmare. It breaks on its own even when nothing is changed, and then after fixing whatever the issue happen to be, breaks again days or weeks later.
- Tape vaulting is completely absent from NetWorker. We had to write a custom in-house program in order to vault tapes for offsite storage.
- Unless you're a Unix administrator you will hate the CLI and its corresponding syntax. Much of the highly needed troubleshooting (and you will be troubleshooting every day) can only be done at the CLI.