Digital Edge

Danger of Building Cross Connects

Michael Petrov — Fri, 11 May 2012 00:00:00

We have no idea how connections from ISP or Telco equipment is delivered to our switches, routers or firewalls. The building management may sound professional and confident – “we will deliver you 2 lines from 2 independent ISPs”.
Well how do we make sure that the delivery mechanism will not use a single point of failure? You may spend time, and design your HSRP/BGP architecture to ensure failover and guarantee up time; but do you know if building management spent the same amount of effort? If you have your redundancy figured out, how can you enforce building management to respect your concerns about high availability, and do the same as you?
Well, we saw the BGP mess, during their building maintenance fiber cross connect infrastructure started going up and down.
Suggestion to IT management: Checks with the building on how to cross connects are delivered. Do not trust words and promises.

Wasteful Spending!

Michael Petrov — Mon, 07 May 2012 00:00:00

I am constantly advising our clients and other IT organizations to consult with us before making any decisions. Questions do not cost anything, however questions are the significance to cost saving. Unfortunately in a lot of cases TCO for IT departments has a cost of stupidity component.

One of the examples as I mentioned in previous blogs, Telco relationship, cross connects, ISPs, even though it sounds very simple may be very complicated, and can cost a lot of money.
I don’t want to just give details about an IT manager trying to save money, and falling to the first Telco cold call. Per se when Telco is talking to me, and quoting me on the price, first thing that comes to mind is how much more in hidden charges the client will be paying at the end. I am just curious why clients don’t ask and negotiate with Telco providers to deliver what they are promising.

So normally the deal unfolds like this:
Telco : we have a great promotion. You have your servers in New York and in Boston. How much do you pay?
IT; $3000.
Telco: we can do it for $2600.
IT: OK. Good. I can save 400.
Telco: (delivers their solution on time if you are lucky).
IT: Wow, great. How do I switch now?
Telco: Connect and test. Then switch.
IT: Where are the ports?
Telco: You have to ask building to cross connect us.
IT: Building management, we need to connect to the Telco.
Building 1: no problem. What are the specs for the cross-connection?
IT: 100Mbps, RJ-45, Cat-6.
Building 1: No problem – 260 per month for one run.
IT: wow, and this is on one end. On another end I know that the building management charges only setup fees. I paid once for that old cross connection so I will pay the same one time?
Building 2: we stopped doing one times charge and start billing on a monthly base, but the good news is that you don’t have to pay one time charge anymore. The charge for this connection is 220. But if you want, we can get you a fiber cross connect so in the future you can push more.
IT: Good idea. Let’s do it.
Building 2: Done – here is your fiber hand off. Now you have to either use a media convertor or have equipment in your LAN that is capable to receive fiber
IT: I don’t have a media convertor, no fiber capable equipment. There will be more investment in hardware…

Certainly it sounds absurd; but this is how it unfolds. This can go on and on during the switch over, and agreement with the building.

So my question to the IT community: Why do most clients let Telco say: connect to us. Why shouldn’t they be accountable to provide a full solution, and deliver it right into your rack?

The second question to the IT community: Why are providers allowed not to disclose all additional charges and related expenses?

Storage Failure- a Huge Success?

Michael Petrov — Mon, 30 Apr 2012 00:00:00

Yesterday Digital Edge encountered an EMC CLARiiON CX-4 failure. It was a major component failure rather than just a disk.

Centralized enterprise storage is the most critical part of any infrastructure, and any Enterprise IT engineer would have an unpleasant feeling, a feeling that would be described as a physical pain concerning storage failure. In certain situations with a bad design "everything" can go down, and who knows for how long and how much data will be corrupted or unrecoverable.

Serious storage failure in most cases ends up with some financial consequences, job loss; money and/or reputation loss etc…

So in this case, EMC had an SPB controller CPU failure, half of its internal architecture. It is as if a human would have had a stroke damaging the brain’s left hemisphere, leaving the person unable to move at all. However not in our case we were up and running, here is why…

The communication from servers to SAN was configured with Multi-Path where volumes were load balanced between SPA and SPB. Each volume had a preferred path with the ability to failover to the failover path. During the failure all volumes paths with SPB preference were migrated to SPA seamlessly and applications didn’t even feel a blip.

This is a result of a good design for Fault Tolerance on each single component the SAN – Servers, Fiber, Switches, SAN itself. In theory we would feel service degradation, but didn’t notice any slowdown as the CLARiiON is a very powerful device – at least for the client’s current needs.

As for recovery, we received the part from EMC, and were ready to fix the failed component in 2 hours. As for industry standards it is a very quick turnaround for hardware manufacturer, its support, and supply organization.

Lastly, I am sure that the failed CPU was not manufactured on US soil and rather produced somewhere on the other side of the world. However shortcoming of the manufacturing quality was greatly compensated by great device, and infrastructure engineering.

Oracle Deployment 11G RAC to New Hardware

Michael Petrov — Tue, 24 Apr 2012 00:00:00

No matter how you plan or how well you’re prepared, upgrading core production components tends to be a difficult task. This is exactly the reason why people are running on older software/hardware for years trying to avoid the upgrades, even though Hardware and Software vendors are pushing hard for upgrades threatening clients that they are not going to be supporting their products any longer.

There are multiple challenges in upgrading to a newer version, but all comes to three major questions.

- Will applications work with the new version without any new bugs or outages?

- Will new version perform the same way or better?

- How are we actually going to do that?

Digital Edge is helping one of its clients move from Oracle 10G RAC to Oracle 11G RAC. Oracle RAC is powering most critical business applications of the client, and is central repository for all its data.

First let me explain WHY we had to do it even though I think it is a bad move.

Why we had to do it:

a. We purchased a new EMC VNX SAN, and originally wanted to migrate Oracle ASM and Oracle cluster ware on the new EMC SAN.

b. The client wanted to get more performance out of Oracle, as they started to bottleneck CPUs. We were thinking about new hardware and started talking to Oracle.

c. Oracle suggested that to be in compliance with licensing we would have to change the CPU architecture. Here is the quote:

“Your current production cluster is based on 2 x R900 servers each with 2 x Quad Core Xeon CPUs each CPU running at 2.4Ghz

The recommended architecture is based on a primary 2 node server cluster for prod and a 2 node cluster for DR.

Each R710 is configured with a single very FAST quad core processor, this was specifically recommended and laid out in the
presentation so you Oracle License costs could be managed, if this was not configured in this manner and you continued
to run servers with 2 x Quad core configuration, you would be looking a license exposure well above $700K.”

So now we ended up with 2 Node Oracle 11G RAC in production, and 2 Node Oracle 11G RAC in DR with a plan for physical Data Guard between them.

Why it is bad:

a. I feel that the servers that we run on are pretty good. Those are Dell 900 with 2 CPUs. We had to get 2 CPUs vs 4 CPUs available to be in compliance with Oracle license.

b. Oracle 10G is on production boxes across the industry for a long time. Has its glitches and problems, but it is a known devil.

c. The main app and all supporting and secondary apps are tuned for 10G performance.

d. In attempt to comply with Oracle new license model we had to move to more powerful processes but have less processing power counting cores.

Here is what we have to do now:

1. Find hardware to deploy testing environment in parallel of PRODUCTION and QA environment that we already have; keeping QA for new releases of software. We also need to have enough hardware to produce adequate load to see how new implementation is configured.

2. Make sure that business users will not confuse TEST environment with actual PRODUCTION as it would look almost the same.

3. Make sure that we run all business cases but at the same time performing stress testing measuring performance base lines.

4. We have to plan for switch over process and measure the time for the switch over process.

5. During the client tests we have to gather base line stats and tune statements per new 11G optimizer.

As everybody knows the hardware cost is more about price of power. Electricity cost has increased in the last two years. So to bring parallel infrastructure up for a month or two, would require additional power, processing capacities, etc...

“This is a great application for public cloud” would jump someone who is relatively new in financial processing – not so fast. The data has to stay secured in the secured perimeter. We cannot just build client servers somewhere outside; also latency between cloud servers to Oracle would invalidate any real stress test.

Certainly our virtualized application layer helped a lot. We had enough resources in our VM clusters to build most of the parallel processing. However we still had to power up a few older physical servers to compensate.

Lastly even with strategic planning this big upgrade put a stop on multiple depending infrastructural project for at least month or two.

By the end of the project I will try to calculate the cost of pre-production test effort to have an idea what the technology upgrades actually cost to the end client and if the whole thing is worth it.

How To Deal With Telco Providers When Ordering Cross Connect For Your Business

Michael Petrov — Tue, 07 Feb 2012 00:00:00

One of the essential and most typically considered “basic” aspects of Enterprise IT is Telecommunication cross connections. For most IT management these connections are “simple”, but these “so-called simple” and crucial IT components have the potential to become the largest environmental pitfall. With hundreds of ISP’s (Internet Service Providers) or Telecommunication vendors, these critical connections become the victims of quoting battles, similar to other IT constituents- the Winner, most likely the glorified “lowest bidder”. However, before selecting the manufacturing vendor that agreed to meet your price-point, some apple to orange comparisons should be made that just might change the adjacent signature to the telecom contract you sign.

First, the BIG PICTURE–

You collocate equipment in a Datacenter and decide to have a private line for speed, security and insulation from the corporate office; Corporate Office- Point A, Colocation- Point Z.

The best situation is to have a Telco provider that has its own equipment in both places (POP – point of presents). This usually drives down cost and implementation time. Once an agreement is made, the vendor selected must configure a “virtual” path between equipment located in the POP of location “A” and “Z”.

From there, the scenario usually goes as follows- “We are ready, we have 2 ports setup, one in Point A and one in Point Z, now connect the switches or routers”. -This exact moment has to be discussed prior to ordering the line - what exactly will be delivered?

Does the provider deliver hand of right to your switches/routers or have a port for you in their equipment that is in the same location as yours?

To go back to the basics for a minute for those of whom made need a refresh, the connection between Telco equipment and your equipment is called a “cross-connect” and in the most cases is managed by the building. Telco from the building management perspective is a regular tenant.

Two huge factors- connection fees and distance: Telco from the building management perspective is a regular tenant, so building charge fees to capitalize as much as possible. For distance, depending on how far, the building might have to use fiber that would add an extra cost, level of complexity, and failure points. I fiber is involved but you agreed with Telco on Ethernet hand off, then there should be 2 media converters on each end of the fiber to convert FC into RJ45. Those converters are additional cost and points of failure.

Cross connect fees can be seen as a one-time installation fee that could vary from $1000 to $3000 or could be monthly ranging from $100 to $500 per month.

Going into a contract with a Telco and hoping to save $500 without accounting for cross connect fees on both ends might transform a mentality for savings into a position for losing money. Also delivery time of cross connects from the buildings can dramatically effect deadlines. Some facilities are forced to use union workers and may cause weeks’ worth of delays.

It is always better to make the Telco provider responsible for cross connects. They know how to deal with facility managers, ordering procedures, and LOA.

Another problem might arise when a Telco that is giving you a perfect deal doesn’t have its own hardware in Point A or Point Z or both. In these cases a so-called Teclo “swap” must take place. Telco’s negotiate between each other at both points to either purchase “last mile” or swap it for services/bandwidth in other locations. These “swaps” cost money, so the Telco that is the “closer” to the client will always have leverage over other competing providers.

If a Telco partner of your choice is planning to do the swap you always have to know about it. You need to know who are the partners in the deal and what chain of swaps are involved. There are known industry horror cases when Telco partners were finger pointing to each other for problems and end client downtime. There are also cases when the Telco NOC (Network Operation Center) would not even know about the swap and would not be able to find circuit ID’s to push a problem ticket to its swapping partner.

Because of the numerous issues that companies face daily, Digital Edge advises the following:

1. Besides the price look for Telco partners that would have equipment in both desired locations.

2. Do not go with a partner that has more than 1 swap deal. The delivery would depend on the swap third party that you would not have control. Only if there is no other option you should go with a Telco that would require more than 1 swap.

3. Negotiate a delivery mechanism and types of hand-off and cross connect responsibilities. Plan and explicitly specify how cross connects will be delivered and who will be paying for them.

4. Understand each cross connection solution, involved parties and NOCs. Keep records of all circuit IDs, NOC numbers, project manager’s and sales people contacts.

5. Explicitly tell your Telco partner to label each circuit with an extra label for example: “Circuit deliver point to pint line for XYZ inc. Do not unplug without authorization of following people:”

Work with the Telco partner to make sure that the label is clear and help avoid outages caused by human mistakes.

Ask Digital Edge Sales Engineering for advice on your future Telco needs; cross connect architecture, MPL architecture and other Enterprise IT aspects.

Enterprise Storage Selection Methodology

Michael Petrov — Fri, 23 Sep 2011 00:00:00

IT space is always overcrowded by all types of vendors. When a company has a need to purchase a new gadget, there will be hundreds of vendors around IT groups preaching products calling the shamelessly not less as the BEST on the market. No matter what they sell. It is hard for CIOs/CTOs/IT directors to make a conquest choice as we all know that vendor offers are sliced differently and you always compare apples to oranges.

The biggest goal of IT groups is to figure out how to compare offerings on a flat place, how to make them equal – make them all apples or oranges.

This is how we did it selecting a SAN solution for an entry-level enterprise implementation.

1. Narrowing the selection

It is hard to come up with an algorithm of how you would narrow the vendor selection. We can easily contact 20 vendors but it would be too much to compare. As an initial selection, we decided to review 3PAR, EMC, NetApp and Compellent. We, as a service provider, didn’t even go into this selection as it was no difference for us if the client will want to review NEC, HP or many other manufacturers. At this point we just got a list from the client who said they want to compare those guys…

After communication between our client and the vendors, the client narrowed the selection between EMC and Compellent. This is where our work as trusted consultants started.

2. Creating a “flat place”

Here is what, in our opinion, counts when you select storage: capacity, power, features.

Capacity:

Capacity analysis should be broken down by:

Raw offered capacity

Raw capacity is easy to calculate number of disks and their sizes. Naturally, it has to be broken down by the class of disks as faster storage costs more and has smaller sizes. IT should never get tricked by marketing talking points like: “our storage software logic makes it so fast that you don’t need many fast disks”. Do they think we are stupid?

Derived usage capacity

In the past it was very easy to calculate usable capacity. Today storage vendors offer different types of RAIDs, pool algorithms and configurations that have certain limitations. Those limitations create lower physical disk utilization, require more hardware and sometimes result in usable space loss or underutilization. Down the road you may easily hit a wall where the array would not allow you to use RAID 5 with 3 disks and would force you to use only RAID 6 with a predefined number of disks. So we would strongly recommend creating a disk usage map, RAID groups and pools at the time of storage selection.

Maximum capacity

Maximum capacity is the number of disks that offered controller configuration can spin before controllers have to be upgraded.

Power:

It is very hard to analyze power. There are many different methodologies to do that. What makes it difficult is that the same application can work differently on different setups. For example, there could be a big debate on what is going to be better to throw? 10 SSD disks in a disk pool or use them as a fast cache. Maybe it is better to use 5 SSDs as a fast cache and 5 SSDs to add to the disk pool? And if you do that, maybe it will be impossible to use 5 SSD disks in the mix with SAS and SATA disks in the disk pool and we need 6 of them.

Here are the parameters that we wanted to use in order to analyze raw power of the storage:

Total Controller RAM

How much RAM controllers have. (ßI’m not sure what he means here.) SAN controllers are computers. Some of them run chopped off a version of Linux, some of them run proprietary OS but they all use RAM. The more RAM controllers have, the faster they work. There is one huge subject – OS version. If SAN OS is 32 bit – the RAM limitation would always be 4G. 64 bit SAN OS would not have this limitation.

Fast Cache

Fast cache plays a role of shocks absorbing I/O spikes and we feel it is a very important part of a SAN implementation. Some vendors don’t have fast cache at all claiming that the algorithm is optimal as it is. So if you select a car, would you get absolutely the same engine without TURBO or with it if the price is the same? The same goes for fast cache. The more of the fast cache is proposed, the faster the system can be.

Total Ports and their aggregate speed

For sure port availability should match. If one SAN offers FC and iSCSI and another one only one of the options, you would probably select the storage that offers both types of ports.

iSCSI could have 2 speeds: 1Gbps and 10Gbps

Fiber could have 4Gbps and 8Gbps.

In a lot of cases you would hear that if you have a high performance application you should go with Fiber. You would ask if you have 1OGbps iSCSI, isn’t it “faster” than fiber? Not necessarily. The problem is in the frame size and the way data is encapsulated. The detailed analysis of the iSCSI vs FC is outside of the scope of this document and should be performed using application specific data. However, the cumulative speed of all ports could give you a good idea about the power of the system.

Maximum throughput on the front end

Technically it is the total port aggregate speed. So if the array has 4x1Gbps of iSCSI ports and 2x4Gbps of FC, the total would be 12Gbps. No matter what you do, you cannot pull more out of the array without changing or expending the controller.

Maximum throughput on the back end

This is the throughput from controllers to the disk enclosures and disks themselves. The speed depends on the architecture of the array and cannot be easily expended.

Number of IOPS (Input output operation)

It is a very important parameter. This is how many input output operations the array can perform in a second. In contrast with throughput, the IOP can be very short but there might be a lot of them. In a lot of cases OLTP database would be an example of when the throughput is not so important but number of IOPS is something that would affect the performance of the system. Number of IOPS depends on number of physical disks (“spindles”) in array (certainly SSDs are not spindles but have their own number of IOPS characteristics).

As an example – 4x600GBx1500RPM SAS disks would give 1.2T of storage. At the same time 8x300GBx1500RPM SAS disks would be the same size BUT WOULD PRODUCE TWICE MORE IOPS. So database systems would perform better on the second configuration.

Important features

All SAN arrays offer different features and comprising is out of the scope of this document. We wanted to point out features that can affect total cost of ownership.

Power consumption:

As power is expensive today, total power consumption is a very critical parameter. It is linearly dependent on the number of physical disks in the array as one single disk can pull up to 0.5Amp power consumption. So overall it is better to have fewer disks which contradict with the IOPS recommendations.

Maximum number of disks

Some SAN vendors would say that they can spin unlimited amount of disks. There is no theoretical limit; however, there are CPU/memory resources needed to control/spin disks. When the number of disks is going up the controllers will slow down. So it is very important to know when there will be a need for an upgrade.

This table shows the comprising of 2 different SAN arrays based on our methodology.

	Compellent	EMC
SSD 100G	0	10
SSD Storage	0	1T
SAS – 15K RPM 600G		41
SAS – 15K RPM 450G	48
SAS – 15K RPM 300G
SAS – 10K RPM 300G
SAS Storage	21.6T on 48 spindles	24.6T on 41 spindles
SATA – 7.2K RPM 2T	12	7
SATA Storage	24T on 12 spindles	14T on 7 spindles

Fast Cache	8G	200G
Max Fast Cache	64G	1T

Disk trays	4	4

Ports FC	8	8
Ports iSCSI	4	4

Max Throughput on front end	64Gbps in FC + 4Gbps on iSCSI	64Gbps in FC + 4Gbps on iSCSI
Max Throughput to back end	96Gpbs	92Gbps
IOPS Based on Drives in array	Could not get	38,640.00

Max number of disks	150 (estimated)	250 (limited)

Power consumption	2.8KW	1.47KW

OS	32-bit	64-bit

3. Comparing technical details

Based on the comprising table we can see that:

EMC provided better disk configuration.

The max throughput is the same from both vendors.

IOPS could not be obtained from Compellent

EMC provided Fast Cache while Compellent claimed that the algorithm of the software would compensate.

EMC is natively 64 bit. Compellent was offered in 32 bit configuration with an ability to upgrade to 64 bit in a few months when it will become available.

4. Cost

At first, Compellent offered a price close to EMC but was around 3% more expensive. When Dell found out that EMC is winning the deal, at the last moment dropped prices almost 100K down. Our client felt it was not ethical for Dell to do that and still wanted to go with EMC, however showed the prices to EMC for comments. EMC provided additional discount but could not come close to the lowballed price of Dell. The client still selected EMC as they felt that Dell may try to recover the loss of revenue through further charges.

Complexity of cleaning up networks with SAN/NAS environments in case of virus outbreak.

Michael Petrov — Wed, 12 Jan 2011 00:00:00

Sometimes clients do get viruses. There are cases when regardless of all security and antivirus measures computers still get infected.

This case introduced here is interesting in multiple ways.

1. The way the client got infected. We had a snow storm on the East Coast. People could not get to work so we started getting requests from management to open VPN connections to the environment for a lot of people. Before the snow storm we had it opened for management only and assumed that people on the VPN are clean and we were letting them into the secured perimeter over encrypted tunnel without any limitations. We did not consider massive VPN connections from home computers and got a user that brought an unknown “creature” to the network.

2. Even protected servers got the infection because the virus managed to disable enterprise antivirus software and infect most of the network. This was a failure on the part of the antivirus software but I don’t want to mention the vendor name here. Unfortunately we didn’t have time to learn the behavior of the virus and figure out all the details of the infection and propagation.

3. We enjoyed our segmentations of functional blocks so we got infections only on terminal servers and it didn’t spread to database servers, mail services and other heavy lifters. So the segmentation philosophy proved itself again. The way we think is that we should manage each functional segment separately versus practices of managing things centrally. Some admins like to manage one big firewall, one big database etc. Even though there are advantages of minimizing managing points, those architectures expose networks and systems to bigger failures. We do know that technology fail where you would not expect no matter how experienced you are. We all had cases when regardless of planning for failures, redundancy, backups, something still happens that nobody could imagine. So segmentation reduces the risk of global failures localizing problematic components.

4. The client has a huge NAS running by EMC Celerra. More than 8 million files are hosted on one single share and we have 7 shares. When antivirus scans through those shares, it takes more than 5 hours. So infected users could plant viruses back on NAS to the areas that were scanned already. So we were constantly chasing our tail. We also use the same Celerra as a native SAN powering SQL Server cluster, Oracle RAC and Exchange cluster. Those servers and volumes were not affected by the virus because of our segmentation policies.

5. Microsoft response. We were actually very surprised by Microsoft help in this situation. It took much longer for the antivirus software vendor to react so we decided to contact Microsoft. After we opened a PPS ticket we got informed that virus outbreaks are Microsoft priority and any fees will be reimbursed back to us. We got an American support team (not offshore). Very knowledgeable person. He told us why we are having such a problem cleaning up – apparently the virus infects DLLs and EXEs and encrypts them. So decryption of such files is a problem. Some antivirus software doesn’t pick them up because of that. The virus is also polymorphic, so its signature changes from time to time. We have explained the way we found successful in preventing virus spread. The engineer told us that we are on the right path but gave us an official “Microsoft” way – a KB article - of preventing spreading like this.

Besides that Microsoft provided us with two tools. One is for online scanning, the other one for offline. Those tools are not auto-updatable, but IT groups can call Microsoft at any time and request fresh copy of those tools as they are updated by Microsoft team every 6 hours.

Online tool is to scan memory and file system of production servers. However if the server has a ROOTKIT then the tool could be useless as ROOTKITs hide viruses. To clean such server, the best way is to rebuild the whole server. Another way is to shut it down, reboot it using a CDROM disk built with the second – offline Microsoft tool and scan the server while OS on local disks is not loaded in the memory.

Another way to detect ROOTKITs is to share disk drives and scan shares remotely from a clean server. ROOTKIT hides viruses and itself because it is patching OS, change memory management and file system. When you are remote, it cannot hide and antivirus on the remote machine would detect infected files including rootkit itself.

We saw cases when remote scan detected infected file, but when we were on the infected server, we could not see this file at all as ROOTKIT was hiding it.

After successful blocking of spreading and offline scanning of servers one by one we could successfully clean up the whole environment. The most important for our client was the fact that they were not down a single minute as our architecture allowed us to keep the system afloat even though we had to do offline scanning on multiple boxes and viruses and virus scans made some production elements slower.

Lessons learned:

We learned that our segmentation philosophy worked great as we kept production of the client uninterrupted.

We also learned that we really need to limit user’s abilities on the network. The users of this client had too many rights and privileges on terminal servers.

We decided on a plan to upgrade the client’s network topology helping preventing viruses from spreading, stealing passwords or data.

We developed a constructive plan to implement better AD policies, terminal services usage policies, VPN and other.

We also are in the process of rebuilding some elements of infrastructure using virtual private cloud elements. Virtual server architecture will allow us to rebuild servers from snapshots quickly rather than cleaning them up.

It is very strange to see some IT and Financial executives’ decision process.

Michael Nikhamov — Thu, 16 Dec 2010 00:00:00

Here is an example. A company is interviewing datacenters partner for co-location and full management services of their network infrastructure and servers. The infrastructure consists of about 100 servers and 40 network devices and appliances. After providing them with our proposal we found out that we are the most expensive provider.

We could not believe this. Our prices are very competitive. We certainly cannot get our hands on our competitor’s proposals but after questioning IT staff of the organization we found out that our competitors offer the same colocation AND full management price as we propose for colocation only. We were in shock. We thought that full management would lack some services and started hassling for more information. Asking if DBA services are included, if change control is included, if patching is included – the answer was YES!!!

Having 140 devices infrastructure and including full support pricing in the colocation cost sounded insane!!! How can you compete with this??? And more over how can anyone deliver those services in such budgetary pressure??? Should just close our operation and go out of business???

Here is how we do this. We cannot provide full management services if we do not deploy the client in our management platform. It includes device repository, monitoring, knowledge management, ticketing and task routing system, scheduled activities, change control, backup and recovery controls and other important management aspects. Only by wrapping our management platform around the client infrastructure we can meet SLAs. We would know how the server is doing 24/7. Each server is like a patient in a hospital connected to sensors by multiple wires so nurses and doctors know what is going on. By deploying clients infrastructure on our management platform will allow clients to be certified by SAS70 or HIPPA or PCI relatively easy and painless.

We are patching servers at least once in a quarter. We feel that each server besides monitoring should be visited buy a human administrator at least once a month to just browse through the log. We are sending our techs to check physical alarms on devices every week. We operating backups to stay close or sometimes exceed requirements of SAS70 or HIPPA or PCI.

Besides basics we know that there will be daily client’s requests for changes, performance bottleneck resolutions, tuning, troubleshooting etc. For infrastructure of this size, there will be a hardware problem once in a quarter. There will be a need for security assessments on regular bases etc.

How can you include all that in the co-location cost?

Well, we were lucky. We got the information from the potential client and figured out what was wrong. The competitor gave them full management including EVERYTHING you can imaging in the IT world but limiting them to only 20 hours per month. It comes to 9 minutes per server per month!!!

If you take into consideration the excitement of the client’s executives that they can get FULL management for that cheap and our way of precisely assessing client infrastructure’s needs and quoting exact services and taking a risk of underestimating needed hours you would understand the disconnect.

Maybe we are just too precise to provide unparalleled consistency. Maybe we are stressing the importance too much?

Through latest sales cycles we can see that our competitors can go away with hazy definitions and still win the business. Get the client in the database canter then figure out how to make money on small things. This looks like novo days strategy of a lot of MSP providers.

I know one thing – the way Digital Edge treat’s its client’s technology – this is the key to the success. This is how IT groups of our clients become successful. This is why Digital Edge shows 100% uptime records for at least last 10 years. And this is why our clients say that their systems are stable and Digital Edge removes the stress from their operations.

Storage, thinking ahead…

Irene Goltsman — Mon, 13 Sep 2010 00:00:00

There is still a need for intelligence behind the technology even though hardware and software vendors are fighting hard to simplify complexity and hide details under the hood of GUI and wizards.

Coming from the system operation and integration business and sitting on sales sessions I see that all sales are done exactly like sales of electronic gadgets. “My hardware can do this, and this and this and even this!!!!!” At the same time if you look at your GPS gadget, you can see the percentage of the features loaded in the device that are actually being used.

The same logic should being applied when we are thinking about the storage. Before having the discussion about the storage, first companies need to access how it will be used.

Below are some simple questions that should be answered before interviewing storage vendors as each small feature will cost additional money and may be not utilized at all.

Do we need fiber, iSCIS, NFS/SIFS? Wrong answer to those questions may result in big additional dollars in licensing, hardware and software. If you are using iSCSI only then there is no need to purchase unified storage. If you need only high performance NFS, there is no need to purchase iSCSI and fiber. And it is not always true that you can gain max performance only out of fiber.
Disk speed. Disks are the most expensive component of any storage system. Same size standard SAS or SATA disk will cost 2, 3 or 4 times less expensive disk from EMC or NetApp or 3PAR. There has to be an intelligent analysis of I/O to purchase what you need. In a lot of cases vendors through in the mix thin provisioning, de-duplication and other techniques that also cost money. De-duplication would be total waist if you store compressed images. De-duplication and thin provisioning cost money as well and needs to be accurately analyzed before purchasing decision. Use of faster disks where it is not necessary is huge waste of money. There is confusion about I/O analysis. If there is a sequential reads then controller and switching speed is more important than disk speed. If you have short I/O, like database transactions, the importance of number if physical spindles (physical disks) much more important than anything else.
Do you need replication? And the most important is what kind of replication you need. There is a simple bit-to-bit replication or complicated mechanisms involving snapshots which will require additional disk space. The application needs must be assessed as we saw cases that our clients were purchasing complex replication mechanisms but application side could be using it. Also a lot of enterprise products have built in replication. Some enterprise storage replication mechanisms are priced per amount of data that needs to be replicated and add significant cost to the solution.