Backup

Did you have a backup data center outside the city? How did that work?

We were running out of California on the back-office system. On the first day of the hurricane, we were able to migrate all the Web site functions to Dallas.
Was that a prearranged plan?

I'd love to tell you 'Yes' -- that I had that foresight. But honestly between you, me and the grand piano, I was trying to migrate away from California and get back in house. And I was very thankful that was one of my timelines that slipped. I (would have been) flipping on the New Orleans migration that week. So I learned a very valuable lesson there.
And that is?

And that is... there is no one hardened environment that is anywhere near as powerful in a disaster as a distributed one. Period.

In other words, I would not say, 'OK, if New Orleans goes away I still got Houston.' It still wouldn't work for me. What happens if Houston's not there when this happens? You're still putting yourself at a single point of failure, is my point. So what you do is think in terms in of pure workflow. What are my critical things? Dispatch. What does that really involve? Well, it might include home data to CAD (Computer Aided Design) data to federal data. And you build a system and a workflow around that. And you can do that via relational databases. You have to have your process flow across those things, various supporting infrastructure, if that makes any sense. It's kind of out there.
Your strategy would have to originate with the CIO, not a vendor?

It has to come from the CIO. The CIO has to be much more of an enabler and less of a keep-the-trains-running kind of guy. The reason I got all the other departments I got is because I was fixing things. You've got to be focused on fixing things, not just keeping things running.
And now you are on your way to pick up an award from the Center for Digital Government and Education?

We were ranked 70th out of 70 major cities (prior to my arrival). We had an all mainframe shop, completely 100%, and a one-page Web site with a picture of the mayor on it -- which, by the way, when I got here, had the wrong mayor on it. We were dead last in everything. What's funny is we had a budget higher then than we have now.

What we did was focus not on gee-whiz stuff, but bang-for-buck stuff, to get the cash. It's like those IBM commercials about things that don't really happen in the real world. I didn't have the luxury of only flipping a switch for this department or that department because I knew I would have to do the back-end integration, and there goes all my savings. So if I flip it all at once, and get voice and data at the same time then I really do only buy one switch. And I really do save the cost of it.

People say, 'Man, you did the largest VoIP in one year. You did 2,500 phones. No city has ever done that. Man, you must really love VoIP.' I say I couldn't care less about VoIP.' So why did I do it? The features? Or the Web browser? Nope, I'll tell you one reason I did it. The same reason we did everything: saving money. Because in the end, we had a $3.2 million budget for phones. And $1.1 million of that was getting the Bell South guys to keep moving the same damn lines back and forth.

They charged me $100 per hour to do that. With VoIP, I plug it in -- and the number follows me. I think I can save $1 million per year doing that. Then we said, 'Well how do we do that? What we can do is get the VoIP. We flip it all at the same time; we count the dead lines. So we turned off 25% of the lines, right there.
And is that the basis for the award, those productivity gains?

No, actually, that's the kind of weird part. That is the stuff I'm kind of most proud of, but no, it was pure functionality on the Web site. We went from zero to 30 online services. We built in a lot of access for the handicapped and whatnot. They don't even know that we built it into a product, which is why we could do the hurricane stuff, why it morphed so quickly.
Your surveillance-camera project got a lot of national attention prior to the storm. Did those survive Hurricane Katrina?

Those stayed up. They stayed up, man. They stayed up in a Class 5 hurricane and not only that -- even the Feds started using them for evacuation. They ended up becoming a signature through this whole thing, because they stayed up. And we're using them now. There's a lot to it. It's not one thing about the cameras that makes them unique. It's the fact that it takes that super high res type camera, combines it with motion detection, separate motion detection software that walks a virtual beat with PTZ, pantone zoom, in conjunction with the way the images are processed and captured. It preserves the chain of custody and limits bandwidth issues while still giving you clarity on the suspect's face, etc. By the way, that one sentence you have no idea how hard that part is to really do.
Get clarity?

Well no, you can get the clarity. But the fact is that with full-motion video on high resolution, I'm going to need build up the Internet the size of Texas to hold all this stuff.

You have to keep the bandwidth down through a series of frame-grabbing things, but also keep your chain of custody clear so the lawyers can't it throw it out. So we had to go through a lot of rigmarole and ACLU guidelines. And then on top of that we … didn't have this huge network to handle that bandwidth. So we had to make them completely mobile and peer to peer, it was really a gumbo of a lot of stuff.
How did you find out they were up?

Well, we were driving around by Office Depot and looked up and said 'Hey man, look there, they're still up.' It was amazing. That's the good part about the fact that we were forced to make them bulletproof. I guess they were hurricane-proof too.
What is the one image that epitomizes Katrina for you?

I think it's when I was handed the phone that I took from a looted Office Depot with the President on the other line. It showed how thin government got.

Air Force One calls and you have to call a number back for security reasons. I said, 'Mr. Mayor I've got Air Force One on the phone that I just stole from Office Depot yesterday.' Stole is probably not the right word -- commandeered -- but that defines it.

The tragedy of it, for me, was that we went through six days of hell and then the guy I was bunking with killed himself. It was both of those things. It sounds cliché but it was really, really a one-of-a- kind triumph and one-of-a-kind tragedy. Actually, there were a lot of moments that I won't forget. There was also, frankly, pulling people from the water. I hate the way this sounds, but I've got two Mercedes and a 60-foot yacht and I've traveled the world, and all that stuff. But there's something about pulling somebody out of the water that is just a wonderful feeling. She had broken ankles. The fact that I could carry a lady with broken ankles and put her in the back of a Humvee... It's that feeling. I won't forget that. I won't forget the bad part. But I won't forget the look when somebody's there and you're pulling them out. You just never get a chance to actually save a life. That's better pay than anything. I've lost a lot of money from lost opportunities -- and just money -- by being a civil servant. But that kind of pay you just can't get anywhere else.
The Web site has morphed in recent days to include press releases, the interactive map showing flood levels and other services. How did you prioritize these?

That is something that came out of my private sector handbook. When we built our Web site, for example, we built our own content management system, verticalized for government. We instinctually did that, instead of just putting up the Web site.

What that allowed us to do, and it's so much easier even than FrontPage, because you literally are able to add functions for credit card costs that really work and take into account all the government factors of doing that. We built a product on that. Before we were low tech; New Orleans had no reputation for tech. Then Steve Ballmer was bringing New Orleans up once a month I heard, talking about Great Plains and our help -- and we were just a stupid little city doing that.

But the Web site doesn't go down and it doesn't crash and we're able to add really complex services back in and out -- because of this content management system we run it on. So we moved that to Dallas (due to Katrina). I've got a handful of Web guys here and they just log in and move objects around. You're going to continue to see that Web site morph from rescue and recovery to now, restoration and things like that. And we're able to do it in the middle of our trimmed-down, army-fatigue-type setting we have here. And just move the objects around.

For instance, we turned on a donation type Web site. People said 'You've got to do one for New Orleans.' And literally 36 hours total, from start to finish, from the mayor saying 'I want to do that,' to us making it live and taking credit cards, we have a Web site up. That takes credit cards. That runs to the government account. That has all these government-oriented ways of doing things. Bureaucracy is kind of built into the product. We're very seamless here. We don't have a rigid customer-vendor thing. It's much more accurate to view the city of New Orleans and our relationship with our contractors as though we were business partners.
New Orleans CIO Greg Meffert and his IT team are due to receive an award this week for having improved their city's portal in a pre-Katrina world. Now the custom-designed features on that portal, and its flexibility, are allowing New Orleans residents to see aerial views of property lots and get the latest information on rescue and rebuilding efforts. Meffert has a private-sector background and a reputation for rewriting his public-sector job description so that he directs several public departments. He has an IT staff of about 100 and a $30 million budget. And now his resume includes Hurricane Katrina, which forever changed the way he looks at disaster recovery, and distributed models. Here Meffert talks with SearchCIO.com about his most harrowing moments in recent weeks and tells us what it's like for a tech guy to be designated to the front lines.

What is the difference between record-keeping of VoIP messages versus traditional telephone messages?

There are going to be different types of records created by telephone calls when you do things digitally. When you do things digitally as opposed to the old-fashioned way, it creates new challenges in terms of retention. For example, if you look at old voicemails, analog form, there wasn't much expectation in the way of preserving them.

With digital voicemail systems and systems that turn voicemails into wave files that then get e-mailed, now you have this whole new possibility and treasure-trove evidence and information that would be potentially subject to preservation obligations, just like any other form of information. The key thing to remember is that the type of media in which the records are stored is largely irrelevant when it comes to determining your obligations to preserve. And, as the types of media that are creating these records with different types of digital information multiply -- for example, records created through VoIP -- it becomes more and more critical for companies to be very focused on their policies and practices regarding information management.
How does a company decide what to retain?

What you need to retain is going to be dictated by subject matter, not by type of media. So, for example, if there are records created by a VoIP system that deals with your 10K, the fact that some records are created by VoIP has no bearing whatsoever on your preservation obligations. You're going to have to figure out a way to deal with that. You can't say, oh well, this is stored in this type of media and these records are created by this type of software application, therefore I don't have to worry about preservation.
When you talk to companies, do you find that many believe they don't have to keep a record of it because it was done over the phone?

Absolutely, there is a lot of uncertainty in terms of what exactly is the extent of preservation obligations with respect to certain types of media. The big issue that still predominates that discussion is backup tapes. While it is entirely possible that at the end of the day the court might say, 'Well, I really don't think it was reasonable to expect you to preserve that type of information,' the way the preservation obligation is generally interpreted is more media neutral. At the end of the day, there might be arguments you could make in terms of burden and cost, as to why you shouldn't have to keep that information, but in the absence of a ruling that says, for example, VOIP is not the kind of information you need to preserve, you'd better preserve it, if it's relevant to subject matter that falls under some preservation obligation.
What are the biggest errors in judgment companies routinely make when dealing with electronic records?

One is keeping information that they're not required to keep. The consequence of that is tremendous cost, when in response to either regulatory investigation or litigation they are required to retrieve and search that information and review it for production. They find they have needlessly multiplied their burden by keeping information that has no business use and wasn't governed by some legal preservation requirement.

No. 2, is not having thoroughly thought out and implemented information management policies and practices. You would be amazed at the big companies with vast sprawling corporate networks generating gigantic amounts of information -- a lot of it very sensitive -- that have not made much headway into implementing policies and practices, so they can have some measure of control and can explain why they have certain information and not other information.

No. 3, is they are not in touch with the de facto information policies -- what actually happens at the company. A lot of what happens is driven by IT people. So, for example, somebody in IT decides that because of storage capacity issues, they are going to purge e-mail on active servers every 90 days. Then a litigation happens, or there is an investigation, and either no one was aware of the purge or thought to communicate with IT that they need to perhaps to suspend the purge.

This gets to the heart of our audience. So CIOs need to be brought into the loop?

Absolutely, the interface between CIOs and lawyers is the story. In all these cases where companies have been punished for losing electronic information, 99% of the time it can be attributed to some kind of communications failure between lawyers and IT people. Not bringing IT into the loop on legal issues is a common and serious mistake. Morgan Stanley is probably the most prominent example. A few years back, there was a case, Keir v. UnumProvident Corp., a big insurance company. The decision gives a fascinating inside look at what happened in terms of the miscommunication between the outside lawyers and the in-house lawyers down to the inside tech people at the company and their vendor, IBM, which handled their backup systems.
What makes VoIP messages such a potential nightmare is that to produce voicemail that has been sent and saved digitally, you have to listen to it real time and transcribe it.

That's right, and the burden involved in that may result in not having to produce it. But it might not, and when you're dealing with regulators, they are less sympathetic to the burden argument.

Now you don't have to create records that wouldn't otherwise exist. If it is not your normal practice to record those oral communications, you're not required to go out and record them and create records just because you have some preservation duty. It doesn't mean I now have to walk around with a tape recorder and anytime I say something to someone that is relevant to a litigation or investigation I now have to tape record it.
Adam I. Cohen is a partner in the litigation department in the New York office of Weil, Gotshal & Manges LLP. Nationally recognized for his work on discovery and document retention issues associated with electronic information, he is the co-author of Electronic Discovery: Law and Practice. The authoritative 2003 primer has already been cited in four landmark e-discovery decisions by federal district courts. SearchCIO.com asked Cohen how CIOs should be treating that murkiest of electronic records -- Voice over Internet Protocol (VoIP) data. The takeaway? Do exactly as company lawyers tell you to.

Why consolidate?

We decided it was insane trying to keep up with maintenance, staffing, heating, power and air conditioning at 10 sites. It wasn't just the cost; there was across-the-board redundancy. For example, in one data center we had good redundancy and in another we had none.

Operationally alone we're going to save $8 million to $10 million per year. That is a savings on technology reductions -- fewer servers, fewer licenses, fewer maintenance agreements, smarter use of technology, including virtualization of our server space -- and then personnel reductions.
Does that savings include heating, cooling and power costs?

No. For example, one of our data centers is in a building that operates five days a week, but the data center doesn't have a separate cooling or heating system, so we're paying to heat or cool a building that is over 100,000 square feet for seven days a week because we don't have a separate data center.
Are you buying new boxes and consolidating the data or are you moving all the current servers to the new site?

A mixture of both. We are centralizing a few boxes [picking up and moving], we are rationalizing as many as possible [taking off of current servers and moving onto common platforms but larger systems].

E-mail is a great example of the rationalization. We are moving from 220 distributed servers to 19 consolidated ones. From a cost of up to $15 per mailbox per month down to $3.
What benefits will you get from this project?

One of the things that is glaring -- one of the 10 data centers built 15 years ago, has planned downtime of 22 hours per year. We've calculated that loss of productivity to be $2.7 million per year. With the new facility, we'll have 1.6 hours per year of planned downtime.

It will be the difference between a tier-one data center and a tier three. We could have made a bigger investment for a tier-four data center, but for the state of Wisconsin, we didn't see that as a need.
What's the difference between a tier-one data center and a tier-four center?

A tier one is a data center that was designed in 1970 with the specs of 28 hours downtime. There is no redundancy in a tier-one data center and only one power path. It was designed for water-cooled mainframes. A tier three and a tier four are comparable. They have redundant components; if the power goes out there is a diesel generator. They have two power feeds from different energy companies. A tier three is designed to be down 1.6 hours and a tier four has no downtime.
Will you have a mirror site?

Our backup site will be a lights-out facility and it will be located 15-20 miles away. There is a limit of 25 miles because we can't effectively mirror beyond that distance.
What will you move first?

We're going to eat our own dog food. The department of administration servers are moving first. We're moving before having to put another department through it.
When Matt Miszewski was named CIO for the state of Wisconsin two years ago, he inherited 10 data centers and 2,500 distributed servers. Recently, Miszewski told SearchCIO.com about his aggressive plan to move the state's servers into a 36,000-square-foot data center -- an eight-month project slated for completion in December.

Symantec Corp.'s Fire Drill storage backup change monitoring software is one example. "There's definitely a lot of attention being paid to this [product] space...

install and maintain security and backup systems rarely interact with the business managers whose day-to-day jobs depend on the services those systems guard...