We don't need no stink'in backups
« on: 2009-11-27 16:33:54 »
I really suspect, we are not able as a species, to hold a thought long enough to actual succeed with technology.
Lack of backup foils Va.‘s new IT system
Source :Richmond times & Dispatch
Author: Peter Bacque
Date: November 21, 2009
RICHMOND, Va. -- Virginia's expensive new state IT system does not have network backups in case connections between its computers fail.
In just five weeks this fall, the Virginia Department of Motor Vehicles suffered 12 computer system outages, putting individual offices out of business for a total of more than 100 hours. One outage lasted 29 hours, another 17.
"The problem of no-redundancy . . . accounts for 90 percent of our outages," said David W. Burhop, the DMV's chief information officer.
During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.
"That means people cannot do their work," said Murali Rao, VDOT's information technology director.
Problems with the state's outsourced computer and communica tions systems can have far-ranging impacts, affecting members of the public trying to do business with state agencies and government employees struggling to do their jobs in the face of computer crashes, lost e-mails and slow repairs.
"We've been literally having problems since July keeping the driver's-license system up," DMV Commissioner D.B. Smit said yesterday in a rare criticism of another state agency. "It's putting my people at risk because customers are angry.
"Every time we're down for an hour, that's about 2,500 people inconvenienced," Smit said. "They're blaming my people for it and [state IT officials] have an obligation to fix it."
In a unique public-private venture, Virginia agreed in 2005 to let the giant defense and information contractor Northrop Grumman run nearly all the state's IT systems.
The 10-year, $2.3 billion project aims to modernize 85 state government agencies' computer networks, PCs, phones, servers and e-mail systems, while holding down costs. The deal also provides IT services to about 1,000 local government customers.
Overseen by the Virginia Information Technologies Agency, the troubled outsourcing deal is the first and largest of its kind in the nation.
George F. Coulter took over as the state's chief information officer in August.
"The first thing I noticed was that the network that Northrop Grumman rolled out didn't have redundancy, backup," Coulter said yesterday. "The contract does not call for redundancy in carriers . . . in the network.
"Why that wasn't put into the network, I don't know," Coulter said. "This is a service we have to have."
Coulter is calling for an emergency meeting of the state's Information Technology Investment Board in the first week of December to deal with the lack of backup in the state's IT system.
"We don't have the pricing on it yet," he said.
"We're going to go after the critical agencies and the critical locations as a priority," Coulter said, pointing to VDOT, DMV and the state Department of Corrections as needing urgent attention.
"It's a major issue," he said. "We've got to get it fixed."
Virginia declared a state of emergency Nov. 11 in the face of record nor'easter rains and winds.
But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event.
"We called at 5:35 in the morning," said Gary Allen, VDOT's chief of technology, research and innovation.
"It took VITA four hours to open the help ticket" and begin to solve the problem.