Correctional Service to hire its own IT after being 'drastically' hit by Shared Services's data centre outage
Botched fire alarm test caused 2017 outage, damaging email and systems including one monitoring offenders
Canada's prison service is so fed up with Ottawa's IT agency that it wants to hire its own computer specialists — which would reverse a 2011 policy that required departments to surrender all their IT work to Shared Services Canada.
The frustrations of Correctional Service Canada (CSC) came to a head last October, when a vital data centre went down for almost 20 hours, cutting off email and more than 80 other key systems.
The Oct. 3, 2017, incident was triggered by a botched fire-alarm test at 6:45 p.m., in which power was cut for three hours because of human error by a third party.
The outage damaged BlackBerry and email systems. So-called "mission-critical" systems were slow to recover. Technicians finally got everything working again by 3 p.m. the following day.
The then-head of the correctional service wrote a blistering letter to Shared Services Canada (SSC), accusing the agency of being asleep at the switch.
"SSC's inability to identify that CSC's … Data Centre was unavailable for an extended period of time … drastically impacted our ability to effectively manage the issue in a timely fashion," Don Head wrote to Ron Parker, Shared Services president on Oct. 24, 2017.
"Neither notices nor alarms were triggered as a result of this shut down, causing staff to struggle in identifying the problem and formulating a recovery plan."
Head, who retired from public service Feb. 3, noted that the hardware that failed was "very old," never having been updated.
"This type of situation has previously been raised with SSC," he said. "For example, past problems relating to a lack of available storage space on CSC's servers, and monitoring of server uptime, could have been more rapidly remediated or avoided with better monitoring by SSC."
Numerous complaints about Shared Services
The letter said the prison service wants to "internalize" some responsibilities that had been handed over to Shared Services when the agency was created in 2011 to take over all federal IT work.
A section of the 2017 federal budget, which came into force in June last year, allows departments to repatriate some IT responsibilities — a measure the Liberal government allowed after getting numerous complaints about Shared Services's poor performance.
CBC News obtained the letter and related documents under the Access to Information Act.
Spokespersons for both agencies say talks are continuing over the transfer of IT work, including who will pay.
Neither notices nor alarms were triggered as a result of this shut down …- Don Head, former commissioner of Correctional Service Canada
CSC's Megan Hooper said managers want to take back control of monitoring, network planning and equipment support, while leaving systems platforms, such as server hardware, under the control of Shared Services.
In a joint statement to CBC, Shared Services Canada President Ron Parker and Anne Kelly, interim commissioner of Correctional Service Canada, said that "CSC's IT services such as email and data centres will continue to be supported by [Shared Services].
"SSC takes its mandate of data centre management and oversight very seriously."
Among the 82 systems knocked out last October, Hooper said, were those used to manage offenders, provide security intelligence and other information to police, and to handle accounting and human resources. All departmental work was either delayed or carried out manually. Landlines were used in place of BlackBerrys and email.
"In the past, unplanned data centre outages have been due to events such as severed network links, air conditioning failure or electrical system failure," she said.
Other agencies and their outages
Neither Shared Services nor CSC would reveal the location of the data centre, used only for prison-service systems, for "security reasons."
Charles Anido, spokesperson for Shared Services, confirmed that talks are underway with CSC to transfer some IT responsibilities. To date, no department has yet taken advantage of the transfer option that was first offered last summer, he said.
"CSC and Shared Services Canada are working collaboratively to move the agency's applications to a modern enterprise data centre that will offer state of the art facilities with multiple redundancies to avoid the service issues encountered in legacy data centres," Anido said.
"There is currently no scheduled closure for this [CSC] data centre."
Other federal departments have been hit with outages linked to Shared Services, including Statistics Canada, Canada Border Services Agency and the RCMP.
Shared Services was also slammed by Canada's auditor general in 2016 for poor service and performance, and in 2017 by U.S.-based consultancy firm Gartner Inc. hired for $1.35 million by Ottawa.
The agency was created in 2011 to consolidate IT spread across 43 departments and agencies, including reducing some 500 data centres to seven and creating a single email system — projects still far from complete.
Follow @DeanBeeby on Twitter
ABOUT THE AUTHOR
