UPDATE – Johnson City officials have confirmed that an apparent ransomware attack impacted the city’s computers starting early Monday morning.
City IT Director Lisa Sagona said an employee in the Geographic Information Systems department showed her the message from hackers early this morning. She said about half of computers tested so far (the city has about 600) are affected and will need to be reimaged or replaced, but added that a $650,000 investment in a “hyperconverged” storage network earlier this year kept the problem from being much worse. That new system just became operational three weeks ago.
All city computer systems should be up by Tuesday, but affected computers won’t be available for use by staff and Sagona said the city will make do with sharing and bringing in extra computers in the interim.
None of the city’s financial or credit card information was compromised in the attack, Sagona said, and there’s no indication any personal information was accessed. IT staff are “methodically bringing operations back online to ensure there are no lingering effects from the threat,” a city release said.
Sagona, meanwhile, said such attacks “are a matter of when, not if,” and said the city investment has made a huge difference in the ability to deal with the situation.
Message on a screen
Sagona said the GIS employee showed Sagona a message on the worker’s screen at 7 a.m. Monday. That message asked for an email in return to unlock the affected files.
“It clearly stated not to even bother with backup because they had encrypted those as well,” Sagona said. “They had encrypted our files.”
Sagona said her staff and support personnel from Bailey Computing Technologies (BCTI) were “quickly able to see exactly what had happened.”
The employee followed protocol, and along with the recently purchased “hyper-converged Storage Area Network,” that prevented a worse outcome, Sagona said. Had the employee sent an email. “That could have led to more encryption or other things we don’t want to speculate about,” Sagona said.
Instead, the city faces an arduous task working through the damage that has been done, along with the expense of either “reimaging” or replacing the affected machines.
All employees were asked to turn off their computers on Monday morning. “We gathered our team of people and some extras and we are looking at every one of our computers … and labeling them either infected or safe. It looks like it’s about 50/50. It’s going to take us some time to assess every one of those.”
System logs show that the encryption began around 5 a.m. At 7 a.m., the city closed all of its systems off to the outside world in a sort of cyber quarantine.
“Had we not had that modern, reliable system we would have had to go to our offsite (hardware) backup,” Sagona said. That would have meant repairing the infected hardware and then restoring from the tape backup, a two or three-day process. The information also would have been a week old, not in real-time. “It saved a couple of days and a week’s worth of information. We lost nothing. We were very fortunate.”
Not if, when
A report from Emsisoft, an anti-malware company, covering the first three quarters of 2019 shows a continued increase in reported ransomware attacks. The first nine months of the year saw 68 reported incidents by state, county and municipal entities and 62 from schools, and 491 from healthcare providers.
Cost estimates for recovering from attacks are difficult to estimate, but Sagona said the city of Atlanta spent more than $8 million “and many months” recovering from a ransomware attack.
Michael Mingle, a senior systems engineer with BCTI, said in the company’s service area here in the Tri-Cities, one to two cyberattacks per month are being reported.
“I feel very fortunate that we were poised and ready to the best of our ability,” Sagona said. “It is not about if it happens, it is about when. It is about detection, not just prevention because you can never prevent every little thing from happening.”
Discovery and recovery
No one will be allowed back on infected computers, but laptops will be in place. User accounts aren’t infected. “We’ll have other means for people to get their work done,” Sagona said, including “round the clock” use of non-affected computers.
Use of training rooms, shared workspaces and flexible scheduling are all possible scenarios as the problem is worked through.
Sagona said the return to business as usual is hard to predict until the total number of affected computers is determined — a process that’s ongoing.
“We have to make a decision whether we’ll reimage those machines if we feel comfortable that the hardware is intact or if we feel like we need to roll out new computers.”
JOHNSON CITY, Tenn. (WJHL) — A “system issue” with the City of Johnson City’s computer system has kept employees from using their computers and email this morning.
Information Technology is working to identify the issue and rectify it, Keisha Shoun of Community Relations said in an email. Updates will be posted on social media.
“Our computers are not working,” city spokeswoman Ann Marie French told WJHL. “They asked everybody to shut down the computers until they can address whatever the issue was.”
French was working to confirm reports that the city’s email server had been shut down so IT could work on the issue, which would prevent employees from accessing email on any computer and thus working from home.
Shoun’s email said that online payments of bills, taxes and citations are functional. Customers who want to pay via phone must have account or citation numbers.
The email said “some electronic operations are being completed via paper” and customers are asked to “allow additional time for any business being conducted in person.”