Coffeehouse Thread

22 posts

A bug that just might kill you

Back to Forum: Coffeehouse
  • User profile image
    lars

    Independent published an article today about how the Sasser Worm crashed UK Coastguard computers.

    It makes one think. It's easy to slip up and leave something vulnerable. But in todays world it may actually end up killing someone.

    /Lars.


  • User profile image
    Knute

    This is one good reason that sysadmins need to make sure that there computers have the latests patches, especially ones that are used in mission critical applications like the ones you state. If
     
    this had been done, none of these computers would be affected as this was patched a month ago.

    ~ Knute

  • User profile image
    Manip

    Sigh.. I live about 25min's away from that coastguard station, and on the local news it was reported more accurately. In reality nobodies life was endangered by this worm, as soon as the computer system died they switched to the traditional (last 100years) system in which all employees are trained and nobody was hurt or rescue hampered.

    From the way it was reported locally it was more of an announce than a danger.

    I think these reports, directed at Microsoft should actually be redirected towards the moron running the networks at these places. With most even basic networks being behind a NAT I don't understand how the network could have gotten into the network and even less so why aren't the admin running the Microsoft Update Services with automatic updating enabled so the machines are updated when the patch becomes available. You don't need to be an MSCE to get all this, it is pretty common sense!

  • User profile image
    curwiler

    Manip,

    I totally agree with you. I'm somewhat protected since I'm sitting behind NAT. I mean, a hospital network that is open to the internet? Sounds like a network admin needs a little security training... there's no reason to have the port(s) open that these viruses (virii?) are using.

    As far as applying patches, I can see the reasoning why this may not be set up to be automatic, as that's a huge drain on resources and such - you want it to be downloaded to a central local server and have it distributed that way.

    Regardless, the virus needs an open port that quite honestly shouldn't be open to the internet.

    -Chuck

  • User profile image
    Manip

    curwiler wrote:

    As far as applying patches, I can see the reasoning why this may not be set up to be automatic, as that's a huge drain on resources and such - you want it to be downloaded to a central local server and have it distributed that way.


    That is what I was saying.. anyone with half a brain cell would install the Microsoft update server + auto update.. link:
    http://www.microsoft.com/windowsserversystem/sus/default.mspx

  • User profile image
    lars

    I'm not that interested in pointing fingers at who is to blame. But on the top of my list is the * that wrote the worm in the first place.

    "I mean, a hospital network that is open to the internet?"

    Come on. My guess is that someone working at home came to work with a laptop and plugged it into the intranet. Stop assuming that every network is a carbon copy of your office network and then go on to jump to conclusions based on that.

    Chances are that the x-rays have an embedded version of the OS. It could even be that noone besides the manufacturer even knows it's running windows. Some patches have sideeffects - do you really want them to be installed silently behind your back on an x-ray machine? Chances are that they may not even touch the software in those machines, since the manufacturer can be held resposible for any malfuctions that lead to patient harm. There could be a zillion more circumstances that we do not know about. So just give the obvious "the admin should have" mantra a rest.

    Can noone se the bigger picture? Everyone slips up sooner or later. Humans are by nature faulty. What has changed is that instead of the slipup being an irritating glitch on your screen, or a trip to the backup tapes - it might actually harm someone in the real world. Abit of humbleness about this wouldn't hurt.

    /Lars.

     


  • User profile image
    Manip

    No.. I think the admin team are personally responsible.

    If we all thought the way you seem to, electricians wouldn't be held responsible for burning down your house when a power surge comes down the line and they haven't wired it up correctly. That is exactly why network admin's are hired and if they don't do that then they need to take responsibility for it.

    Just because companies can hire anyone with a CV and copy of the Windows 2k Resource kit is not my problem, more fool them. Admin's are network professionals and therefore are treated as such.

    PS. If a laptop can be plugged into the network then there is something wrong there already.

  • User profile image
    Knute

    Yes an unpatched laptop would never be authorized to be plugged into our network. They must meet certain standards, have all the current patches and the latest virus updates.

    I couldn't agree more that there is someone responsible for the network. I get really tired of hearing excuses when a network is brought down, when if the network admins had been on top of things it would have never come down.

    ~ Knute

  • User profile image
    GooberDLX

    I hear ya lars!

    Jake

  • User profile image
    lars

    Manip wrote:

    No.. I think the admin team are personally responsible.


    Pointing the finger and assigning blame. Problem solved. Exactly what my post was not about.

    *sigh*

    The sysadmin is a moron with half a brain cell. He should be hung from an oak at high noon. Everyone happy.

    /Lars.


     

  • User profile image
    GooberDLX

    You should search for the book..

    "The Case of the Killer Robot".. might put you into a different perspective..

    Jake

  • User profile image
    sbc

    I think there really should be a job just dedicated to installing, testing and distributing patches (i.e. System Patcher) - the time spent patching could be done doing more productive things like system development and maintentance as well as the usual mundane tasks that still have to be done (fixing printers, installing software).

    Of course you could always use SUS - but many PC's are still running 98, or the bandwidth is not available, or more often than not, the technicians don't know how to use it (the excuse is they don't have time - but you have to spend time to save time in the long run).

  • User profile image
    Jeremy W

    As someone who works in a very large hospital, maybe I can shed some light here.

    First, hospital networks are in some of the oldest buildings you'll find. Many aren't even running on CAT5, nevermind any kind of standard. Most hospital admins don't have large amounts of money to spend for decent firewalls, nevermind locking down ports from new machines (mac address assignments).

    Next, because of the way medical apps are, there are an incredible number of open ports, and most hospital DMZ's are a complete mess, so it's not really beyond the realm of comprehension that someone might make a blind connection through the firewalls and DMZ's.

    Between old networking, lack of money for decent tools and lack of true best practices it's easy to see how something could get through. So, don't blame the sysadmins in this case.

    Also, there are some issues with putting a SUS server in, such as how in the world you set thousands of desktops to point to it. Many hospital admins aren't aware of advanced login script stuff or KiXtart, so basically resign themselves to not updating desktops, nevermind servers.

    SBC covered this well. The time, bandwidth and training required to run SUS in a large organization is significant. It's worth it if you can spare it, but if you can't you've got a big issue.

    Again, most hospitals aren't even running 100Mb Ethernet. Try deploying SUS in that kind of environment, especially if desktops have never been updated?

    I shudder, mainly because we are only doing our SUS rollout in the next few weeks and I know we're going to have this issue, so I'm staggering the KiXtart script so that our entire network doesn't crumble.

    So many of the things inherent to proper protection are so new that hospitals haven't been able to react. Patch Management, for instance, is so new that only 2 of the 13 hospitals in our city have any kind of patch management solution. Nevermind firewalls, properly configured NAT, desktop and asset management...

    Give the sysadmins a break. If you really want to blame someone, blame your provincial / state / federal government for not placing a priority on IT or on healthcare.

    The fact of the matter is that governments don't want to see healthcare dollars going to ancillary or support services. They want to see it going to primary patient care services. The fact that we've been able to get the tools and software we have is a testament to our management and to our region's CIO. Most hospitals don't have these benefits.

  • User profile image
    Manip

    IBM and other big boy would have us think protecting a network costs money.. but I could build a pretty mean firewall with a P500, copy of Linux and three NIC cards for a very low amount of money. You are there complaining about money but if you where hit by this worm somehow you must have found the cash for XP/2K workstations. If the network is a problem then first you need to report the problems up the chain, and next you need to start coming in on weekends and encourage your team to do the same, even pro bono. Work out a plan of action and carry it out.

    If you feel that things are dangerous (being in a hospital) and that lives are in danger, cut the internet line... it is the only moral thing to do.. Make sure you have the network problems documented and the managements unwillingness to do anything all documented. I wouldn't do anything so drastic in any normal company but in a hospital I just might.

  • User profile image
    Jeremy W

    This'll sound much worse than I mean it to: You don't know what you're talking about.

    Some quick points:

    - no amount of pro bono work can change a 10Mb switch to a 100Mb switch
    - the amount of time required to upgrade 530 switches is incredible, especially given that we're running copper in some places
    - it isn't IS/IT that spends the money for new desktops, it's departments. The fact that those departments can spend 1500$ on a desktop doesn't help us when we need to spend more than 1M$ to upgrade the network
    - cutting off the Internet connection isn't an option BECAUSE of patient care. Many hospital apps are internet-based these days, as is all our virus, spam, desktop updates, etc. Cutting off the Internett connection merely "pauses" a situation. When you have 8000 desktops you can't just go around one by one
    - dropping in a new firewall doesn't solve anything when systems and vendors need the same access, you need to rearchitect your entire DMZ and forward-facing security. We have 4 40K$ firewalls. It's not for lack of hardware.

    As I said earlier, hospitals aren't something most people understand. When changing an IP address on a switch can cause the cardiac reporting system to go offline, turn off the control access ports for one of the datacenters and cause printing to go off-line in 5 buildings... Things aren't simple.

    The point of my whole post was that hospital networks aren't simple. The politics aren't simple. Simple solutions like "if I was you I'd just do [x]" tend to make me close my ears, because the individual obviously doesn't understand the situation.

    10,000 employees. 8,000 desktops, 300 servers on 5 OS's, 8 mainframes, a phone system more complex than most towns of less than 50,000 people, covering an area 30 square blocks large.... This isn't a simple environment. Heck, we even have our own power plant.

  • User profile image
    Jeremy W

    Also, just to be clear, we weren't hit by this worm. But, I can easily understand how other hospital sysadmins could have been. Having no money, no staff, being unable to hire new staff or go on training... Your options really are few.

    Thankfully we have fantastic management here, and our biggest issue is prioritizing what we do. So, this month we're:

    - setting up SUS services
    - consolidating a dozen servers
    - redoing our DMZ
    - redoing VPN
    - launching CITRIX services
    - putting in new proxy services (likely appliances)
    - rolling out ZENWorks (like SMS) to another 1000 desktops
    - implementing official patch management for servers (including a new test lab)

    It's not like we're sitting idle, but even these things won't solve a lot, especially when we have more than 1000 Win: 3.11, 95, 98 and ME boxes.

  • User profile image
    Manip

    Your right when you say I don't know what I'm talking about as fare as not knowing the network in question. But even after you have given a little better explanation I still don't see any excuses for an insecure network. The problem is your building a building on a shaky foundation, I mean you can run ANY size network on 10 Mb switches, it is just about placing your switches and high speed links in strategic locations on your network. $40 firewall appliance? Could you possibly be throwing money down the drain faster?! That is just an insane waste of money..I think your entire department has serious problems with planning and resource management.

    Do you have a network diagram handy?

  • User profile image
    Jeremy W

    "Strategic locations"... You do know that this is 30 city blocks, right? The switches are in each and every building because they have to be. Walking to 200+ closets and upgrading the gear without any downtime, in an organization that has to be 24/7 isn't a simple task.

    No, 40K for firewalls. We're getting proxy appliances (8K each, plus another 5K for load balancing equipment).

    I'm not sure how this is a waste of money. You find me a P3500 that'll handle millions upon millions of requests per minute and you can keep the change. Provided, of course, that it integrates into AD, eDirectory, has full reporting, remote management (including able to wake on LAN, even remotely), etc.

    I am quite impressed with how this department does planning and resource management considering the scarcity of resources.

    My feeling is that you've never worked in an enterprise environment, so anything over 2500$ for hardware probably sounds like a lot to you. A single server is 8-12K here, so buying a pair of appliances for 8K is actually cheaper (nevermind that the software is better and we get fantastic manufacturer support, something that's very important in a large company with not enough resources).

    Again, I'm really not trying to knock you. If this wasn't the company it was I'd be nodding along right with you. But it is. It's healthcare, it's a whole other breed. This is one of the reasons that Microsoft is actually hiring a not insignificant amount of healthcare It people, because it's something that's very hard to understand. It's kind of like telecom in that way.

    As far as a network diagram, no. I could get you the ideal for our network diagram which will be accurate once the network upgrade project finishes in 2 years (going all 100Mb, replacing all the closets and all the switches without any downtime) in terms of very high level architecture. But even then it's only one part, that wouldn't include things like the backup network, SAN fibre-sharing, mainframe networks or large internal divisions like labs and our university.

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.