| View previous topic :: View next topic |
| Author |
Message |
Hardware
Clan Ghost Bear
Star Colonel
Joined: 04-Feb-2002 00:00
Posts: 605
Location: United States
|
 Posted: 15-Mar-2004 13:58 Post subject: Where's Hardware? |
  |
|
As many of you have no doubt noticed I've not been around much of late. I've been putting in a lot of hours at work. For the last month we have been busily migrating data out of an old share on a VMS system scheduled to go away by next year. This particular share is accessible from a desktop PC on the Windows 2000 network.
It was supposed to be shut down, all the data migrated off of it a year ago. Of course once the data was off it was supposed to be locked out. Management never gave the okay to lock it out, so it stayed available. Of course users started using it again. Even worse some business units started using it for processes, since it is a free share with 0 security. Anybody can access any directory or file with full rights on this share.
Well, starting two weeks ago we have been getting hammered with a series of viruses. Bagle, Mydoom, Netsky, all of them have destructive payloads that randomly write copies of infected files to available shares. Including this share in VMS land, since 2000 desktops can access them. Even worse one of the varients will infect available MS office files on shares. This whole share is available and there are plenty of office files on it.
We're finally getting the upper hand in our battle with the viruses and getting all the users anti-virus updated. Infection seems to be slowing down judging by the volume of infected files found every day. Better still we are migrating data out and the business units that are using this share for critical proccesses promise to migrate those to secured shares.
Friday at 3pm we lose power from the local utility due to some tree limbs across the street falling on the power lines. The lights flicker and then go out in the computer center but the computers and monitors are still powered by the UPS. There are three of us in the Operations center and we are all holding our breath or telling the computers not to crash when everything goes down. A second later our diesel generator kicks in. Too late, the damage has been done.
We find out later that our UPS has three dead or nearly dead batteries. Even worse the maintenance company didn't notice this 2 months ago when they were there to service the batteries. Just to add salt to the wounds we can't switch back to street power once it comes back since the UPS is down it won't condition the power and the generator/street switchover is sure to cause a surge and spike that will scrag some more hardware. So we have a huge diesel generator roaring away in the parking lot.
Long story short. We spend 5 hours running around like madmen bringing up essential services just so the stores can process credit and we can monitor the WAN. The tree limbs ripped out one of our provider's circuits to the operations center but we have no way to know that since our monitoring servers are down and when they come back the geniuses on the networking team have defaulted the operations account as not having access. The entire networking team was in another state (okay, New Jersey, just a half hour away) working to bring up a new warehousing facility and they can't even begin to work on the problem until they get back. Oh, did I mention that their cell phones don't work in this part of New Jersey and the warehouse doesn't have phones yet? Better still, instead of 8 lines in the operations center we are down to 2 since our PBX went down with all the other systems. We were reduced to a cordless emergency phone and our operations wireless.
Our storage area network is toast. The NT team has just spent 24 hours rebuilding it. The RAID cluster the share I have been working on for a month resides on loses two drives essentially scragging it. We have to restore from backups. The full backups from the last two weeks are considered compromised due to viruses and cannot be used. The full backup from 3 weeks ago is incomplete due to an issue that has been outstanding for a year. Fortunately we have a full backup from 4 weeks ago. The day before we started our project.
If I had another job I would walk.
_________________
The more I get to know people the more I like my dog.
|
|
| Back to top |
 
|
Oafman
Draconis Combine
Tai-sho
Joined: 18-Nov-2003 00:00
Posts: 1657
Location: United States
|
| Posted: 15-Mar-2004 14:28 Post subject: RE: Where's Hardware? |
|
|
All I have to say is; "Bummer dude."
Anything else would just take up too much of your already precious time. That just bites.
_________________
Festina Lente!
|
|
| Back to top |
|
Seraph
Blighted Sun Battalion
2nd Company
"Seraph's Slaughter"
Major
Joined: 11-Mar-2004 00:00
Posts: 1744
|
| Posted: 15-Mar-2004 16:10 Post subject: RE: Where's Hardware? |
|
|
Glad I'm just a welder.
Guess that's what a field general feels like when this start going wrong and he loses his comm network.
_________________
If ignorance is bliss, then why are you so miserable?
|
|
| Back to top |
|
AWAD
Draconis Combine
Chu-sa
Joined: 06-Feb-2002 00:00
Posts: 766
|
| Posted: 15-Mar-2004 20:01 Post subject: RE: Where's Hardware? |
|
|
This is when you talk to your boss about a raise or you are walking. Yeah they will be pissed and try to axe you later, but roll in teh dough for awhile and start looking to jump ship.
Plus thumb your nose at them for being stupid. When you flip the switch, you do it all the way.
AWAD- Whose top of the line chemical and optics company is IT stone age and just damn lucky
|
|
| Back to top |
|
Nightmare
Lyran Alliance
Kommandant-General
Joined: 03-May-2002 00:00
Posts: 2214
|
| Posted: 15-Mar-2004 23:37 Post subject: RE: Where's Hardware? |
|
|
Surprisingly many companies ignore the IT side as long as it works. I see a lot of these things in my work as a guard, although I'm not required to know what to do about them. One company had the hardware service outsourced, to another firm. One night there was a power loss, their UPS failed and some of the main servers shut down, resulting in some interesting problems. Electrical doors didn't open, gates stayed shut and so on, since those machines handled the electronic door cards etc. I had to open the stuff manually in the morning so people could get to work. (most people only have key cards, I carry one of the very few real keys)
Later it turned out the service guys had noted the servers failed, but they didn't bother coming around to investigate it in the middle of the night! The money spent on that firm could as well have been spent on beer! 
_________________
A tree fall in the forest, and no one is around, and it hits a mime. Does anyone care?
|
|
| Back to top |
|
Sir Henry
Team Bansai
Senior Tech Specialist
Joined: 04-Feb-2002 00:00
Posts: 4899
Location: United States
|
| Posted: 16-Mar-2004 06:25 Post subject: RE: Where's Hardware? |
|
|
You have no Idea how right you are Nightmare. I'm in IT and we can't get a full time facility or a Full Time HR person in our building. There are over a hundred of us and we still can't get them....
Hardware, I share your pain. We're doing HIPAA conversions and our guys have been working for about a month, of seven days a week, 12 - 16 hour days. It sucks. We order new Servers and Desktops to handle the new system. We got the New servers and old Desktops. With little memory and small harddrives, exactly opposite of the ones we paid for....
The April 1 deadline looms...... 
_________________
Sir Henry
A Dragon in the disguise of a bunny, is still a Dragon.
|
|
| Back to top |
|
-Mud
ex-Jade Falcon
Bounty Hunter
Joined: 04-Nov-2003 00:00
Posts: 1082
|
| Posted: 16-Mar-2004 06:44 Post subject: RE: Where's Hardware? |
|
|
The money spent on that firm WAS probably spent on beer...ultimately
|
|
| Back to top |
|
Horhiro
Draconis Combine
Samurai
Joined: 04-Feb-2002 00:00
Posts: 1625
Location: United States
|
| Posted: 16-Mar-2004 19:30 Post subject: RE: Where's Hardware? |
|
|
Well Hardware, it seems it is finally time to pack your bags and move your arse back up here to Massachusetts where you belong!
I'm sure you could find some work in Worcester or Hampshire County and you would be close enough to Mordel, Chihawk and myself to get some serious BTech games rolling. (not to mention routinely partaking in some homemade Pierogi and lamb shish-kebabs!) 
Just convince K that the winters up here aren't really that bad....although it is snowing right now and we're expecting about 8 inches...aaauuuugghhhhh! You'll just have to use those embellishing skills you've learned all these years rpging.
GM: "okay hardware, to convince her to move north you'll need to roll against your coersion/intimidation skill.....let's see....with all the modifiers......ummmm......yes, you'll still have to roll a natural 20!" 
_________________
"I have lived my life trying to be a virtuous man. The Dragon admires tenacity, and the code of the samurai upholds it as well." -Minobu Tetsuhara
|
|
| Back to top |
|
Sir Henry
Team Bansai
Senior Tech Specialist
Joined: 04-Feb-2002 00:00
Posts: 4899
Location: United States
|
| Posted: 17-Mar-2004 06:38 Post subject: RE: Where's Hardware? |
|
|
It's more like a 23.... 
_________________
Sir Henry
A Dragon in the disguise of a bunny, is still a Dragon.
|
|
| Back to top |
|
Havoc~Ronin
Federated Suns
Major
Joined: 13-Nov-2003 00:00
Posts: 427
|
| Posted: 17-Mar-2004 10:49 Post subject: RE: Where's Hardware? |
|
|
I read your story and wept. I used to be Special Projects manager for on of the leading natural gas and energy production corporation where the Ivory Tower decided that IT ad too many resources. So, they saddled me with the task of letting people go ( a task I fought against every single management meeting trying to make understand their short sigthed stupidity). After cutting the required amount of jobs possible, they gave me the golden handshake and said ciao.
The next week I found out that they were beginning thier data conversions! Needless to see without a Project Manager or the staff they just let go the conversion failed and sytems crashed. Put it this way, they put a technical manager who used to screw up his Outlook in charge of IT projects.
I partied that night. 
So, I feel your pain and fully uinderstand the lack of intelligence at a higher level. Now I work for the "MAN" and they are always looking for people...... Remember, you have transportable skills.
_________________
"Ours is not to question why, ours is just to do...."
|
|
| Back to top |
|
|
|
