Dulling of the Edge

As is my nature, now that the migration has passed and settled, I was dreaming up new projects that the company could benefit from. Since boss is opposed to housing tapes offsite (post), I wondered if we could use our west coast office as a disaster recovery site and vice-versa. I could rebuild both the older servers, send one there, and both could serve that purpose. Using the old hardware, this could be accomplished at relatively low cost.

Then another thought hit me. It was unconscious at first. I just didn't want to approach boss with the idea. I didn't want to start any preparatory research to present the idea to him. I didn't want to bring it up at all. Time and again I have heard boss tell me I'm wasting my time. I did just that when I spoke with an offsite storage facility for the backup tapes. I also did that when I intended to view a 30-minute WebEx (site) demo from a Kintera competitor. Luckily he shot me an email when he found out, demanding I cancel the appointment. It was irrelevant why I wanted to view the demo. Knowing if there is a better product out there, or knowing if Kintera is comperatively worthwhile, really is a waste of time.

So I dropped the disaster recovery idea, despite thinking it is still a good one. I realized that in a few short months boss had worn me down. This is not who I am. I don't give up on an idea because of this but, as is obvious above, I've become bitter. I used to do the homework so I could provide any other decision-makers a full understanding of what, why, and how. Now I don't want to do it because it's too difficult to gauge whether this will be another "waste of time." While I still pride myself on my work, I'm starting to feel the dulling of my edge. I think he's won and I'm very disappointed.

I'm Exhausted

I'm exhausted.

I don't bitch about personal stuff much on this site but my life is clearly affecting my work situation as well as my upkeep of this blog. Between work, part-time college, a pregnant wife, and being VP of my coop board, I'm out of fuel.

It helps that I'm experiencing some post migration downtime, even though I'm dealing with a lot of trivial, tedious stuff. My mind is just trying to rest a little so my days are spent working on workstation rebuilds, documentation, and vague preparation for projects unconfirmed. We do have one fund raising event in the near future but most of that work is done and what remains is a few weeks away.

My initial budget for 2007 was due this week. Pretty much nothing special going on there. May not get the servers I wanted but looks like the software and workstation I requested will make it. Telephony, all of it, is being moved into my department, which means more responsibility that I don't want. It's something I've handled in the past but am not looking forward to it right now.

Ugh. Too tired to sound coherent. I hope I'm in better shape by the next blog. There's my complaint. Pass the vitamins.

Month Three Review

Taking ownership of this network through the migration (post) has helped, in my mind, solidify my position in the organization. The urge to leave has lessened and though relations with my boss are still rough, the friction ebbs. I've come further than I expected. The last three months have felt like an eternity and it surprises me that I'm only getting off the new employee probationary period now.

Much of this last month went into preparation: login scripts, folder topology, hardware and settings details, staff and vendor relations, and overall project management. A lot of time was spent communicating with staff (post) to make sure the transition was as easy as possible, as well as hashing out details so staff were minimally impacted. All that work fit nicely into a staff email (post) and a post-op summary (post).

Three days in the thick of migration activity led to an overall successful end result, even though I was bumming the whole way through the process (post). It is my way. In the end two new servers replaced two aging ones. File, print, Exchange, and anti-virus services were moved over. Backup services were consolidated on to one server instead of separate server installs on each server, and a new tape drive was connected. Some cabling was replaced and employees were provided new, uniform, drive mappings and logical printer names, all distributed through group policy and login script.

Debugging followed. One of the lingering issues was broken POP3 access for branch offices (post). Oddly, despite all that I had checked and rechecked, it turned out that the POP3 virtual server had at some point shutdown. Easy fix that took me forever to figure out, but it's all done now.

The legacy of my predecessor made a few days hell as manually set DNS settings (post) and a randomly assigned login script changing DNS settings to manual kept causing workstation connectivity to flake. Also an easy fix but time consuming.

XP/2003 home directory synchronization breaks from time to time and there is an odd message that appears on startup saying that network drives couldn't be connected, even though they are, but other than that there are far fewer gremlins.

While I do think a company grows as its systems stabilize, I wonder if this much change in three months will propel the company or maintain status quo as people get used to things. Certainly the company has been waiting for upgrades, so maybe they will start running out the gate now that there is so much more network stability. I like watching companies grow and hopefully I've facilitated that (post). Onward to Kintera (ugh) (post), fundraising events, and workstation rebuilds.

Purchases, Sales Reps, and Gloating About Great Deals

Remember my rant about not paying retail prices (post)? Of course not! ;)

Well, the point was to get a dedicated sales rep and a business account when possible. Not only can they offer better pricing but they also provide personalized customer service. I've had opportunity to purchase several items recently for the server migration and for general upgrades/needs. I wanted to show you why having a sales rep helps.

One of the Dell (site) servers we purchased spec'ed out like this: Single Xeon, RAID 5+1, 2 GB RAM, 146 GB SCSI HDs, rack chassis, redundant power supply, no OS, standard warranty. Dell website price: $3300 Sales rep price: $2700. It helped I was buying a machine that was being replaced by a newer model, the PowerEdge 2950.

Purchased several lengths of ethernet cables from PC Connection (site). One type was back ordered so I called my sales rep and requested a replacement that would ship sooner. By accident both shipped. PC Connection credited my company for the extra cables and let us keep them anyway. Sales rep said it was for the inconvenience; I think it was because it would cost them more to have them returned. Still, one rep is easier to deal with than getting a new person on the phone each time. Your sales rep has incentive to take care of problems so follow-up is relatively minimal.

CDW (site) sent me Backup Exec licenses for Linux instead of Windows. We had to repurchase the Windows ones but the CDW rep explained that whether Symantec refunds the cost of the erroneous licenses or not, CDW will definitely credit us. Like the Dell rep, I've been dealing with this CDW rep for years. He values my business as much as I value his. In the end I can get these things done with less hassle.

Back to Dell. I just order two workstations. This deal impresses me most. Optiplex GX620s, Pentium D, 1 GB RAM, 80 GB SATA HD, DVD+/-RW and a DVD-ROM, two (TWO!!) digital 17" flat panel monitors, two 1 GB USB pen drives, standard warranty, XP Pro, no office suite: $1080. On the site it would have cost me $1400. My officemate is tired of hearing me talk about.

(Why two monitors? Boss wanted to capitalize the workstations so they had to cost over a grand. My initial quote was a little under so I figured I would add a few necessary items. My sales rep is so good that no matter what I added, he just kept giving me the same price. Then I wrote him and said I have this much to spend, help me spend it. He squeezed in some really nice monitors that will help my company out. I wish I always had these problems!)

When the prices didn't work for me, I used NewEgg (site). I've found them dependable and cost effective. They often have rebates specific to them. There is very little customer support but I've seldom had problems. Limited staff, many no-name brands and OEM products, and a limited return policy help them keep their prices down. Returning some items does incur a restocking fee of 15% (oy!) and some items cannot be returned, only repaired through the manufacturer's warranty. I would advise when purchasing through NewEgg, read everything carefully. Customer reviews on products help a lot.

J&R (site) is also kind to non-profits. I've used them in the past but not yet with my current company. They aren't my top retailer but they are in my suite of sales outlets to keep handy.

Breakin'

Stuff keeps breakin'!

This week, when we shutdown the old domain controller, everything broke. Many staff couldn't do anything. I realized quickly that all of their computers had their DNS manually set. The old server no longer existed and nothing made sense to the workstations anymore. This easy fix wasn't so easy because for some PCs DNS settings would revert back to manual. Eventually I realized that the old IT Manager had implemented a login script after all, he had just done it randomly so when I had checked a few accounts I didn't see it. For no reason that I could discern certain staff had the script running including the ED, one of the receptionists, and the advocate. I've screamed my predecessor's name several times this week. His specter continues to haunt my network.

Gremlins

So a week after server migration success some problems are cropping up. Notably, when accessing webmail users cannot just login with their username and password. Trying that, the login screen comes back filled in with mail.domain.org/username in the username field. This has to be manually changed to domain/username to actually work. It's an easy work-around but very inconvenient and unnecessary. I am researching the matter, however, am unsure where to begin to find a fix.

Additionally I've had POP3 issues and VPN issues but do not know if this is the result of a power outage this weekend that knocked out my firewall. Perfect timing. My remote users are becoming cranky.

Deploying Network Printers in Windows 2003 R2

The consultant provided me with a link to a page from WindowsNetworking.com (site) to deploy network printers via Group Policy (link). It's a pretty straight forward setup. The only part that didn't work for me was adding the PushPrinterConnection.exe login script toward the end. I'm not sure if having a script in place already was confusing the matter. My solution was to copy the executable into sysvol/scripts and include a line in my script to execute the program:

\\domain.org\sysvol\scripts\pushprinterconnection.exe

This worked great for me and saved me a few minutes at each workstation. Topics of interest at the site:

Deploying Printers with Group Policy in Windows Server 2003 R2 (link)
Managing Printers with Windows Server 2003 R2 (link)
Managing Printers Using Group Policy (Part 1) (link)
Managing Printers Using Group Policy (Part 2) (link)

End of Project Summary

An end of project summary is a good thing to send to staff, especially if you've needed their assistance with any part of it. We all know that staff are primarily concerned with their desktop first, and rightly so. Therefore if the project doesn't involve speeding up their workstation, to them you've not made a huge difference. They'll always notice something but the enormity of your effort will be lost if you don't explain it to them.

Why is it important to let staff know? Because in the process you also wind up telling your bosses what you did. Additionally a favorable impression from staff has an impact on your supervisors' impression of you. Lastly, staff understand what you do, why you do it, and how much their cooperation is necessary in these projects. If they understand that their efforts weren't squandered, they'll be more receptive to your future requests. Below is what I wrote. After I sent it the Executive Director came to my office and thanked me for writing the email and said she felt it's good to keep staff informed about such matters.

Hello All,

I wanted to thank everyone for their assistance during this migration. Everyone getting rid of unnecessary files on the network helped tremendously. We probably dumped several gigabytes of data that we weren’t using or was redundant, which shaved a lot of time off the migration and allowed us to focus on other tasks. Overall the migration went smoothly. Usually there are glitches to a process as big as this and we’ve had a few, but overall minor and I believe most of us have not experienced much interruption to our work. The remaining issues should be resolved soon. If you’re interested, following is a summary of what was done and where our company network stands today.

We have two new servers: Alpha and Beta. Alpha keeps all the files and Beta keeps all the email. We also have a new backup system, so we will only use one tape a night instead of 6. Additionally we have upgraded our antivirus software to Symantec 10.1 that should also fight spyware. Everyone’s computer is now up-to-date with the latest security patches so our workstations are far more secure. The files on the network have been organized and we implemented uniform drive mappings, printers have been renamed logically, and security settings have been reset so certain folders are only accessible to certain people. You’ll also notice a new Webmail interface.

The ultimate goal was to replace some of our aging server hardware, stabilize email, and consolidate a fragmented and quirky file and security structure. I think we have met our goals. By migration standards ours went very well.

Again, thank you all for your assistance. If you have any questions or are having any problems, please contact me.

IT Manager

Turning Point

A handful of staff stopped me in the hall or came by my office to not only congratulate me on a completed job but also to express astonishment at how little impact this migration had on their workflow. I'll take that as high praise, thank you very much. Even my boss, in his way, complimented me, by forwarding on to me favorable remarks by other staff. I haven't even written my summary yet explaining what really went on and how much I appreciate staff cooperation.

They had a right to be stunned. It's amazing the scars bad IT Managers and Admins leave on staff. They were all preparing for a doomsday scenario with email out for days and files deleted. Apparently something of this nature happened last year but the details are sketchy.

I hope with more emphasis on security and stability the staff can perform their jobs a little easier so the company can expand what it does. There is room to do much more. If we can get Kintera working as it should and have it sync with Raiser's Edge I think a few paths will appear. Specifically the company should be able to market itself better by using statistical data gathered through Kintera. I would love to see the company triple it's online revenue in a year and I think it's possible. Currently the haphazard Kintera setup is generating decent revenue; a well managed Kintera could produce much more.

I see this migration as a transitional point. It was a very necessary step for this company. I hope the staff take the new setup and stability for granted because they should not have to worry about losing anything or expect network resources to be unavailable. That kind of computer infrastructure went out with the 80s...

...well, it should have anyway.

Migration Summary - Chillin

As much as I love projects, they do make me stress and get critical of myself. I received a couple of messages of support today such as one commenter writing "if the users are happy, servers are migrated, and you accomplished your goals, it sounds like success to me" (post). The chaos I expected today lasted only an hour or two in the morning. The consultant hadn't been able to fix the permissions issue remotely as he thought, so he came in early at my request and finally resolved the matter. Ah, Microsoft and its checkboxes. Half of being certified is knowing where to find the right checkboxes.

Once he found the issue, everything worked as it should. After which there were a couple of minor issues here and there but overall the day went well. The staff took to the new folder and printer schemes quite well. Email was not an issue and everyone was already upgraded to the new antivirus. The new installation of Backup Exec 10.1 and the LTO drive are working great. The login script works as planned, the rewiring looks good, and as a bonus, all the desktops are completely patched. Everyone was back in business today with minimal disruption. So I guess the migration was a pretty big success after all and I desperately need to learn to chill.

Thanks for the support, all!

Migration Day 3 - I'm Out

When I envisioned this migration being documented on this blog, I expect a well planned success story with me as the hero. Man, does reality suck. And embarrassingly so.

I spent all day fighting with...well, I don't know what I was fighting with but apparently it was trivial. The consultant remotely fixed the permission issue after I left work today. He explained what he thought was the problem, which I didn't quite understand, but hopefully Tuesday is only minor chaos.

My day involved a lot of back and forth. While I did deploy the network printers and drive mappings, setup every users' desktop profile, and rewired the server room, the permissions issue monopolized my time. At one point, after I stopped and reset all the shares, everyone suddenly had permissions to write to every folder. No user was responding to the explicit rights I had set. Until that moment I had assumed all staff had rights to modify every folder because every folder had been setup to allow the Everyone group to modify. I had not checked memberships because I felt any changes prior to migration would possibly break something. I found out today that all staff who started prior to my arrival had been given administrator privileges. Holy crap! What the hell was the previous IT Manager thinking?! And Holy Crap! Why hadn't I checked this before?!

If the consultant is right and he fixed the issue, I'm thinking Tuesday should be relatively smooth. The impact on staff has been light, though I do expect confusion because no one will know where to find their files, but hopefully everything works the way it should. Yet I'm having a hard time calling this a success. I feel like I made way too many mistakes and I'm certainly feeling the limitations of my knowledge. I know no migration goes off without a hitch and I did miss 3 days this week in training, yet I feel this could have/should have gone better. The rest of the week is going to be spent doing cleanup so we'll see what comes.

Migration Day 2 - Strike 2

This migration isn't going well. Stuff that should be easy isn't. All I'm working on now is the file server, copying the files from one server to the other, then setting new rights. All users have been give full access to their personal folders and yet none can write to those folders including my profile, which is a domain admin. What could I have possibly done wrong with the simplest piece of server creation?

I'm back in the office tomorrow trying to sort this out. I've created and shared folders since I first touched a computer. The problem seems so impossible that I don't even know where to begin. It's times like these I really start questioning my abilities. File and folder setup should not be an issue.

On the plus side I did manage to deploy network printers through the combination of login script and group policy. For some reasons group policies never work perfectly for me, even for some simple things. Yet other times they work exactly as expected. The printer deployment half worked through group policy but I needed the login script to save the day. I wonder if this exemplifies my ingenuity or my lack of ability. Today it feels like the latter.

Migration Day 1

That the mid-week training sapped preparation time from me is true but I really can't lay the entire blame on today's situation on that. I like having all the i's dotted but I missed a few too many things this time. While the consultant with me wasted no time, today was more work than it should have been. Considering all that thought on preparation, I should not have had to spend hours updating some computers today. I had requested from staff for a few pieces of information, namely whether they were running Outlook 2000. A few people responded and I took care of them by updating them to the latest Office service pack. I really had no excuse not going one-by-one but simply trusted that everyone else was on Outlook 2003. Today I discovered that some people didn't even have XP SP2 installed. I knew all these problems existed and yet I still let it slip. I get very upset at inefficiencies by my own hand. There's just no excuse when I've been thinking and writing about this stuff for so long.

The migration went fine regardless. We moved Exchange control to the new server and the other new server took over as domain controller. The Symantecs (Backup Exec and Antivirus) were setup and deployed. Tomorrow I brave the storm and head over to rewire, transfer files, implement rights, move staff into a new OU with group policy, and rework a login script. If I have the time or energy I would like to setup each user's desktop so on Tuesday there is very little hysteria.

Compared to the NT 4.0 to 2000 migration I experienced a few years ago, this was a breeze. That was a week-long nightmare, this will be a couple of days and so far nothing is broken. Should have been breezier though. But at least some of the problem desktop issues also got handled today. Guess I needed that day without staff and a little confrontation with my weaknesses.