Apple and enterpise deployments Pt 1

Hello everyone.  This will mark part 1 of my 3 part blog series about being an Apple System Administrator in a large enterprise type environment.  I will first talk aobut my transition from Novell/eDirectory and Windows 2003 server/Active Directory over to OS X and Open Directory.  The pros, the cons, what I like and what I have experienced.  The second part will be about working in a large laptop deployment, like a 1:1.  I work for a school system which gives every student a laptop, so we are a complete 1:1.  The second part will also be about working in education and how it directly affects everyone involved and why you would consider doing a 1:1.  The third and last installment in my 3 part blog will be about using third party enterprise solutions with OS X and Apple products.  This will mainly consist of my 1 year working with JAMF and the Casper Suite, but it will contain some more insight into a few other products. Ok, now where to start.  I suppose I should give a brief background so people who read this (if anyone does read this) know and understand where I am coming from.  Jump back nearly a decade ago to 1999.  I was 18 years old and worked my first IT job straight out of highschool while going to college.  I worked at a retail store that did warranty service, repairs, custom builds and so forth for both business and consumer clients.  I worked back in the tech shop and basically did just about anything you can think of hardware wise, as well as helped maintained the companies internal equipment.  I started off a PC fanboy no doubt, and was a gamer.  In fact gaming is what made me want to learn networking to begin with.  When I was 15 I was playing DOOM II over modem play with friends and taught myself how to use a modem from the command line in Windows 95 solely for the purpose of playing co-op video games with my neighbor down the street.  So, at my first IT job we also had Macs, supported and repaired Macs, and of course sold them.  I didn't know much about them really and didn't care much either.  I hated OS 8 which was the current Mac OS at the time. After working there about maybe three months or so our Mac guy quit.  My boss tried to hire a new Mac guy but back then they were even more rare to find.  So, my boss got fed up and had a meeting and told all of us we are all now both Mac and PC techs.  I was pretty new and had just started working there, so I think I got picked on a bit more when this first happened.  I walk back to my work bench and lone and behold there is a tangerine CRT iMac, that had been dropped and the CRT had imploded.  So I took it a part and became a Mac tech because my boss made me.  He told me that was my very first task at becoming a Mac tech; was to open up that Tangerine iMac, order the right parts for it, replace them and make sure it works.  Before then I had only touched a hand full of Macs and they were all beige case Power Macs and a few Performas.  I didn't really know much about them at all. Fast forward a year or two and OS X 10.0 was released and I had already been somewhat of a Linux user in the past, so it did appeal to me a lot more than the classic OS.  I still dislike the classic OS.  I started getting more into it, had some Apple certs at this time and by default pretty much became the default Mac guy at my job from here on at this point.  It was because I mainly didn't care if I had to work on them.  My co-workers often complained and bitched if they had to work on a Mac, and others would refuse to learn it because it was a Mac.  That I never understood.  The more you know is vital to your job skills I think.  I saw it as an opportunity to just broaden my IT and tech skills onto a new platform.  I wasn't going to give up Windows or PCs at any point in time.  I was just also going to use Macs is all.   I started learning OS X and started tinkering with it more and more every day at work and started liking it the more and more I used it. Now we get to the enterprise stuff.  After I put in about 6 years of entry level work doing all sorts of things I landed a Network Tech job at a school district.  This is actually a previous school district to the one I work for now.  We had about a 300 Mac deployment out of 10,000 PCs.  So roughly, 3% give or take.  There I helped manage the Macs, do all hardware level repair on all the Macs, as well as PCs and our servers which were almost all HP Proliants.  We ran a mixed envrionment of eDirectory, Netware, SuSe, Windows 2003 Server, OS X server, Windows XP Pro and OS X.  We did have some Linux boxes here and there but those were almost always someone who worked in the software development department and they wrote code for a living.  They maintained their own systems.  During this time period I had the joy of using Netrestore (now retired) from Mike Bombich.  This is where I first started Netbooting Macs to image them and automating the process with shell scripting.  I also networked the Macs with the PCs and tried to integrate them into eDirectory.  Which me and the other guy that worked on Macs had working in our sandbox before I left working there.  I have no idea if they expanded that into deployment or if they even still have all those Macs at my last job.  It wasn't an easy task either.  However, we did get OS X server to bind to a Netware server and we did get it to authenticate while managing the machine via group policy on OS X server.  This would have been about 3 or more years ago perhaps.  There was not a lot of documenation out there and our Apple SE was putting us in touch with other gurus around the nation that had done similar things. During that time period I also was a sub contractor for a Company called Pro-Onsite which no longer exists.  They got bought out by a company called Conexio.  They were testing out the Mac market and had some how heard of who I was around town and offered me a side gig.  I did all their on site work for consumers and small businesses.  Overall, I didn't like being a sub contractor and they kind of jerked me around a lot.  It was a second job on the side.  I ended up keeping a few clients on side and I still maintain their Macs, but other than that I no longer sub contract. We managed to get the iPrint client to work with the Macs as well.  Which brings me up to my next topic, which is desktop management.  Since, we didn't really have any OS X production servers in our network we had to manage everything with local policy.  The one thing I love about Unix is that with standards like POSIX it allows me to use the shell to execute commands to change ownership and permissions of folders and sub folders fairly easily.  Also, it allows me to allow or disallow access to items I want managed on my client machines.  I didn't want anyone to have access to apple script, because there has always been some security issues related to apple script and I didn't want users Google searching how to exploit Macs by writing Apple Scripts.  So, I was able to use Unix at the command line in OS X to forbid non admin users access to folders and items by simply setting the ownership and permissions.  We did this on all three hundred Macs.  Which means we had to do it in our images.  We had about 5 or so images with everything in them, the largest of them being over 50 gigabytes in size.  That was our Final Cut Pro image for the labs that used that software.  This is where Apple on the enterprise got kind of ridiculous.  I was managing 300 Macs with 5 or 6 different images and imaging them over target disk mode since I didn't have netboot servers at every building they were in, and I was never going to try to image a Mac over our WAN, not that the Cisco guy would ever even let me configure any of the routers to allow that, but it would be ungodly slow.  AFP performance for data throughput on imaging was also flaky and a bit of a problem child.  When I talk more about JAMF and The Casper Suite later on I will touch on imaging and data throughput with AFP.  I kept graphs this summer when we reimaged 6,000 Macbooks in 5 weeks of AFP data throughput.  I think our record was just under 500 Macbooks in one 12 hour day that we imaged.  I actually have some data to work with on those thoughts. So, one mistake in the image and we had to ARD admin out all the fixes via Unix based commands.  The guy who also helped me with the Macs did not know Unix at all, but was a Mac guy.  Actually, he technically was above me and manged the Macs and I was technically his side kick at this job.  This was annoying because already our imaging time took for ever, we had little to no time to test every aspect of it, and if we made one mistake we then had to fix it remotely with ARD Admin.  Trust me, high school kids get creative on how to exploit a machine just so they can play video games and no matter how smart you think you are, there is always an exception that gets past you.  Plus in public educatoin you must manage the machines more for many reasons.  One being the schools don't want the be blamed by the parents for anything.  Plus with things like CIPA there are more regulatons and guide lines we must follow, that is just one thing we are regulated by and there are many others.  So, we already need to lock a machine down more so our work is generally harder and we didn't have a robust group policy or Active Directory like environment for the Macs. So we have to deal in a highly managed environment, with users constantly trying to exploit machines to play video games and hop on things like myspace.com, and we have to also make sure thousands and thousands of machines are running at any given time with a skeleton crew.  My friend who works for a very large bank works in IT.  He tells me that they have maybe at all their district branches about 400 to 500 client machines.  They have 100 people on their IT staff.  Granted, they are dealing with financial data which is worth a ton of money, but we have 6,542 Mac clients at my current job and we have maybe 7 people that deal with Macs.  The rest of my department is PC admins and techs, a Cisco guy and some software developers.  All of them are busy too, but we can occassionally pull them off their tasks to get them to help with big projects.  So, in my enterprise environment I am way under staffed. Stepping back to my Olathe days before I get ahead of myself, we did use some technologies to deploy machines in lab environments.  Right after the Intel Macs and BootCamp came out, we of course, had a user request that her lab be a dual boot imaged lab running both Windows XP and OS X.  She had enough of a reason for us to do this so we used Bombich's Netrestore (now retired) and used it with some shell scripts to resize the drive's partition and slap down a Windows XP image on about 50 iMacs for a lab.  I was interviewed by CIO magazine at the time after the author had read my posts on the Internet trying to get this deployment to work.  We got it to work and it was managed in the environment but it was still a lot of leg work, OS X and OS X Server were still missing some key components to let us manage them easier.  The article itself is old and out dated and of course I was only quoted for like one tiny tid bit of our conversation which wasn't really relevant to what I was actually doing.  That I guess is what happens when people are writers first and tech people second.  I don't fault them for that though. Now on to my current deployment.  Last year at the end of October 2007 I was hired on at my current job to help with the school system's 1:1 student to Macbook deployment.  We deployed about a week and a half later.  So I had to hit the ground running and have pretty much been running ever since.  We have 6,000 Macbooks and every highschool student and teacher and administrator have one.  I have about 30 Xserves all running OS X Server 10.5.5 in a pure Open Directory environment.  I use OS X Server, The Casper Suite, and Server Tools to mainly do my job at work.  I use some third party apps along wtih ARD Admin, but for the most part I use those three every day.  While Apple tries to transcend the easy to use GUI into their enterprise products they do leave some things out that I think would be wanted by almost any sys admin.  I willl start off by saying I think that Mac makes an excellent end user product.  The productivity a user can perform is great.  I am easily more productive on my Mac than I am on my PCs.  I get more work done quicker it seems.  I also work in a Mac environment so I kind of have to work on a Mac all day every day while at work.  However, I used to work on a PC all day every day for years before helping run our 6,000 Macbook deployment.  I feel now I am more productive.  Perhaps that I am now more experienced plays into it, I am not sure, but I do feel more productive on my Mac with OS X.   So, now on to the good stuff.  How does it work and more importantly, how well does it work?  Well, my environment is unique in that it is pretty large, and almost all of my users are laptops out in user space all over our wireless VLANs.  So, my experiences may differ from those who have different type environments.  I will just start off down an unsorted list for most parts and explain what each things is that Apple or third party offer that I use and how and why I use it.  Then how effective it really is in a live environment. Apple Open Directory and LDAP This is pretty straight forward.  Apple follows most of the open source LDAP standards and I have never had a problem importing or exporting LDAPs from non Apple to Apple and vice versa.  Work Group Manager is the native tool you will use to import/export users, create and enforce group policy, and many other things you can do to manage OS X clients.  I do use a third party application called Passenger, and it really is a godsend.  I highly recommend it for any OS X System Administrator.  You can create batch import users from spread sheets and plain text, batch edit permissions and ownserhip and save out what you do in the GUI to shell scripts for cron jobs later on.  It also does plenty of other things.  If you are serious about having OS X Server and managed Mac clients in your environment you should seriously consider getting Passenger.  There is a file, often referred to as MCX, which lives for each user and it gets updated at log in/out.  This file is udpated by the policies and settings that you create for Work Group manager.  The file is located on the client machine, and gets updated every time a user logs in/out or reboots.  Managed Client Policy will over write any local policy you may have.  So, even if logged in as a local admin if your group policy does not allow you access to /Applications/Utilities, then even as local admin you can't run those applications from that folder.  However, with local admin rights you can easily of course change it things around since you do have admin rights, and super user rights as well.  The point of me bringing that up is, how it works is you basically have three levels of policy in MCX (managed client OS X).  This is how I have it set up at least.  I have one nested group of all sub groups that I want to have the same settings.  Nested group policies get applied first.  Then you have individual group policies which are applied next and will over write any nested group policy.  So, if you have one particular group of users you want to add policy to, you can keep them in the nested group, but the actual group policy will take over if you want that particular group extra access or what not.  The last way I apply policy is by the individual user, which trumps all group policy.  Any policy given to an individual user in Work Group Manager is the final say.  It will over write all group policies.  Most Directory Services are like this, and Apple did good in that manner.  Using Work Group Manager as an actual tool can be frusturating though.  I find it unresponsive at times.  It locks and crashes, and has undoubtly caused corruption in my LDAP and BSD databases.  10.5.5 Server Tools did fix it, and it does really fix a lot of bugs I encountered, however they are not all fixed.  I did submit an enterprise ticket with Apple recently (as in the last month) about a bug where WGM generates negative UIDs (user IDs) to freshly created user accounts.  This causes a whole sortment of problems where the account can't log in, it can't synchronize, it won't authenticate to AFP shares, or it won't even show up in the directory at all.   In the last case you should use inspector in WGM, which is an advanced feature that will allow you to edit the raw data of a user account.  If you come across any negative UIDs, manually change it over to a working UID and the account most of the time then works.  All in all the intuitiveness of Apple is here in WGM, but the features and stability are lacking in comparison to it's competitors.  Active Directory and eDirectory are their components are a better product when it comes to things like this.  Apple is not horrid, but there is plenty of room for improvement.  Later on I think I am going to compile a wish list of features I wish to see in updates and in new versions of OS X.  Anyone interested in running seriuos Apple Enterprise products on their own, be forewarned that you will have to learn the Apple way (which isn't hard) and be paitent.  There is no easy button to hit for these kinds of things that everyone thinks Apple can do.  This isn't the end user OS.   I really hope Apple fixes the frality of WGM and makes it more stable.  Just the other day I had to create a few new user accounts and got negative UIDs, along with my co-workers in their respective buildings as well.  I know the fix and am the most experienced so I get to manually do all the user account creating.  Manual data entry is a pain, and no one in IT wants to do it.  They want to do imports and exports.  WGM is the weakest link of Open Directory and LDAP when it comes to OS X on the enterprise.   Managing OS X Servers OS X Server gives you a set of native tools called Server Tools.  I know, clever name huh?  You'd figure it would be called like iServer or something right?  These tools include the following:  Work Group Manager, Server Admin, Server Assistant, Server Preferences (only works for simple server setups), XGrid, Server Monitor, and System Imaging Utility.  OS X Server does have some of it's items and services streamlined for easy use and easy function.  I will go right out and say that I think that Server Admin is probably the best tool.  It never crashes, runs like a champ, and allows me to manage 30 servers remotely from my desktop in my office.  Applications like that are a must for enterprise deployments, and Apple did a pretty good job with Server Admin.  Server Admin also allows you to control and configure share points, network mounts, services, ACLs, so on and so forth.  System Imaging utility is not that robust.  While I haven't really used it all that much I have no reason to.  Since I am using Casper, Composer, and InstaDMG.  Apple has always been lacking on image Deployment and with 10.5 Server we are seeing their first attempt of System Imaging Utility.  While, for a while not too many companies had an out of the box Netboot and Imaging solution, they are all catching up.  Windows now has WDS which I saw a live demo of at the Vista and Server 2008 road show a few years ago.  It looked decent enough at the road show, and allowed system administrators to drag and drop images on to machines in the directory.  Apple's imaging utility can't hold a stick to that.  I also worked with Zen Works at my old job from Novell, and while I didn't find it the most intuitive set up, it did work.  I wrote a few menu based shell scripts to automate the imaging process when I worked at the Olathe School District and we were able to image 10,000 computers over one summer.  Which is pretty impressive.  So out of the box, Apple doesn't offer anything super special for imaging.  I recommend you look into third party like Casper, or DeployStudio which is FOSS.  I haven't exensively looked at DeployStudio but I would highly recommend it just by reading up on it.  For one, it supports PXE booting, so it could feasibly image your Windows and Linux boxes as well.  I mean if you really look at the history of OS X Server, it was Bombich's Netrestore which really allowed System Admins to deploy software and images over the Network.  Server Monitor is OK, I don't use it because the port it uses is blocked at my work.  I don't really care to bug our Cisco guy to open up that port just for one application that may not even be that useful.  I have run it on a few servers locally and well to be honest, while it does offer some nice features, it doesn't really offer anything else that Server Admin, Activity Monitor, and ARD Admin reports can't do.  So I never bothered to get that port open.  We do have a network monitor service that spams my blackberry every time a server goes down, so that is really all I wanted to use Server Monitor for to begin with.  You may find more use for it than I did.  Server Setup allows you to create .plist files so you can drag and drop them onto multiple servers over your network for automated set up.  Me personally, it only takes an hour to load OS X Server if not less time, and my server set ups are pretty simple and straight forward.  I'd rather drop in a simple shell script or a few lines of code over ARD admin to configure my servers.  I like to keep it simple and keep only a few technologies and services running on each server and then try to load balance amongst them.  I have had some hiccups with LDAP and Portable Home Directories and OS X Server and they are finally starting to simmer down a bit with the release of 10.5.5, but I am still having issues.  If Apple wants to be serious I need my users to be able to log in and synchronize their mobile accounts with no problems.  Logging in is probably the most basic and essential function of a Directory Service network deployment.  If my users can't log in then they can't use their computers and it makes all of this wonderful technology all pointless.  I will touch on PHDs (portable home directories) in the next part.  They kind of deserve their own section. Portalbe Home Directories Portable Home Directories (PHDs) are something that is great if you have users that are mobile or if you are a huge laptop deployment.  They synchronize a mobile account to the machine locally for many many advantages.  For one, they can use that account off site.  It caches the account to the machine locally, and when the user is off off site or somewhere in user space that has no WiFi connection they can choose to synchronize their account when they get back on line.  This is not a back up, this is a sync, and they are two different and distinct things.  If you remove a file off the machine you are on locally, then sync your home account, it will see that the file on the local account is removed and remove it from the network folder.  Some times users do not quite understand the difference between back up and sync.  This is a concept that is very hard to get across to users as well.  I can't tell you how hard it is to fully explain to that users.  I would not rely on PHDs as a back up method, they are more of a mobile account tool than anything else.  Now, if you are worried about security there are some cool things you can do.  You can use file vault which encrypts the home directory, so if other users have access (that is also part of the nature of PHDs) they can't just exploit or root the machine and then have your way with other user's data.  It also uses by default, I believe AES 128bit encryption.  While, anyone who really gets into security knows that encryption doesn't mean it is impossible to get the data, but it would most likely stop me from getting it.  You can also choose to keep your PHD on an USB thumb drive and plug it in from machine to machine and log in that way.  This would be another great option for those users who travel amongst computers and need a data sync.  We had so many sync isses with Server 10.5.4 and it was such a pain.  Since then Apple has released two specific updates to fix this.  While, it is still not perfect, and I still have some accounts that fail for no reason, it has vastly improved from what it previously was.  10.5.6 is suppose to have even more fixes for this.  Out of all the things that can go wrong and you might let slide when it comes to problems on your production servers, being able to log in is not one of them.  I think that is completely unacceptable.  I will say this to every system admin that wants to use PHDs in a large deployment.  Have a back up plan.  I have a managed local account which is basic, which can access the machine.  That is my back up.  I have had accounts not work on Monday and then magically work on Wednesday.  For no reason.  I didn't change anything.  My guess is that in the LDAP replication of OS X Server's Open Directory, there are some times glitches, where after maybe a full new replcation it fixes the issue.  That is my only guess as to what it could possibly be.  While it is a head ache it is not quite a nightmare.  I would say 90% to 95% of my accounts work.  In some cases I have rebuilt a few replica servers and I have done a few imports of existing users to over write possible corrupted data.  I think that there is a bit of frality in their product and it just needs that final polishing to be solid. Part 1 is not finished but for now I am going to take a break and go over some more of my notes to finish up some thoughts of it.  I hope you enjoyed the read and if you have any questions or comments please contact me or comment here on the blog.  I will touch up on printing, ARD Admin, some more client management, and other native products that come with OS X Server.  I would like to bring up a few third party items as well.  I would like to do a whole section on Passenger because of how neat of a tool I really think it is.  So, stay tuned for continuation of part 1 in the next week or so.  This has been based on my 1 year experience of running a 6,000 Macbook deployment managing them in a pure Mac environment.