Recently Merlin Mann, Mac productivity and time management expert extraordinaire of MacBreak Weekly, 43 Folders and 5ives, asked the question, "how do you manage the constant growth of files when your primary computer is a notebook like the MacBook Pro." After several discussions with Merlin, it became clear that TechDigs.net should cover the topic with a How-To. Techdigs.net readers tend to be substantial consumers of digital media files, use their network to store files and often are faced with storage and backup quandaries. 

Read more for details. 

Merlin's specific challenge was for the Mac, so we've focused this article on a Mac solution. However, Windows users can achieve the same results using SyncBackSE in place of ChronoSync. In future articles, we'll dedicate time to Windows-specific details. We also plan to review services currently available for Windows such as Mozy and Carbonite. There are dozens of approaches to solving this problem, but this is how I do it. While the same theories can be applied to Windows machines, this article gives the details for a Mac solution.

If you're like me, your home and entertainment storage needs are growing exponentially. Music libraries, podcasts, movies, photographs and more are taking over your hard drive(s). Depending on whether you're primarily a notebook user or a desktop user, the problems you confront will vary. To give you an idea of the problems I face... in my home we currently have about 85GB dedicated to music and podcasts, 110GB to photographs, and about 300GB to video. Obviously, with nearly a half terabyte of ever-increasing data, it’s not possible to carry all of my entertainment files around with me inside my MacBook Pro. The classic finger-in-the-dike approach is to buy the latest 120GB, 200GB, or 400GB drive from Maxtor, Seagate or LaCie and spread files over several drives. It’s like buying bread by the slice.

The Problem

I have both Mac and Windows computers on my home network that all want/need access to these shared files for different purposes. Most video files are watched in the theater where we have a Media Center PC connected to a DLP projector. The photo files are used by all of the machines on the network for both viewing and editing. At home, most music is listened to over a Sonos music system, and three iPods are synced on Mac and Windows computers. However, when I travel I still want to be able to download podcasts on my MacBook Pro and sync them onto my iPod. The problem became clear with the realization that my life now simply demanded managing hundreds of gigabytes of data. It was time to move to the next level.

The Solution

My solution is to manage all these music, photograph and video files by centrally storing them on a network attached storage (NAS) device.

What is a NAS?

A NAS is a dedicated data storage device connected directly to an Ethernet network that provides centralized data storage and access. Technically, most NAS devices are specialized computers with multiple redundant discs. My current favorite NAS device is the Infrant ReadyNAS NV. With sizes ranging from 250MB up to 3TB (yes, terabytes), the Infrant ReadyNAS NV gives you storage with peace of mind. The redundant disks work together so that if you lose one of the four drives, the other three take over until you can physically replace the bad one. The downside is that four drives provide only the storage capacity of three, since they're also providing redundancy. For me, that's a small price to pay for no worries. My recommendation: get a NAS with individual hard drives that are no smaller than 500GB. Because all the drives in the NAS must be the same size and should be the same type, if you can't quite swing the ~$1,800 for the 2TB (4-500GB drive) model, Infrant will let you start with one, two or three drives until you can afford to add more.

While there are other NAS devices available for less money, I prefer the Infrant ReadyNAS NV for several reasons:

1. Performance: The Infrant ReadyNAS is a really nice performer. While not as fast as the latest built-in hard drives, on a gigabit network it's quite snappy – usually exceeding USB drives. For copying large files (hundreds of megabytes), my read speed is about 20 megabytes-per-second and write speed is about 17 megabytes-per-second. Your mileage may vary.

2. Multiple filing protocol support: Including CIFS (standard Windows), AFP (Apple Filing Protocol), HTTP/S (web-based). The AFP support includes complete support for AFP.

3. Any SATA drive will work. While the drive must be the same size and speed as the other drives in the unit, you can buy off-the-shelf SATA drives from sources other than Infrant. Very handy if a drive dies.

4. Infrant supports RAID 0, 1 and 5 (my favorite). Plus they support their own X-RAID technology which is like a RAID 5 with the ability to add new drives on the fly.

I won't get into the details of setting up a NAS as it varies per product and configuration. A good tip, however, is that while many NAS devices (including Infrant) support AFP and the extensive supported naming conventions, you might want to consider using CIFS. That means you'll need to adhere to a stricter file naming convention than you would normally on a Mac, but the benefit is that your files will more easily back up to a remote backup service (more on that in the backup section).

Keeping iTunes Synced

When storing music on a NAS, one of the challenges is managing iTunes on a notebook. The exact method may vary depending on your gear, however, here's what works well for me.

1. Dedicate a home-bound computer for iTunes CD ripping. Assuming you have two computers, set up iTunes on a Mac that is always connected to your network. Set the iTunes Music folder location to point to a dedicated music folder on your NAS. Since this iTunes setup will place all music, podcasts, audiobooks, TV shows and movies on the NAS, this computer should also be used as your CD ripping computer. Use the screen shot below as a guide:

   

2. On your Mac notebook, set the iTunes Music folder location to point to your local Music folder – the default setting. Make sure you uncheck Keep iTunes Folder organized and Copy files to iTunes Music folder when adding to library. You don't want the music files on the NAS to be copied to your notebook. Use the screen shot below as a guide:

   

3. Use this Mac (your notebook Mac) as the sync machine for the iPods you want to be able to sync when you travel. That way you can sync new podcasts or other content while on the road. In order to make all the iTunes files on the NAS available in this copy of iTunes, use the Add to Library... function and point to the NAS folder where you put all your music in step #1 above.

4. When you're on the road, you can sync your iPod to add new content – which is downloaded to your local Music folder. While you'll get an error message saying that it couldn't sync files because they weren't available, they won't be deleted in iTunes or from your iPod. Once you get back to your home network, everything will sync normally. Whatever files you downloaded on the road will be on the notebook computer only unless you manually move them to the NAS.

Keeping iPhoto Synced

I don't use iPhoto for my photo management. I'm a big fan of PhotoMechanic because it doesn't use a separate database to manage the photos. Instead, PhotoMechanic relies on folder hierarchy and camera data and keywords imbedded in the photo files themselves. For me that works better. If you use iPhoto, the same rules used for iTunes should be applied to iPhoto. Basically dedicate one machine for pull files from your camera and move its iPhoto library to the NAS. On the notebook, make sure you uncheck Copy files to iPhoto Library folder when adding to library in Advanced Preferences. That way, it'll just point to the NAS photos and work like iTunes does in the scenario above.

But what about Backups?

You might think that the NAS negates the need for backups. Not so fast. Yes, the NAS provides a bit more redundancy than a standard hard drive, but it doesn't necessarily protect you from accidental file deletion, natural disasters and physical destruction (fire, flood), theft or unnatural disasters like your 2-year-old watering it to keep it green. To protect yourself from these catastrophic problems, you still need to back up – and that's no small chore. Full confession: recently, I made a very stupid mistake that caused one of my UNIX scripts to go wild and delete all of the photos on our NAS – over 100GB of digital photograph masters spanning nearly ten years. Since I mirror my photographs daily, I was able to recover all of them without losing a single photo. While some would say I was lucky, I happen to know it was good planning.

Backup Options

There are four primary options for backup.

• Tape backup. I'm not even going to cover tape backups. Most people who want to go that route understand the options and strategies used. I can't recommend tapes for most users.

• DVD backup. With growing storage needs, 4.4GB or even the larger 9GB dual layer DVDs are just too small for most regular backups. While they can be used to archive projects (make sure you make at least two copies of verified DVDs), it's way too painful to regularly back up hundreds of gigabytes onto dozens or hundreds of DVDs.

• Hard drive backup. On a regular basis, you can back up to another hard drive or rotate a hard drive. To protect against natural disasters, you'll have to store at least one backup drive off-site or in a 2^nd floor fire-safe. The larger your active data set gets, the more unwieldy this practice becomes. While LaCie makes a 1TB USB drive, it's not cheap at $450. And, you'll find a rotation strategy warrants at least two drives – even more cost.

• Online backup service. Until recently, online backup services were too expensive to consider. However, Amazon's S3 service has reduced that price to something most people can afford. At $0.20 per GB upload and $0.15 per GB storage, my 200GB of storage costs me about $30 per month after the initial $40 to upload. There are other services out there, however, my experience is with Amazon's S3 and the JungleDisk client software for S3.

  There are a few downsides to Amazon S3 backups beyond just cost.

  1. Since your data is now on a server over which you don't have full control, you should encrypt highly personal information (like financial data) before backing it up. While JungleDisk provides encryption for all data that is sent to S3 and I use it, I still make sure my financial data is encrypted on my local disk before sending it.

  2. Most home Internet connections have slow upload speeds. Mine is about 750Kbits (about 94Kbytes) per second. So, it's important that you set up a backup process that only uploads changes. It takes patience to get your data set up on the server. We'll talk more about that later.

  3. Since home download speeds are usually five to ten times faster, restoring data is pretty speedy. However, it's still slower than if you were restoring from a local hard drive.

My daily method of choice is online backup. Even with its share of warts and a monthly fee, I find it gives me peace mind to know that all my files exist outside my home. Even with a natural disaster or theft, I won't lose irreplaceable photographs, and files. As a backup to the backup, I mirror the same files to a secondary hard disk in my home every month or so.

Backup Software

I used to recommend backup software such as Dantz Retrospect. Software like Retrospect was required when going to tapes since it managed tapes so well. These days, with such huge sets of data, I've revised my backup strategies and software recommendations:

1. While not backup software, you'll need to use JungleDisk with Amazon's S3 service. JungleDisk gives you the ability to mount a remote drive on Amazon's S3 WebDav service. JungleDisk and S3 work like a drive in .mac – only unlimited in size. JungleDisk is not without its share of problems. It will crash once in a while and has trouble with files that don't adhere to its stricter non-Mac naming conventions. Also, it can't back up files larger than 2GB in size (which I wouldn't recommend anyway unless you have a really fat upload pipe). Those files should be backed up using a local method such as DVD or hard drive.

2. For financial software such as Quicken or MoneyDance, I recommend maintaining encrypted copies and configuring the software to make backups each time you quit (keeping 10 or so). That way you'll end up with versions in case you have to go back in time.

3. For mirroring data from the NAS to a local disc or the remote JungleDisk/Amazon S3 service, I recommend ChronoSync on the Mac. Unix geeks may prefer rsync, however, unless you really know what you're doing, you're better off with ChronoSync. I'll cover the specific ChronoSync setups below.

On a Mac which is always connected to your network, setup JungleDisk and ChronoSync.

JungleDisk Configuration

1. Sign-up for an Amazon S3 account. There's no cost if you don't use it.

2. Download the JungleDisk software from jungledisk.com.

3. Install the JungleDisk software.

4. Configure the JungleDisk software with your S3 Access Key ID and Secret Access Key.

5. Set JungleDisk to Encrypt files using my AWS Secret Key. I have found that encryption doesn't slow anything down. The bottleneck is your upload speed.

6. Uncheck Upload files in the background for faster performance. It doesn't actually change performance, it just moves the status and progress meter from ChronoSync to JungleDisk – this leaves you with less control to abort, etc.

7. Mount the JungleDisk drive using APPLE-K (also known as COMMAND-K). If you setup using the defaults, the server address will be: http://localhost:2667/.

ChronoSync Configuration

1. Unfortunately, ChronoSync isn't set up by default to synchronize files in place. This is important, because if you don't sync using this method it will take twice as long (first syncs the file, then essentially moves it). In discussions with the ChronoSync developer, we determined that there is a hack that will work. Hopefully in future versions, you will be able set this configuration using the standard interface. For now, you must do the following (to watch a 3MB online video of this process, click here):

   • After installing the ChronoSync software, locate the ChronoSync icon in the Applications folder.

   • CONTROL-Click the icon and select Show Package Contents.

   • Open the Contents folder, then the Resources folder.

   • Find the file ChronoSync.plist.

   • Open the file by CONTROL-clicking it and open with TextEdit (you may have to select Open with Other, depending on your Mac).

   • Change the following text (make no other changes):
     <key>SkipSafeCopy</key>
     <false/>
     to:
     <key>SkipSafeCopy</key>
     <true/>

   • Save the file ChronoSync.plist.

2. Set the ChronoSync Preferences to match your needs. I like to uncheck Run scheduled items synchronously so that I can mirror two settings files at once.
   

3. Create a settings file for each volume or folder of your NAS that you want to back up. I keep those ChronoSync settings files in a folder called ChronoSync in my user folder. Because of the time it would require and the fact that I don't shoot much video that can't be replicated, I have chosen not to back up my video files using JungleDisk. My backups are:

   • Storage: A master folder where all my non-photo/video/music files and documents which are stored on the NAS are kept.

   • Music: A folder where all music is kept on the NAS. The sub-folders are broken up by artist-album.

   • Photos: A folder where all photos are stored on the NAS. The sub-folders are broken up by event or location.

   • NetBackup: A folder where I sync files from any other computer on the network, including my notebook, that I want backed up regularly. I use ChronoSync on the notebook to get these files to the NAS for backup.

4. The settings for most directory mirrors are the same -- I'll use my photos configuration as an example:

   • Setup. I use mirror instead of backup because I want to update only what is different based on file size. I do not have synchronize deletions checked because in general I want to make sure that an accidental deletion of photographs doesn't get propagated to the mirror. Once in awhile, I manually run it with synchronize deletions checked, but I never have that setting run automatically.
     

   • Options. The Reporting & Error Handling configurations are set that way because this runs at night, and I don't want it to stop on an error. The other settings are important because of JungleDisk/Amazon S3. You want to update only if the file sizes are different, and you don't want to verify as that would take too long.
     

   • Rules. I recommend setting up rules using the advanced mode. For photos, only a few exclusion rules makes sense. For Music, you may also want to exclude some folders such as Podcasts, Audible, etc.. Regardless, you should exclude all files larger than 2000MB (an Amazon S3 limitation) as shown below.
     

Mirroring Your NAS

It should be called the daily mirror, because that's what you're doing when you schedule ChronoSync to make sure data exists both locally and on JungleDisk/Amazon S3. The first time you do this it's going to be a long slow process. It'll require patience and you'll have to babysit it. You can abort a mirror at anytime, and whatever you have mirrored won't have to be re-mirrored unless the files are updated locally. Until you have a full mirror set online, I don't recommend scheduling the daily sessions. For example, if want to mirror 50GB of data, your upstream Internet connection is 750,000 bits-per-second, and you'll dedicate 14 hours-per-day for the process; it will take about 11 days to get that 50GB mirrored. Now, because the type of data we're talking about doesn't change often, after the initial mirror is complete the time spent per day is dramatically reduced. In general, my 200GB of mirrored data takes less than two hours per day to keep updated. Obviously after a vacation full of photos or ripping a few CDs, it'll take a bit longer. The reason I allocate only 14 hours or so per-day is because Internet connection is very slow while the mirroring takes place.

Because it initially takes so long to mirror, I recommend breaking it into projects:

1. Photographs. In general, photographs can't easily be re-created if lost, so they are my #1 priority.

2. Storage. Next in line is Storage. While some files can be recreated, others such as financial data are much more complicated. So this is my #2 priority. These files also tend to be smaller so this is a short process.

3. Music. While music files can generally be recreated by re-ripping, who wants to go through that hassle? It's #3 priority.

4. NetBackup. Next in line is NetBackup. While these may be important files, they're already a backup of a backup, so I prioritize them #4.

As you get an initial mirror of each group above, it's time to schedule a daily mirror for that group. Mine are set to start at midnight. If you're not at home during the day, you may want to start yours at 8am. Set up the mirror schedule to match your schedule.

A few other tips worth considering:

1. Make sure your NAS has email notification set up for errors.

2. Use a UPS connected to your NAS to reduce potential problems in power outages. A small UPS such as the $60 (street ~ $50) APC BackUPS BE500R is fine.

3. Check on your NAS every week (at a minimum) to be sure you have enough space on the volumes you set up and that the files you expect to be there are in fact there.

4. Every month or so, run each ChronoSync profile with delete turned on. Make sure it's set to immediately delete. Do this only after you have made sure that all the files you expect to exist on the NAS are there, because any missing files will be deleted from the JungleDisk/Amazon S3 mirror.

5. Make sure your network is ready for the task. Gigabit Ethernet is a must. See our article: Make Your Home Network Fast!

Now, you can sleep at night.

TrackBack URI for this entry

Comments (72)

Subscribe to this comment's feed

Show/Hide comments

Paul W said:

Excellent guide and very useful for me as I'm currently deciding how to store and serve my media around my house. I have one question in relation to the management of iTunes with a notebook. I have a Macbook and have almost decided on purchasing a Mac Mini for my front room to provide media to my TV and act as the 'base' media computer for the house. If I reside the media on a NAS with the Mac Mini managing the iTunes library, I would also still like the Macbook to see a 'current' iTunes library. The only way I see this working with the Macbook is via continually 'Adding to Library' to capture the new content ... now I can manage that for new additions that I initiate (eg. newly ripped CDs/donwloaded TV/movies) but do I have to do it for the podcasts as well or is some auto syncing possible? Thanks again ...

January 25, 2007

TechDigs.net said:

Paul W: Another option would be to setup the Infrant NAS an iTunes server. It then looks like a shared iTunes server (of course you could do that with the Mini too). If you don't do that, then I suggest that you keep your notebook iTunes library local and then 'add to library' the NAS data. Then, for podcasts, you can subscribe to the podcasts on your notebook also so they get downloaded to your notebook. It's not optimal, but short of a Chronosync script or similar that updates your notebook iTunes library (keeping the Mac Mini as the master), there aren't too many options.

January 25, 2007

Shawn said:

...thanks a million, man....perfect setup. I wonder if there's a slightly 'less' crazy system, using a pair of LaCie's ethernet disc mini's (currently available on lacie refurbed for $99/250 gb). I'm wondering if having two of these 250 gb drives would give me a way to connect them to the router, and then a software solution that would essentially give me a raid-1/mirrored drive, accessible by every one of my networked computers. Researching now....if you've got input, feel free. I realize that your solution is more robust, but noticeably more expensive.

January 25, 2007

TechDigs.net said:

Shawn: We're working on reviews of a few lower cost NAS products such as the Buffalo Linkstation. It's 200GB to 500GB NAS that isn't RAID. They're about $225 each. However, I think a NAS is a better way to go than using drives connected to routers. They're just about the same price and dedicated to the process. Bottom line, any network accessible drive can replace the NAS in this article -- with all the obvious caveats. More to come...

January 25, 2007

rob said:

Almost exactly my setup, but I let my ReadyNAS NV act as an iTunes Server (daapd) for other devices to see. Seems more elegant. Granted, the daapd implementation is a little old in the tooth. I prefer SuperDuper! to Retrospect.

January 25, 2007

TechDigs.net said:

Rob: Totally agree about SuperDuper. However doesn't work well over WebDAV for the data you want to use remotely. Chronosync is better for that. Using the NAS as the iTunes server is actually something I do also, but it's not quite as friendly. It's something people can look into.

January 25, 2007

Jonathan said:

Why do you uncheck "Keeps iTunes music organized"?

January 25, 2007

TechDigs.net said:

Jonathan: On the notebook (the non-master) you don't want it to organize or import the data, just add it to the database. You only want the master (say a Mac Mini or similar) to keep the iTunes library organized. Make sense?

January 25, 2007

Tim Jarrett said:

Killer article. I'm getting ready to cut the tether and put my external RAID array directly on my network so I can 86 the failing PowerBook they're connected to. I'm currently waiting for reviews of the new AirPort Extreme with NAS support. I'm afraid that it won't be fast enough over 802.11g to support using iTunes with the music files on the NAS, as you describe here, and Apple hasn't said anything about 802.11n support for older (e.g. first generation MacBook Pro) machines...

January 25, 2007

TechDigs.net said:

Tim Jarrett: Thanks! I generally don't like Wireless for permanent entertainment connections. Yes it works, but it's not perfect. That said, the 802.11g actually does fine if you only want to playback content. However, I would definitely have the master iTunes machine hardwired. Not sure how I'll feel when 802.11n is more real. Could be interesting, but who knows. In terms of a drive on the new Extreme, that could be interesting, but if you don't need that to do wireless and you don't need that for a low-cost option. We're going to review some other solutions as well.

January 25, 2007

Tom Boucher said:

802.11g is plenty fast for day to day listening to music. I used this method with an airport express for years. The only time it bogged down was when you were originally copying the files over, or you did something like 'scan for gapless playback' or the sound leveling feature where it hits every file. Plug into a network when you do that.

January 25, 2007

Paul Freeman said:

A great article, but one thing I've heard people who deal with RAID arrays day to day say a lot, is that it's a good idea not to use drives of the same make. - Yes, you need to have the same size and spec, but by having different brands in place you avoid the problem of a defect in a single batch that suddenly causes several disks to fail all at the same time. There are probably advocates of both methods, but the different brand line of though makes sense to me (Since I've seen the mass death of several drives wipe out arrays in the past)

January 25, 2007

TechDigs.net said:

Paul Freeman: THANKS! Your concept makes sense and it's a very good thought, however more people have had trouble with mismatched drives than with bad batches. That said, last time I built a NAS I did order drives over a few weeks just in case. Either way works though, as long as the drive is approved by the vendor (in this case Infrant).

January 25, 2007

Dustin said:

Thanks for the post. I am actually in the process of trying to do something quite similar. I wanted to give an alternative to using Amazon or any other vaulting services. I was thinking if one had a web server account of one way or another, this person could map an ftp drive to a location on the server and then use the sync software to make the necessary copies from the NAS to the ftp service. Why, you may ask, is this better? Well it might not be; it is another option though. I am planning to use dreamhost.com to do so which will give me plenty of space, and the space which your account is allocated grows weekly. This option would also be cheaper ($7-$20 per month - including web hosting).

January 25, 2007

TechDigs.net said:

Dustin: You're welcome. Amazon is a necessary evil in my mind, still to flaky to be perfect and it can get expensive. There are some new solutions coming and we'll be tracking those.. Any solution that accomplishes things in a similar way with WebDAV would be a reasonable solution. Thanks for the suggestion.

January 25, 2007

Neil'os said:

I know the need for GB of secure data is very appealing to the inner geek. But, it's akin to suffering caused by attachment. My entire computer setup could be wiped tomorrow (around 250Gb inc. OS) and I'd miss some photos and mp3's that I can't replace. I'd have to re-find some email addresses. But life would go on. Being attached to material possessions is bad enough, admitting an attachment to Gb's of data seems doomed. Why not identify about 650Mb tops of really very significant, can never ever be replaced stuff and just burn it to a CD?

January 25, 2007

TechDigs.net said:

Neil'os: Thanks for the comment. Each person will end up having their own perspective and threshold regarding their own needs. Your approach could work very well for you and others, but not for everyone. For those who keep lots of data, it won't. That's why we showed how to do this. Your choice to do this or not is yours as it is everyone's. We're sure not going to judge someone for keeping too little or too much data.

January 25, 2007

nalenb said:

Similar to what Paul W said above, how do you manage the addition of new iTunes content? Do you just manually add to each machines/accounts iTunes or is there some automated way to scan for new content daily or something? Ideally I'd like some content locally (favorite songs and podcasts) and TV shows & movies on the NAS. I assume that I can toggle the copy to library switch as I get each type of content and it would work.

January 25, 2007

Adam Masri said:

I disagree with a lot of the recommendations in this article. I think backing up terabytes of data over the Internet is rather insane. It is costly, & would tie up your fast Internet connection for a ridiculously long period of time. If your credit card expired or there was another billing or customer service hiccup, your backup could be obliterated, requiring re-upload and associated time & cost. It would be better to have a large FireWire RAID 1 or 5 HD, connected to the mini you already have in this scenario. The mini running OS X can do Windows & Mac filesharing. A RAID HD should cost less than a NAS, as there's no additional computer inside the box for the filesharing task. Most importantly, over the years, I've seen NAS products corrupt files when an upgrade to the NAS or MacOS comes out. You don't want several TB of data corrupted due to some new file format or filesystem format. I would auto-launch iPhoto & iTunes at startup on the mini, and share both their libraries. It's easy to share iPhoto pictures, just drag-copy from one library to the other. For iTunes, it's pretty easy: just open the foreign iTunes Library folder via filesharing, and drag-copy items into your iTunes window. This will copy those items over. IMO this would be far simpler to keep track of what was stored on each machine. For backup, I'd use a real backup app like bru, and backup to an external large HD like LaCie's Bigger Disk 2TB. Then keep the backup drive offsite, perhaps at a friend's house. Update the backup once a month. - Adam

January 25, 2007

TechDigs.net said:

nalenb: No real automated way to do it without custom software to manage the iTunes library file - would be ugly to write. However the suggestions we made above to Paul W. Thanks for the question.

January 25, 2007

TechDigs.net said:

Adam Masri: Thanks for taking the time to read and the comment. I agree with you, terabytes over the internet is too much. However, for the key files you don't want to lose in a real disaster (tornado, hurricane, earthquake, etc.), the Internet is one of the easiest ways to get it off site. Photos are the best example of this. Music isn't worth it for many --- that's what insurance is for. As a backup, the odds of you losing both your local and your disaster recovery back up at the same time due to credit card, etc., is outrageously low. Even lower than getting hit by a hurricane. Good point on the RAID connected to the mini, it would work the same way. However, I am not as concerned as you about updates from reputable vendors. We've run Linux-based RAIDs for years and had no more trouble than we've seen from driver update problems with RAIDS and technical problems with software RAID reliability (although those problems are generally gone these days). Of course there's nothing like a good backup. Using iTunes sharing is a great idea and something we like. Not everyone does because, but it's a good one and we agree that it should be considered and probably included in this article. Your backup idea is a good one and one that businesses (storing offsite regularly) have used for years. For those with the discipline to do it, it's a great way to go. For those who don't have the discipline and just want something automatic for those files that they can't be without, the Internet is not a bad way to do it. Again, thanks for your good suggestions. Each person should tweak these ideas as they see fit (including doing something different if it doesn't work for them). It all depends on their threshold for loss and their ability to manage it. Cheers!

January 25, 2007

Mark said:

there is a wonderful NAS solution called NASLite (http://www.serverelements.com/) It installs SMB on a dedicated machine to function as a NAS appliance, dedicated to serving disk space. Any junk PC hardware will work fine, and NASLite is free.

January 26, 2007

nalenb said:

iTunes sharing isn't that great because you can't build local playlists off the shared library and the different media types don't show up in your library list, i.e. TV shows don't show under the TV shows item. Sharing out a drive from one computer is okay (although the small drives on minis and laptops won't hold all the info) and I can treat it like the NAS, but what happens when my kids are on the mini that is sharing and they reboot while I'm importing a movie file? The NAS (or other network drive) is a better solution becuase the uptime should be much higher where you don't have a lot of control over the other computers. Backing up and storing drives at a friends doesn't really protect you from natural disasters if you live close. If you don't live close then you end up shipping the drive back and forth. The ideas presented here seem to be aimed at solving two problems: backing up large amounts of data and accessing more data than can fit on a mini or laptop drive. It's great to see someone layout a solution to both problems.

January 26, 2007

Morten said:

Noise level. What about noise from the Infrant ReadyNAS NV, is it silent enough for your living room?

January 26, 2007

gregkeene said:

Morten: No, I don't think the ReadyNAS NV is the right device for the living room, too much noise (but I like a dead-silent living room). The ReadyNAS NV+ is supposedly a bit quieter, but we haven't tested. Infrant also has a product we plan to review in the future called Repertoire that is dead silent, but it's expensive. My suggestion is to put it the NAS another room and run Ethernet (preferably Gigabit). That's how my home network is setup.

January 26, 2007

Adam Masri said:

The key problem I have with NAS devices is that I remember when the Mac got upgraded from HFS to HFS+ filesystem, some NAS devices broke in that transition. Saving files from an HFS+ Mac to an HFS NAS could destroy files on the NAS. After that, I never trusted the technology again. A future filesystem upgrade on Macs or Windows could cause something like this again. But when you upgrade the OS on your computer, you may have a false expectation that nothing will go wrong with the NAS, when in reality you really need to backup both systems. People treat NAS like an appliance, one that doesn't need any babysitting. It's easy to forget about it, and not do backups of the NAS. I'd rather have a full backup application running on a computer, so I can easily backup my data to any hard drive I please, and be able to upgrade (perhaps to SATA) as my needs change. Last, the key reason I don't like this solution is because iTunes & iPhoto data files are not designed for multi-user use. Having them out on a fileserver opens the possibility that two Macs could hit the same datafiles simultaneously, damaging them. I'd rather use these apps the way Apple intended, on a single Mac, sharing their data out to the network.

January 27, 2007

Adam Masri said:

TechDigs: "I agree with you, terabytes over the internet is too much. However, for the key files you don't want to lose in a real disaster (tornado, hurricane, earthquake, etc.), the Internet is one of the easiest ways to get it off site." True, for a small number of important files, such as accounting data, important databases, etc. You could use .mac for that. However, just as you mentioned, the need for storage is growing at a staggering rate -- parents are prolific photographers, and every picture is a precious irretrievable moment of their baby's & family's development. I've worked as a Mac consultant for almost 15 years. I recently had a situation where a client wasn't doing backups, and thought she had lost her HD. She was crying on the phone. It's easy for a callous person to blow this off as a "tie to material possessions," but pictures of baby growing up are more important to these people than material possessions. Stuff can be replaced, memories cannot. I have clients with thousands, some with TENS of thousands, of pictures in their iPhoto libraries. These are easily 50GB sized libraries, and continue to grow at an outrageous rate -- first it's mom. Then it's dad taking a few pics of the family. Then at some point the kids create accounts in OS X, have their own workspace & their own iPhoto, and start taking pictures of their own. Data storage needs skyrocket quickly in this scenario, and a fast, easy to use backup solution is paramount.

January 27, 2007

Adam Masri said:

For these people, I tend to recommend a large external HD, such as those from LaCie or ROCSTOR, and CMS' BounceBack Pro. BounceBack is great because you plug the drive in, the software sees the drive, launches, does a mirror backup, and quits. The client drags the backup volume to the trash, disconnects the drive, and puts it away. If it isn't fast & easy, it won't get done. This is about as fast & easy as it gets. It doesn't handle multiple versions (snapshots) of a single file thru time, but since it only backs up the changes since the previous backup, future incremental backups only take a minute or two. http://www.cmsproducts.com/product_bounceback_software.htm Now, I'm not advocating this solution for this group. Obviously the needs of those reading this article are greater, and the technical proficiency of this group allows more complex data backup solutions. That's why I think a real backup app like bru makes the most sense to backup to the off-site backup. http://www.tolisgroup.com/ If the concern is possible loss of primary & backup storage, why not put the backup HD in a safe deposit box at the bank? This is about as safe as you could get. This is what I do. The safe deposit box is at a bank about 30 minutes away from home. The chances that a natural disaster would wipe out such a large geographic area is virtually impossible. You could have two backup HDs, one in the bank vault, the other at home. Backup nightly to the on-site backup, then swap monthly w/the offsite backup. You could never lose more than a month's worth of data. Swap more often if you need to.

January 27, 2007

Adam Masri said:

nalenb: "iTunes sharing isn't that great because you can't build local playlists off the shared library..." Well, you may not be able to add songs from a shared library to a local playlist, but you could just create playlists on the mini. Use the mini as your store of all music & video, and use it to sync iPods. You already know you can't store everything locally on your laptop, so don't try. Instead, copy over the music you know you want on your laptop. Or, just use an iPod to play music and save all that space on your laptop for other uses. You'll still have full access to your music library when at home thru iTunes sharing. "and the different media types don't show up in your library list, i.e. TV shows don't show under the TV shows item." They don't show up under the TV shows item in the source column, but if you click the shared library, you get a disclosure triangle. Click that, and you see a TV Shows playlist inside the shared library. Music, video, anything you've stored in iTunes is shared and will stream even across a wireless network. So you have access to everything when you're home, or if you use network bridging to gain access to your home network from elsewhere on the Internet. You could do that with a VPN (IPSec) or possibly even ssh port forwarding type setup, having your laptop gain an IP on your home network. When you're out, you can use your laptop's copies of iTunes & iPhoto. When you get home, add those items back to the mini. That way, the mini becomes your storehouse of data, giving access to other devices around the home. Slim's software will import iTunes libraries & playlists.

January 27, 2007

Adam Masri said:

"what happens when my kids are on the mini that is sharing and they reboot while I'm importing a movie file? The NAS (or other network drive) is a better solution becuase the uptime should be much higher where you don't have a lot of control over the other computers." The mini becomes a server, the same way the NAS would have. This is not a computer used for playing games, it has one purpose and that is to serve media around the house. If you get a newer dual-core Intel mini, you'll have a buff enough machine that you could also connect it to your TV & use it for Front Row to watch your media on TV. Maybe even add DVR functionality. http://www.miglia.com/ If you want things quieter, put the mini & RAID elsewhere & hook up an Apple TV to your TV to watch what's on the mini. Hook it up to a big RAID, and you've got a single box to play back your media, serve it around the house, & back it up. - Adam

January 27, 2007

Adam Masri said:

Oh yeah... One last thing. I noticed on Infrant's Website that their XRAID technology doesn't work w/a hot spare. If your data is important enough for RAID, I don't see how you can depend on a box w/o a hot spare. I know from personal experience that either due to failure or human error, it is possible to lose two drives fairly close together. A hot spare keeps your RAID humming along. - Adam

January 27, 2007

JE said:

I've got a very similar setup on a PC based home-network using a Buffalo TeraStation. Even with this, I like to back up very important stuff (Quicken & such) online and was disappointed that Mozy (and Carbonite and others) all do not support network drives, including those on the NAS. All of our data resides on the NAS, so to use Mozy I have to schedule a SyncBack profile to copy some stuff back to the local drive periodically so that Mozy can pick it up. Anyone know a Mozy-like solution that will read network drives?

January 27, 2007

Michael said:

Adam what Raid would you recommend for the mini?

January 27, 2007

nalenb said:

Adam, good points. My plan right now is to use a large usb drive off the new Airport base station as a network storage drive. I'll put the movies and TV shows on that drive. Then on each iTunes I'll turn off copy to library, add them and then turn on copy to library. That way the large video files are only linked. I have DVD backups of the purchased videos which is good enough for me. If they are destroyed it's not that big of a loss. iTunes isn't meant to have a shared library, but as I took the article, it's fine to have multiple local libraries sharing networked files (they are just files after all). I'll probably keep a copy of the music files on the network drive just for ease of importing them to each machine (our music is probably < 10 gigs). We all have different play lists setup, so having 1 machine do all the iPod functions won't work well. Plus I haven't found headphones I'd like to have on all day. For photos, I plan to keep them all on the Mini for now since I have < 5 gigs. I'll back them up to the network drive and to S3 for safe keeping. iPhoto sharing is good enough since I can drag photos out of the Mini's library and into mine if needed (which is rare for me). I'll have nightly sync's of user data from each machine to the network drive and probably to S3 as well. It's great that we have so many solutions now that we can tailor to our particular needs.

January 27, 2007

TechDigs.net said:

Adam: Too much to respond to everything, however a few key things. - To each his own. I think replicating 50GB over the Internet is fine if it's important enough to you. If you don't, then I don't think you should. - I would not recommend using X-RAID (Infrant's proprietery RAID), instead use RAID-5 -- which is standard on the ReadyNAS. - If you are concerned about future compatibility, then don't use AFP, use SMB. SMB is a standard that's not going to change and is widely used. Naming isn't so restrictive that most Mac users will have no trouble. Regards! TechDigs.net

January 27, 2007

TechDigs.net said:

Michael: My suggestion is that you don't run a software RAID directly on the Mini. Using multiple drives connected via firewire or USB-based RAID on the mini is likely to be far less reliable than a NAS, period. It's not what the Mini is designed to do. A SATA-based RAID in a MacPro is a different animal, but a Mini - wouldn't recommend it. For those unitiated, a dedicated NAS RAID solution is a computer with Linux configured to be a RAID server usually with a web interface. Think basic processing power, fast Ethernet and a SATA back-plan. Therefore designed to do it from the start. Don't let FUD stop you from investigating them. They're a very reliable solution used extensively. The key is to remember to treat them like any other computer with some care and assure that you set them up properly with notifications for drive failures etc. Tying up your multipurpose media machine with storage and back-up duties doesn't make a lot of sense if you have lots of data to store to be used throughout your house. It all depends on your need. They're not for everyone, but will work well for many.

January 27, 2007

TechDigs.net said:

nalenb: A piece of advice. Before you start relying heavily on the new Airport extreme with an external drive, bear in mind that the early adopters will have some trouble and might lose some data depending on which external drive they use, etc. I'm too am counting on that product being cool, but I wouldn't put my important information on it until it's had a few weeks or months in the wild and drive compatibility is established.

January 27, 2007

Kernos said:

Fascinating article. Thank you. I am sharing the link with all of my OS X groups. I have my own domains hosted with Dreamhost.com, shared hosting, with 500 gigs of disk space and 5 tb of bandwidth. I have used both Chronosynch and RBrowser to synch local data to my remote domain. I generally use RBrowser because I know how to use ssh with Chronosynch. This is a lot cheaper than using Amazon and I do not backup video remotely. Is there any problem using one's own domain's disk space for backup? Kernos

January 28, 2007

TechDigs.net said:

Kernos: Good idea and thoughts. No, there's no problem using your own domain's disk space for backup. The only issue is encryption and security. SSH will encrypt while sending, but it maybe unencrypted when it's stored at your host. Depending on what you are backing up, you should gauge the importance of encrypting it.

January 28, 2007

Michael said:

I think this is a great article and plan on using part from it and parts from the comments. I like the adding to itunes part as I have a Mac Pro that I use and need the media to pull into iMovie or Final Cut. The iTunes Sharing does not work for this. I also have a Mac Mini as my media center hooked up to my TV. Adam has a point about staying all Mac. I have tried different NAS setup with poor results. I have an old Mac Mini and an old Power Mac G5. I am looking for a hardware Raid 5 setup. I have 8 500 gig LaCie drive and a 2tb LaCie Drive. I looked and the Apple Raid but $6,000 for home use is a bit much. My plan was to use my intel Mac Mini to run a firewire Raid for LaCie. I wanted to use the mini to save space in my 13U rack I have under the stairs in my basement. Since I have been looking I have seen a lot of talk about SATA Raid boxes and know I need and Card for that so I would have to use the Power Mac G5. What would someone recommend? Thanks

January 28, 2007

Mitch said:

Congratulations on having found the time not only to get your backups under control, but to share it with the world via such a detailed article. I sadly haven't found the time yet (still messing about with DVDs) but was looking at using the following NAS simply because it's much cheaper: http://www.dlink.com/products/?pid=509 It is only a two bay RAID 1 solution but combined with an off-site mirror it will be adequate for my needs. I plan to buy 2 and keep one at my parent's house. I then plan to sync my backups to that drive over Hamachi: http://www.hamachi.cc http://homepage.mac.com/lxr/homepage/spaceants/hamachix/index.html I have to manage my parents network anyway so I may as well get some use out of it. Using them as an off-site storage location seems ideal. I'm waiting to see the details of any backup software built into Leopard, but until then I will likely continue to use Apple Backup. It's simple and is adequate for my needs. Mitch

January 28, 2007

TechDigs.net said:

Mitch: Thanks. The good news, we have that D-Link unit on its way for review and we'll be able to contrast the pros/cons of it with other devices we have in the queue for testing. More on that soon. TechDigs.net is a huge fan of Hamachi - having used it for the last 10 months or so. As you probably know, Hamachi won't give you direct access to a remote NAS, instead it will give you access to a Windows, Mac or Linux machine that has access to a NAS. It's something we've suggested to Infrant and hope other NAS manufacturers consider - secure remote access. For backup I too am interested in Time Machine. On the surface it looks more like a local drive backup solution as opposed to remote. For remote backup, I really urge you to look at Chronosync. For incremental mirroring over the Internet it's cleaner and faster than than Apple Backup. For remote backup it's more efficient than SuperDuper.

January 28, 2007

Bit said:

Absolutely AWESOME article. Bravo. I guess this issue is really starting to come to the fore. I have been working through many of the same issues discussed by the author and I settled on the ReadyNAS NV+. I have not regretted the decision for a second. Everything comes down to the fact that the people reading this article are (most likely) responsible for protecting some seriously important data. Imagine the look in your wife's eyes if you had to tell her that you lost the pictures of your wedding, etc. IMHO there are too many other things in life to worry about so I do everything I can to make sure I can go to sleep at night knowing my data (memories, etc.) are safe. People are amassing digital data faster than ever before (and the rate of accumulation is increasing). It is freaking scary.

January 29, 2007

Tom said:

- I would not recommend using X-RAID (Infrant's proprietery RAID), instead use RAID-5 -- which is standard on the ReadyNAS. Actually, the Infrant runs RAID 0,1,5,JBOD and I think 0+1. X-Raid is the embedded software that does the standard RAID. One advantage of it is the ability to change the RAID setup. Typically you have a RAID with n disks of all the same x size. When you run out of space you have a few options: 1) backup the current RAID, create a new RAID with more disks or larger disks. Maybe convert RAID level. 2) get another RAID along side it with new disks 3) get another server X-Raid lets you convert, on the fly, between RAID levels. Start with a RAID 1 of 2 drives. Add a 3rd drive and convert to RAID 5 for 50% more space. I don't know if X-raid will let you increase the size of the drives (convert 250s to 500s for example).

January 30, 2007

TechDigs.net said:

Tom: Theoretically X-Raid will let you increase the size of the drives. What you would have to do is replace one drive, let it rebuild itself, wait until it's complete and repeat the process until all the drives are swapped. It's not sanctioned by Infrant, but they say it will work. If you are starting with 4 drives, I recommend RAID 5. Only if you are starting with less does X-RAID make sense.

January 30, 2007

Duncan said:

I know this sounds like a gold-plated solution for home use, but just out of curiosity, has anyone used Apple's RAID box for this? What Apple-specific technology would it offer that other NAS devices don't? (I'm thinking of its ability to serve media directly using built-in OS-X applications.) I'm wondering if it might be reasonable to buy a used unit (or new, if that's all one can find), populate with a minimum number of disks for today's storage needs, and grow the RAID volume over time.

January 31, 2007

Cody said:

How does the Infrant ReadyNAS NV(+) compare to the Thecus N5200? Thecus says its OSX compatible but has anyone actually worked with one in a multi OS environment? http://www.thecus.com/products_over.php?cid=1&pid=8

February 01, 2007

Christopher said:

Great article! I stumbled into the party late but it has given me some food for thought as I am looking into S3 for backup. Your setup is quite similar to mine but I have a couple of thoughts: - We both have a media centre computer in the living room so I decided long ago that it would be a linux box running Myth.. essentially this gives me complete control over the hardware and a wide choice of services. (For example I advertise the music files so iTunes users that just need to listen can connect easily, I can even create playlists on the server) Do you think there are any benefits to the NAS route? Hardware lust aside I think there is nothing to gain. I like that I can chuck in disks and storage cards but then linux servers are my thing so I concede my setup doesn't have the flick a switch ease of use. Anyway in the end it's all linux serving up data on a raid.. - I already use rsync to mirror data to another server I run on the other side of the ocean so that would be the most likely method for me to run the backups to S3. One advantage to this is rsync's optional rate limiting.. I can set it to 30-40% of my upstream bandwidth and not notice any slowdown, other than the speed of the transfer of course. - I'm looking into S3 mainly for reliability as I don't want to have to worry if my remote server goes down even though it is also a linux server with raid. I'm still in the planning stage but you've given me some good ideas.

February 07, 2007

Bernard said:

After reading the article and going through the comments, I still have one question. How would you encrypt your financial data stored offsite? What do you recommend or use? Thanks.

February 08, 2007

TechDigs.net said:

Duncan: Definitely gold-plated. Most media solutions don't need s streaming server and a fast network server would be enough. Although the full-blown OS would be cool for certain functions. Nothing wrong with it, but the cost probably wouldn't justify the gains.

February 08, 2007

TechDigs.net said:

Cody: Looks like the Thecus handles five drives (something I wish the Infrant did). Infrant is widely distributed and Thecus doesn't appear to be yet. Beyond that, don't have much info.

February 08, 2007

TechDigs.net said:

Christopher: As a Linux command-line guy, rsync is perfect for you. Chronosync doesn't by a good command-liner much. S3 would be a good solution for you as would a service like Dreamhost (also supports Webdav).

February 08, 2007

TechDigs.net said:

Bernard: If your financial software is already encrypting the file locally, then you're fine and you have nothing to think about. Regardless, if you use JungleDisk and S3, there's an option in the software to assure that you encrypt everything you send to S3. Just be sure it's turned on.

February 08, 2007

Kevin Plumb said:

Great article, I'm going to implement it very soon to deal with my ever growing backup needs. I don't understand the part about Chronosync to default to synchronize files in place. Can someone elaborate as to what this achieves? Thanks

February 09, 2007

TechDigs.net said:

Kevin: Thanks. Synchronize files in place comes from an option in unix utility called rsync (which is Chronosync's functionality is based on). Normally when rysnc makes a copy on the remote drive (if if it's just updating it) it makes a copy working copy not using the final name, then renames the file it to the final file name. The reason it does this is in case stop mid-sync the old file is still there in tact until the new one arrives. Most llocal file systems don't have that trouble because renames happen nearly instantly. This doesn't work with services like Amazon S3 and other webdav services. The reason is the act of renaming takes as long as the first copy (essentially having to re-upload the file the second time), so you'll completely double your effort. It'll send the file, then re-send it when it renames it. So, if do the copy/sync in-place, rysnc only sends it once using the final file name. Chronsync is basically the same, but you have to manually change the 'SkipSafeCopy' flag in the preferences file.

February 09, 2007

Isaac said:

Thanks for the great article. Two questions for the average home use who doesn't have tetrabytes, just looking for a reasonable offsite backup solution: 1. It is clear how ChronoSync pushes only the incremental changes to the Local Drive created by JungleDisk. But when Jungle disk then turns around to send the files from its cache, over the net, to Amazon S3, is it incremental? i.e. won't it still send the whole file, as opposed to just the changed bytes, because it doesn't DO incremental? 2. What would be the benefits or drawbacks of using Interarchy (or any other client that supports Amazon s3) in the place of JungleDisk? You can make