Recommend a storage solution... - DPChallenge Forums

Nikon AF-S NIKKOR 24-70mm f/2.8G ED

09/24/2007 09:01:46 PM · #1

I've had 3 hard drives go out on me in the last 6 months. It's enough to drive me crazy! I finally decided that the way to go, for reliability, was a RAID 5 system. Now ... I'm not so sure!

I purchased a Buffalo TeraStation Live 1TB system last week and I'm just about ready to send it back. Performance on this thing absolutely sucks!

I shoot weddings using multiple cameras. So the first thing I do, after copying the files onto the hard drive, is sort the files in chronological order and then renumber them starting with IMG_0001. This intermixes each of the camera's files so that I have a logical sequence of images. This process is normally fairly quick. As an example, I shot 1700 images at Friday's wedding. A normal rename of 1700 files would have taken about a minute to finish. The TeraStation took about 10 minutes to perform the rename and stopped twice with errors! (10 minutes was the length of time on the successful run!)

After renaming the files, my next step is to load them into Lightroom. Now this step can potentially take awhile. On a wedding with 1700 images it may take 15-20 minutes to load them all. (it runs faster if I have recently deleted the lightroom database, slower if I have a few weddings already in the database)

Well... after about 30 minutes, Lightroom had finished loading 14% of Friday's wedding. So I left the thing running all night. I got up the next morning at 4am and it *still* had not finished loading all of the image.

That is simply unacceptable!

So my questions are three-fold:

1) If you have the Buffalo TeraStation, are you happy with it? Did you have to do anything to speed it up? Add some kind of caching system? Anything?

2) Do you have ANY experience with a RAID 5 system that was NOT slow? I'm concerned now that RAID 5 simply will not give me the performance I need. I'm open to hear stories to the contrary. What do you have? What has been your experience? What do you recommend?

3) If RAID 5 isn't the answer ...... what is?!?

--------------------

Background info:
I've confirmed that the Terastation, the switch, and my laptop are all at 1Gbps speeds and that no errors are occurring. I've played with different ethernet frame sizes. I've played with many different parameters. Nothing appears to have much of an effect on the performance of this system.

ben4345

Nikon D700

09/24/2007 09:28:21 PM · #2

As my long experience troubleshooting, upgrading, optimizing, and etc... PC and mac systems.. I have always ended up in the conclusion "Simple is Safe" No RAID, no storage station.. just internal drives for a desktop and externals for notebooks.

What I suggest for most clients of mine, that are in a similar deleama. Assuming you are using a desktop with some free 3.5" bays.
Is to do something like this:

A fast for your primary drive *used for your current work flow* (i.e. Western Digital Raptor 150GB recommended)
followed by multiple 500GB or larger hard drives (secondary)*storage for previous work flows, and archiving

wavelength

Nikon AF Nikkor 50mm f/1.8D

09/24/2007 09:32:19 PM · #3

Are you using the Terastation as a pure NAS solution?

If it has a USB2.0 connetor, that would be faster. A LOT faster.

wavelength

Nikon AF Nikkor 50mm f/1.8D

09/24/2007 09:36:00 PM · #4

Oh, being a NAS system the RAID5 has nothing to do with the speed. You're purely limited here by the system-system connection (the Terastation is running Linux, I believe)

If you installed RAID5 into your home system INSIDE the box, you'd not be able to tell a difference, except that when you lose a hard drive you don't lose all your data. Actually, with hard drives being so cheap, I'd do 1-1 Mirroring instead.

09/24/2007 09:37:07 PM · #5

Originally posted by ben4345:

A fast for your primary drive *used for your current work flow* (i.e. Western Digital Raptor 150GB recommended)
followed by multiple 500GB or larger hard drives (secondary)*storage for previous work flows, and archiving

That's similar to what I was doing before the desktop died. I replaced it with a laptop. So it's external all the way now.

Basically, I've kept doing something similar to what I was doing with the desktop. I've had a fast Firewire attached drive as my main drive, and a 2nd drive which I manually mirrored all files to whenever new files were put onto the main drive. So these two drives are always mirrors of each other. In some ways, it was "luck" that my latest drive to go out was my main drive ... BECAUSE it was backed up in just this manner. Any other drive and I would have been hurting.

Then... as my main drive fills up, I move inactive weddings off to other drives. These drives are turned on long enough to copy the files, and then turned back off. So they do NOT remain spinning any longer than necessary. I have *many* such drives!

I was hoping to replace my main+mirror pair with a raid system. Now I'm not so sure it's any better than my manual approach.

09/24/2007 09:37:39 PM · #6

I think RAID 5 is pretty good. Provided you set it up inside your PC not in external unit. You can also go crazy and setup raid 10.

Another thing you can do is get 2 hard drives or 2 raid 0 and then use syncronization software to sync them together. I do that for backup. I use SyncToy from Microsoft.

Hope it helps.

Nick

09/24/2007 09:38:09 PM · #7

Originally posted by wavelength:

Are you using the Terastation as a pure NAS solution?

If it has a USB2.0 connetor, that would be faster. A LOT faster.

Yes, it's a pure NAS solution. USB 2.0 is faster than gigabit ethernet?

09/24/2007 09:40:26 PM · #8

Originally posted by wavelength:

If you installed RAID5 into your home system INSIDE the box, you'd not be able to tell a difference, except that when you lose a hard drive you don't lose all your data.

I have heard (no experience except for this Terastation, but I have heard) that Raid 5 is marginally slower on reads, and noticeably slower on writes due to striping the data across multiple drives.

I've also heard that more drives is faster, but also increases the likelyhood of failure. This system has 4 250Gb drives in it.

violinist123

Canon EOS-6D

09/24/2007 09:41:54 PM · #9

Probably you have network issues causing the performance to crap out. At gigabit speeds, you can rack up errors fast and eachone requires the data to be sent across again.

Assuming nothing is wrong with the box, they didn't load some piece of crap controller with 2MB of ram on it, etc, etc. There's a lot to it.

On another note, constant powering on/off will send your drives to heaven quicker than leaving them on. My workstation drives go tits up every few years. My server drives have been spinning for 4 years straight, a few extended power outages not withstanding. Scsi drives, so literally they are -always- spinning.

Anyway, NAS is one piece of the system. Lots of things could be causing your headache.

edit: what you post about raid 5 read/write speeds is true, however they are talking about differences which are not really noticeble by us humans.

Message edited by author 2007-09-24 21:42:47.

09/24/2007 09:42:52 PM · #10

Originally posted by Nikolai1024:

I think RAID 5 is pretty good. Provided you set it up inside your PC not in external unit. You can also go crazy and setup raid 10.

Not sure how that'll work with a laptop.... :(

Originally posted by Nikolai1024:

Another thing you can do is get 2 hard drives or 2 raid 0 and then use syncronization software to sync them together. I do that for backup. I use SyncToy from Microsoft.

Yeah, that sounds kinda like what I was doing before. Manually syncing two external drives with software designed for that purpose. I was hoping the raid 5 solution would get me out of the manual sync process... but it's not going to work if performance is this bad.

09/24/2007 09:43:51 PM · #11

Originally posted by dwterry:

Originally posted by wavelength:
If you installed RAID5 into your home system INSIDE the box, you'd not be able to tell a difference, except that when you lose a hard drive you don't lose all your data.

I have heard (no experience except for this Terastation, but I have heard) that Raid 5 is marginally slower on reads, and noticeably slower on writes due to striping the data across multiple drives.

I've also heard that more drives is faster, but also increases the likelyhood of failure. This system has 4 250Gb drives in it.

It depends on RAID controller. Better controller handles requests faster but costs more.

09/24/2007 09:45:15 PM · #12

Originally posted by routerguy666:

Probably you have network issues causing the performance to crap out. At gigabit speeds, you can rack up errors fast and eachone requires the data to be sent across again.

See, I thought about that too. But both the interface on the computer and the one on the NAS are reporting 0 errors. I'm assuming if it had to do a retransmission, it would count it as an error....

09/24/2007 09:46:08 PM · #13

Can you plug it in with USB2?

09/24/2007 09:47:58 PM · #14

Originally posted by Nikolai1024:

Can you plug it in with USB2?

Nope... it has two USB ports on the back, but they are for adding external drives to the NAS system. So I could augment the storage being offered by the Terastation by adding external drives. They would not participate in the raid 5 array in anyway, they would simply be made available through the same "network share" system that the NAS is using to share the array.

09/24/2007 09:50:33 PM · #15

Then you have to connect to the device directly without a switch. Do you have network cross over cable?

I'll guess that you don't. In that case you should buy one or make one. You can redo old cable to be cross over. Let me know and I'll look for a diagram. It is pretty easy actually.

violinist123

Canon EOS-6D

09/24/2007 09:51:07 PM · #16

Originally posted by dwterry:

Originally posted by routerguy666:
Probably you have network issues causing the performance to crap out. At gigabit speeds, you can rack up errors fast and eachone requires the data to be sent across again.

See, I thought about that too. But both the interface on the computer and the one on the NAS are reporting 0 errors. I'm assuming if it had to do a retransmission, it would count it as an error....

I've only ever had two issues with nas appliances. One is network errors. Second is controller issues. I would call them up if you think the network is clean.

09/24/2007 09:52:32 PM · #17

Originally posted by Nikolai1024:

Then you have to connect to the device directly without a switch. Do you have network cross over cable?

I don't... will it really be any faster than not using the switch? I'd like to try it ... but I'm not so good with playing with wiring (no tools or skills to go about taking about my cable and putting it back together again .. at least, not if I want it to still work).

09/24/2007 09:55:54 PM · #18

Originally posted by dwterry:

Originally posted by Nikolai1024:
Then you have to connect to the device directly without a switch. Do you have network cross over cable?

I don't... will it really be any faster than not using the switch? I'd like to try it ... but I'm not so good with playing with wiring (no tools or skills to go about taking about my cable and putting it back together again .. at least, not if I want it to still work).

Test it and see. I just found out you should have cross over swtich on the back. Do you see it?

violinist123

Canon EOS-6D

09/24/2007 09:56:44 PM · #19

It will eliminate your switch as a source of errors. Pick up the xover cable at the store. Making one will frustrate you the first dozen (or hundred) times, and takes a crimper and an rj45 end to do.

09/24/2007 09:58:53 PM · #20

Originally posted by Nikolai1024:

Originally posted by dwterry:

Originally posted by Nikolai1024:
Then you have to connect to the device directly without a switch. Do you have network cross over cable?

I don't... will it really be any faster than not using the switch? I'd like to try it ... but I'm not so good with playing with wiring (no tools or skills to go about taking about my cable and putting it back together again .. at least, not if I want it to still work).

Test it and see. I just found out you should have cross over swtich on the back. Do you see it?

Wait a minute... wrong model... you may not find it there. Anyway I think you should get crossover cable and give it a shot.

09/24/2007 09:59:55 PM · #21

Originally posted by routerguy666:

...
Making one will frustrate you the first dozen (or hundred) times
...

true that...

09/24/2007 10:07:02 PM · #22

Originally posted by Nikolai1024:

Test it and see. I just found out you should have cross over swtich on the back. Do you see it?

Well now.... this is interesting.

There is no switch on the back, instead it auto-senses. I have plugged it directly into my laptop (I had to hard code an IP address) and it's up and running. At first blush ... it appears to be faster. I'm making a copy of Friday's wedding right now, just to give it a "real world test".

This is going to take awhile, it has copied about 100 images out of 1700 so far.