DPChallenge: A Digital Photography Contest You are not logged in. (log in or register
 

DPChallenge Forums >> Hardware and Software >> Recommend a storage solution...
Pages:  
Showing posts 26 - 50 of 63, (reverse)
AuthorThread
09/26/2007 05:27:47 AM · #26
My suggestion, and its worked for me for 8 or so years. I have lost only 2 drives in that time and both to power problem Large case, ALOT of cooling, and turn the thing off when your not using it. I replace towers not drives, and I run 850gig on one machine, and 2tb on the video machine.
For me keeping things cool seemed to help, or i am just lucky.
09/26/2007 06:39:34 AM · #27
Originally posted by paddles:

Forgive me if this is a silly question, but is your operating system or software using part of the RAID drive cluster as a swap file or for disk caching?


No question is silly. :-)

The laptop's internal drive is used for everything except storage of pictures.

Originally posted by paddles:

If you were renaming a whole lot of files at once, this would count as a sequence of very small writes.


It makes sense. But doesn't help the situation. :( Still debating whether to send the thing back. Right now I think my only choice is to process everything locally on drive C: and then move the files out to the RAID drive for storage. That seems awfully dumb.... but at least it is 10 times faster.

Originally posted by paddles:

A RAID 1+0 (aka RAID 10) configuration might be more suitable? Or, as you said earlier, perhaps a non-RAID configuration.


I don't think it does RAID 10. And the whole point to going to a RAID configuration was to hopefully gain some reliability. I just didn't realize the performance cost. :(

09/26/2007 06:40:27 AM · #28
Originally posted by VisiBlanco:

Check out //www.drobo.com/


I was considering that one before I purchased the Buffalo product. I really thought this was the better choice. Now I'm not so sure any RAID system is going to offer the performance I need....

09/26/2007 06:43:03 AM · #29
Originally posted by liltritter:

My suggestion, and its worked for me for 8 or so years. I have lost only 2 drives in that time and both to power problem Large case, ALOT of cooling, and turn the thing off when your not using it. I replace towers not drives, and I run 850gig on one machine, and 2tb on the video machine.
For me keeping things cool seemed to help, or i am just lucky.


In all my years of computing, these are the first three drives I've ever lost and they all happened within the last 6 months. I don't know if it's simply statistics (I had 7 drives online at once ... and many of these have been in use for several years) or if there is some "bigger problem" at work. Everything is run off of a UPS so the power should be clean. I'm downstairs in a cool room, so it's not heat. I just don't know what it could be.

09/26/2007 07:38:08 AM · #30
It wouldn't surprise me if the system is using it's cpu to carry out the necessary parity xor calculations (I can't find any information from their website, or any reviews). This is what happens when you set up a motherboard to use raid 5.

Ideally it would use a dedicated xor chip to do all the calculations as it is much more efficient, but that adds cost
09/26/2007 07:42:16 AM · #31
Originally posted by relent:

It wouldn't surprise me if the system is using it's cpu to carry out the necessary parity xor calculations (I can't find any information from their website, or any reviews). This is what happens when you set up a motherboard to use raid 5.

Ideally it would use a dedicated xor chip to do all the calculations as it is much more efficient, but that adds cost


It's a NAS (network attached storage). The laptop sees it as a network drive ... so there is no local processing done on the raid, all of that is internal to the raid box itself.


09/26/2007 08:11:27 AM · #32
I have a similar setup (different NAS box) and it with LR is far too slow (it was okay with RSP but much slower then a local drive). I have it running with a gigabit card & switch, so it's more then the network.

I also picked RAID5 but I think what is happening is that it's software rather then hardware RAID5 so it's VERY slow writing. Reading is much better BUT still slow.

RAID10 would likely be faster but takes more space to run. I like the R5 setup but am changing the way I use it.... I am changing my drives around on the PC (will add another 400Gb) and plan to use it as the active work drive and push stuff to the R5 box over time.... I have to figure out how LR or it's replacement will deal with that, so no answer yet.
09/26/2007 10:08:54 AM · #33
Originally posted by dwterry:

Originally posted by relent:

It wouldn't surprise me if the system is using it's cpu to carry out the necessary parity xor calculations (I can't find any information from their website, or any reviews). This is what happens when you set up a motherboard to use raid 5.

Ideally it would use a dedicated xor chip to do all the calculations as it is much more efficient, but that adds cost


It's a NAS (network attached storage). The laptop sees it as a network drive ... so there is no local processing done on the raid, all of that is internal to the raid box itself.


The NAS is effectively a computer in its own right - the cpu of that may slow it down
09/26/2007 10:16:32 AM · #34
Originally posted by relent:

The NAS is effectively a computer in its own right - the cpu of that may slow it down


I suppose ....... but that does mean it's a "dedicated CPU" as it has no other tasks to process.

09/26/2007 10:17:18 AM · #35
Originally posted by robs:

I have a similar setup (different NAS box) and it with LR is far too slow (it was okay with RSP but much slower then a local drive). I have it running with a gigabit card & switch, so it's more then the network


Thank you. That tells me a LOT right there! It's definitely time to look for a different solution.

09/26/2007 10:31:31 AM · #36
Originally posted by dwterry:

Originally posted by robs:

I have a similar setup (different NAS box) and it with LR is far too slow (it was okay with RSP but much slower then a local drive). I have it running with a gigabit card & switch, so it's more then the network


Thank you. That tells me a LOT right there! It's definitely time to look for a different solution.


I suggest a PC with internal drives on SATA.
09/27/2007 11:37:27 AM · #37
you can get Firewire Caddies with RAID capabilities - much faster than a Gigabit network will ever give you- hopefully your laptop has built in firewire?

RAID 5
//www.firewiredirect.com/product/19/

RAID 1
//www.synetic.net/Synetic-Products/SyneRAID-Units/SyneRAID-XFW-XSC.htm
//sewelldirect.com/araid-external-firewire-raid-1.asp
//www.meritline.com/ams-venus-ds3r-3-5-firewire-enclosure.html
//www.cwol.com/firewire-drives/caldigit-firewirevr.htm
09/27/2007 11:39:09 AM · #38
Firewire is 800Mb/s, gig-e is 1000Mb/s.
09/27/2007 05:24:22 PM · #39
Originally posted by routerguy666:

Firewire is 800Mb/s, gig-e is 1000Mb/s.

Ethernet has a lot of overhead. You will never see 1000Mbit/s data transmission... in fact the most you can get with a very good router, proper network cards (pure hardware) and cables is halt of that (mem-to-mem copy). From there start subtracting... IP headers, protocol overhead, message size and a lot of I/O. Divided by 8 (bits) you have about 25-40 MByte/s max. Firewire and USB don't have this overhead... so you get about the same as 400Mbit/s USB :)

OP posted about a test... 24 Mbytes/s for read is normal.. but <1MB/s for write is absurd. Did you create the raid-5 or it was pre-formated? I ask this because if you formatted the drive than it will take some time until the process completes in the background. It may take up to a day, or if you are using it extensively for writes it may take more. Leave the box on for a day without writing anything to it than try it again. Also, if you have a second computer try creating a share from one to the other and see the throughput it gets using the same switch and the same cables.

09/27/2007 06:05:28 PM · #40
on a 1Gbit link, you'll be lucky to see anything faster than about 30Mb/sec. 10Mb/sec is more common, due to the various overheads in a system. You can tweak it up to about 70Mb/sec but not for most realistic applications.

A good SATA drive in your machine will run about 300Mb/sec. So expect a NAS to be about 10-30x slower than a local hard drive.

I use a terabyte NAS on a 1Gbit link, but I cache/ duplicate any images I'm actually working on, on a local drive. Basically the NAS is a large back-up system, I don't work from it, unless I'm pulling something out of the 'archives'

Message edited by author 2007-09-27 18:05:55.
09/27/2007 07:07:44 PM · #41
guys - he has a laptop!

Originally posted by dwterry:

That's similar to what I was doing before the desktop died. I replaced it with a laptop. So it's external all the way now.


why can't u guys read these threads fully before posting about SATA drives etc!?

Message edited by author 2007-09-27 19:08:55.
09/27/2007 08:44:24 PM · #42
For more information about these NAS devices, check out this site:
//www.smallnetbuilder.com/content/view/85/93/

They have reviews and benchmarks for pretty much every "small network NAS" on the market. What you will find in the reviews is that there is often little or no difference between running on 100mbps vs. "Gigabit" ethernet for these devices. The reason is mainly the processor and bus on these boxes can not keep up with the potential throughput of 100mbps, let alone gigabit connections. (with the exception of a very few high end models) In layman's terms...these small NAS devices are SLOW (especially writing if using software RAID 5) and will never compare to the speed of even a single internal drive, no matter how fast your network is. (unless you get into datacenter level SAN or fiber channel equipment)

These devices are perfect if you are backing up data, streaming moderate bandwidth multi-media (read), or occasionally accessing an archived file. But they are not good for IO intensive real-time tasks. (like you are trying to do)

If you are on a workstation, 2 750GB drives in RAID 1 configuration, installed internally will give you the same storage space, MUCH faster performance, and possibly better data security (I've experienced corruption and recovery problems with Raid 5 arrays on cheep software based controllers). If you have to use the laptop, the external firewire raid caddies suggested by Bobster would probably be your best solution if you don't want to risk using your internal drive. (some very high-end "workstation laptops" have dual drives that can be setup in raid 0 or 1 configurations internally)

If you hang onto that raid box (can't return it), use it as a backup and archive device. This could allow you to get away with smaller, faster drives on your Desktop or Laptop to do your work on, while having a secure place to store your photos while you are not working on them.

Hope that was helpful.

sh0rty :P

Message edited by author 2007-09-27 20:50:12.
09/27/2007 08:54:44 PM · #43
Originally posted by nikolaos:

Originally posted by routerguy666:

Firewire is 800Mb/s, gig-e is 1000Mb/s.

Ethernet has a lot of overhead. You will never see 1000Mbit/s data transmission... in fact the most you can get with a very good router, proper network cards (pure hardware) and cables is halt of that (mem-to-mem copy). From there start subtracting... IP headers, protocol overhead, message size and a lot of I/O. Divided by 8 (bits) you have about 25-40 MByte/s max. Firewire and USB don't have this overhead... so you get about the same as 400Mbit/s USB :)


Wow... good info! I'm amazed this kind of info isn't more common knowledge (maybe I just don't run in the right circles?).

Originally posted by nikolaos:

OP posted about a test... 24 Mbytes/s for read is normal.. but <1MB/s for write is absurd. Did you create the raid-5 or it was pre-formated? I ask this because if you formatted the drive than it will take some time until the process completes in the background. It may take up to a day, or if you are using it extensively for writes it may take more. Leave the box on for a day without writing anything to it than try it again. Also, if you have a second computer try creating a share from one to the other and see the throughput it gets using the same switch and the same cables.


It came pre-formatted. I did get an increase in speed when I directly connected the NAS to the laptop (bypassing the switch). Jumping from 24mb/s to 33mb/s for reading. Write times barely nudged up to 0.9mb/s.

09/27/2007 08:56:36 PM · #44
Thanks sh0rty, lots of good info there! I really appreciate the help.
09/27/2007 09:22:42 PM · #45
Originally posted by sh0rty:

If you hang onto that raid box (can't return it), use it as a backup and archive device. This could allow you to get away with smaller, faster drives on your Desktop or Laptop to do your work on, while having a secure place to store your photos while you are not working on them.

That's the key to NAS. Work locally and archive/backup across the network. If you're working remotely, it's no surprise. This is one reason I hate how Lightroom works. It makes image management across multiple disks too difficult.
09/27/2007 09:45:24 PM · #46
Originally posted by _eug:

This is one reason I hate how Lightroom works. It makes image management across multiple disks too difficult.


I have LR always create the .XMP files as it imports and keep them up to date as I update the images. That way, I can backup my images AND their changes together at once (something you can't do if the changes are stored in the flippin' database!). This also means I can blow away the database at a moments notice and not lose anything. And it also means I can move the images to any drive and be up and running with LR with minimal effort.

I wish I could just turn the database off. It does absolutely nothing for me except slow LR down.

09/27/2007 09:59:13 PM · #47
Originally posted by _eug:

This is one reason I hate how Lightroom works. It makes image management across multiple disks too difficult.


The funny thing is, that's the one thing it is designed to do and does really well, compared to say 'bridge' Particularly if those are removable or networked drives.

Message edited by author 2007-09-27 21:59:30.
09/28/2007 01:26:22 PM · #48
Windows Home Server may be a solution, if you can wait. See //www.microsoft.com/windows/products/winfamily/windowshomeserver/default.mspx

Windows SBS (Small Business Server) is another solution, but you will need to invest time to manage the server.

SBS allows you to have software RAID, or add the RAID controller of your choice.

You can run an anti-virus on the server, in which case you don't need to run it on your PC for the networked drives--the result is faster execution when reading files from networked drives.

You can buy the level of processor you need, and add RAM to the server, and otherwise upgrade just as you'd upgrade a PC.
09/28/2007 02:20:24 PM · #49
I have to second the Drobo. I had all your same problems and switched to it about a year ago and simply love it. It's fast and recognizes it's own problems. I wouldn't go back unless I went to a full server setup.
09/28/2007 02:38:33 PM · #50
Just saw this today: //www.engadget.com/2007/09/28/fusion-ios-iodrive-puts-power-of-a-san-on-a-pcie-card/

If money is no object, and ultimate speed is the goal, they claim "nearly a thousand times faster than any existing disk drive". It is solid state too (no moving parts), so hardware failure is unlikly. ;-)

sh0rty :P
Pages:  
Current Server Time: 03/28/2024 08:23:08 AM

Please log in or register to post to the forums.


Home - Challenges - Community - League - Photos - Cameras - Lenses - Learn - Prints! - Help - Terms of Use - Privacy - Top ^
DPChallenge, and website content and design, Copyright © 2001-2024 Challenging Technologies, LLC.
All digital photo copyrights belong to the photographers and may not be used without permission.
Current Server Time: 03/28/2024 08:23:08 AM EDT.