| Author | Thread |
|
|
11/25/2007 01:54:29 AM · #1 |
| Hey thanx David..... Might have to have another look at sidecars :-/ btw.... I noticed some speed improvements in 1.3 but it might just be something else on the machine and it was minor. |
|
|
|
11/23/2007 03:18:57 PM · #2 |
Originally posted by robs:
Could you walt thru the steps you take to move the file to the NAS when they are not been worked on? I know you said you kick out the sidecar filoes, so do you just copy both to the NAS box? Do you bother have it update the LR catalogue so it nows where they are now?
I am running from the LR-DB alone as the performance with sidecar files was just pathetic (and that convert from RSP was never really a good option - too little, too late, so never needed to play with the sidecars in LR). |
I delete the LR database fairly frequently ... I haven't had LR 1.3 long enough to know if the performance has changed. But deleting the database was the only way to keep the performance up to acceptable speeds.
And the main reason I use the sidecar files is because it allows all of my settings for each file to be saved "together" with the raw files, no matter where I store the raw files. (ultimately, everything gets moved off to an external drive and turned off)
I use Super Flexible File Synchronizer to move my files for me. It helps guarantee that any changes I make on the internal drive get mirrored out to the NAS.
So... for example, this is how I handle a wedding:
1) Copy all files to the wedding folder for the couple (on the internal drive)
2) Use ACDSee to view the files in modified date order. Then, while sorted thusly, I rename the files starting at IMG_0001. Note that I synchronize the clocks on my cameras. This way I can keep multiple cameras running and later bring the images together in sequence simply by renumbering them.
3) This is now the first point at which I synchronize the internal drive to the NAS.
4) Then I import the files into Lightroom and have it generate the sidecar and preview files at the same time. This could take a little while. I'll find something else to do while it's busy.
Note that it's applying a "preset" as it's importing. Many of my images are just about the way I like them right at this point, with only minor adjustments necessary.
5) Now I begin using LR to sort through the keepers and apply adjustments (wb, exposure, cropping).
6) When done, I export the files to jpegs. And synchronize again. This time it synchronizes any sidecars that have changed as well as the newly generated jpegs.
7) And finally, I begin uploading the jpegs to smugmug so that I have an external backup.
As far as LR's catalog goes .. I simply don't use it. All of my settings, including ratings, color labels, adjustments, etc, are stored in the sidecars, so I have no qualms about deleting the library.
|
|
|
|
11/23/2007 02:59:23 PM · #3 |
Originally posted by dwterry:
FWIW, that's *exactly* what I am doing. Shoots that are actively being worked on are stored on the internal drive, but frequently copied/mirrored out to the NAS. Later, I'll do individual edits directly on the NAS and find the performance to be acceptable there. So it's only the "bulk" operations that I'm doing on the internal drive. |
Could you walt thru the steps you take to move the file to the NAS when they are not been worked on? I know you said you kick out the sidecar filoes, so do you just copy both to the NAS box? Do you bother have it update the LR catalogue so it nows where they are now?
I am running from the LR-DB alone as the performance with sidecar files was just pathetic (and that convert from RSP was never really a good option - too little, too late, so never needed to play with the sidecars in LR).
Message edited by author 2007-11-23 15:00:28. |
|
|
|
11/23/2007 02:48:16 PM · #4 |
Originally posted by sh0rty:
My recomendation remains; work with files localy, backup and archive on the NAS device. |
FWIW, that's *exactly* what I am doing. Shoots that are actively being worked on are stored on the internal drive, but frequently copied/mirrored out to the NAS. Later, I'll do individual edits directly on the NAS and find the performance to be acceptable there. So it's only the "bulk" operations that I'm doing on the internal drive.
|
|
|
|
11/23/2007 01:48:32 PM · #5 |
Originally posted by nikolaos:
With 1000BASE-T (copper cables) the max you can get _is_ 400mbit/s without protocol overhead. I _have_ tested this.
|
When I read this review, it reminded me of this old thread:
//www.smallnetbuilder.com/content/view/30195/51/1/1/
SmallNetBuilder.com did a review of cheep (~$100) gigabit switches that are capable of "jumbo frames", and found that many where capable of near line-speed performance (with 2 computers attached)...averaging over 700mbps! (>500mbps even without "jumbo frames")
With performance capabilities like this on cheep gigabit networking equipment, I don't think the network is the slow point for the original poster. I maintain it is the performance of the NAS box.
My recomendation remains; work with files localy, backup and archive on the NAS device.
sh0rty :P
|
|
|
|
10/06/2007 07:45:24 PM · #6 |
I know old thread but there was no point in making a new one.
After some further thinking about it, and reading up on things again, I think you should have gone with one of the newer eSAAT drives intstead.
The DriveStation Quattro HD for example.
Not sure that really helps at this point :/
Or, you could get out the Big Guns
Message edited by author 2007-10-06 19:48:21.
|
|
|
|
09/29/2007 04:17:09 AM · #7 |
Originally posted by routerguy666:
Originally posted by dwterry:
Originally posted by nikolaos:
Originally posted by routerguy666:
Firewire is 800Mb/s, gig-e is 1000Mb/s. |
Ethernet has a lot of overhead. You will never see 1000Mbit/s data transmission... in fact the most you can get with a very good router, proper network cards (pure hardware) and cables is halt of that (mem-to-mem copy). From there start subtracting... IP headers, protocol overhead, message size and a lot of I/O. Divided by 8 (bits) you have about 25-40 MByte/s max. Firewire and USB don't have this overhead... so you get about the same as 400Mbit/s USB :) |
Wow... good info! I'm amazed this kind of info isn't more common knowledge (maybe I just don't run in the right circles?). |
It's not common knowledge because it's wrong and reflects an understanding of ethernet technology circa 1994 when you were at the mercy of your collision inducing hub. |
Hmm... I've been a little behind in technology? I was referring to TCP/IP, protocol and ethernet overhead not collisions. I'd like to see a 1000mb/s _data_ transfer from a GE.
With 1000BASE-T (copper cables) the max you can get _is_ 400mbit/s without protocol overhead. I _have_ tested this.
Now, if you get 1000BASE-SX (fiber) which costs 500-700$ a card and a few thousand dollars Fiber Switch you may get close to the theoretical limit, but I haven't tested that.
There are a lot of technical reasons why a 1000mb/s connection transfers data at 10MByte/s but it still is a 1000mb/s connection for the marketing department.
Bottom line is that if a GE transfers at 10MB/s I'll live with it... at 20MB/s I'll be happy.. and over it I'll be broke.
-N.
|
|
|
|
09/28/2007 06:19:49 PM · #8 |
Originally posted by sh0rty:
I just re-read the original post...you say 3 drives fail in 6 months!! Where they all in the same USB enclosure? If so, I'd throw out that enclosure, there is probably something wrong with it (ie. bad power supply causing spikes), as it is not normal for 3 drives to fail that fast in even the worst conditions. :-(
sh0rty :P |
No... nothing that simple to diagnose. One was an internal drive, and the other two were in enclosures (different brand enclosures, and, if I remember right, different brand drives, but definitely different models and sizes).
|
|
|
|
09/28/2007 04:28:07 PM · #9 |
Originally posted by dwterry:
Originally posted by nikolaos:
Originally posted by routerguy666:
Firewire is 800Mb/s, gig-e is 1000Mb/s. |
Ethernet has a lot of overhead. You will never see 1000Mbit/s data transmission... in fact the most you can get with a very good router, proper network cards (pure hardware) and cables is halt of that (mem-to-mem copy). From there start subtracting... IP headers, protocol overhead, message size and a lot of I/O. Divided by 8 (bits) you have about 25-40 MByte/s max. Firewire and USB don't have this overhead... so you get about the same as 400Mbit/s USB :) |
Wow... good info! I'm amazed this kind of info isn't more common knowledge (maybe I just don't run in the right circles?). |
It's not common knowledge because it's wrong and reflects an understanding of ethernet technology circa 1994 when you were at the mercy of your collision inducing hub.
|
|
|
|
09/28/2007 03:55:35 PM · #10 |
I just re-read the original post...you say 3 drives fail in 6 months!! Where they all in the same USB enclosure? If so, I'd throw out that enclosure, there is probably something wrong with it (ie. bad power supply causing spikes), as it is not normal for 3 drives to fail that fast in even the worst conditions. :-(
sh0rty :P
Message edited by author 2007-09-28 16:03:38. |
|
|
|
09/28/2007 03:17:54 PM · #11 |
Originally posted by option:
what kind of hard drives are you using that fail so often?
I use seagate SATA drives (with esata/usb 2.0 enclosures)and have never had one fail on me... |
Sorry. The latest was a Seagate in a USB 2.0 enclosure. :-)
No manufacturer is immune....
|
|
|
|
09/28/2007 03:17:43 PM · #12 |
Originally posted by sh0rty:
If money is no object...
|
I REALLY meant that part...FusionIO's "target retail price" is $30/gb...that is $19,200 for the 620Gb device!!!
//www.fusionio.com/faq.html
I guess 1000x the performance for 100x the price (compared to similar sized SATA disk drive) is a relative bargain! ;-)
sh0rty :P
Message edited by author 2007-09-28 15:20:32. |
|
|
|
09/28/2007 02:53:28 PM · #13 |
what kind of hard drives are you using that fail so often?
I use seagate SATA drives (with esata/usb 2.0 enclosures)and have never had one fail on me... |
|
|
|
09/28/2007 02:38:33 PM · #14 |
Just saw this today: //www.engadget.com/2007/09/28/fusion-ios-iodrive-puts-power-of-a-san-on-a-pcie-card/
If money is no object, and ultimate speed is the goal, they claim "nearly a thousand times faster than any existing disk drive". It is solid state too (no moving parts), so hardware failure is unlikly. ;-)
sh0rty :P |
|
|
|
09/28/2007 02:20:24 PM · #15 |
I have to second the Drobo. I had all your same problems and switched to it about a year ago and simply love it. It's fast and recognizes it's own problems. I wouldn't go back unless I went to a full server setup.
|
|
|
|
09/28/2007 01:26:22 PM · #16 |
Windows Home Server may be a solution, if you can wait. See //www.microsoft.com/windows/products/winfamily/windowshomeserver/default.mspx
Windows SBS (Small Business Server) is another solution, but you will need to invest time to manage the server.
SBS allows you to have software RAID, or add the RAID controller of your choice.
You can run an anti-virus on the server, in which case you don't need to run it on your PC for the networked drives--the result is faster execution when reading files from networked drives.
You can buy the level of processor you need, and add RAM to the server, and otherwise upgrade just as you'd upgrade a PC.
|
|
|
|
09/27/2007 09:59:13 PM · #17 |
Originally posted by _eug:
This is one reason I hate how Lightroom works. It makes image management across multiple disks too difficult. |
The funny thing is, that's the one thing it is designed to do and does really well, compared to say 'bridge' Particularly if those are removable or networked drives.
Message edited by author 2007-09-27 21:59:30.
|
|
|
|
09/27/2007 09:45:24 PM · #18 |
Originally posted by _eug:
This is one reason I hate how Lightroom works. It makes image management across multiple disks too difficult. |
I have LR always create the .XMP files as it imports and keep them up to date as I update the images. That way, I can backup my images AND their changes together at once (something you can't do if the changes are stored in the flippin' database!). This also means I can blow away the database at a moments notice and not lose anything. And it also means I can move the images to any drive and be up and running with LR with minimal effort.
I wish I could just turn the database off. It does absolutely nothing for me except slow LR down.
|
|
|
|
09/27/2007 09:22:42 PM · #19 |
Originally posted by sh0rty:
If you hang onto that raid box (can't return it), use it as a backup and archive device. This could allow you to get away with smaller, faster drives on your Desktop or Laptop to do your work on, while having a secure place to store your photos while you are not working on them. |
That's the key to NAS. Work locally and archive/backup across the network. If you're working remotely, it's no surprise. This is one reason I hate how Lightroom works. It makes image management across multiple disks too difficult. |
|
|
|
09/27/2007 08:56:36 PM · #20 |
Thanks sh0rty, lots of good info there! I really appreciate the help.
|
|
|
|
09/27/2007 08:54:44 PM · #21 |
Originally posted by nikolaos:
Originally posted by routerguy666:
Firewire is 800Mb/s, gig-e is 1000Mb/s. |
Ethernet has a lot of overhead. You will never see 1000Mbit/s data transmission... in fact the most you can get with a very good router, proper network cards (pure hardware) and cables is halt of that (mem-to-mem copy). From there start subtracting... IP headers, protocol overhead, message size and a lot of I/O. Divided by 8 (bits) you have about 25-40 MByte/s max. Firewire and USB don't have this overhead... so you get about the same as 400Mbit/s USB :) |
Wow... good info! I'm amazed this kind of info isn't more common knowledge (maybe I just don't run in the right circles?).
Originally posted by nikolaos:
OP posted about a test... 24 Mbytes/s for read is normal.. but <1MB/s for write is absurd. Did you create the raid-5 or it was pre-formated? I ask this because if you formatted the drive than it will take some time until the process completes in the background. It may take up to a day, or if you are using it extensively for writes it may take more. Leave the box on for a day without writing anything to it than try it again. Also, if you have a second computer try creating a share from one to the other and see the throughput it gets using the same switch and the same cables. |
It came pre-formatted. I did get an increase in speed when I directly connected the NAS to the laptop (bypassing the switch). Jumping from 24mb/s to 33mb/s for reading. Write times barely nudged up to 0.9mb/s.
|
|
|
|
09/27/2007 08:44:24 PM · #22 |
For more information about these NAS devices, check out this site:
//www.smallnetbuilder.com/content/view/85/93/
They have reviews and benchmarks for pretty much every "small network NAS" on the market. What you will find in the reviews is that there is often little or no difference between running on 100mbps vs. "Gigabit" ethernet for these devices. The reason is mainly the processor and bus on these boxes can not keep up with the potential throughput of 100mbps, let alone gigabit connections. (with the exception of a very few high end models) In layman's terms...these small NAS devices are SLOW (especially writing if using software RAID 5) and will never compare to the speed of even a single internal drive, no matter how fast your network is. (unless you get into datacenter level SAN or fiber channel equipment)
These devices are perfect if you are backing up data, streaming moderate bandwidth multi-media (read), or occasionally accessing an archived file. But they are not good for IO intensive real-time tasks. (like you are trying to do)
If you are on a workstation, 2 750GB drives in RAID 1 configuration, installed internally will give you the same storage space, MUCH faster performance, and possibly better data security (I've experienced corruption and recovery problems with Raid 5 arrays on cheep software based controllers). If you have to use the laptop, the external firewire raid caddies suggested by Bobster would probably be your best solution if you don't want to risk using your internal drive. (some very high-end "workstation laptops" have dual drives that can be setup in raid 0 or 1 configurations internally)
If you hang onto that raid box (can't return it), use it as a backup and archive device. This could allow you to get away with smaller, faster drives on your Desktop or Laptop to do your work on, while having a secure place to store your photos while you are not working on them.
Hope that was helpful.
sh0rty :P
Message edited by author 2007-09-27 20:50:12. |
|
|
|
09/27/2007 07:07:44 PM · #23 |
guys - he has a laptop!
Originally posted by dwterry:
That's similar to what I was doing before the desktop died. I replaced it with a laptop. So it's external all the way now. |
why can't u guys read these threads fully before posting about SATA drives etc!?
Message edited by author 2007-09-27 19:08:55. |
|
|
|
09/27/2007 06:05:28 PM · #24 |
on a 1Gbit link, you'll be lucky to see anything faster than about 30Mb/sec. 10Mb/sec is more common, due to the various overheads in a system. You can tweak it up to about 70Mb/sec but not for most realistic applications.
A good SATA drive in your machine will run about 300Mb/sec. So expect a NAS to be about 10-30x slower than a local hard drive.
I use a terabyte NAS on a 1Gbit link, but I cache/ duplicate any images I'm actually working on, on a local drive. Basically the NAS is a large back-up system, I don't work from it, unless I'm pulling something out of the 'archives'
Message edited by author 2007-09-27 18:05:55.
|
|
|
|
09/27/2007 05:24:22 PM · #25 |
Originally posted by routerguy666:
Firewire is 800Mb/s, gig-e is 1000Mb/s. |
Ethernet has a lot of overhead. You will never see 1000Mbit/s data transmission... in fact the most you can get with a very good router, proper network cards (pure hardware) and cables is halt of that (mem-to-mem copy). From there start subtracting... IP headers, protocol overhead, message size and a lot of I/O. Divided by 8 (bits) you have about 25-40 MByte/s max. Firewire and USB don't have this overhead... so you get about the same as 400Mbit/s USB :)
OP posted about a test... 24 Mbytes/s for read is normal.. but <1MB/s for write is absurd. Did you create the raid-5 or it was pre-formated? I ask this because if you formatted the drive than it will take some time until the process completes in the background. It may take up to a day, or if you are using it extensively for writes it may take more. Leave the box on for a day without writing anything to it than try it again. Also, if you have a second computer try creating a share from one to the other and see the throughput it gets using the same switch and the same cables.
|
|
