|
| Author | Thread |
|
|
06/02/2008 05:34:36 PM · #1 |
Hi all -
When I first switched from my PC to a Mac, I imported all my (active) photos into iPhoto and started keywording. I have over 33,000 images on the MacBook and iPhoto doesn't handle it very well.
I have recently been using Lightroom for my RAW processing and as I am shooting more RAW images, they start in Lightroom and get key word organized there.
I am considering dumping iPhoto completely and using just Lightroom for my organizing/rating but I am concerned with bogging it down with a large catalog.
How big is your catalog? What is your workflow like?
Thanks in advance.
Dave
|
|
|
|
06/02/2008 06:10:12 PM · #2 |
10 1/2Gb and 35 1/2K files... Can you say s.l.o.w... I see others running larger cats with no issues, so I just don't know why it's a pig on this machine.
|
|
|
|
06/02/2008 06:18:05 PM · #3 |
I have one catalog as large as 15,000, but I group most of my catalogs by shoot (a specific wedding, a specific concert, etc), and I probably average around 1,500-2,000 per catalog.
robs, how are your catalogs and files laid out (local storage, external/network storage, etc)? Have you tried to purge your thumbnails and optimize your large catalogs? As a matter of course, I go through and purge my thumbnails on any galleries I don't use in a while, and always run validations and optimizations on catalogs (with a catalog backup) before and after large changes or imports. Are you using sidecar files? There was a serious performance bug with earlier versions of LR when using sidecar files on large catalogs.
|
|
|
|
06/02/2008 06:30:47 PM · #4 |
I gave up using sidecar files VERY fast (was used to that concept in RSP). I used to have them on a NAS and it was just pathetic (even though the NAS read is relatively fast as they go) so it's a second drive now... although images and LR cat are on the same drive (I don't see too must waiting on that drive but it's usage is not low). I do the reorgs when I think of it and that helps a little. I have 1.5-ish Gb memory, so it should be enough.... the holdup is somewhere else.. graphics card possibly - this machine is @4yrs old now.
What does purging the thumbs do? Would it not have to rebuild them next time you touch the image?
|
|
|
|
06/04/2008 12:15:00 AM · #5 |
Originally posted by robs:
What does purging the thumbs do? Would it not have to rebuild them next time you touch the image? |
It would indeed. LR creates a series of thumbnails (I guess "previews" in the correct LR term) for images, depending on how you view them. When you make changes, they (usually) trickle-down and previews get regenerated from the largest available "up to date" preview, etc.
While I don't have any direct metrics to point to, I believe from my own experience with large catalogs, that the preview cache becomes very cluttered over time. From a more technical view, on ntfs filesystems (assuming you're on a winblows box) become fragmented with large numbers of small file deletions/creations. Unless you regularly defragment the fs your previews are in, you'll eventually see a performance impact.
I'm a stickler for maximum performance for three reasons I can think of --
First, I was/am an "old school" CSC major -- back in my college days I spent hours pouring through assembly just to eek out a few milliseconds of performance. I contributed to the early (pre-distribution) days of the linux kernel. In other words, I was an uber-geek. I am still largely uber-geeky, and I just love to tune things. So, some of my concern over performance is just part of my personality.
Second, I actually run Linux native on my hardware. I run winblows for things like LR in a virtual machine, so I'm already "one foot in the performance grave" so-to-speak, because I'm running in a virtual machine. So, any small incremental performance improvement I can make on my winblows vm, I can usually notice perceptible improvements.
Third, I heard a ham radio (another of my hobbies) story a few years ago, and it stuck with me. Short version -- In radio, we measure signal strength in decibels (dB). Minor improvements in transmission strength might only be measurable in much smaller units, like centibels. You use a high-quality solder connector here vs. some cheap crimp-on connector, maybe you get 1 centibel improvement. You use a better transmission line you get a couple more centibels. Etc., etc., etc... It eventually adds up and you get a whole 1-2 dB of additional signal strength, just by making a series of very small improvements. And that 1-2 dB is what and make or break a DX contact. Mathematically, that holds true (perhaps even more-so) in computers, too. You make small improvements in the right places, and you can see perceivable improvements elsewhere.
Wow, okay, this has turned into a novelette. Sorry. I'll try to dial-back the verbosity a little...
My own advice for tuning-up LR:
- Store your catalogs database on fast, local disk (even RAM disk if possible - much longer discussion there).
- Store your actual media files on equally fast, local disk whenever possible, but a separate physical disk than your catalog.
- Consider your location of the ACR cache (default \Documents and Settings\userid\Local Settings\Application Data\Adobe\CameraRaw\Cache) and the data-path between your media and that device. Each time you enter the develop module for an image, (assuming it isn't in the cache already) it will be copied from the media device to that device. You want this transfer as fast as possible.
- For my really huge catalogs (which simply can't fit on local disk and are on my NAS), I copy or move groups of files back and forth between the NAS and local storage. For example, when I first import a batch of pictures into a catalog which contains mostly NAS-stored files, I first import them to a local disk area and do my initial editing/exporting/etc. When I'm satisfied I've done the bulk of my work, I simply use the folder view in LR and move the files to the NAS area. If I later need to do a lot of editing on some batch of files, I copy them (not *move* but just copy) to local disk, tell LR where they are, do my business, then delete the local copy, and finally update LR to point back to the NAS files.
- Don't use sidecar files unless you legitimately have a need for them. The time to stat and sync the files and sync metadata changes is a performance impact, even with the newer versions of LR. If your media fs is ntfs, you'll also start down the path of introducing fragmentation. I prefer to make my media devices read-only. Back up your database after every large edit, and every so many days, of course.
- "Every so often" (which is obviously up to you), clean house. You can do these steps for all of your catalogs at once, or certain catalogs; whatever is right for you.
My personal clean-house steps:
- Run the "Optimize Catalog" on your catalogs.
- Shut down LR (duh -- but worth mentioning)
- Remove the preview cache for your catalogs. (This is the "catalogname Previews.lrdata" directory inside each of your "catalogname" directory. Just destroy the entire directory. All that should remain in your "catalogname" directory is the file "catalogname.lrcat". If you use the Recycle-Bin to do this, empty the recycle bin.
- Defragment the fs which holds your catalogs. If using the included defragger, you'll need to run it multiple times in a row before it finally settles on how much defragging it's going to do for you. I recommend a third-party defragger rather than the winblows version.
- When you restart LR, it will take some time to rebuild previews when you view images. If you have a set of images you know you'll be working with, consider just rendering 1:1 previews of them manually.
My personal general LR performance tips:
- Only view the photos you want. When you use the "all pictures" folder or other large/wide scope views, LR will start statting all the files to check for file validity, meta data changes (even if you don't *write* sidecars automatically, it will still look to see if there are any sidecars available to *read*), etc. By keeping you active set small, you reduce the calls to your media filesystem(s) -- this is HUGE performance hit over network filesystems CIFS or NFS -- stats and metadata checks are expensive.
- If you are going to enter the develop module, consider which images you'll need to develop, and "hit them all first" to get them cached before you start editing heavily. Your cache will probably hold around 10-15 raw files before it is overrun; so keep that in mind as well, and try not to keep overrunning your cache.
- Remember that export or manual generation of 1:1 previews will hit your cache, too, so again if you'll be doing a lot of developing/editing, separating your developing from your exporting might save you some cache overruns.
- When importing, if you know you'll be making substantial changes, consider only rendering minimal previews vs. standard or 1:1. The fewer "bad" previews you generate up front, the fewer fs updates will be made later once you start re-rendering previews. Likewise, if you make any really broad develop setting changes, consider doing the "clean house" for that catalog, then re-rendering 1:1 previews afterward.
WOW, that turned into a post from hell. Sorry. If you're still reading, I pity you and deeply apologize, but I hope some of this info is helpful. :P
|
|
|
|
06/10/2008 09:50:51 PM · #6 |
| Hey - Just coming back to this. That was helpful (but then I started as a prog in the late 80's so know where you are coming from:). Yeah it's NTFS & cleaned up often.... I will get another internal drive but am waiting for the next machine as this one is too far gone now days. The RAM disk caught my eye... have not used one for years and didn't know it was still possible - worth a thought (I am not a windoze person). |
|
|
|
06/10/2008 11:11:52 PM · #7 |
Originally posted by robs:
Hey - Just coming back to this. That was helpful (but then I started as a prog in the late 80's so know where you are coming from:). Yeah it's NTFS & cleaned up often.... I will get another internal drive but am waiting for the next machine as this one is too far gone now days. The RAM disk caught my eye... have not used one for years and didn't know it was still possible - worth a thought (I am not a windoze person). |
HA! And here I thought I had killed another thread. Just call me ThreadKillah'!
I did some playing with a ram disk on my native Linux box (since it runs 64-bit and can access all my memory), then translated the disk to the xp session. I didn't see great performance, but I expect that is from the "internal network" translation of the linux disk to the xp vm. I didn't play with it much after that, but perhaps this will inspire me t----ugg, I have way too much other work to do.
Anyway, a little googling led me to:
MS article about their skeleton ram disk driver (you already have this, but probably didn't know it)
A little more about how to use the above driver step-by-step
A commercial ram disk product for XP - this looks pretty cool, and allows you to load and save disk images, etc. It's $50 for the 32-bit version and $100 for the 64-bit version.
I'm sure there's more out there.
That would be the trick, if you had the ram, though (and a 64-bit os to see it) -- put your working set of images and your adobe raw cache on ram disk -- talk about some freakin' performance... Would be safe, too -- keep your actual db on local disk and work with a copy of your image files on the ram disk; if you crash and lose the disk, who cares? Just populate it again from your actual disk files. The relatively small up front hit to populate the drive would be uuuuuber-offset by the performance later. You'd just have to keep your ram disk and working set sized appropriately, obviously constrained by your physical memory.
|
|
|
|
06/11/2008 05:10:54 AM · #8 |
| Great info cdrice, thanks . I'm new to LR and still am trying to get acquainted with it. |
|
|
|
Current Server Time: 08/21/2025 03:34:26 PM  |
Home -
Challenges -
Community -
League -
Photos -
Cameras -
Lenses -
Learn -
Help -
Terms of Use -
Privacy -
Top ^
DPChallenge, and website content and design, Copyright © 2001-2025 Challenging Technologies, LLC.
All digital photo copyrights belong to the photographers and may not be used without permission.
Current Server Time: 08/21/2025 03:34:26 PM EDT.
|
