Workflow: Setting up a Master Photography Archive

Sooner or later, every photographer finds a reason to split up their RAW catalog, change RAW tools, or even use multiple RAW editors for different projects. Even if they continue to use the same RAW tool, projects will start being split between catalogs, by year, or month, or project to combat the tool grinding to a halt. Regardless of the reasons, setting up a workflow that allows for different RAW management tools is a really good idea. One key element of that is to maintain a master photography archive.

Background

This is a part of a long running workflow series. If you came here directly and some of the terminology or reasoning seems strange, see the previous posts. This is a part of a larger whole. For background see these previous posts:

• Breaking Vendor Lock-In
• Splitting Up Catalogs
• Project Based Workflow

Master Photography Archive – Problem Solver

A master photography archive is a collection of “digital prints” (i.e. JPG images). It organizes all of your past photography shoots so they are at your fingertips. There are lots of ways you could organize your archive, but I have one that I like a lot. It is what we are going to explore here.

Regardless of the layout you set up, a master photography archive should solve a couple of really fundamental problems:

• Easy Addition: It should be easy to add new shoots to the archive.
• Browsing: Looking through images by shoot, date, or everything.
• Searching: Easily find all images of a particular subject/location/theme/rating/tag/etc.
• Origin: Easily allow location of the RAW file regardless of how many or few RAW tools and catalogs you use.
• Shareability: Images in the archive should be easy to use, for social sharing, for basic printing and even for customer fulfillment.
• Mobility: It should be easy to move the master archive around between drives as needed to manage hard drive space.

Some of these goals will be more related to the software you use to work with your archive than the actual directory structure or naming conventions. However, the best tools do not ignore that structure, but build on top of it. Further, by encoding as much of the crucial information into the way the images are organized, it makes it possible to switch tools or even find things without a tool.

What to Store in Your Master Archive

Should you export all images or just a subset? The main problem a master photography archive solves is that it gives you one easy place to find images from your entire photographic history.

Once upon a time, I exported and kept all images in my master archive. For the last few years I’ve switch to only storing 1-star or higher images. I retain all images in RAW form, but only place 1-star and higher rated images in my master archive. What you should do depends on how you expect to use images and how you rate them. If you rate like me though, 1-star and higher is probably good enough.

Full Quality JPGs

In general, I want my master archive to handle most of my day-to-day archive related work. My goal is that I (almost) never need to go back to the RAW project once I finish and export the images from that project. The exception to this is if I want to re-edit the RAW, or if I need absolute quality and even a 95% quality JPG isn’t good enough. For most things, including prints, that isn’t needed.

One big advantage of this is that full quality retouched images can live in my master archive. That’s handy when I decide to tweak an images years later. I don’t have to go back to the original catalog to save them.

Social Sharing JPGs

An alternative to storing full resolution JPGs in your archive is to store social sharing size images. Doing so would reduce the space required, possibly dramatically. The cost is that you will have to return to your RAW tool (which might be defunct, no longer rented, no longer running on your OS, etc…) to get a full res copy.

Unlike a master photography archive that stores full quality JPGs, you can’t use your master photography archive as a place to park retouched files down the road. To save the full quality version of an image, you need to return to the RAW editor project or catalog.

For me, that trade off isn’t worth it. Space is cheap, and the convenience of having the full quality images at my fingertips is worth the extra storage space.

Structuring Your Master Archive

How you structure your archive will in part depend on how your existing RAW images are organized. In my case, I currently use a Project Based Workflow with most things edit inside Capture One. Early in my photographic career I use Lightroom with a single Monolithic catalog. The good news is, that you can pretend you have a project based workflow and it is flexible enough to handle other types of workflows (such as Monolithic or Semi-Monolithic workflows).

Before we start, lets define a term:

A Project is one unit of photography.

For me as a professional, a wedding is a project, even if it is multiple days. Engagement photos are a separate project from the couple’s wedding, mostly because they sometimes occur months apart. A vacation or hiking trip is a project also.

Project Centered

My master photography archive is project centered. Each project lives in a directory inside my archive. Those directories are named using the date of the project (or the first day for multi-day projects).

Here are some examples of project names:

• 2010-12-14 New Zealand
• 2015-05-18 Colorado Bend State Park
• 2018-04-21 Taylor Animal Shelter
• 2018-09-07 Joshua Tree Scouting

Before we start to talk about tools for using your master archive, the directory structure itself has already solved two of the problems: it allows you to easily find the RAW file and we can perform a basic search based on the date or project name.

If you use a monolithic catalog, you know where the RAW is, but having the date is helpful. For semi-monolithic workflows, the project date and name will tell you which catalog to look into. The name of a project base catalog is the same as the name of the directory in the master archive. Easy.

Image File Names

What should you name the images in your master archive? There are a couple of general options:

• Descriptive Name: Name them for the project with a counter. Ex: New Zealand 2010 – 003.JPG
• Original Name: Leave them named the same as the source RAW file. Ex: _DSC0947.JPG
• Original Hybrid: A new name that includes the original name and additional information. Ex: 2018-04-01__DSC0947.jpg

Avoid Descriptive Names

Right off, I’m going to recommend that you do not use the Descriptive Name example above. While, using this type of naming for delivery to clients is a good idea, it is not a good choice for an archive. Changing the name of the image breaks the link back to the original RAW file. The new name contains no useful information that isn’t also already in the project name. Why loose information for no gain?

Original Image Name

Keeping the original name is a good strategy except for one caveat: if you re-export the images, or create multiple versions of an image, there will be naming collisions. Most software (Lightroom and Capture One included) handle collisions sanely. They simply add a trailing number for each new version exported. If you create new versions by hand, you can simply follow the same convention. Just be careful with new software. Make sure it doesn’t overwrite existing files if there is a collision.

Hybrid Original Name

The system I use is a hybrid of the original name. My naming scheme is YYYY-MM-DD_Original Name. The date used here is the export date, the day I produced the JPG. The file name encodes two important pieces of information: when the JPG was produced and the original file name. If I have multiple versions of an images, I can easily tell when each version was created just from the file name.

Directory Structure

I personally place individual project archive directories into folders based on the year. My structure looks like this:

• Master Archive
  • 2018
    • 2018-04-21 Taylor Animal Shelter
      • JPG Images…
    • 2018-09-07 Joshua Tree Scouting
      • JPG Images…
    • …
  • 2017
    • 2017-03-12 Enchanted Rock Startrails
      • JPG Images…
    • 2017-07-04 4th of July Fireworks
      • JPG Images…
    • …
  • …

If you use a semi-monolithic or monolithic catalog system, you might choose to insert a directory between master photography archive and the year or replace the year, to identify which catalog the projects live in. I just depends on how your catalogs are setup.

• Master Archive
  • 2018 Professional
    • 2018-04-21 Taylor Animal Shelter
      • JPG Images…
    • …
  • 2018 Personal
    • 2018-09-07 Joshua Tree Scouting
      • JPG Images…
    • …

Flexible Storage

Eventually, your master photography archive will get too large to just live entirely on your main computer. The longer you shoot, the larger it will get.

My master photography archive spanning back to 2009 (just JPGs) totals about 850GB. Because I treat projects as atomic units of photography, it is easy to shift older projects from the master archive directory on my laptop, to a master photography archive directory on a shared network drive or external hard drive.

Keep Recent Locally

In practice, what I do is keep about the last year of my master photography archive on my local SSD. It is the part of the archive I’m mostly likely to use. Keeping it local makes it available anytime I have my laptop, and makes it fast because it is on the internal SSD.

The remainder of my archive lives on an external USB drive, and previously on a network drive. If I had a larger hard drive (for example, if my master photography archive were on a large workstation with a multi-terabyte drive in it), I would keep the entire thing on the main drive. The key is that I can easily shift projects (or entire years of projects) between drives or even computers as needed.

• Local Master Archive (most recent 1 year)
  • 2018
    • 2018-04-21 Taylor Animal Shelter
      • JPG Images…
    • 2018-09-07 Joshua Tree Scouting
      • JPG Images…
    • …
• USB Master Archive (older stuff)
  • 2017
    • 2017-03-12 Enchanted Rock Startrails
      • JPG Images…
    • 2017-07-04 4th of July Fireworks
      • JPG Images…
    • …
  • …

Master Photography Archive Tools

Establishing what you store, and how is only 1 part of the problem. The second part is to choose a tool, or tools, to use to access your archive. Without a good tool, it is difficult to solve some of the problems we want to solve: browseablity, searchability, and shareable.

Using Capture One or Lightroom

There are a lot of options for photo managers, including things like Capture One and Lightroom. Both can work with an arbitrary directory structure, and will read the image metadata. To use one of these commercial tools, set up a catalog called “master archive” or some other variant of that name, and Add (Lightroom terminology) or Import in Current Location (Capture One terminology) your entire master archive.

One of our goals was easy of adding new work. Using Lightroom and Capture One make this easy, but not trivial. Both require you to do an import of the directory (even if it is through synchronizing the folder in their library panels). Neither tool sees new images in diretories it manages without your intervention.

The other drawback is that they are both commercial paid tools. That isn’t a show stopper, but maintaining access requires ongoing payments.

Digikam: An Open Source Tool

There are a variety of tools for managing photos in the open source world. I use Digikam and it is the one I’m going to talk about here.

Digikam maintains a database of photos. So far so similar to Lightroom and Capture One. The main difference is that it automatically updates the database from the file system. If you add images, it notices them. If you move or remove images, it also notices that. That makes adding new photos trivial.

Digikam provides powerful filtering through tags, ratings, text and and meta-data search, as well as map based searches and facial recognition. It also easily supports catalogs spanning multiple drives and understand that network and usb drives may be connected and disconnected.

Digikam has good tools for enhancing the meta-data in your photos (adding tags, description, location, and other extended metadata). And, Digikam can save all of that information back into the image files, or use XMP sidecar files. It will interoperate with other tools that use the image meta-data directly.

Look for a future post about using Digikam, but for now, it is a reasonable option that won’t lock you in.

Other Tools

There are a ton of photo managers out there, including things options from the big three: Apple Photos, Google Photos, Microsoft Photos. Commercial tools like ACDSee. I haven’t tested most of these so I can’t speak to their usability as a master photography archive tool. My best advice is to test carefully in a sandbox before you us them on your real archive.

Here is a full article on using Google Photos as a Master Archive.

Conclusions

Setting up a master photography archive can be one of the most powerful steps a photographer can take towards using their body of work. It allows for easy finding and sharing. A master photography archive is a central hub for your photography. It ties together projects from years past.

A master photography archive does not need to be complicated, but small decisions, such as the directory structure, have long reaching affect. By planning carefully, and following in the footsteps of others, your photographic life can be simplified for years to come.