Don’t be the turkey // a “belt and suspenders” approach to Personal Digital Asset Management

Back in the old days, if you had a photo collection, you could stuff it in a box and put it under the stairs in the basement and you knew where it was. Nowadays, that’s not the world we live in anymore — nowadays, everything has gone digital, across multiple devices. I remember my first digital camera in the early 2000s — it was clunky and produced poor quality images… since then, with the advent of the iPhone and the mobile revolution, it’s well more than a decade and probably closer to 2 decades since all of our image creation has gone digital — although film photography does seem to be experiencing a resurgence, especially with Gen Z.

In regards to managing a photo collection, the digital revolution presents both advantages and disadvantages. In the old days, everything was safe in the box under the stairs — unless your house burned down, in which case everything was gone (see What to do with those boxes full of old photos?). Whereas now that things are digital, you can easily have multiple copies of assets, and therefore you’re not as vulnerable to a single catastrophic loss. This idea of redundancy is very important and it’s something we will come back to here below, but for now let me just say that it’s a double edged sword and needs to be properly managed.

Digital collections do present their own issues: because there are so many copies of things, the challenge is now how to keep track of everything, since it’s not just in one box under the stairs. Your photos can be in multiple places, and as you go on through the years, it can get out of control and turn into a hot mess (see What to do with those old hard drives full of photos?). So the question is, how do you properly manage your photo collection in today’s digital age?

In today’s cloud-based and ecosystem-based computing environment, there are a number of touted “one-stop-shop” solutions for managing your photo collection. There’s Google Photos, Apple Photos/iCloud, Amazon Photos, and Adobe Lightroom — in regards to the latter, I’ll come back to this because I do use Adobe Lightroom, but I use Lightroom Classic as opposed to the main Lightroom product (formerly “Lightroom CC”), and will circle back on the difference between these two products below. Even Dropbox tries to get you to automatically upload your photos to it (I actually do use Dropbox in my workflow, as will be explained in further detail below — but not as a “one-stop-shop” photo storage tool).

All of the aforementioned are great products with useful features, and the choice mostly depends on which ecosystem has you within its gravitational field (Apple, Google, Amazon, etc). By design, these products are not interoperable, but rather they are designed as “one-stop-shop” solutions and moreover each of the ecosystems has a vested interest to keep you safely inside their walled garden. For basic photo users, the convenience and seamlessness of such products is compelling.

But I’ve been around long enough that I don’t really trust anybody or any one product entirely. There are several great books about risk and probability, including these following, among others >>

• Nicholas Taleb >> The Black Swan: The Impact of the Highly Improbable
• Peter L. Bernstein >> Against the Gods: The Remarkable Story of Risk
• Michael Blastland + David Spiegelhalter >> The Norm Chronicles: Stories and Numbers About Danger and Death
• Nate Silver >> The Signal and the Noise: Why So Many Predictions Fail — but Some Don’t

Figure 1 // Nicholas Taleb, The Black Swan: The Impact of the Highly Improbable

In a nutshell, a proper understanding of risk means not only identifying the likelihood of different scenarios, but also understanding whether you can tolerate the cost of certain negative “tail” outcomes. As traders and poker players will explain: you win some, you lose some, but you have to stay in the game — if a drawdown wipes you out, then it’s all over. That’s the whole reason the insurance industry exists: you’re paying out more in premiums than your loss payout on a probability-weighted, expected-value basis, and meanwhile the insurance company is making money on its portfolio of contracts because of this — but we nonetheless still buy insurance because, if your house burns down, that just might be too big of a cost to swallow (whereas, for the insurance company, the risk is diversified within its vast portfolio of contracts) — you cover yourself from the catastrophic loss that you can’t tolerate by buying insurance. So… It might be extremely unlikely that Apple iCloud or Google Photos loses your photo collection, but… what if, somehow, that were to happen? Can you live with that risk, however minimal it might be? Only you can answer that question.

The “inductivist turkey”

Bertrand Russell tells the story of the “inductivist turkey” (often retold, including by Nicholas Taleb and Pedro Domingos, among others)… it’s a simple illustration of what is known as David Hume’s “problem of induction”. Day after day, the turkey wakes up to a life of luxury which, for an animal, means that he is fed and protected from predators… no need to forage for food, he is fed all he can eat by these wonderfully generous, caring humans. So he gets used to this and thinks “well, life is pretty great” until, after a thousand days of this, he wakes up to discover the real reason he was being fed and cared for all this time!

Figure 2 // the “inductivist turkey”

The moral of the story: just because everything has been fine so far, does that mean things can’t go wrong? What happens if one of these services fails? What happens if, for some reason, Apple iCloud goes down, or even just glitches and one or all of your photos are gone? What happens if your profile gets hacked? Accidentally deleted? We don’t even need to go into more doomsday scenarios like a solar storm or a nuclear war that wipes out the Internet, there are plenty of more prosaic scenarios where you could lose your photo collection if you have all your eggs in one basket.

As with many things, there may not be a one-size-fits-all approach, and the optimal toolkit may vary for different users, depending on the volume of their assets, their level of tech savvy, and their risk tolerance, among other factors. I therefore think it makes sense to present things in terms of a framework and strategy for Personal (💜🌈) Digital Asset Management (“PDAM”), rather than one specific methodology and set of tools. But let me first describe and explain the specific PDAM toolkit and workflow I use to manage my own assets, and then I will present a more generalized PDAM framework.

💜🌈 // It is appropriate to include the modifier “personal” to distinguish the tools we are discussing here, since the term Digital Asset Management (“DAM”) typically refers to enterprise-level solutions that catalog large volumes of photography assets, as well as managing permissions/access across a large set of users.

I use Lightroom Classic as my main PDAM tool, and I have it setup to operate entirely within Dropbox for seamless redundancy, complemented with periodic external snapshot backups. I call it the “belt and suspenders” approach (a throwback from my former life as a corporate attorney specialized in syndicated bank lending).

Figure 3 // the “belt and suspenders” approach

Who is ASK?

Following is a video that recaps my background in digital asset management >>

Lightroom

Adobe Lightroom is the tool at the center of my PDAM workflow. It’s an incredible and powerful photography software tool — no other photo tool even comes close. Following is a basic presentation of Lightroom; for more details kindly refer to Diving deeper on Lightroom + setting up LrC to reside inside Dropbox.

Lightroom delivers 2 main pillars of functionality:

• asset management // photos + videos
  - storing
  - organizing, selecting, curating
• asset manipulation // photos only

Lightroom delivers this functionality across 3 separate tools/interfaces:

• desktop // figure 4
• mobile // figure 5
• web // figure 6

Figure 4 // LrC-desktop

Figure 5 // Lr-mobile

Figure 6 // Lr-web

These 3 tools/interfaces remain synced together via a cloud-based asset library (“Lr-cloud”).

The difference between Adobe Lightroom and Adobe Lightroom Classic is as follows (figure 7):

• Lightroom (“Lr”) // formerly “Lightroom CC” >> desktop client relies on the cloud-based library Lr-cloud to store the assets
• Lightroom Classic (“LrC”) >> desktop client maintains its own asset library/catalog on the desktop system drive

Lightroom isn’t really just one product but rather several tools and products, and it can get somewhat confusing as Adobe keeps changing the branding — the fully cloud-based product is now just “Lightroom” instead of “Lightroom CC”. The mobile and web clients (“Lr-mobile” and “Lr-web”) are common to both Lightroom and Lightroom Classic, the difference resides in the desktop client. I use Lightroom Classic in my PDAM setup because I prefer the added redundancy and security of having my valuable assets stored on my desktop system (“LrC-desktop”), as opposed to relying entirely on Lr-cloud.

Lightroom costs $10-$20/mo depending on the package. For more details kindly refer to Diving deeper on Lightroom + setting up LrC to reside inside Dropbox.

Figure 7 // differences between LIghtroom vs. Lightroom Classic // source Adobe

Lightroom Classic // desktop application

With LrC-desktop, still and video assets are imported into a “catalog” file representing the entire “library” database of assets (as well as previews), however the “original” assets remain stored as files within folders transparently on the system drive of the desktop computer — the latter fact constitutes a keystone of my PDAM workflow, since it allows for the redundancy that I consider crucial. Once imported into Lightroom, the assets can be organized into different “collections” (ie, “albums”) for different projects, trips, themes or any other grouping as desired, without affecting or moving the files on the desktop system drive. It doesn’t matter what sort of folder structure one may use to organize the asset files on the desktop drive — I have top-level folders to separate personal, art, commercial, and video assets, and then within these I tend to subdivide by year, and then by month or project. You can organize and reorganize the folder structure at any time — any moving and shuffling of assets between folders as well as renaming files is done directly within LrC instead of Finder.

Lightroom // mobile app

The next part of the Lightroom system is Lr-mobile, ie. the Lightroom app on your iPhone or other mobile device. While I create many different types of digital assets, these days most of the photo and video assets are captured directly on my iPhone. Over the years, I have previously used digital SLR cameras and obviously for professional uses, they still exist for a reason. But, after being exposed to photography across many domains from fine art to fashion, my personal takeaway is that the best images are about the creative eye and Henri Cartier-Bresson’s “decisive moment” rather than the lens used to capture. This is especially true as the quality of iPhone captures has greatly improved in recent years. I must also recognize a personal preference for abstract photography as well as manipulated imagery, further lessening the importance to me of capture quality. This is of course a subjective opinion, but to me it matters less what I use to capture an image weighed against the convenience of the iPhone, which is the camera I always have with me.

Following is an outline of my asset ingestion process via Lr-mobile:

1. The Lr-mobile app on my iPhone automatically imports photos and videos captured on my iPhone
2. From Lr-mobile, new assets are synced up to Lr-cloud
3. From Lr-cloud, new assets are synced down to my “INbox” folder in LrC-desktop, thereby being automatically added to >> a/ the LrC-desktop catalog and b/ the file and folder structure on my desktop computer
4. The final step is to move new assets out of my “INbox” folder and distribute them into other folders in my file system — as the name “INbox” would suggest, this is a temporary holding folder.

Lightroom sync

With Lightroom Classic, it is important to understand that between the 3 tools/interfaces described above (LrC-desktop + Lr-mobile+ Lr-web), there are really 2 independent systems here (not 3):

• LrC-desktop
• Lr-cloud

Let’s back up for a moment. Assets ingested into Lr-mobile are automatically synced up to Lr-cloud and then automatically synced down to LrC-desktop, but the reverse is not automatic: you can import new folders of assets on your local desktop drive (as well as external drives) into the LrC-desktop library, but this does not automatically sync them up into Lr-cloud.

Assets can however be induced to sync up from LrC-desktop to Lr-cloud by adding them to a collection that is toggled as “Sync with Lightroom” — which means “sync up to Lr-cloud”. Syncing up from LrC-desktop to Lr-cloud may be desired for multiple reasons:

• collections that you may wish to share and/or publish as public Lightroom albums
  - asymptoticSystemKey >> https://adobe.ly/4fU1ciJ
  - studioLeoVprintingHouse >> https://adobe.ly/46n9QTw
• asset manipulation on Lr-mobile // photos only
• asset selection on Lr-mobile
• having one more redundant copy of your assets in Lr-cloud?

I therefore have a collection called “syncMobile” where I can drag assets ingested in LrC-desktop that I may wish to edit on mobile (manipulation+selection) — you can save a lot of time by getting some editing done while the kid’s at swimming lessons. But for the most part, I don’t worry too much about keeping everything in Lr-cloud, since I use it mostly as an asset ingestion mechanism/conduit, for importing newly created photo+video assets from my iPhone via Lr-mobile.

LrC-desktop and Lr-cloud must therefore be understood as two separate systems, bridged together by a sync mechanism. But this sync mechanism is not completely robust and, over the years, I have experienced many ebbs and flows in its stability and integrity. Several times I have faced accumulating sync errors and other sync-related issues. On the other hand, the separation of these two systems also presents the solution to such problems. Because of the 3P cloud redundancy I have in place with Dropbox (which I am about to describe in the next section), in the event something ever goes wrong with the Lr-cloud library or if the integrity of its sync with LrC-desktop gets corrupted, I simply log into Lr-cloud on the web and delete the Lr-cloud library (“wiping Lr-cloud”), without affecting either the LrC-desktop library or catalog, which remain intact although no longer synced.

Sadly, I have had to resort to “wiping Lr-cloud” multiple times over the years. For this reason, I tremble at the thought of trusting all my eggs in one basket. Some engineer from Adobe will likely chime in that the potential for sync issues is greater with 2 separate systems, whereas there might be less potential for issues with one system that relies entirely on Lr-cloud. Alas, for all its power and efficiency, the main Lightroom product (fully cloud-based, formerly “Lightroom CC”) cannot be considered infallible — like any other software. All it takes is a left-tail incident once in a while, the occasional black swan where things go wrong and you learn that you can’t ever fully trust any one single app or platform.

Dropbox

Here’s where the redundancy part comes in. As explained, Lr-mobile pulls new photo+video assets from my iPhone, syncs them up to the Lr-cloud, then syncs them down to LrC-desktop on my Mac, at which point the assets are stored in folders on my system drive. So there’s redundancy right there, because I don’t rely on Lr-cloud, but have my asset files redundantly on the desktop system.

But that’s not all. To ensure full redundancy (in case I need to wipe Lr-cloud), I store my entire asset library (including not only all my asset files, but also the LrC catalog) in the Dropbox folder on my system drive:

• The top level folders containing my stills+video assets
• The LrC catalog
• The “INbox” folder where new assets are synced down from Lr-cloud

Storing not only all my assets, but also the LrC-desktop catalog machinery in Dropbox ensures continual background sync to the Dropbox-cloud.

Because I have such a large volume of assets under management, I am on a Dropbox Advanced plan, which provides 15 TB of storage, which is more than I need. And if ever I needed more, you can keep upgrading up to 1,000 TB — for all intents and purposes, infinite storage. The cost of such a Dropbox setup depends on your volume of assets — from $10/month for 2 TB, up to $72/month for 15 TB, and upwards from there for more users/storage.

For more details on how I have setup Lightroom to reside inside Dropbox, kindly refer to Diving deeper on Lightroom + setting up LrC to reside inside Dropbox.

Why Dropbox? I have battle tested multiple different cloud storage systems, both via web interface and through native desktop app — for me it’s crucial for the cloud service to integrate seamlessly with Finder, not only for convenience, but also to provide continual, automatic background backups of my entire digital asset library.

• Dropbox >> by far the fastest, and most reliable. I have been using Dropbox for well over a decade and have never faced any sync issues, where all the other services have failed me. Moreover, Dropbox is noticeably and substantially faster than the other services. With my fiber internet connection of 300 Mbps, the speed bottleneck is the cloud server as opposed to my internet connection. Dropbox’s faster performance is noticeable.
• iCloud+ >> was the biggest disappointment — since it’s an Apple product, you would expect it to perform the best at integrating with macOS, but it was noticeably slower than Dropbox and when faced with large volumes, led to occasional sync errors. The capacity caps out at 2 TB, but if you combine accounts with Family Sharing, you can extend this up to 12 TB.
• Google Drive >> generally I find the Google ecosystem confusing and lacking transparency. Full disclosure, I have not tested the native macOS Finder integration.
• Box >> I had to use this service when working with Gap Inc.’s internal photo studio, and the performance was horrendous. I would never recommend this product.

External Snapshots

As my asset library has grown larger, its total volume actually exceeds the available storage capacity of my local system drive. Thanks to Dropbox’s selective sync feature, I can hive off certain assets from my local drive, such that they remain consolidated (as placeholders) within the file system on my local drive (within my Dropbox folder), but their contents are only pulled down from Dropbox-cloud when/as needed. Therefore, Dropbox-cloud in fact constitutes the only single fully comprehensive copy of my asset library, since the copy on my local drive is not complete (and Lr-cloud is at best partial).

Therefore, I must protect against the possibility that Dropbox could be wiped out by a black swan. To borrow a term from the world of finance, we could think of this as “counterparty risk”. Cloud providers have redundancy built-in to their networks such that, if one of their server farms went down, your data is (probably?) not lost. But it would be foolish to ignore the possibility that Dropbox itself could blow up entirely — pretty unlikely for sure, but it could happen (eg. nuclear war, or an intense solar storm — see Diving deeper on External Snapshot Backups).

Moreover, there’s also what you could call “account risk” (within the “counterparty risk”)… eg. there’s an urban legend of some guy who merged his Google accounts and lost all the photos he had been storing for years in Google Photos. That’s definitely a potential risk… I feel it’s probably more of a risk with the “one-stop-shop” photo products like Google Photos that you don’t realize is somehow dependent on your Google account — as opposed to data stored in Dropbox-cloud as a literal file/folder structure, transparently mirroring what you have on the system drive… probably less likely to accidentally lose access to everything you have in Dropbox through some unintended account change since cloud storage is the core product offered by Dropbox — as opposed to something like Google Photos, which is just an ancillary product within a massively complex ecosystem.

Then, there’s also the risk of malware. For years I had been operating complacently, under the false rumor that Macs were immune to the viruses that commonly plague Windows-based PCs… until 4 years ago my system got infected with Safe Finder malware and, after failed efforts to remove it, had to resort to erasing my entire system drive and rebuilding from scratch. While Macs apparently are not susceptible to viruses per se, malware is another story — so at the end of the day it’s just semantics: malware is a general term for malicious software, while a virus is a specific type of malware. Any system can be susceptible to malicious hacking, so long as the user succumbs to social engineering and clicks on a link or attachment. FWIW, anti-virus products are no solution — on some level, the whole antiviral industry is junkware. Following is the gold standard Apple Support Community treatise on Mac security >> John Galt: Effective defenses against malware and other threats.

Furthermore, to the extent malware can infect image files — malware can be hidden in images through a variety of techniques — and your image files are backed up in Dropbox-cloud, then it’s not inconceivable that LrC-desktop, Lr-cloud and the Dropbox-cloud backup could all be compromised simultaneously (along with any continuously connected external backups). Dropbox does have version history and “rewind” features so, in theory, it should be possible to go back to before the point of infection and pull things back out of Dropbox. But, you never know… what if Dropbox somehow fails to deliver? In case of catastrophic loss, I feel it would be prudent to have the option of resorting to a clean, uninfected external backup that had remained physically and temporally isolated since prior to the time of infection.

To protect against such risks (however unlikely), I therefore make periodic “snapshot” backups to external solid state drives (“SSD”). When I say “snapshot”, I mean that I make a backup to the external drive, and then I seal it and never touch it again. I’m wary of regularly connected external backup systems (such as Apple’s Time Machine), since one of the risks one needs to protect against with an external drive is that the entire system could be compromised by malware/virus. So physical and temporal separation is important. Since the snapshots are only periodic, there would admittedly be some residual “gap” risk, ie. data loss in the time interval since the last snapshot. But, since I already have triple redundancy in place between Dropbox + local system drive + Lr-cloud, the external drive is more of a last resort in case of a catastrophic, simultaneous loss of data integrity across the other 3 systems. The size of drive depends on your asset pipeline, but for me 2 TB per year for around $150 gets the job done. For more details on my external backup workflow, kindly refer to Diving deeper on External Snapshot Backups.

CRR framework >> Consolidated + Redundant + Routine // aka. “Cheesy Rainbow Rave”

So that’s how I do it. But now, to recap, let’s zoom back out to present a more generalized framework. In the abstract, I think there are three pillars to it (“CRR”):

1. Consolidated
2. Redundant
3. Routine

But, before we dig into what each of those means, I admit it all sounds a bit boring, and you will probably want to forget about all of this before you’ve even read it. So, to make it a little more memorable, I have come up with a cute (ie. “dorky”) little mnemonic for you: “Cheesy Rainbow Rave”!

Figure 8 // “CRR” framework >> Consolidated + Redundant + Routine // aka. “Cheesy Rainbow Rave”!

Now let’s dive into what each of these means.

1/ Consolidated

This means having all your assets in one single system. Not to mean that there aren’t multiple copies of assets (quite to the contrary, as we saw), but that the multiple redundant copies should all be within the context of one unified and consolidated collection (that might have multiple redundancies) — as opposed to having your collection fragmented with some assets in one system and other assets in another system because then, over the years, the risk is that you will lose track of some of them.

2/ Redundant

As discussed above, it is important to have one’s asset library redundantly spread across more than one system, to protect against black swan events — ie. the “belt and suspenders” approach (figure 3). As the old saying goes, “don’t put all your eggs in one basket”. To me this speaks to the main problem with the several “one-stop-shop” photo collection services on the market: Google Photos, Apple Photos/iCloud, Amazon Photos, and Adobe’s main Lightroom product (formerly “Lightroom CC”) — the first 2 are great products with convenient features, the latter a very powerful tool — but what if something goes wrong?

3/ Routine

This third and last element is sort of like the icing on the cake — or rather, more like good dental hygiene — which is the opposite of icing on cake? For the most part, the workflow I’ve described above is automated… photos and videos captured on the iPhone automatically import into Lr-mobile, then automatically sync up into Lr-cloud, then automatically download from there onto my desktop system and LrC-desktop which resides inside of my Dropbox folder, and hence are automatically uploaded to Dropbox-cloud.

But, while automatic, the process still does need to be managed — or at least overseen. In order for the photos to import into Lightroom, the Lr-mobile app needs to be open on the iPhone. For the assets to upload to Lr-cloud, the iPhone needs to connect to the internet. And, in order for the assets in Lr-cloud to then sync down into LrC-desktop, the desktop computer needs to be turned on and the LrC-desktop app needs to be opened for this to happen. Sometimes, this person I know (who definitely isn’t my wife) forgets to do some or all of the aforementioned for weeks on end, and then there’s a huge volume of assets being ingested. To make matters worse, if the iPhone‘s memory gets full, there’s no memory available for Lr-mobile to import those assets, which requires available iPhone system memory. And sometimes said unnamed person also forgets to turn on her desktop computer and open LrC-desktop such that, when she eventually does, there are thousands of assets downloading into the LrC-desktop INbox folder, which can cause its own problems.

The whole process of asset ingestion in my PDAM workflow is automatic, but it’s in fact not fully automatic. You do need to stay on top of it. So I’ve chosen the moniker “Routine” instead of automatic.

💜🌈💜🍭🔲💜🔳

So, there you have it: my Personal Digital Asset Management workflow using Lightroom Classic + Dropbox + periodic external backups, along with the “Cheesy Rainbow Rave” framework to achieve a “belt and suspenders” approach to risk management.

I’m a huge fan of the Apple ecosystem and would love to rely solely on Apple Photos and iCloud because that would be so very convenient. Alas, as much as I love Apple, when it comes to photos however, there’s a lot missing.

Adobe Lightroom is an incredible and powerful tool — no other photo tool even comes close. Sadly though, it has let me down over the years and I don’t fully trust it with my precious collection… and, more generally, I feel it’s unwise to trust any one single tool, hence the “belt and suspenders” approach I have devised. If someone didn’t have it in them to implement my Lightroom Classic + Dropbox system, my next best choice would be the fully cloud-based Lightroom product — delivering most of the powerful features and functionalities of Lightroom, with the important caveat however that this would mean putting all your eggs in one basket, without the redundancy that I believe is crucial.

Please feel free to reach out with any questions or comments >>

atomic space kitten