Internals

Workflow

@startuml
card Indexer {
}
card Extractor {
}
card Builder {
}
card Server {
}

Indexer -> Extractor
Extractor -> Builder
Builder -> Server
@enduml

The Indexer creates a file index from a directory to detect quickly file changes and to calculate unique file IDs
The Extractor calculates preview files and extracts meta data based on the file index and unique file IDs. Duplicated files (same binary content) are extracted only once. All preview files and meta data are stored in a storage directory
The database builder creates a database from the extracted preview files and meta data for the WebApp
The server serves the database with preview files from the storage directory

Note

Before the server can serve the database all preview images and videos needs to be calculated. This can be time consuming depending on the amount of media files and the processing maschine.

Building Blocks

Indexer

@startuml
folder Files {
}
card Indexer [
Indexer
]
database FileIndex {
}

Files -right-> Indexer
Indexer -right-> FileIndex
@enduml

Responsibilities:

Create and update a file index from a diretory to detect file changes quickly such as new, deleted files and moved files
Create unique file identifier for each file (SHA1 sum)

For the first time the creation of a file index can be time consuming due the calculation of SHA1 checksum where each byte of a file needs to be processed. Depending on the total file size this process can take hours.

On file index update known files do not need to be updated. Known files are detected by file path and inode. Other strategies are supported, too.

Note

The file index is used to have a filtered view on your files of your source directories. The gallery handles only files which are listed in the file index. To exclude files or folders in the gallery it is sufficient to add exclude patterns on the source directories.

Extractor

@startuml
folder Files {
}
database FileIndex {
}
card Extractor [
Extractor
]
folder StorageDir {
}

Files -down-> Extractor
FileIndex -down-> Extractor
Extractor -down-> StorageDir
@enduml

Responsibilities:

Calculate preview images and videos based on the file index and the uniq file identifier
Extract meta data original files, preview data or other meta data
Store all data to the storage directory

The work of the extractor is explorative and the raw output data from extractors is stored.

The calculation of preview files and extracting meta data is time consuming. The calculation of preview videos takes also long. Depending on image and video count and used PC this process can take weeks for the intial run. Updates compute only new media (detected by the file index).

Database Builder

@startuml
database FileIndex {
}
folder StorageDir {
}
card Builder [
Builder
]
database Database {
}

FileIndex -down-> Builder
StorageDir -down-> Builder
Builder -down-> Database
@enduml

Responsibilities:

Create a database from preview files extracted meta data for the WebApp

The database builder prepares the data for the (mobile) browser and collects the important data for the presentational WebApp

Events

User inputs such as add or remove tags are handled and stored as events. These events are applied on the database entries.

Server

@startuml
database Storage {
}
database Database {
}
database Events {
}
card Server [
Server
]
card WebApp [
WebApp
]

Storage -down-> Server
Database -down-> Server
Events <-down-> Server
Server <-down-> WebApp
@enduml

Responsibilities:

Serve the WebApp
Serve the preview images and videos to the WebApp
Handle user events like add or remove tags

Since the database is loaded into the browser, the server acts mainly as a static webserver. The main logic such as filtering and sorting is executed in the WebApp

Export Meta Data

@startuml
database FileIndex {
}
database Database {
}
database Events {
}
card Export [
Export Meta
]
folder Files {
}

FileIndex -down-> Export
Database -down-> Export
Events -down-> Export
Export -down-> Files
@enduml

Responsibilities:

Export meta data to XMP sidecar files

Fetch

@startuml
card Server [
Server
]
card Fetch [
Fetch
]
folder Storage {
}
database Database {
}
database Events {
}

Server -down-> Fetch
Fetch -down-> Storage
Fetch -down-> Database
Fetch -down-> Events
@enduml

Responsibilities:

Fetch a (subset) of a remote gallery and merge it into the local gallery

Export Static Gallery

@startuml
database Storage {
}
database Database {
}
database Events {
}
card Export [
Export Static
]
folder "Static site" {
}

Storage -down-> Export
Database -down-> Export
Events -down-> Export
Export -down-> "Static site"
@enduml

Responsibilities:

Export static web site from a subset
Static export requires only an HTTP server such Apache2 or Nginx

Design Decisions

For this gallery I assume a average private photo collection of 100,000 media, 200,000 files and a recent mobile phone.

Web Gallery

Today (almost) every thing can be presented and edited by a web page. Mobile phones are omnipresent and have enough power to process complex web applications. So it makes tatal sence to offer a web gallery for the family where everyone can consume the gallery on their mobile phone.

Prerendering Previews

The HomeGallery uses precalulated preview images and videos. There are pro and cons regarding prerendering and on demand rendering:

Pros of prerendering

Some feature require preview images. The similarity search as core feature plays only well if (almost) all similarity data are available
All previews can be served immediately and gives good user experience
Server can be simple such as a SoC like a Raspberry Pi
Server can be static for the static site export
Original files can be offline (and safe) after preview and meta data extraction

Cons of prerendering

Prerendering taks time. Preview image calculation of 20,000 image takes few hours. 200,000 require days. Video previews needs weeks - depending on the used system
Prerendered images and videos consume about 15% of the orginal size, depending on image/video ratio
Gallery can be used after preview calculation is done

Pros of on demand previews

Initialization of the gallery (time to first usage) takes less time
Only requested previews are calculated and saves preview storage

Cons of on demand previews

Access to original files is required
Image previews need a powerful host for good user experience
Videos requires supported hardware to trancode videos just in time
Some features can not be suppored (e.g. similiarity search)

The prerendered previews is choosen due the core feature of similarity search, SoC targets and decoupeling of (offline) originials. I do not think that storage consumption of the previews is an issue to the private storage space. Further, the time of the preview calculation can be splited in several chunk steps (e.g. by years) to use the gallery quickly until all previews are calculated.

JSON

The main data structure is encoded in JSON format. JSON is the common data exchange format of the web. It can be read by machines but also by human, which helps debugging problems.

It might be stored plain, compressed by gzip or as newline-delimited JSON in the HomeGallery.

Database

There is no database, just a JSON array. And the main gallery database is loaded completly into the browser. Today devices - even mobile phones and SoC - are fast enough to process and filter 100k entries quickly (below a second and less). So why use a backend database which is slowed down by client-server requests?

For a private media collection of 100,000 files the execution times are good enough.

Javascript

I like Javascript and Typescript and I think these are awesome languages. The eco system with node packages is wide supported. The language helps me prototyping the features quickly in the backend and frontend or both. Perfect for a pet project.

Offline File Index

HomeGallery uses an offline file index which stores meta data a directory tree. It keeps a state of a file such as name, file time and inodes. With it, a change can be detected quickly by comparing the stored state with the current state.

This method is choosen over live filesystem notifications like inotify because it offers more flexibility. A live notification needs to be run while the gallery application runs. Any change outside the lifetime of the application are not recognized. This drawback is crucial.

Further a live notification is not required since the time when files or directory change are mostly well known. E.g. a user copies new files from the camara memory card or a user renames some files or folders. For all these scenarios an update happens infrequently and it can be performed fast engough.

An offline file index offers also the possibilty to load the directory tree of an offline media source and operations can still be performed on existing previews or meta data on the storage.

File identifier

As file or media identifier a SHA1 checksum is used. This checksum is calculated over the file content and if a single bit changes, the checksum changes. SHA1 algorithm is used because git uses it and is good enough where the gallery has not more than one million files. Due the usage of the file index, an calculation of the checksum happens only for unknown file such as new or changed files.

A renamed or moved file might trigger a recalculation of the checksum as identifier but results to a known identifier. So already calculated preview files or meta data can be kept.

It has also some security properties: The identifier is not guessable. As long the identifier is not known, the image can not be retrived. A storage can hold images for several user galleries.

Note

Any change on the file (also embedding new meta data such as GEO data, IPTC or XMP) leads to a new identifier and to new preview file generations. Therefore it is recommended to use side car files such as .xmp files.

Data Structures

File index

The file index holds a state of the filesystem including

Base directory
Filename (relative path from base directory)
Filetype
Filesize
Timestamps
Inode
File checksum

It is stored as gziped JSON file. This data is generated and can be recalculated

To inspect an file index the tool jq is recommended.

$ zcat index.idx | jq .
{
"type": "home-gallery/fileindex@1.0",
"created": "2020-09-26T21:10:15.269Z",
"base": "/home/user/Pictures",
"data": [
   {
      "dev": 65026,
      "mode": 33188,
      "nlink": 1,
      "uid": 1000,
      "gid": 1000,
      "rdev": 0,
      "blksize": 4096,
      "ino": 691414,
      "size": 3408759,
      "blocks": 6664,
      "atimeMs": 1601154468012.836,
      "mtimeMs": 1513937811000,
      "ctimeMs": 1578598308510.7227,
      "birthtimeMs": 1578598308382.723,
      "atime": "2020-09-26T21:07:48.013Z",
      "mtime": "2017-12-22T10:16:51.000Z",
      "ctime": "2020-01-09T19:31:48.511Z",
      "birthtime": "2020-01-09T19:31:48.383Z",
      "filename": "preview-test/files/camera/IMG_20171222_111649.jpg",
      "sha1sum": "f29f407905f8af94ece4720be997fd291adea487",
      "sha1sumDate": "2020-09-26T21:10:07.757Z",
      "isDirectory": false,
      "isFile": true,
      "isSymbolicLink": false,
      "isOther": false,
      "fileType": "f"
   },
   {...},
   ...
]
}

Storage directory

The storage directory is a directory to store generated preview images, preview videos and extracted meta such as EXIF data, geo addresses or similarity data.

It is a simple object storage where the key is the checksum of the file with given suffixes. Most data have JSON or binary format.

...
|-- f2
|   `-- 9f
|       |-- 407905f8af94ece4720be997fd291adea487-exif.json
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-1280.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-128.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-1920.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-320.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-800.jpg
|       `-- 407905f8af94ece4720be997fd291adea487-similarity-embeddings.json
|-- fc
|   `-- 0b
|       |-- 1fd17f3eab9c7caf15f3ff4a567573a8ac79-exif.json
|       ...
...

Note

In most cases the storage directory does not need to be deleted. It is assumed that the disk space at home is available (and the costs for that is cheap). The files in the storage are reused on file renames ore file moves to save CPU time.

Database

The database is the main structure for the web app and holds all important information of each media. This data is generated and can be recalculated.

The database is stored as a gzip compressed JSON object.

$ zcat database.db | jq .
{
  "type": "home-gallery/database@1.0",
  "created": "2020-09-26T21:31:19.028Z",
  "data": [
    ...
    {
      "id": "f29f407905f8af94ece4720be997fd291adea487",
      "type": "image",
      "date": "2017-12-22T10:16:51.820Z",
      "files": [
        {
          "id": "f29f407905f8af94ece4720be997fd291adea487",
          "index": "index",
          "type": "image",
          "size": 3408759,
          "filename": "preview-test/files/camera/IMG_20171222_111649.jpg"
        }
      ],
      "previews": [
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-128.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-1280.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-1920.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-320.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-800.jpg"
      ],
      "year": 2017,
      "month": 12,
      "day": 22,
      "width": 4864,
      "height": 2736,
      "orientation": 1,
      "duration": 0,
      "make": "LEAGOO",
      "model": "T5",
      "iso": 1056,
      "exposureMode": "Auto",
      "focalLength": 3.5,
      "focalLength33mm": -1,
      "latitude": 0,
      "longitude": 0,
      "altitude": 0,
      "whiteBalance": "Auto",
      "similarityHash": "KuSqiWXWVVpqXGmJWU2JGlJula1epWlWaWmVJVQKaUZqIpWklFVJpoliaWqWWFoIZtqakN2VqFiSmWVFVGpilmKlWRYRdJplila3VirmiahlSyU5SaA="
    },
    ...
  ]
}

Events

All user interaction (currently limited to image tagging) are stored in a event database.

Events are stored as plain newline-delimited JSON (AKA ndjson). This data contains only manual actions and should be treated with care.

$ cat events.db | jq .
{
  "type": "home-gallery/events@1.0",
  "created": "2020-09-06T06:57:17.507Z"
}
{
  "id": "541c203a-bccc-455c-babd-4bcd7858f3b9",
  "type": "userAction",
  "targetIds": [
    "f29f407905f8af94ece4720be997fd291adea487"
  ],
  "actions": [
    {
      "action": "addTag",
      "value": "awessome"
    }
  ],
  "date": "2020-10-07T07:04:46.912Z"
}
...

Note

Please export your meta data to standardized XMP sidecar files so that other photo library applications can read your tags from HomeGallery

Note

While the database can be recreated, the events holds your manual work which gets lost on deletion. Delete the events file only if you know what you are doing.