Internals

Workflow

@startuml
card Indexer {
}
card Extractor {
}
card Builder {
}
card Server {
}

Indexer -> Extractor
Extractor -> Builder
Builder -> Server
@enduml

  • The Indexer creates a file index from a directory to detect quickly file changes and to calculate unique file IDs

  • The Extractor calculates preview files and extracts meta data based on the file index and unique file IDs. Duplicated files (same binary content) are extracted only once. All preview files and meta data are stored in a storage directory

  • The database builder creates a database from the extracted preview files and meta data for the WebApp

  • The server serves the database with preview files from the storage directory

Note

Before the server can serve the database all preview images and videos needs to be calculated. This can be time consuming depending on the amount of media files and the processing maschine.

Building Blocks

Indexer

@startuml
folder Files {
}
card Indexer [
  Indexer
]
database FileIndex {
}

Files -right-> Indexer
Indexer -right-> FileIndex
@enduml

Responsibilities:

  • Create and update a file index from a diretory to detect file changes quickly such as new, deleted files and moved files

  • Create unique file identifier for each file (SHA1 sum)

For the first time the creation of a file index can be time consuming due the calculation of SHA1 checksum where each byte of a file needs to be processed. Depending on the total file size this process can take hours.

On file index update known files do not need to be updated. Known files are detected by file path and inode. Other strategies are supported, too.

Extractor

@startuml
folder Files {
}
database FileIndex {
}
card Extractor [
  Extractor
]
folder StorageDir {
}

Files -down-> Extractor
FileIndex -down-> Extractor
Extractor -down-> StorageDir
@enduml

Responsibilities:

  • Calculate preview images and videos based on the file index and the uniq file identifier

  • Extract meta data original files, preview data or other meta data

  • Store all data to the storage directory

The work of the extractor is explorative and the raw output data from extractors is stored.

The calculation of preview files and extracting meta data is time consuming. The calculation of preview videos takes also long. Depending on image and video count and used PC this process can take weeks for the intial run. Updates compute only new media (detected by the file index).

Database Builder

@startuml
database FileIndex {
}
folder StorageDir {
}
card Builder [
  Builder
]
database Database {
}

FileIndex -down-> Builder
StorageDir -down-> Builder
Builder -down-> Database
@enduml

Responsibilities:

  • Create a database from preview files extracted meta data for the WebApp

The database builder prepares the data for the (mobile) browser and collects the important data for the presentational WebApp

Events

User inputs such as add or remove tags are handled and stored as events. These events are applied on the database entries.

Server

@startuml
database Storage {
}
database Database {
}
database Events {
}
card Server [
  Server
]
card WebApp [
  WebApp
]

Storage -down-> Server
Database -down-> Server
Events <-down-> Server
Server <-down-> WebApp
@enduml

Responsibilities:

  • Serve the WebApp

  • Serve the preview images and videos to the WebApp

  • Handle user events like add or remove tags

Since the database is loaded into the browser, the server acts mainly as a static webserver. The main logic such as filtering and sorting is executed in the WebApp

Export

@startuml
database Storage {
}
database Database {
}
database Events {
}
card Export [
  Export
]
folder "Static site" {
}

Storage -down-> Export
Database -down-> Export
Events -down-> Export
Export -down-> "Static site"
@enduml

Responsibilities:

  • Export static web site from a subset

Design Decisions

For this gallery I assume a average private photo collection of 100,000 media, 200,000 files and a recent mobile phone.

Prerendering Previews

The HomeGallery uses precalulated preview images and videos. There are pro and cons regarding prerendering and on demand rendering:

Pros of prerendering

  • Some feature require preview images. The similarity search as core feature plays only well if (almost) all similarity data are available

  • All previews can be served immediately and gives good user experience

  • Server can be simple such as a SoC like a Raspberry Pi

  • Server can be static for the static site export

  • Original files can be offline (and safe) after preview and meta data extraction

Cons of prerendering

  • Prerendering taks time. Preview image calculation of 20,000 image takes few hours. 200,000 require days. Video previews needs weeks - depending on the used system

  • Prerendered images and videos consume about 15% of the orginal size, depending on image/video ratio

  • Gallery can be used after preview calculation is done

Pros of on demand previews

  • Initialization of the gallery (time to first usage) takes less time

  • Only requested previews are calculated and saves preview storage

Cons of on demand previews

  • Access to original files is required

  • Image previews need a powerful host for good user experience

  • Videos requires supported hardware to trancode videos just in time

  • Some features can not be suppored (e.g. similiarity search)

The prerendered previews is choosen due the core feature of similarity search, SoC targets and decoupeling of (offline) originials. I do not think that storage consumption of the previews is an issue to the private storage space. Further, the time of the preview calculation can be splited in several chunk steps (e.g. by years) to use the gallery quickly until all previews are calculated.

JSON

The main data structure is encoded in JSON format. JSON is the common data exchange format of the web. It can be read by machines but also by human, which helps debugging problems.

It might be stored plain, compressed by gzip or as line-delimited JSON in the HomeGallery.

Database

There is no database, just a JSON array. And the main gallery database is loaded completly into the browser. Today devices - even mobile phones and SoC - are fast enough to process and filter 100k entries quickly (below a second and less). So why use a backend database which is slowed down by client-server requests?

For a private media collection of 100,000 files the execution times are good enough.

Javascript

I like Javascript and Typescript and I think these are awesome languages. The eco system with node packages is wide supported. The language helps me prototyping the features quickly in the backend and frontend or both. Perfect for a pet project.

Offline File Index

HomeGallery uses an offline file index which stores meta data a directory tree. It keeps a state of a file such as name, file time and inodes. With it, a change can be detected quickly by comparing the stored state with the current state.

This method is choosen over live filesystem notifications like inotify because it offers more flexibility. A live notification needs to be run while the gallery application runs. Any change outside the lifetime of the application are not recognized. This drawback is crucial.

Further a live notification is not required since the time when files or directory change are mostly well known. E.g. a user copies new files from the camara memory card or a user renames some files or folders. For all these scenarios an update happens infrequently and it can be performed fast engough.

An offline file index offers also the possibilty to load the directory tree of an offline media source and operations can still be performed on existing previews or meta data on the storage.

File identifier

As file or media identifier a SHA1 checksum is used. This checksum is calculated over the file content and if a single bit changes, the checksum changes. SHA1 algorithm is used because git uses it and is good enough where the gallery has not more than one million files. Due the usage of the file index, an calculation of the checksum happens only for unknown file such as new or changed files.

A renamed or moved file might trigger a recalculation of the checksum as identifier but results to a known identifier. So already calculated preview files or meta data can be kept.

It has also some security properties: The identifier is not guessable. As long the identifier is not known, the image can not be retrived. A storage can hold images for several user galleries.

Note

Any change on the file (also embedding new meta data such as GEO data, IPTC or XMP) leads to a new identifier and to new preview file generations. Therefore it is recommended to use side car files such as .xmp files.

Data Structures

File index

The file index holds a state of the filesystem including

  • Base directory

  • Filename (relative path from base directory)

  • Filetype

  • Filesize

  • Timestamps

  • Inode

  • File checksum

It is stored as gziped JSON file. This data is generated and can be recalculated

To inspect an file index the tool jq is recommended.

$ zcat index.idx | jq .
{
"type": "home-gallery/fileindex@1.0",
"created": "2020-09-26T21:10:15.269Z",
"base": "/home/user/Pictures",
"data": [
   {
      "dev": 65026,
      "mode": 33188,
      "nlink": 1,
      "uid": 1000,
      "gid": 1000,
      "rdev": 0,
      "blksize": 4096,
      "ino": 691414,
      "size": 3408759,
      "blocks": 6664,
      "atimeMs": 1601154468012.836,
      "mtimeMs": 1513937811000,
      "ctimeMs": 1578598308510.7227,
      "birthtimeMs": 1578598308382.723,
      "atime": "2020-09-26T21:07:48.013Z",
      "mtime": "2017-12-22T10:16:51.000Z",
      "ctime": "2020-01-09T19:31:48.511Z",
      "birthtime": "2020-01-09T19:31:48.383Z",
      "filename": "preview-test/files/camera/IMG_20171222_111649.jpg",
      "sha1sum": "f29f407905f8af94ece4720be997fd291adea487",
      "sha1sumDate": "2020-09-26T21:10:07.757Z",
      "isDirectory": false,
      "isFile": true,
      "isSymbolicLink": false,
      "isOther": false,
      "fileType": "f"
   },
   {...},
   ...
]
}

Storage directory

The storage directory is a directory to store generated preview images, preview videos and extracted meta such as EXIF data, geo addresses or similarity data.

It is a simple object storage where the key is the checksum of the file with given suffixes. Most data have JSON or binary format.

...
|-- f2
|   `-- 9f
|       |-- 407905f8af94ece4720be997fd291adea487-exif.json
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-1280.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-128.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-1920.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-320.jpg
|       |-- 407905f8af94ece4720be997fd291adea487-image-preview-800.jpg
|       `-- 407905f8af94ece4720be997fd291adea487-similarity-embeddings.json
|-- fc
|   `-- 0b
|       |-- 1fd17f3eab9c7caf15f3ff4a567573a8ac79-exif.json
|       ...
...

Database

The database is the main structure for the web app and holds all important information of each media. This data is generated and can be recalculated.

The database is stored as a gzip compressed JSON object.

$ zcat database.db | jq .
{
  "type": "home-gallery/database@1.0",
  "created": "2020-09-26T21:31:19.028Z",
  "data": [
    ...
    {
      "id": "f29f407905f8af94ece4720be997fd291adea487",
      "type": "image",
      "date": "2017-12-22T10:16:51.820Z",
      "files": [
        {
          "id": "f29f407905f8af94ece4720be997fd291adea487",
          "index": "index",
          "type": "image",
          "size": 3408759,
          "filename": "preview-test/files/camera/IMG_20171222_111649.jpg"
        }
      ],
      "previews": [
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-128.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-1280.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-1920.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-320.jpg",
        "f2/9f/407905f8af94ece4720be997fd291adea487-image-preview-800.jpg"
      ],
      "year": 2017,
      "month": 12,
      "day": 22,
      "width": 4864,
      "height": 2736,
      "orientation": 1,
      "duration": 0,
      "make": "LEAGOO",
      "model": "T5",
      "iso": 1056,
      "exposureMode": "Auto",
      "focalLength": 3.5,
      "focalLength33mm": -1,
      "latitude": 0,
      "longitude": 0,
      "altitude": 0,
      "whiteBalance": "Auto",
      "similarityHash": "KuSqiWXWVVpqXGmJWU2JGlJula1epWlWaWmVJVQKaUZqIpWklFVJpoliaWqWWFoIZtqakN2VqFiSmWVFVGpilmKlWRYRdJplila3VirmiahlSyU5SaA="
    },
    ...
  ]
}

Events

All user interaction (currently limited to image tagging) are stored in a event database.

Events are stored as plain line-delimited JSON. This data contains only manual actions and should be treated with care.

$ cat events.db | jq .
{
  "type": "home-gallery/events@1.0",
  "created": "2020-09-06T06:57:17.507Z"
}
{
  "id": "541c203a-bccc-455c-babd-4bcd7858f3b9",
  "type": "userAction",
  "targetIds": [
    "f29f407905f8af94ece4720be997fd291adea487"
  ],
  "actions": [
    {
      "action": "addTag",
      "value": "awessome"
    }
  ],
  "date": "2020-10-07T07:04:46.912Z"
}
...