Darktable Tags (changed, applied styles and filetype) are written in the wrong section in XMP

[rekcodocker]

Describe the bug

Since version 5, DT is writing tags in these two sections:

   <dc:subject>
    <rdf:Bag>
     <rdf:li>1. Develop Filmic</rdf:li>
     <rdf:li>changed</rdf:li>
     <rdf:li>rw2</rdf:li>
    </rdf:Bag>
   </dc:subject>
   <lr:hierarchicalSubject>
    <rdf:Bag>
     <rdf:li>darktable|changed</rdf:li>
     <rdf:li>darktable|format|rw2</rdf:li>
     <rdf:li>darktable|style|1. Develop Filmic</rdf:li>
    </rdf:Bag>
   </lr:hierarchicalSubject>

(emphasis mine)

The problem is: dc:subject and lr:hierarchical subject are used for user-tagging. Tags by me.
I apply tags to photos in another program (Digikam) and this is the section where they are stored.

The edit-information from DT now:

• shows in Digikam, mixed in with the subject-tags
• when exported, shows up in viewer-software, mixed in with the subject-tags

I think these tags should not be in this section at all. They are not 'subjects'.

Steps to reproduce

Open an image file.
Make some modifications. Apply a style.
Close DT.
Open the XMP file and see that there are now tags added to the subject and LR-hierarchical subject sections.

Expected behavior

Put this editing-information in another section of the XMP file, not in subject or LR:hierarchical subject.

Logfile | Screenshot | Screencast

No response

Commit

No response

Where did you obtain darktable from?

downloaded from www.darktable.org

darktable version

5.0

What OS are you using?

Linux

What is the version of your OS?

Linux Mint

Describe your system?

Laptop Intel based

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

Intel

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[wpferguson]

The problem is: dc:subject and lr:hierarchical subject are used for user-tagging. Tags by me.

Where is this defined? Is there a reference document or specification somewhere?

Can't digikam filter tags?

See #18154.

[rekcodocker]

Yes, there is a definition. XMP is defined in the ISO standard ISO 16684-2:2014.

From that document:

Purpose of the dc:subject Field
Field Name: dc:subject
Definition: A list of keywords or tags describing the content of the resource.
Type: Ordered array (list) of text values.
Use Case: Improves discoverability of the resource by associating it with searchable terms or thematic keywords.

'describing the content' and 'improve discoverability' - so it is not a section where proprietary editing-information or use of styles (which can be arbitrarily named) should end up.

Many other programs use this 'subject' section to display what the photo is about. It is not just that digikam should filter tags that Darktable suddenly decided to put there: This field is also used by multiple image viewers, Immich, Nextcloud, ACDC and many others. It is really interfering with their use.

This information is Darktable specifc, and not interesting for any other software and it interferes with them.

[wpferguson]

So, where should they go....

[rekcodocker]

The section

<darktable:history>
    <rdf:Seq>
     <rdf:li> 

already exists.

Option 1: put them in there, dedicated name and all. If the style can be found, you're golden.
Option 2: Not at all. Just add the actual history and not the name of the style (just the operations) or type of the rawfile (that's known already)
Option 3: Close to, next to or under darktable:history, for example a new one darktable:styleinfo

[wpferguson]

history is the history stack, and processed to restore the history. If we start messing with that, we can corrupt all image processing on import.

There are more darktable tags than just styles, that was an example. For instance there are darktable|group and darktable|group_leader to fix #16827.

After thinking, probably just darktable:tags.

What about lr:hierarchicalSubject. Do you use that?

   <lr:hierarchicalSubject>
    <rdf:Bag>
     <rdf:li>darktable|format|cr2</rdf:li>
     <rdf:li>darktable|group|277a1dad-0734-4718-8ff0-f621be63b06f</rdf:li>
    </rdf:Bag>
   </lr:hierarchicalSubject>

[rekcodocker]

history is the history stack, and processed to restore the history. If we start messing with that, we can corrupt all image processing on import.

Just meant it as an example. Better not mess with it then.

Again, history works fine. I don't know what problem is solved by storing the name of a style that only exists on my PC.
It means it has no value on someone elses PC.

There are more darktable tags than just styles, that was an example. For instance there are darktable|group and darktable|group_leader to fix #16827.

After thinking, probably just darktable:tags.

Sounds like a good idea.
The tags we are talking about are (1) 100% Darktable specific and (2) very often user- or even PC specific.
Any Darktable-unique section is fine.

What about lr:hierarchicalSubject. Do you use that?

I think it is short for "LightRoom: hierachicalSubject".
As far as I know, DT also reads or writes the subject in that field. I do know DigiKam views it as a subject-field.
I would not use any existing generic (non DT-specific) field to store Darktable-specific information.

[revifr]

From what I can see, the two fields mentioned are consistent. So no bug there. Darktable wants to add those processing items as keywords, that may not be ideal, but it's not a bug, either, imo.

What I do find less than ideal is that darktable adds tags in the xmp in a way that will pollute the tag hierarchy in another program (after all, one use for sidecars is exchange of metadata between programs).

note for @rekcodocker : you might want to check the order in which digikam looks for keyword tags in the sidecars. I have Xmp.digikam.tagslist as the first item in those for both read and write, so I do not get the darktable-specific keywords transferred (but that means you won't transfer keywords from darktable to digikam at all)

[rekcodocker]

I checked Digikam and its imports.

One problem is that each new version of Digikam overwrites the settings of the old version, so it was not set correctly.
I had to correct that first. However, after restricting the fields that it reads and writes, ( according to this page: https://userbase.kde.org/Digikam/Tutorials/Setup_of_digiKam_for_Windows_compatibility ) I still have the same problem.

Darktable inserts tags like the raw-format, 'changed' and the applied styles in a section meant to describe the image:

   <dc:subject>
    <rdf:Bag>
     <rdf:li>1. Develop Filmic</rdf:li>             *** Darktable ***
     <rdf:li>People/Family/Derrick</rdf:li>    *** Mine, this is a subject-tag *** 
     <rdf:li>Swimming</rdf:li>     *** Mine, this is a subject-tag *** 
     <rdf:li>changed</rdf:li>   *** Darktable ***
     <rdf:li>orf</rdf:li>   *** Darktable ***
    </rdf:Bag>
   </dc:subject>
   <lr:hierarchicalSubject>
    <rdf:Bag>
     <rdf:li>Activity|Sport|Swimming</rdf:li>   *** Mine, this is a subject-tag *** 
     <rdf:li>People/Family/Derrick</rdf:li>   *** Darktable ***
     <rdf:li>darktable|changed</rdf:li>   *** Darktable ***
     <rdf:li>darktable|format|orf</rdf:li>   *** Darktable ***
     <rdf:li>darktable|style|1. Develop Filmic</rdf:li>   *** Darktable ***
    </rdf:Bag>
   </lr:hierarchicalSubject>

These are so deeply in to the hierarchy that I can not filter them out.

From what I can see, the two fields mentioned are consistent. So no bug there. Darktable wants to add those processing items as keywords, that may not be ideal, but it's not a bug, either, imo.

XML (or XMP in this case) leaves a lot of room for a lot of hierarchy and fields. So why do they have to be in a place that, according to the specification, is meant for something else?
Include the tags, by all means. By why here? This is against specification.

What I do find less than ideal is that darktable adds tags in the xmp in a way that will pollute the tag hierarchy in another program (after all, one use for sidecars is exchange of metadata between programs).

Indeed, that is causing the problem. It is however not just another program.
As far as I can see the other program is following the XMP-specification.
'Subject' really means something different according to the XMP specification.
The solution is not to say 'oh but darktable finds it very important'.

[revifr]

On dimanche 2 février 2025 14:21:02 heure normale d’Europe centrale rekcodocker wrote: I checked Digikam and its imports. One problem is that each new version of Digikam overwrites the settings of the old version, so it was not set correctly. However, after restricting the fields that it reads and writes, I still have the same problem. Darktable inserts tags like the raw-format, 'changed' and the applied styles in a section meant to describe the image: ``` <dc:subject> <rdf:Bag> **<rdf:li>1. Develop Filmic</rdf:li>** <rdf:li>People/Family/Derrick</rdf:li> <rdf:li>Swimming</rdf:li> **<rdf:li>changed</rdf:li> <rdf:li>orf</rdf:li>** </rdf:Bag> </dc:subject> <lr:hierarchicalSubject> <rdf:Bag> <rdf:li>Activity|Sport|Swimming</rdf:li> <rdf:li>People/Family/Derrick</rdf:li> **<rdf:li>darktable|changed</rdf:li> <rdf:li>darktable|format|orf</rdf:li> <rdf:li>darktable|style|1. Develop Filmic</rdf:li>** </rdf:Bag> </lr:hierarchicalSubject> ``` These are so deeply in to this field that I can not filter them out. > From what I can see, the two fields mentioned are consistent. So no bug > there. Darktable wants to add those processing items as keywords, that > may not be ideal, but it's not a bug, either, imo. XML (or XMP in this case) leaves a lot of room for a lot of hierarchy and fields. So why do they have to be in a place that, according to the specification, is meant for something else? Include the tags, by all means. By why here? This is against specification.

It's not, those fields like "<rdf:li>changed</rdf:li>" are keywords and added as such. That's perfectly alright as far as the XML specification is concerned... And, as they are keywords, they should be within those XML tags.

>What I do find less than ideal is that darktable adds tags in the xmp in a >way that will pollute the tag hierarchy in another program (after all, one >use for sidecars is exchange of metadata between programs). Indeed, that is causing the problem. It is however not just another program. As far as I can see the other program is following the XMP-specification. 'Subject' really means something different according to the XMP specification. The solution is not to say 'oh but darktable finds it very important'.

Again, adding keywords is *not* against the XML specification, nor, I suspect, against the specification for this field. And perhaps XML defines a tag "Subject", but that is irrelevant here, as we are talking about the tag "dc:subject", i.e. a new tag defined in a namespace "dc:"; that namespace was defined earlier as 'xmlns:dc="http://purl.org/dc/elements/1.1/"'... That you don't like darktable adding keywords is perfectly fine, but it doesn't make the addition "against specification." At least with digikam, you can prevent import of the "dc:subject" field, with the "lr:hierarchicalSubject", you'll get all darktable's tags in a separate tree, with root "darktable". Not too horrible as pollutions go. Remco

[github-actions]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[rekcodocker]

I feel we're going back and forth trying to be 'right' or trying to 'win'.
It really is an issue for me and I can not fix it in Digikam. This is the only section I can use to send descriptive tags back and forth between these to programs.

If I sum up, you argue two things: 1 - it is (or may be) according the specification and (2) look to Digikam for the solution.

(1) How do you read the XML spec?
Discussion between developers makes it clear: They are using the XMP as a last resort backup for the database.
I am happy it works that way.
But this XMP section exists for "keywords or tags describing the content" and to "improve discoverability". That's the spec. That's the literal spec.
This section is not for database backups.
The spec states (literal quote): A list of keywords or tags describing the content of the resource. Use Case: Improves discoverability of the resource by associating it with searchable terms or thematic keywords.

I have a picture of an apple. It's labelled 'food' and 'apple'. During editing Darktable assigns a tag "styles|enhanceColor".
This does not help finding the image.

(2) Removing the import from digikam is not possible. This section is essential for content-describing tags.
It is the only section that both Digikam and Darktable read for content-describing tags.
Now Darktable is using it as a backup.

Most frustrating are two things:

1. Darktable already has its own section: darktable:history. It's used to record all operations on an image.
   If I apply exposure compensation, it goes in 'darktable:history'.
   But if I apply a via a style, it suddenly goes into the description of the subject of the image.
   How does this make sense? Both are the same operation on the image. Why does it make a difference how I applied it?

2. Why is the style_name_ remembered at all? Styles are extremely specific to the computer that the image was edited on. Open the file on another computer and that style means nothing. Change the style name, or the style itself - and you don't know what the style "style|enhanceColor" refers to.
   Style is like a macro: It's a number of operations in modules. Remember the operations, but not the name of the macro.

[wpferguson]

darktable XMP files are written by darktable, for darktable.

Their purpose (from darktable's point of view) is:

• to be the database backup of last resort. If you lose your database, you can rebuild it by reimporting the images and their XMP files.
• a way to transfer images with their corresponding information between darktable instances. I use this. I have a current release of darktable that stores all my processed images. I also ride master and do all my image development their to test the code and find bugs. When the next release occurs I update the current darktable, then move my images and XMP files over to it and import them.
• a way to transfer image processing information (PlayRaws)
• in some instances a debugging tool.

The XMP files may be used for other purposes, such as metadata management, timelapse generation etc.

---

Why is the style_name_ remembered at all? Styles are extremely specific to the computer that the image was edited on

darktable ships with 534 styles included. The code that applies those styles checks the style tag to see if the style has already been applied, so that it's not applied twice.

Darktable already has its own section: darktable:history.

This is processing information. It is used to reconstruct the processed image.

Why is the style_name_ remembered at all?

This is state information. State information is used to keep information about the image such as track of styles applied, grouping, processing status, etc. The only way to keep state information between darktable instances is using tags.

---

Why hasn't this been "fixed" yet?

• it's not a problem for any of the developers.
• Nobody knows how to fix it
• Nobody has time to fix it.

I'll speak for myself.

• While I don't use tagging, I do care about the state information and need it to make the code work.
• I don't know the spec, and therefore I don't have any proposed hierarchy to make this work. Once I have a hierarchy, then I have to figure out where to put it, how to put in, how to make it work, how to maintain backwards compatibility, how to...
• I don't have time. I'm booked solid until August. I'm in the midst of porting a 12 year old rails application. There's 20 weeks of work left, but it's needed in 10 so every moment I have is spent working on that. Once that's done I'll start on the backlog of darktable work I shelved and then I can work on this.