Add org-cite support

[bdarcus]

Native org citation support, called the org-cite module, has almost wrapped up, and the feature will be merged to master "any day."

For org-roam, OC introduces citation and citation-reference objects to org-element, and the main oc.el module file includes functions like org-cite-list-keys that will return all citation keys in a document. So indexing these will be trivial.

Beyond that, OC is designed around flexible and modular "capabilities"; currently:

• activate (fontification, previews, etc.; relevant for org-roam; see csl-activate, whose functionality is likely to be integrated into the core oc-csl processor)
• follow (actions tied to org-open-at-point; this is the one most relevant to org-roam-bibtex)
• insert
• export

One configures these "processors" like so:

(setq org-cite-activate-processor 'basic)

So it will be a highly flexible, modular, system.

Here, for those curious doom users, is my draft PR to add support to its biblio module:

doomemacs/doomemacs#5212

PS - I recently learned that org-ref requires both ivy and helm, which I find really problematic, as I don't use either. I guess you will have a decision to make about its status going forward. I would hope that if you continue to support it, it becomes optional in ORB.

[myshevchuk]

Hi, thank you. I can't promise that I will be able to look at it until the org-cite feature is merged into master, but once it is, ORB will definitely accommodate for it. Moreover, I believe there is very little that needs to be done on the ORB side to support org-cite. So to mimic Org-ref's behaviour, one just has to add an activate processor and point it to orb-edit-note, which as of now can be readily done on the user side - there is no need to change anything within ORB. But surely ORB should take care of it automatically and define such a processor itself.

Now, the question of really supporting org-cite is a question to Org-roam @jethrokuan. Org-roam should be able to recognize org-cite elements and index them as "ref" links. As was discussed on Discourse some time ago, there is not too much work to be done, but that work means changing internals of Org-roam. The question whether to continue to support Org-ref citations is therefore also a question to Org-roam. Also, Org-ref is not a hard dependency of Org-roam. One is not forced to to install it to be able to use Org-roam, but one of course won't be able to elevate Org-ref citations to "ref" links. I believe it would be possible to keep support for both Org-ref and org-cite, and Org-ref is already optional.

Then of course there is little sense for ORB to support org-cite (to define an activate processor) until the real support is introduced into Org-roam.

[bdarcus]

Yeah, I was wondering how much really needed to be done.

The main oc.el module file includes functions like org-cite-list-keys that will return all citation keys in a document. So indexing will be trivial.

I guess I should have posted this on the main repo tracker.

WDYT @jethrokuan? Feel free to move this if you think better.

So to mimic Org-ref's behaviour, one just has to add an activate processor and point it to orb-edit-note, which as of now can be readily done on the user side.

To be sure, this piece will also be trivial.

But it would actually be a "follow" processor.

I will be adding one that wraps the bibtex-completion stuff, including it's open-notes function, which works with OR/B.

emacs-citar/citar#156

The basic idea there is those then get bound to org-open-at-point when on a citation, or citation-reference.

PS - I edited the main post with some of this info.

[bdarcus]

Org-ref is not a hard dependency of Org-roam.

No, but it is in ORB. That's what I was meaning here.

[myshevchuk]

Org-ref is not a hard dependency of Org-roam.

No, but it is in ORB. That's what I was meaning here.

No, ORB doesn't really depend on Org-ref. Well, it's complicated. There is no (require 'org-ref) statement in ORB code (a hard dependency), this is delegated to the user. In Package-Requires header, Org-ref is indeed listed as dependency but this was introduced to automate installation of Org-ref because some users were not aware it was required by Org-roam to enable cite:links "refs". The only Org-ref function that ORB uses, apart from what is necessary to "integrate" Org-ref, is org-ref-format-entry, which is just a formatting function and can easily be replaced.

So what is happening here is that ORB depends on Org-roam being able to recognize Org-ref cite:links as "refs". But Org-roam does not depend explicitly on Org-ref, so ORB is forced to propagate it as a dependency, although conceptually it's not. So rest assured, Org-ref is not and will not be a strict logical dependency for ORB. As soon a Org-roam does not intrinsically depend on Org-ref, ORB will immediately make it absolutely optional.

[bdarcus]

Just by way of update, @myshevchuk @jethrokuan, org-cite is now in org-mode master.

[bdarcus]

Looking again at org-cite-list-keys, I realize now it is currently only for the export process.

But that code is super simple:

(defun org-cite-list-keys (info)
  (delete-dups
   (org-element-map (org-cite-list-citations info) 'citation-reference
     (lambda (r) (org-element-property :key r))
     info)))

[bdarcus]

Tentative discussion of plans for oc-focused org-ref 3.0.

jkitchin/org-ref#885 (comment)

[myshevchuk]

@bdarcus Thank you! I may not always respond in time but I'm trying to follow the updates regularly. Unfortunately, I'm really very busy now - I have to finish my PhD thesis by the end of August - and will probably have very little time for new ORB features until then. On the other hand, reviewing and merging pull requests should not be that much time consuming, so you are welcome to submit them!

[bdarcus]

Best of luck with that!

I'd like to hear from @jethrokuan on what he's thinking for org-roam proper. It seems that's the first step?

IUC, it really just means indexing oc citation-references?

Thinking more, with everything rapidly coalescing around org-cite, I'm not sure: are any changes needed in ORB?

Open notes functionality already works with bibtex-actions-org-cite, and that simply wraps bibtex-completion-open-notes.

But on this, one easy change is to remove this dependency. I really don't want that package installed for me, particularly when it, in turn, installs helm AND ivy, and will likely soon be obsolete anyway.

In Package-Requires header, Org-ref is indeed listed as dependency but this was introduced to automate installation of Org-ref because some users were not aware it was required by Org-roam to enable cite:links "refs".

[bdarcus]

Updated video on org-ref functionality for oc.

https://www.youtube.com/watch?v=3TBYg1MZ6iY

And new repo:

https://github.com/jkitchin/org-ref-cite

He recently added a function to update org-ref cite links to org-cite citations.

[bdarcus]

Now that OR has OC support, I guess the question is what ORB might add.

So org-ref-cite (whose future is now unclear) has a hydra-based "follow processor", and bibtex-actions an embark-based one.

Each provides access to functions that take one-or-more keys-at-point as input. I expect to some degree the functions could be shared between them, though I haven't yet tried myself.

I have also made my bibtex-action-open-notes function rely on a defcustom for that function, so that a user could swap org-edit-notes etc.

[myshevchuk]

What needs to be updated in ORB is its ability to treat Org-cite citations in ROAM_REFS in the same way it treats Org-ref citations. That is it should be able to recognize Org-cite entries in Org-roam database and determine whether to create a new note or take the user to the existing one.

It should also be able to provide its "note actions" for an Org-cite-powered note as well as create new notes and put Org-cite citations in ROAM_REFS.

It should also provide a function for an Org-cite follow processor to create/open a note.

The above is more of a roadmap for myself. I'll look at it as soon as possible.

[OmarAshkar]

@bdarcus what is the workflow of using bibtex-action with orb? How to make note editing go through org-roam templates?

[bdarcus]

@bdarcus what is the workflow of using bibtex-action with orb? How to make note editing go through org-roam templates?

Discussed in the README, but:

(setq bibtex-actions-file-open-note-function 'orb-bibtex-actions-edit-note)

But the default notes function does work with org-roam. It just doesn't use its template system (uses the internal one).

I've contemplated adding a notes function for org-roam using its template system, but don't have the knowledge or time ATM.

[OmarAshkar]

@bdarcus I worked fine with a bothersome issue. Sometimes, when pressing C-; then n, It asks for capturing again. If I just choose a different file, a new note is created. If I used a custom template for the bibliography, a new chunk of text is inserted. Is there is any way to avoid that?

[myshevchuk]

@OAshkar Can you reproduce this issue or does it just happen randomly?

[OmarAshkar]

@myshevchuk It happens every time I select the note option from embark-act. I got redirected to the capture templates and after I choose a template, it appends to the created note.

[chuxubank]

@OAshkar Did you try org-roam-bibtex-mode ?

[OmarAshkar]

@chuxubank yes, I check it is running and still same thing.

[myshevchuk]

@OAshkar Most probably some configuration issue on your side. I'll need more information to debug it. It is in any case unrelated to Org-cite support, so please either open a new issue or post the question on Discourse

[OmarAshkar]

@myshevchuk Thank you. I will check my configs. If I failed to find the problem, I will open an issue with my config.

[myshevchuk]

Orb note actions now recognize Org-ref v3 [[cite: &links]] and Org-cite [cite:@elements] in addition to the old Org-ref v2 plain cite:links. See #220 and #222.

[myshevchuk]

Another piece of Org-cite support here: #224

[samspo]

Hello @myshevchuk ,
I use the org-cite format [cite:@elements]
orb-note-action works
orb-insert-link works too
However, org-open-at-point brings me to an hydra menu (from org-ref, I suppose) and it's not able to perform most of the actions as it says:

Bibtex-completion couldn't find entry with key "@elements".

It would work with the following format cite:elements, however it is not the org-cite format.
Maybe I'm doing something wrong. Any idea?

[myshevchuk]

Hi, @SampSo.

orb-note-action works
orb-insert-link works too

That's it about Org-cite support in ORB.

Maybe I'm doing something wrong. Any idea?

Yes, you're trying to use Org-cite citations with Org-ref. Org-ref does not support Org-cite citations.

However, org-open-at-point brings me to an hydra menu

That's not ORB's task. You can take a look at Citar, which provides a comprehensive support for Org-cite citations, in particular the familiar Org-ref-like behavior. I believe, you also need Embark for this, but please consult Citar's docs for more details.

if you use Org-ref + Helm/Ivy-bibtex + Org-roam, ORB provides integration for that.
If you use Org-cite + Citar + Org-roam, ORB provides integration for that too.

[samspo]

Thanks a lot for the follow up.
I should then switch to org-cite + citar (I tried before, but I wasn’t able to configure it correctly ; I’ll try again)
Also, in my config I have org-ref :disabled (with use package), and I don’t know why it is loaded. I’ll have to check that (I’m wondering if org roam bibtex don’t load it somewhere)
Thanks again!

[myshevchuk]

I’m wondering if org roam bibtex don’t load it somewhere

ORB does attempt to load Org-ref when org-roam-bibtex-mode is turned on. But I added this only yesterday. It wouldn't load Org-ref in earlier versions.

[bdarcus]

FWIW, it's not a problem to use org-ref with org-cite.

But if you use org-cite and citar, ORB itself becomes optional.

[myshevchuk]

it's not a problem to use org-ref with org-cite.

That's true.

But if you use org-cite and citar, ORB itself becomes optional.

That's true also for the case of the Org-ref and Helm/Ivy-bibtex tandem. All these packages, Citar included, primarily target BibTeX bibliography management and citation management in Org files. They also have their own capabilities for note management, which is rather rudimentary compared to Org-roam's though. ORB's main task is to force all these packages to use Org-roam for note management instead of their built-in mechanisms. ORB's main feature is enabling expansion of BibTeX keywords in Org-roam templates and none of the above packages can do that.

All other features of ORB like inserting links/citations or accessing additional actions on a note are secondary convenience features resulting mainly from user requests. They also attempt not to duplicate the existing functionalities of other packages but rather to utilized them. So I'd say it depends on the workflow. For a note-centered workflow, ORB is a natural extension to connect Org-roam with bibliography/citation management packages. For a citation/BibTeX-centered workflow, both ORB and Org-roam become optional, since all these packages have their own note management systems.

[bdarcus]

ORB's main feature is enabling expansion of BibTeX keywords in Org-roam templates and none of the above packages can do that.

Yes, citar doesn't use org-roam templates. It could, but it's not a priority, in part because, as you say, ORB provides that.

For a citation/BibTeX-centered workflow, both ORB and Org-roam become optional, since all these packages have their own note management systems.

I would separate ORB and org-roam though. I'd say:

• use org-roam if you want backlinks etc that include citations (now including org-cite)
• use ORB if you want to use org-roam's template system

[samspo]

Thanks for the clarifications!

[premopie]

Hello everyone! First of all, thank you very much for the ORB! I would like to share a small idea regarding the discussion between @myshevchuk and @samspo.

That's not ORB's task. You can take a look at Citar, which provides a comprehensive support for Org-cite citations, in particular the familiar Org-ref-like behavior. I believe, you also need Embark for this, but please consult Citar's docs for more details.

if you use Org-ref + Helm/Ivy-bibtex + Org-roam, ORB provides integration for that. If you use Org-cite + Citar + Org-roam, ORB provides integration for that too.

Maybe, to have a nice environment utilizing Org-cite + Helm/Ivy-BibTeX + Org-roam + ORB it would be enough to have the orb-note-actions recognizing the citation at point?

Suppose I have an Org document containing Org-cite element

[cite:@some_paper]

Having a citation at point orb-note-actions search for the note file associated with the citekey instead of referring to ROAM_REFS property?

If someone finds it a bit confusing (orb-note-action becomes context-sensitive) maybe adding an additional function (let's say orb-cite-action) would be OK?

[myshevchuk]

Hi @premopie, I'm glad that you like it!

Having a citation at point orb-note-actions search for the note file associated with the citekey instead of referring to ROAM_REFS property?

That would be easy to implement, but the idea of the orb-note-actions command is that, as its name implies, it provides actions for an Org-roam-bibtex note. I'm reluctant to change that and would rather stick to the DRY principle. There is Org-ref for cite:citekey citations and Citar for Org-cite citations. So if you want some actions on [cite:@some_paper] citations, take a look at Citar.

[myshevchuk]

You can also use Citar as a replacement for Helm/Ivy-bibtex.

[premopie]

Thanks for prompt reply.

That would be easy to implement, but the idea of the orb-note-actions command is that, as its name implies, it provides actions for an Org-roam-bibtex note. I'm reluctant to change that and would rather stick to the DRY principle.

Your approach is reasonable.

I don't know Citar well; as far as I can see the user is recommended to use Vertico or Selectrum which means that I would have to duplicate the completion ecosystem. What I need at the moment is to reach a roam note associated with a given citation quickly, for this reason, I wrote something like this

(defun pr/orb-open-associated-note ()
  (let*
      ((element (org-element-context))
       (citekey
	(cond
	 ((eq (org-element-type element) 'citation)
	  (org-element-property :key (car (org-cite-get-references element))))
	 ((eq (org-element-type element) 'citation-reference)
	  (org-element-property :key element))
	 (t
	  (user-error "Cursor outside a citation element!")))))
    (orb-edit-note citekey)))

and bound it conveniently with some key. Now I'm able to jump quickly to (or create new) note and optionally call orb-note-action there for additional actions.

All in all, I can confirm that ORB already offers basic support for org-cite.

In my opinion, there is some void space between Org-cite + Helm/Ivy-BibTeX + ORB ecosystem for implementing Org-cite processors.

[bdarcus]

In my opinion, there is some void space between Org-cite + Helm/Ivy-BibTeX + ORB ecosystem for implementing Org-cite processors.

Absolutely true. It should be pretty trivial to fill that void (namely writing "ivy-bibtex" and "helm-bibtex" follow processors) , but alas, nobody has yet done so.

Probably the easiest approach would be to just borrow some code from the org-ref-cite follow processor, and the citar code.

[bdarcus]

Oh, and on this:

... as far as I can see the user is recommended to use Vertico or Selectrum which means that I would have to duplicate the completion ecosystem.

Citar is based on the built-in completing-read API, and is designed the exploit and fit in well with the new generation of packages also based on that API (Embark, etc).

For people committed to ivy or helm, though, it's maybe not the best choice.

Still, it's a pretty cool ecosystem of packages worth giving a spin.

[myshevchuk]

In my opinion, there is some void space between Org-cite + Helm/Ivy-BibTeX + ORB ecosystem for implementing Org-cite processors.

What I need at the moment is to reach a roam note associated with a given citation quickly, for this reason, I wrote something like this

Looking at it from a different perspective, now I see why such a function may be included in ORB. Like for people who don't have Citar installed for this or another reason. Since ORB is about Org-roam notes and it cares about people being able to reach their notes from everywhere and since Org-roam and ORB do recognize Org-cite citations as valid "ROAM_REFS", I think it would be perfectly OK to include your function in ORB for the time being.

In principle, the concept of ORB Note Actions may also be extended to citations. Should it be the case, however, Org-ref citations should be supported as well - like providing a common interface to both citation types.

In any case, you are welcome to submit your function in a PR.

I don't know Citar well; as far as I can see the user is recommended to use Vertico or Selectrum which means that I would have to duplicate the completion ecosystem.

Citar conceptually consists of two parts, maybe even more, I don't know it well enough yet: 1) a completion interface for BibTeX entries like Helm/Ivy-bibtex; 2) citation processor like Org-ref but for Org-cite citations. Vertico or similar packages are recommended if you want an enhanced experience for part 1), and yet even so they are optional. You can use Citar without them. For part 2) you only need Embark, although I'm not 100% sure it is absolutely required. Embark is an interesting package in itself and may be useful in other contexts. I don't use it much myself though. So you may give Citar a try anyway. It greatly extends default Org-mode Org-cite processors in part 2.