Memory consumption

Help
21

It begins to start getting philosophical :sweat_smile:.

Basically, this is an issue which cannot be solved easily.

One could rewrite in Rust. There are cool desktop applications writen in Rust. Such as Samba - a Music library interface for dancesport-oriented music libraries (and more).

However, this is not doable with the current project setting. Maybe, if we fully commercialize JabRef, there could be resources for that.

Besides a full rewrite, it is also possible to work on the architecture of JabRef. Currently, all entries are hold in memory from the loading of the library until closing of a library. With a latex free representation and other derived information. We can work on reducing this, maybe by starting to use an embedded SQL datatabase.

However, this is also a huge effort, maybe taking 5 years or more (with the current staffing).

All in all, it is a matter of priorities. Should we fix some of the 100 bugs? Should we work on one of the 50 issues with groups? Should we stop supporting external contributors? That means to stop all activities written down at JabRef and Software Engineering Training | Developer Documentation. Should we stop to provide support in this forum?

Currently, we are more working towards fixing issues and bringing in new features. We have some ground work in mind and try to fix architectural issues while implementing features (and fixing bugs). However, we cannot promise to make fast progress in a whole rewrite of the data handling.

What would help us to free resources for “tough” work, is to support us in two areas:

  1. documentation writing: We have 63 open issues: Issues · JabRef/user-documentation · GitHub. We have hosted the documentation page on GitBook to enable easy WYSIWG editing. See user-documentation/en/contributing/how-to-improve-the-help-page.md at main · JabRef/user-documentation · GitHub for detalis.
  2. issue refinement. We collect issues for external contributors at Candidates for University Projects · GitHub. All of these issues are “ready to start” . It takes from 15 minutes to several hours to get such an issue ready. (Several hours “only” if JabRef code needs to be adapted to be newcomer friendly). Most issues “just” need a good description about the intended behavior of JabRef. This takes time. If one wants to support here, please get in touch with us.
22

Great answer, @koppor. Thank you.

Basically, this is an issue which cannot be solved easily [and] … Besides a full rewrite, it is also possible to work on the architecture of JabRef.

This is how I understood the situation initially, though I have sometimes wondered if something else was going on, such as issues due to Wayland + Nvidia + fractional scaling on my system. What I meant to communicate was not a complaint but that it can be tricky for non-developers to identify the source of perceived performance issues and hence, know what to investigate and when to report a possible bug.

Currently, all entries are hold in memory from the loading of the library until closing of a library.

That’s a lot of data in my case, so it looks like JabRef is working as expected.

Also, I was probably wrong about JabRef becoming unresponsive when other apps did not. I now believe the real issue was a problem with the akonadi service.

What would help us to free resources for “tough” work

I am working on this by degrees. I’ve read all the linked resources and followed the dev setup. I don’t work as a developer and large contributions are not feasible at the moment, but I keep an eye on the issues and try to contribute what I can. So far, those “contributions” have surely taken more time than they have given. :person_shrugging:

2 Likes
23

(e.g. if you can see that RAM usage is at 95 - 100% and lots of HDD/SDD read and write operation exist and your swap space is used as well), then you might want to look into ways to acquire more RAM or cut down on the applications in active/passive usage.

FYI, my specs look like this:

Core™ i7-8550U CPU @ 1.80GHz (4 cores, 8 threads)
Intel UHD Graphics 620 + NVIDIA GeForce MX150
Western Digital Black sn850 SSD

Typical conditions are like this, with a 5.5 MB library open in JabRef. CPU load varies, of course, and swap usage is 0%. Obviously, library size is responsible for JabRef’s memory consumption here.

image
image1760×1758 256 KB

24

For a systematic approach, one would need to to work on org.jabref.benchmarks.Benchmarks. Meaning: Benchmark the creation of BibEntry object systmatically. How does it behave with LaTeX commands etc? Then benchmark a whole library: Investigate “properties” of a Bibdatabase object. This way, hopefully, the first point(s) of optimization can be identified.

An initial hint was given at java - Count metrics with JMH - Stack Overflow, but still not a full-detail answer.

I am hoping that one can solve the issues. In general, we try to use different JVMs. GraalVM, because it promises higher execution speeds (Native compile [GraalVM] by calixtus · Pull Request #11298 · JabRef/jabref · GitHub) and Eclipse J9, because its inclusion is less effort than GraalVM but also promising lower footprint (Switch to Eclipse OpenJ9 (to gain performance) by koppor · Pull Request #481 · koppor/jabref · GitHub). – I am fearing a bit that the garbage collection of objects does not work “properly”, but let’s see how experiments with JMH and other JVMs continue.

25

I figured out that search-groups and any searches that act on the whole (large) library are the main difficulty in my case. If I move a subset of the entries into an explicit-selection group, the editor becomes responsive and I can search within the group without difficulty. Working in all entries is also OK if there is no search in operation.

This makes sense, considering the size of the library and in comparison to other applications, which also struggle with searches of large files, especially when regular expressions are involved.