Unlinked File Importer: Does not import all importable files

Help
1

Hey there,

there seems to be a problem with importing unlinked local files in bulk.

JabRef 5.4–2021-12-20–ab44182
Linux 5.11.0-44-generic amd64
Java 16.0.2
JavaFX 17.0.1+1

If I want to import a lot of unlinked local files, I open the search for unlinked local files dialog, select the directory and click search. It then tells me there are 760 Documents (I select all). After clicking on ‘Import’ it imports a few files while the loading bar fills up and it reports (in the Import Results) that all files have been imported. BUT the group I imported them into has only grown by 19 documents instead of 760…
If I now click on search again in the still open search for unlinked local files dialog, it reports that there are still 741 Files left for importing…

Did anyone else encounter this bug? It really annoys me because i have to import ~80.000 Documents from ~100-150 directories… it is just frustrating

best regards
Christian

2

I’ve done some digging in the code with a debugger:
I tried importing 125 files
the call of importFilesInBackground() in ImportHandler returns an array of 125(126) items while Jabref shows just 80 new entries

after some further digging I believe it may be a concurrency issue

3

Thanks a lot for your report and the investigation! Would you mind opening an issue at our bug tracker so we can keep track of it ?

the call of importFilesInBackground() in ImportHandler returns an array of 125(126) items while Jabref shows just 80 new entries

And you verified that the dialog shows they have been imported successfully?
Do you get any other errors?

4

Yes of course. When I’m at work on Monday I’ll write a ticket. And maybe some screenshots of my investigation

I’m not sure. There may have been some importing errors but not that many. But as I said: I’ll report the issue in the bug tracker in more detail

Btw: this bug exists in the previous version and the respective windows versions as well

2 Likes
5

Took a bit longer but it’s done now: Unlinked File Importer: Does not import all importable files · Issue #8444 · JabRef/jabref · GitHub

I’m not sure if that’s detailed enough. I’d like to invest more time into this bug but can only do so at work. Just ask If you need more infos.

greetings

1 Like
6

If by any chance, the files are named after a common scheme AND you do have the corresponding entries in your library, you could try to cut down the workload by using regular expression search and autolink files.

E.g.

image
image870×133 4.55 KB

Help for regular expression search:

Of course, this would not solve the main problem you reported…

7

I solved that Problem prelimary by generating the bib file my self (iterating through the filesystem and creating dummy entries with just the file location, the group and the name of the file using a bash script)

As long as the full text search still works it’s acceptable. It is not optimal though

8

Oh that’s smart! Nice.

Would you mind sharing the bash script?

For all those that have to deal with similar problems, until Jabref gets fixed?

9

Yes, I can do that.

Note that the declaration of the groups itself is not part of the script.

It has some issues but the entries for most (>90%) documents were generated.

Unfortunately I cannot upload since I’m a new user. So I just use pastebin: RDS_generate_bib_entries.sh - Pastebin.com

1 Like