I can confirm this (also on Win7) with the recent master build. When you have that option ticked and double-click on another bib file, a second process spawns and the bib file is opened in the first process (as intended). After this has happened, the second process should shutdown on its own, which it does not. I also get the message "Arguments passed on to running JabRef instance. Shutting down." in the log, but the code that follows this message seems to be insufficient to stop the process.
The feature was working fine when I last looked at it in the 3.X builds. Might this be related to the switch to JavaFx? Maybe some GUI-bootstrapping that does not finish on its own?
Can another person reproduce this behavior?