Collection management tools

[gantoine]

In the past few weeks, there have been a number of requests related to managing ROM files, existing collection, saves, patches, firmware, etc.

#251 #228 #227 #186 #187

@nickl- wrote a great post on the value of datafiles, which is what prompted me to start this discussion.

Since there are already a small number of ROM managers under active development out there, I feel this project should avoid trying to be a ROM manager (i.e. adding internal functionality), and instead defer to one of these existing tools.

For those who want to manage their files within RomM, we could (maybe?) wrap a command line tool like igir or oxymorom, building actions and settings into the UI that map to them, and calling them via shell subprocesses. There are also tools like RomPatcher.js which would allow us to patch files directly in the browser, but lack the management aspect of other tools.

All that being said, I'm very curious to hear people's thoughts on this approach, and their own ideas on how RomM can best help manage collections.

cc @XargonWan @binarygeek119 @IngwiePhoenix

[IngwiePhoenix]

I was confused about the sudden ping, but I appreciate it! :)

I looked at the other rom managers you linked to, and the first thing that struck me is: They just dont look good.

The reason I even started looking into managing my roms better was in part due to having nothing but a boring file structure. A rom manager could help me extend both the metadata I can use im something like Playnite or just to gather some other useful information together that is independent of the emulator I use or system I am on.

And that is another important thing for me; being independent of the system I use. I have a laptop, a desktop PC, a phone and my TV is rooted as well and I plan to buy a handheld at some point and maybe a tablet - all for their very own usecase. Having a central ROM manager means I have a much cleaner way to access my files; possibly by some API.

One thing that I always end up looking up again are the RPCS3 settings for Drakengard 3. Having a place where I can both store my dump of the game and potentially adjacend files would be super lovely - and the ability to look them up just as fast as well.

So, while I get your idea of not wanting to be "just another rom manager", I would personally love to see more capabilities added that facilitate parts of this. As a short little list, this is what comes to my mind (at 3.30am here in germany):

• Associate games with extra files or text (like notes on the Drakengard 3 settings to use)
• Add file conversion (convert PSX games to CHD, Gamecube to RVZ and alike. This'd be especially useful on tasks that take much longer; they can just chug along on a homeserver and when you get back to it, it might be done by then!)
• Add save file support. My biggest gripe with all current emulators is the sheer lack of save syncing. If romM had the feature to add save files, it might also be possible to retrive them later or update them as a pseudo-sync-system. i.e.: rclone sync + webdav endpoint?

Hope this jibberish makes some sense to you :) Thanks for this awesome project! I haven't sat down to add my ROMs yet, and I have to probably write a whole lot of mappings since I keep most of my system folder names in uppercase...

Full example.

ingwie@bigboi:/mnt/f/Retro$ tree -d -L 2
.
├── BIOS
│   └── (...)
├── Emulatoren
│   ├── ANESE
│   ├── Citra
│   ├── DesMuME
│   ├── Dolphin
│   ├── EKA2L1
│   ├── Flash Player Projector
│   ├── PCSX2
│   ├── PPSSPP
│   ├── RPCS3
│   ├── RetroArch
│   ├── VBA-M
│   ├── Vita3K
│   ├── bgb_1.5.7
│   ├── bsnes_hd_beta_10_3_windows
│   ├── ePSXe205
│   ├── mGBA-0.8.4
│   ├── mupen64plus
│   ├── touchHLE
│   └── yuzu
├── FirmwareFiles
│   ├── PS3
│   ├── PSP
│   └── PSV
├── ROMs
│   ├── 3DS
│   ├── Dreamcast
│   ├── GB
│   ├── GBA
│   ├── GBC
│   ├── GameCube
│   ├── N64
│   ├── NDS
│   ├── NES
│   ├── PC-98
│   ├── PS1
│   ├── PS2
│   ├── PS3
│   ├── PSP
│   ├── PSV
│   ├── SNES
│   ├── SWF
│   ├── Sega Saturn
│   ├── Switch
│   └── Wii
└── SaveData
    └── (...)

[gantoine]

Gather some other useful information together that is independent of the emulator I use or system I am on

What kind of information would you find useful, beyond the metadata that can be fetched from IGDB/no-intro?

Having a central ROM manager means I have a much cleaner way to access my files; possibly by some API.

It sounds like you're also looking for a quick and clean way to access those files, without using a browser. How do you see a theoretical API working?

Having a place where I can both store my dump of the game and potentially adjacent files would be super lovely

So when you search for a game and open that game's details page, you'd also see other files alongside it (patches, saves, manuals, etc.), Seems we'll need built-in file upload and/or a folder structure that can present those extra files in the UI (and make them available for download). And maybe a section for personal notes...

If romM had the feature to add save files, it might also be possible to retrive them later or update them as a pseudo-sync-system

File syncing of this nature would require something beyond the browser, like a client installed on each system. If you want save file sync today and you're savvy enough, you should be able to jerry-rig something (like rclone + samba)

Hope this jibberish makes some sense to you :) Thanks for this awesome project!

You can thank @zurdi15 for 99.9% of the work done so far, it's really impressive 👏🏼

[gantoine]

@IngwiePhoenix You'll be happy to hear some of what you've been asking for is coming soon, namely save and state file support! Without automation/sync of course, but it's a step in the right direction.

[zurdi15]

The idea of RomM was born out of the need to have a centralized and beautiful place (as I give much importance to the visuals in this kind of tools) to manage all of my roms in a very basic way, because as @IngwiePhoenix said the only thing I had was a boring folder structure + syncthing.

So apart from that, RomM should be integrated with tools like RetroMan or RomPatcher.js. I see RomM as the beautiful web client that wraps all those tools + having its own features of course.

@nickl- wrote a #102 on the value of datafiles, which is what prompted me to start this discussion.

Also this, he said that the software out there seems to be fragmented and I think RomM can be that link as he described here:

"These two worlds of organization vs manageability has remained in disconnect, which impairs the archivability of retro games into a singular source library by any definition. Hopefully RomM is here to finally make sense of all the chaos."

[XargonWan]

My two cents: it's true that we got other rom managers.... But it's not true... I mean, I tried almost all of those and they're not satisfying nor complete.
I am here because I think that RomM is the best one around and sincerly, because I couldn't do RetroMan as I wished to.

So yes, I feel like that it's ok to add more functions as in my opinion RomM can be THE solution in the future (as it's already for me.).

I know that this is the principle of the fragmentation, that I don't like, but I think that RomM is unique in its kind, expecialy if we want to improve it and make it the definitive Rom Manager of our dreams (at least mines :P).

About the syncronization issue I think it's doable with Node.js, Electron or WebRTC, am I wrong?

---

Said so, here is my two cents about how a collection system could work.

The collection is a list of one or more roms.
If the device support is introduced, these can be easly moved or removed from or to a device as the user can sync entire collections.
A rom can be in multiple collection.
The rom cannot be duplicated in a single collection.

The collection adding can be done on the rom(s) itself by "Add this rom to the collection" and then flagging one or multiple collections.
Or it can be done by searching and selecting the the roms in the collection screen and pressing the "add to this collection".

[nickl-]

At the very least RomM needs to be able to identify roms.

Rom identification concerns:

1. Roms are uniquely identified by their hashes, this is what is important to us (not their filenames, archives, collections, etc)
2. The only reliable identification metadata for roms are the dat files (for obtaining name, region, etc. mapped to hash keys)
3. Only then can additional metadata (like boxart, banners, logos, etc) be retrieved from online databases.

At the very least the following needs should be satisfied:

1. Extract roms from archives (zip, chd, z7, rar, etc)
2. Calculate hashes
3. Parse dat files
4. Map hashes to identification metadata

Perhaps this is out of scope for RomM, and will be better managed as a separate project, in the linux spirit of do one thing well. But neglecting to properly identify roms and instead enforcing a required archive naming convention for identification will be a nuisance, and is not sustainable. Deferring to third party libraries therefor does not relieve RomM from its responsibility to properly identify roms based on hashes.

Because these are time consuming, they need to be multi-processed to fully utilize available resources. At which point other rom management concerns like repackaging, collection management, naming conventions, duplicate identification, etc. also becomes important. Since they too require the above processes to first be completed, and retaining a cache of extracted archives is not feasible. Making it increasingly difficult to defend an argument that these requirements are out of scope for RomM.

[gantoine]

At the very least RomM needs to be able to identify roms.

I'm going to play devil's advocate here a bit (even though I agree with the spirit of your post). RomM, as it currently stands, is a ROM manager, an organizer, a library for your game files (and soon your saves, states, etc.). Collection management (for media outside of games) has historically been a fairly manual task. Even today you wouldn't give Sonarr an .mkv file called killers_movie_indian_2023 and expect it to understand that it's a copy of Killers Of The Flower Moon (2023) by looking at the file hash/metadata/pixels (well at least not yet!). You need to name it something like Killers.Of.The.Flower.Moon.2023.REPACK.1080p-FLUX.mkv to make heads or tails of it.

Now an external, self-hosted service, whether built by this team or someone else (@michael-j-green 👀) that manager .DAT files and offers an API to query file hashes, is a lot more reasonable. In the spirit of Linux, this project doesn't have to concern itself with .DAT files if another one can do it better, and we can defer to it. If the app is self-hosted, RomM wouldn't even have to extract ZIP files and calculate hashes; just point to the file and let that tool do the work. At that point things like collection management, naming conventions and duplicate identification can be accomplishes without having to interact with the content of each file.

I appreciate your thoughts @nickl-!, and I'm curious to know what you think of mine!

[michael-j-green]

As someone running their own ROM manager project, I think I can chime in here.

I’ve built DAT lookups in my project, and while it certainly improves matching, it’s a huge barrier to entry for new users. On more than a few occasions, I’ve had to guide users through loading in the DAT’s from TOSEC and a few other locations despite documenting it.

I’ve concluded from this that trying to do DAT’s on the endpoint server is not a good idea. It’s great for those users who are quite capable, but for your average punter who just wants to play retro games, it starts to be a big ask - especially after we’ve asked them to get IGDB keys and run them through docker deployment.

I agree with @gantoine in that DAT lookups should be an external service.

@gantoine called me out here specifically because I’ve been working on such a service. Sadly with life getting in the way, it’s not ready for release yet.

[nickl-]

I appreciate your thoughts @nickl-!, and I'm curious to know what you think of mine!

Okay I'll bite...

I understand this project takes inspiration from managing other media (like movies, books, comics, etc.) but roms aren't the same as other media.

Roms are data dumps, and even if you go back to the source you will be able to reproduce a file with exactly the same hash every time. This is why we rely on hashes to identify roms, regardless of your naming convention (or lack thereof).

You cannot scan the same document twice to reproduce the same data file, nor will you be able to reproduce the same music or video encoding from source again. You may be able to identify a specific release by hash, but even just changing the container or format will produce an unidentifiable copy.

With roms it is important to ensure data integrity, as these are applications which are executed, any modifications to the original can have serious repercussions (security, malware, virus, etc). This is not the case for the other media types, save some exploits due to bugs in rendering software.

To conclude, other media cannot rely on hashes, whereas roms must depend on hashes, to properly identify. Therefore they should not be treated as the same.

[gantoine]

To conclude, other media cannot rely on hashes, whereas roms must depend on hashes, to properly identify.

I think we're in agreement on this point, and the argument is whether RomM (the project) should be tasked with .DAT file management and hash comparison, or if it should be delegated to another project/code base. Today I'm of the opinion that another project could do this better, cleaner, in a more isolated way, without being burdened by the structure of this one.

That being said, if someone were to open a PR adding good .DAT file management and hash comparison support, I would not turn it down out of principle, as any support is better then no support.

[I3lackshirts]

Just started my unraid server so very new to the whole idea of storing my media. I have used plex over the years but have not cared much about the organization until now. Since I have finally got that up and running I was looking for something similar for my roms and came across your project. Looks very exciting and something I will be using going forward :).

I found this discussion while doing research and wanted to provide some of my input as well. I come from the Plex side of things (instead of Jellyfin as you pointed out) so I am not familiar with if the file structure is similar or not. But what I am looking for is somewhere I can save all roms in a set (ex. NES nointro romset) regardless of region, rev, or editions. That way I can store and download the games as needed from my server rather than looking them up again on Google. Right now its seems RomM is best used as a 1G1R type of data base. Which is great but its sort of what I am doing already on my windows machine in a directory.

I wish there was a way to do something sort of like Plex for multiple versions (see: https://support.plex.tv/articles/200381043-multi-version-movies/). As you have it right now it does work to have a "Dr. Mario" folder with all the versions of roms in them. There is just no way to tag a "default" rom file as it is now. I don't know if there are any "best practices" for structuring rom files as it seems to vary, but I would propose a way similar to how you may be working on save states? Still maintaining the same structure of files, but creating a "Versions" folder that could be linked to all other versions or the same rom? I find it useful to keep everything because I have had to go back in the past to find a different version or rev for RetroAchievements or compatibility and have been keeping all files since then. Would love to keep all that on the server and pull it as needed to each device.

I did see others mentioning ROM hacks too, that I am not sure what you have planned for that (if anything yet) but it would be awesome to have some sort of way to link those too. I personally get the updated roster Tecmo Bowl games every year and it would be nice to have all the years of games I have under Tecmo Bowl.

Also if there would be a way to tag files within RomM it would make it much easier to manage and sort my games. I currently divide my ROM sets manually into 3 different folders Favs, Playable, and Rest of Set. Favs is pretty obvious and just the ones I play often or are popular that I want to try. Playable would essentially be a "1G1R" set (personally these are all English with a few in other languages as I see fit rather than a complete set). Then rest of set folder would be all the different versions or games that I keep as a backup. Right now I may just update the file names to get the "Favs" tag on them but it would be nice to have it done all inside RomM and not need to tag files by file name.

It also seems like there is a lot of solutions out there to manage ROM's that I have not heard of before so it may not be feasible or useful for emulators to do things like this easily but that would be what I would be looking for and would find the most useful for a ROM management platform. Something I can store locally and have as much of a complete set as possible, then move files to as needed.

[gantoine]

here is just no way to tag a "default" rom file as it is now

there's a ticket for this, and we do plan to work on it in the new year #404

I did see others mentioning ROM hacks too

work is ongoing on save/state support, and eventually we will tackle bios files and rom hacks, but it's a long-term feature in our backlog

if there would be a way to tag files within RomM it would make it much easier to manage and sort my games

there's an issue for that too, but you'll need to reorganize your files to match the required structure #219

sorry for the very succinct responses, as it's the holidays, but i wanted to get back to you asap before i forgot!

[bfenty]

I just found this software today and I'm currently scanning my archive of roms. I don't actually know how many I have but it's 10k+ easily.

I love the design of the software; it's beautiful. I've been looking for something like this for a very long time. I think you're doing a great job and on the right track. This is definitely going to help me organize my collection and actually find what I'm looking for.

A couple things I'd really like to see:

• The ability to modify metadata in the UI. Things like a system being detected but not matched could be easily fixed with a context menu that allows you to search the system (much like you do now with Rom matching).
• Also, hiding files/folders that are detected but shouldn't be (such as bios files).
• On that topic, might be cool to recognize bios files and link them to the system instead of as a game?
• it would be nice to be able to combine multiple instances of a rom into one entry. Maybe it's a multidisk, maybe there's a zip and a unzipped copy, maybe it's a multi part rar file.
• This is more of a dream wishlist, but would be REALLY cool if you could link to retroarch (maybe via a retroarch plugin) and use romm as the storage for retroarch. So like I could have RA on multiple clients and they all play off the network copy and save files back to romm. This is probably a lot of work and maybe out of scope; just think it would be cool.
• Edit to add one more wish list item: adding game manuals. I mean, I would assume I would need to supply that myself, but having it linked in the front end ui so I can read the manual (likely a pdf) or at least download it would be super useful.

Thanks again for making a great project!

[zurdi15]

First of all, Happy New Year!

The ability to modify metadata in the UI. Things like a system being detected but not matched could be easily fixed with a context menu that allows you to search the system (much like you do now with Rom matching).

We have plans to make platforms more editable/manageable from the ui

Also, hiding files/folders that are detected but shouldn't be (such as bios files).

This can be currently done throught the config.yml file. You have here an example of how to do it!. In any case, hidding them from the ui would be very nice tho.

On that topic, might be cool to recognize bios files and link them to the system instead of as a game?

This is on the making, RomM is going to support savefiles, states and bios, each one in it's own section!

it would be nice to be able to combine multiple instances of a rom into one entry. Maybe it's a multidisk, maybe there's a zip and a unzipped copy, maybe it's a multi part rar file.

This is currently supported aswel. You can have a multi-disc/multi-part game as a single entry if you place them under the same folder, for example: roms/ps2/final fantasy VII/CD1.iso and roms/ps2/final fantasy VII/CD2.iso. As long as the files are under the same folder, not matter what kind of files, RomM will detect them as a single entry.

This is more of a dream wishlist, but would be REALLY cool if you could link to retroarch (maybe via a retroarch plugin) and use romm as the storage for retroarch. So like I could have RA on multiple clients and they all play off the network copy and save files back to romm. This is probably a lot of work and maybe out of scope; just think it would be cool.

This is still far from where we are now, but definetly planned to support things like this. We already support webRcade, but in the future RomM will have more ways to play integrated in it!

Edit to add one more wish list item: adding game manuals. I mean, I would assume I would need to supply that myself, but having it linked in the front end ui so I can read the manual (likely a pdf) or at least download it would be super useful.

Didn't thought about this, but we will definetly add it, liked the idea. Would be super useful if you can open an issue for it.

[michael-j-green]

This is still far from where we are now, but definetly planned to support things like this. We already support webRcade, but in the future RomM will have more ways to play integrated in it!

In my project I'm using EmulatorJS for playing ROMs in the browser. It works pretty well, and isn't difficult to integrate. The dev's are super responsive too.

[bfenty]

Thanks for the quick reply!

This can be currently done throught the config.yml file. You have here an example of how to do it!. In any case, hidding them from the ui would be very nice tho.

This is currently supported aswel. You can have a multi-disc/multi-part game as a single entry if you place them under the same folder, for example: roms/ps2/final fantasy VII/CD1.iso and roms/ps2/final fantasy VII/CD2.iso. As long as the files are under the same folder, not matter what kind of files, RomM will detect them as a single entry.

Yes, for both of these I was thinking of a way to do it in the UI as opposed to the base file structure. I realize this is more work, but hey I'm writing a wishlist, right?

Didn't thought about this, but we will definetly add it, liked the idea. Would be super useful if you can open an issue for it.

I opened an issue for this here. I think that would be an easy one and would add a lot of value.

This is still far from where we are now, but definetly planned to support things like this. We already support webRcade, but in the future RomM will have more ways to play integrated in it!

Very excited about this one. I realize that's a long ways off but would be super useful to be able to use RomM as a sort of all-in-one backend for various emulators eventually. This is something that's sorely needed in the space and I think you're on the right track already.

Again, I want to be clear-I really like where it's at already. I'm terrible at python but if you're open to it I may try to submit some pull requests if I can work on a few of the issues.

[gantoine]

PRs are always welcome! And you're welcome to join the Discord and ask any questions in the #web-ui channel.