Implement mosaic server as rust executable

[mhkeller]

I saw that you recently switched from a node js server to a python one due to issues with the arrow node library. I was wondering if you would consider packaging the server as a rust executable.

The advantage I see is that it would make it very easy to include in projects since the user wouldn’t have to manage a separate python environment or bundle dependencies.

My use case is running mosaic in an electron app and so not having to bring in a different language and dependencies would be a plus.

[domoritz]

I thought about that and it would be nice to try. A big reason for switching was the stability of the node bindings for DuckDB and I suspect the rust bindings may not be as well supported as Python. But it's worth a try. Also, we may be able to refactor the Jupyter widget code at some pint to reuse the logic from the server so if we do play with rust it would be nice to also experiment with embedding the rust server in python for the widget.

So overall I'm supportive and we could always start with an experimental package in rust. I'd be very curious about the performance differences. For example, I would like to see whether we can switch to HTTP2/3 and how that would improve performance.

Also, unrelatedly I'd love to see how mosaic works with layer cake. It's such a neat library.

[domoritz]

I'm not saying Rust has less support but I just haven't used it to know whether it has the same level as support as C++/Python. I think it would be cool to try building a rust duckdb server for mosaic on top of e.g. warp.

[mhkeller]

I appreciate all the effort!

[domoritz]

Would you like to try building a rust server? I have some scaffolding locally that I can push so you can add to it but it's just a start.

[mhkeller]

I am a rust fan but not a practitioner. It would be an interesting exercise to learn how but it would likely take a while.

[domoritz]

Totally. I just pushed what I played with which is just the scaffolding for now: #332.

[jheer]

I will just add that the Node.js server is still a possible option (npm run server:node in the Mosaic repo). As of DuckDB v0.10 it now seems to be working quite well. Hopefully that remains true.

[mhkeller]

Ah that's good to hear that they're working on some of the problems. I was going to ask if any of the bugs you had seen were being tracked in issues / tests somewhere so I could know when things were good to use.

I think the overhead of packaging the python server is a bit too much for my project and since I still have other parts of it that need developing in the meantime, hopefully, by the time I finish, the node or rust routes remain/become good solutions.

Thanks again

[domoritz]

We can have multiple implementations in parallel.

[jheer]

We've been filing bugs on DuckDB / Arrow extension as things break in the node client. If we find more maybe we can track them in parallel in Mosaic issues. Fortunately, there are no pending issues for the node client that I'm aware of! It's a relief not seeing regular crashes or weird internal cache failures :)

And FWIW while I also dislike dealing with Python package management, I have found working with hatch to be a relief. Once you install hatch everything else seems to work, without additional management, virtualenv configs, and so on. But it is also nice just to fire up the node server and have everything in JS land.

[mhkeller]

Thanks good to know!