lib,src,test,doc: add node:sqlite module

[cjihrig]

#53264 has been open for over a month with no objections, so I am opening this PR with an initial node:sqlite module. There is other functionality that could potentially be exposed in the future, but I believe this is enough for an experimental MVP.

Fixes: #53264

Summary: Node.js now includes a built-in sqlite module (require('node:sqlite')) that becomes available when using the --experimental-sqlite flag

The following example shows the basic usage of the node:sqlite module to open
an in-memory database, write data to the database, and then read the data back.

import { DatabaseSync } from 'node:sqlite';
const database = new DatabaseSync(':memory:');

// Execute SQL statements from strings.
database.exec(`
  CREATE TABLE data(
    key INTEGER PRIMARY KEY,
    value TEXT
  ) STRICT
`);
// Create a prepared statement to insert data into the database.
const insert = database.prepare('INSERT INTO data (key, value) VALUES (?, ?)');
// Execute the prepared statement with bound values.
insert.run(1, 'hello');
insert.run(2, 'world');
// Create a prepared statement to read data from the database.
const query = database.prepare('SELECT * FROM data ORDER BY key');
// Execute the prepared statement and log the result set.
console.log(query.all());
// Prints: [ { key: 1, value: 'hello' }, { key: 2, value: 'world' } ]

[nodejs-github-bot]

Review requested:

• @nodejs/gyp
• @nodejs/loaders
• @nodejs/startup

[targos]

I can't review the C++ code, but API and tests LGTM for a MVP.

[benjamingr]

Yay, happy to see this come to fruition. Let's land as experimental and iterate 🎉

[benjamingr]

It would be nifty to also support Symbol.dispose here but that can be in a follow up PR.

[benjamingr]

This is a lot more feature complete and ready than I was expecting, good job Colin!

[RedYetiDev]

This is adding a new feature, so shouldn't be semver-minor? (and with this kind of change, notable-change as well?)

[ShogunPanda]

Love it! Amazing job @cjihrig

[mcollina]

lgtm

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/60156/

[Qard]

Should we be concerned at all about the performance impact of people using these sync APIs within requests and blocking the event loop? With sync being the only option here I feel like the risk is increased.

If the concern is simplicity, I feel like only async would be better than only sync. Especially given top-level await. What reason is there to favour sync over async here?

I won't block on that concern, and in fact I think the work done so far is great. I would be happy landing what's here now. I just wanted to share my concern with going for sync first rather than async first.

[benjamingr]

Should we be concerned at all about the performance impact of people using these sync APIs within requests and blocking the event loop? With sync being the only option here I feel like the risk is increased.

SQLite runs in process and is often CPU bound, we do have async APIs for some CPU intensive stuff (like in crypto) but I suspect for most people this API is the better one performance wise (that said, in some cases an async API is preferable and we should expose that as well)

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @RafaelGSS.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[RafaelGSS]

@cjihrig I took the liberty to create a summary in the PR description for the notable changes section in the upcoming release. Please feel free to adjust it.

[cjihrig]

Thanks @RafaelGSS.

how difficult would it be to support transferring DatabaseSync instances between worker threads, particularly :memory: instances?

@tlhunter the database could be made transferrable. I think we would want to add extra bookkeeping to clean up resources like finalizing prepared statements derived from the connection.

There is also a serialization API (not currently exposed in Node) if we just wanted to transfer the contents of the database, but I don't think that's what your question is about.

[tlhunter]

@cjihrig I'm mostly curious about using it as a shared data storage, for example if many threads wanted to simultaneously write to a shared log.

Of course this could also be achieved by message passing and having a dedicated thread do the DB writes but it seems like sharing a database might make for cleaner patterns in app code. Theoretically writing to the same in-memory DB could even result in higher throughput assuming whatever locking mechanism sqlite has is more efficient than message passing overhead.

[cjihrig]

for example if many threads wanted to simultaneously write to a shared log.

SQLite itself supports different threading modes. I believe that pattern would only be supported in serialized mode. If we made the connection transferrable, JavaScript would not allow the object to be used in multiple contexts though (transferring a prepared statement could be an option though).

You might be able to use SQLite shared cache for this in-memory use case, but I'm not 100% sure and shared cache is discouraged. Another option is to create a file-backed database in the temp directory and opening it from each thread.

[aduh95]

There are some conflicts with v22.x-staging, this would need a backport PR if we want it in 22.x.

[targos]

I found the conflicts trivial to fix. Pushed to v22.x-staging at 26a2c3a

[RedYetiDev]

@targos @aduh95 is the appropriate label backported-to-xyz, or none because a specific backport PR was not needed? (For future reference)

Thanks!

[targos]

none because a specific backport PR was not needed

[karlhorky]

If anyone wants to try out node:sqlite interactively in a sandbox, I've set up a CodeSandbox with @hemanth's node-nightly here:

CodeSandbox demo: https://codesandbox.io/p/devbox/node-sqlite-with-node-js-nightly-vygw6p?file=%2Findex.js

[karimfromjordan]

I just tested this and it's very slow and appears to ship with SQLite 3.46. Is this expected in the Nightly build?

[bholmesdev]

@karimfromjordan I think that's a limitation of codesandbox. I was unable to run the sandbox on my machine, and I was also unable to get node-nightly to succeed. Luckily, I found these instructions on how to install the latest nightly release using nvm. This worked for me:

NVM_NODEJS_ORG_MIRROR=https://nodejs.org/download/nightly/ nvm i node

Running against an index.mjs file with the snippet from those PR notes ran very fast for me. Your machine may vary of course.

[karimfromjordan]

@bholmesdev I tested it locally. I ran 1000 inserts, with WAL enabled and it completes in about 1 - 1.5 secs which is really slow. better-sqlite3 finishes the same statements in 50 ms.

[RedYetiDev]

This PR is about the addition of SQLite into Node.js. If you have concerns about its usage, please visit the help repo.

If you have concerns specific to the sandbox environment, please reach out to @karlhorky privately.

[cjihrig]

@karimfromjordan thanks for sharing. I can confirm a slowdown, but I'm not seeing anything near 20x - more like 4x. Please note this is not optimized at all yet. When I start working on this again, I'll look into improving the perf if no one beats me to it.

[mpnkhan]

This is exciting. I always loved the PHP+MySql combo. And this one is one more reason for me to love nodeJS more.

[cjihrig]

From a quick test, compiling SQLite in Node with the same settings used by better-sqlite3 yielded roughly the same performance. There may be other things to optimize as well, but it's good to know we aren't doing anything too awful 😄.