Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite node module handling (npm plugin) #874

Draft
wants to merge 41 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
f3fbb1b
Add foundation for a new npm plugin
marvinhagemeister Sep 11, 2021
fde2f51
Acorn: Add missing exportDefaultDeclaration
marvinhagemeister Sep 11, 2021
5908aed
Add support for commonjs default exports
marvinhagemeister Sep 11, 2021
4682b9e
Add support for legacy sub packages
marvinhagemeister Sep 12, 2021
f1dee81
Add support for auto installing npm dependencies
marvinhagemeister Sep 12, 2021
46bb3f2
Add basic support for "exports" field
marvinhagemeister Sep 12, 2021
bd516a1
Fix unable to resolve scoped packages
marvinhagemeister Sep 12, 2021
7624518
Watcher: Ignore `.cache/` folder
marvinhagemeister Sep 12, 2021
34c4f08
Cleanup npm-plugin logging
marvinhagemeister Sep 12, 2021
4f153e8
Experiment with npm autoInstall
marvinhagemeister Sep 12, 2021
7b52e2c
Fix incorrect npm auto-install cache directory
marvinhagemeister Sep 12, 2021
da3bb3b
Fix commonjs rewriting non "module.exports" assginments
marvinhagemeister Sep 12, 2021
ad65d49
Add support for commonjs proxy modules
marvinhagemeister Sep 12, 2021
2139c51
Fix duplicate download requeusts
marvinhagemeister Sep 12, 2021
ac47f47
Drop if-statement if it's unreachable
marvinhagemeister Sep 12, 2021
cea027b
Remove single top level IIFE in CommonJS bundles
marvinhagemeister Sep 12, 2021
121c772
Switch to a non-string based transpiler for commonjs
marvinhagemeister Sep 16, 2021
560c362
Fix CommonJS file not being detected
marvinhagemeister Sep 16, 2021
86ce5cb
Include json in npm bundles
marvinhagemeister Sep 19, 2021
a0aa34a
Improve npm module bundling
marvinhagemeister Sep 19, 2021
9b1d76e
NPM: Bring back disk cache
marvinhagemeister Sep 19, 2021
31419e6
Update to zecorn 0.8.1
marvinhagemeister Sep 19, 2021
d5b5a2a
Use `writeFile` helper
marvinhagemeister Sep 19, 2021
7aa7999
Remove `setCwd` hack
marvinhagemeister Sep 19, 2021
1862bbf
Fix unable to load json in commonjs package
marvinhagemeister Sep 16, 2021
65104d9
NPM: Remove file dependencies on old plugin
marvinhagemeister Sep 29, 2021
b8dd4ac
NPM: Support loading assets from node modules
marvinhagemeister Sep 29, 2021
944d74e
NPM: Add back size warning plugin
marvinhagemeister Sep 29, 2021
4a13701
NPM: Remove old npm plugin
marvinhagemeister Sep 29, 2021
c2adc8a
Bring back etag caching for npm packages
marvinhagemeister Sep 29, 2021
b791f9b
Upgrade zecorn to fix codegen issues
marvinhagemeister Oct 4, 2021
300d7fc
Allow loading assets from auto-installed packages
marvinhagemeister Oct 6, 2021
755826e
NPM: Use already extracted package if available
marvinhagemeister Oct 6, 2021
ee489af
Support auto installing versioned packages
marvinhagemeister Oct 6, 2021
151def0
Fix incorrect CLI argument casing
marvinhagemeister Oct 6, 2021
a9e22b7
Specify custom npm registry via `--registry`
marvinhagemeister Oct 6, 2021
efe3a9e
Add changeset
marvinhagemeister Oct 6, 2021
bedc4da
NPM: Default to `index.js` if no entry point is found
marvinhagemeister Oct 6, 2021
52950ec
Update zecorn to 0.9.5
marvinhagemeister Oct 6, 2021
bbe841c
Reduce test CLI noise
marvinhagemeister Oct 6, 2021
3043967
WIP
marvinhagemeister Oct 7, 2021
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add support for commonjs proxy modules
  • Loading branch information
marvinhagemeister committed Oct 6, 2021
commit ad65d4989a5b6d5f30c3665cb41b82805a1f495f
1 change: 1 addition & 0 deletions packages/wmr/src/lib/acorn-traverse.js
Original file line number Diff line number Diff line change
Expand Up @@ -835,6 +835,7 @@ const TYPES = {
importDefaultSpecifier: local => ({ type: 'ImportDefaultSpecifier', local }),
importNamespaceSpecifier: local => ({ type: 'ImportNamespaceSpecifier', local }),
exportDefaultDeclaration: declaration => ({ type: 'ExportDefaultDeclaration', declaration }),
exportAllDeclaration: (source, exported = null) => ({ type: 'ExportAllDeclaration', source, exported }),
assignmentExpression: (operator, left, right) => ({ type: 'AssignmentExpression', operator, left, right }),
variableDeclaration: (kind, declarations) => ({ type: 'VariableDeclaration', kind, declarations }),
variableDeclarator: (id, init) => ({ type: 'VariableDeclarator', id, init }),
Expand Down
2 changes: 1 addition & 1 deletion packages/wmr/src/lib/plugins.js
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ export function getPlugins(options) {
// Only transpile CommonJS in node_modules and explicit .cjs files:
include: /(^npm\/|[/\\]node_modules[/\\]|\.cjs$)/
}),
npmPlugin2({ cwd, autoInstall }),
npmPlugin2({ cwd, autoInstall, production }),
resolveExtensionsPlugin({
extensions: ['.ts', '.tsx', '.js', '.cjs'],
index: true
Expand Down
132 changes: 109 additions & 23 deletions packages/wmr/src/plugins/npm-plugin-2/commonjs.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,46 +5,132 @@ const CJS_KEYWORDS = /\b(module\.exports|exports)\b/;
export const ESM_KEYWORDS =
/(\bimport\s*(\{.*?\}\s*from|\s[\w$]+\s+from)?\s*['"]|[\s;]export(\s+(default|const|var|let|function|class)[^\w$]|\s*\{))/;

function acornCjs({ types: t }) {
return {
name: 'commonjs-transform',
visitor: {
Program(path) {
for (let i = 0; i < path.node.body.length; i++) {
const stmt = path.node.body[i];
if (
t.isExpressionStatement(stmt) &&
t.isAssignmentExpression(stmt.expression) &&
t.isMemberExpression(stmt.expression.left) &&
path.get(`body.${i}.expression.left`).getSource() === 'module.exports'
) {
path
.get(`body.${i}`)
.replaceWith(
t.variableDeclaration('const', [t.variableDeclarator(t.identifier('__default'), stmt.expression.right)])
);
path.get(`body.${i}`).appendString(`export default __default;`);
/**
* @param {{ production: boolean}} options
*/
function acornCjs(options) {
return ({ types: t }) => {
return {
name: 'commonjs-transform',
visitor: {
Program: {
exit(path) {
for (let i = 0; i < path.node.body.length; i++) {
const stmt = path.node.body[i];

if (
t.isExpressionStatement(stmt) &&
t.isAssignmentExpression(stmt.expression) &&
t.isMemberExpression(stmt.expression.left) &&
t.isIdentifier(stmt.expression.left.object) &&
stmt.expression.left.object.name === 'module' &&
t.isIdentifier(stmt.expression.left.property) &&
stmt.expression.left.property.name === 'exports'
) {
let replacement = null;

// Detect: `module.exports = require("...");`
if (
t.isCallExpression(stmt.expression.right) &&
t.isIdentifier(stmt.expression.right.callee) &&
stmt.expression.right.callee.name === 'require'
) {
replacement = t.exportAllDeclaration(t.clone(stmt.expression.right.arguments[0]));

path.get(`body.${i}`).replaceWith(replacement);
}
// Otherwise use a variable instead
else {
replacement = t.variableDeclaration('const', [
t.variableDeclarator(t.identifier('__default'), t.cloneDeep(stmt.expression.right))
]);

path.get(`body.${i}`).replaceWith(replacement);
path.get(`body.${i}`).appendString(`export default __default;`);
}
}
}
}
},
ExpressionStatement(path) {
// Remove "use strict" directive
if (t.isLiteral(path.node.expression) && path.node.expression.value === 'use strict') {
path.remove();
}
},
IfStatement(path) {
// path.replaceWithString(`const foo = 42;`);
// return;
// Detect proxy modules. They usually have the form:
//
// if (process.env.NODE_ENV === 'production') {
// module.exports = require('./prod.js');
// } else {
// module.exports = require('./dev.js');
// }
if (t.isBinaryExpression(path.node.test)) {
const node = path.node.test;
const { operator } = path.node.test;

let envMember = null;
let literal = null;
if (t.isMemberExpression(node.left) && t.isLiteral(node.right) && typeof node.right.value === 'string') {
envMember = path.get(`test.left`);
literal = path.get(`test.right`);
} else if (
t.isMemberExpression(node.right) &&
t.isLiteral(node.left) &&
typeof node.left.value === 'string'
) {
envMember = path.get(`test.right`);
literal = path.get(`test.left`);
}

if (envMember && literal && envMember.getSource() === 'process.env.NODE_ENV') {
const actual = options.production ? 'production' : 'development';
const expected = literal.node.value;

let replacement = null;
if ((operator === '===' && actual === expected) || (operator === '!==' && actual !== expected)) {
replacement = path.node.consequent;
} else {
replacement = path.node.alternate;
}

if (replacement !== null) {
if (t.isBlockStatement(replacement)) {
// TODO: Once we have a more stable parser
// in place we can prependItems
path.replaceWith(t.cloneDeep(replacement.body[0]));
} else {
path.replaceWith(t.cloneDeep(replacement));
}
}
}
}
}
}
}
};
};
}

/**
* Attempt to convert CommonJS modules to ESM
* @param {object} options
* @param {boolean} options.production
* @returns {import('rollup').Plugin}
*/
export function commonjsPlugin() {
export function commonjsPlugin({ production }) {
return {
name: 'commonjs',
transform(code, id) {
async transform(code, id) {
const hasCjsKeywords = CJS_KEYWORDS.test(code);
const hasEsmKeywords = ESM_KEYWORDS.test(code);
if (!hasCjsKeywords || hasEsmKeywords) return;

const result = transform(code, {
parse: this.parse,
plugins: [acornCjs]
plugins: [acornCjs({ production })]
});

return {
Expand Down
5 changes: 3 additions & 2 deletions packages/wmr/src/plugins/npm-plugin-2/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ import { isValidPackageName } from './utils.js';
* @param {object} options
* @param {string} options.cwd
* @param {boolean} options.autoInstall
* @param {boolean} options.production
* @returns {import('rollup').Plugin}
*/
export function npmPlugin2({ cwd, autoInstall }) {
export function npmPlugin2({ cwd, autoInstall, production }) {
const PREFIX = '\0npm:';

// FIXME: Buffer for assets
Expand All @@ -26,7 +27,7 @@ export function npmPlugin2({ cwd, autoInstall }) {

// TODO: Caching

let result = await npmBundle(cwd, id, { autoInstall });
let result = await npmBundle(cwd, id, { autoInstall, production });

result.output.forEach(chunkOrAsset => {
if (chunkOrAsset.fileName === 'virtual-entry.js') {
Expand Down
5 changes: 3 additions & 2 deletions packages/wmr/src/plugins/npm-plugin-2/npm-bundle.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,9 @@ import { npmAutoInstall } from './npm-auto-install.js';
* @param {string} requestId
* @param {object} options
* @param {boolean} options.autoInstall
* @param {boolean} options.production
*/
export async function npmBundle(cwd, requestId, { autoInstall }) {
export async function npmBundle(cwd, requestId, { autoInstall, production }) {
const meta = getPackageInfo(requestId);
const pkgName = meta.name;

Expand All @@ -41,7 +42,7 @@ export async function npmBundle(cwd, requestId, { autoInstall }) {
!process.env.DISABLE_LOCAL_NPM && npmLocalPackage({ root: cwd }),
autoInstall && npmAutoInstall({ cwd }),
npmLoad({ browserReplacement }),
commonjsPlugin(),
commonjsPlugin({ production }),
subPackageLegacy({ rootId: requestId })
]
});
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions packages/wmr/test/fixtures/npm-commonjs-proxy/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
<h1>it doesn't work</h1>
<script src="./index.js" type="module"></script>
3 changes: 3 additions & 0 deletions packages/wmr/test/fixtures/npm-commonjs-proxy/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import { value } from 'foo';

document.querySelector('h1').textContent = value;
9 changes: 9 additions & 0 deletions packages/wmr/test/npm.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,15 @@ describe('node modules', () => {
expect(text).toMatch(/it works/);
});
});

it('should inline proxy modules based on `process.env.NODE_ENV`', async () => {
await loadFixture('npm-commonjs-proxy', env);
instance = await runWmrFast(env.tmp.path);
await withLog(instance.output, async () => {
const text = await getOutput(env, instance);
expect(text).toMatch(/This is development/);
});
});
});

describe('auto install', () => {
Expand Down