Allow `allowJs` and `declaration` to be used together #32372

weswigham · 2019-07-12T01:27:03Z

Which, in turn, should enable incremental builds and project-based builds for projects containing .js and .json files.

This introduces a new symbol-based declaration emitter (meaning it's highly coupled to the checker and is indeed part of the node builder) - currently this is only used for JSON and JavaScript, as the output is likely worse than what the other declaration emitter is capable of (minimally, it'll likely never respect input declaration order). In addition, it is still incomplete - it does not yet support serializing namespaces.

Things yet to do:

Still, unfinished as it is, this is probably able to be used in most scenarios already, so getting some implementation feedback (and output feedback) would be welcome.

Fixes #7546

cc @DanielRosenwasser you dumped this into the 3.6 release, but never assigned it to anyone - so I started on it at the start of the week. Think we could squeeze it into the beta?

brendankenny · 2019-07-12T22:09:08Z

very exciting!

so getting some implementation feedback (and output feedback) would be welcome.

hopefully you meant this :)

Running it against lighthouse (adding declaration, removing noEmit, and adding emitDeclarationOnly to our tsconfig), the output seems reasonable, but I only get three d.ts files before it hits an Error: Debug Failure. Unhandled class member kind! 32.

I might be holding it wrong, though, I've never actually used --declaration.

JSDoc comments are also not preserved on the signatures. Will that be possible?

sandersn

Some initial comments. Nothing too useful yet, just observations.

Still need to look at the tests and the meat of the code.

src/compiler/utilities.ts

src/testRunner/unittests/tscWatch/programUpdates.ts

src/compiler/checker.ts

weswigham · 2019-07-12T23:26:33Z

JSDoc comments are also not preserved on the signatures. Will that be possible?

Should be - the work item in the OP to do with "using setOriginalNode where possible" should cover that.

Running it against lighthouse (adding declaration, removing noEmit, and adding emitDeclarationOnly to our tsconfig), the output seems reasonable, but I only get three d.ts files before it hits an Error: Debug Failure. Unhandled class member kind! 32.

Hoho, thanks for the already setup real-world codebase. I had some bugs with base class serialization which are now fixed, so one of those assertions is gone, and the other shouldn't be possible to trigger so long as you can't write namespace Cls { interface X {} } class Cls {} in JS (plus a small bug exposed in the normal declaration emitter to do with confusion of a type as a type parameter).

In any case, lighthouse no longer crashes with this PR and those settings, which is good, however we do produce some invalid declaration files 😦 . I'll need to look into which symbol patterns produce the invalid output and patch up their emit. Looks like @typedef's in a file with a module.exports = {...} is a big offender, which makes sense - the way this works at all is very strange within the checker.

DanielRosenwasser · 2019-07-12T23:59:12Z

Part of the idea was to have infrastructure in place to fuzz-test and ensure that we don't just crash on random .js files. It seems like we're already running into some and producing incorrect results, so I think if anything we can put this into typescript@experimental`.

weswigham · 2019-07-13T00:35:03Z

I think if anything we can put this into typescript@experimental

To what end? Do we have partners willing to provide the same kind of and quality of feedback we'd get from the month long beta period we now have? I though the entire point of a beta is that it can be semistable and have features that may not make the final release cut (because they didn't stabilize enough over the beta timeframe) - it's not an RC, after all; a beta's not meant to be in a state where it's promotable to a full release.

sandersn

More comments, up through getinternalSymbolName

src/compiler/checker.ts

DanielRosenwasser · 2019-07-17T23:39:47Z

@typescript-bot pack this

weswigham · 2019-07-18T03:34:09Z

@typescript-bot pack this pls - where you at

typescript-bot · 2019-07-18T03:34:14Z

Heya @weswigham, I've started to run the tarball bundle task on this PR at fd9648a. You can monitor the build here. It should now contribute to this PR's status checks.

typescript-bot · 2019-07-18T03:39:59Z

Hey @weswigham, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/36910/artifacts?artifactName=tgz&fileId=BF842F98B0BE069E93F283BACE70B6730B39DC4117B82B30B115803058C7555202&fileName=/typescript-3.6.0-insiders.20190718.tgz"
    }
}

and then running npm install.

matthewmueller · 2019-07-24T11:13:05Z

Works pretty well! I'm seeing a couple issues:

1) error in tsconfig.json that I just want to bring up to make sure it's covered:

This might just be due to my vscode's typescript being different than the project's typescript

2) It doesn't seem to understand the following type definitions

/**
 * Imports
 *
 * @typedef {import("../timer")} Timer
 * @typedef {import("../hook")} Hook
 * @typedef {import("../hook").HookHandler} HookHandler
 */

/**
 * Input type definition
 *
 * @typedef {Object} Input
 * @prop {Timer} timer
 * @prop {Hook} hook
 */

/**
 * New `Context`
 *
 * @class
 * @param {Input} input
 */

function Context(input) {
  if (!(this instanceof Context)) {
    return new Context(input)
  }
  this.state = this.construct(input)
}

Where I'm importing one typedef from another file into this file. In this case I see:

weswigham · 2019-07-24T17:03:52Z

What's the shape of Hook (input and output) And what TS version is your IDE using? I likewise had to make declaration parsing a taaaad more permissive in some ways to represent some JS patterns, so there's the possibility that something isn't an error with this PR but is without it.

weswigham · 2019-07-24T17:05:49Z

Oooo, and that's a JS-style psuedoclass - I haven't even tested those yet, but it definitely seems like their output is a bit wonky. Will have to work on that. Afaik the only way to emit them into a declaration file is with a type alias declaring both call and construct signatures.

matthewmueller · 2019-07-24T18:03:56Z

Sorry for an incomplete example! Hook contains the following:

../hook.js

/**
 * Imports
 *
 * @typedef {import("../reporters").Reporter} Reporter
 * @typedef {import("../context")} Context
 * @typedef {import("../timer")} Timer
 */

/**
 * Hook handler
 *
 * @typedef {(t: Context) => void|Promise<void>} HookHandler
 */

/**
 * Input type definition
 *
 * @typedef {Object} Input
 * @prop {Reporter} reporter
 * @prop {string} filename
 * @prop {string} title
 * @prop {string} cwd
 * @prop {Timer} timer
 */

/**
 * New `Hook`
 *
 * @class
 * @param {Input} input
 */

function Hook(input) {
  if (!(this instanceof Hook)) {
    return new Hook(input)
  }
}

/**
 * Exports
 */

module.exports = Hook

I should mention, everything works great from a type-checking perspective. Just when you generate typedefs, I'm seeing errors.

And I'm using Typescript 3.5.2 in VScode. I've started using the workspace version 3.6.0-insiders.20190708 and the tsconfig.json seems to work :-)

Lastly, I should mention that this unfinished PR alone is already super helpful. I wasn't really sure how to write typedefs and just having example outputs allows me to fix things by hand. Thanks for your hard work so far!

FWIW, I just needed to make 3 fixes to make to the example above work (with the red squiggles):

import(...) with a old-school constructor and @class needs to be typeof import(...)
Types (State, Input, etc.) need to be placed inside namespace test { }
References to these types need to be changed from State to test.State

Update

Change (1.) turned out to be wrong, I needed to turn it into a class declare class Hook in the definition file, then it does work with import("./hook")

weswigham · 2019-07-24T19:37:16Z

OK, yup, definitely looks like it comes down to not serializing the types of JS functions-as-classes yet. It'll be one of the next things I look into - how they work internally is all a bit ad-hoc, so recognizing them and serializing them in a compatible way will be... interesting.

matthewmueller · 2019-07-24T19:53:12Z

Awesome! Let me know if I can help in any other way. One last thing I ran into and it might just be the odd way I'm doing things. This is not a bug, but more of a possible UX improvement.

I use the following to make a local reference to typedefs so I don't have to constantly import this type over and over in the code.

/**
 * @typedef {import("./context")} Context
 */

Currently for the JS functions-as-classes exported with module.exports = Context correctly uses imports. The generated index.d.ts has the following:

type Context = import('./context')

But for other typedefs, it actually inlines the whole type. Sometimes these types are huge so it's probably not what you want.

reporter.js

/**
 * Reporter type definition
 *
 * @typedef {Object} Reporter
 * @prop {(event: Event) => Promise<void>} report
 * @prop {() => Promise<void>} flush
 * @prop {() => Promise<void>} close
 */

context.js

/**
 * @typedef {import("./reporter").Reporter} Reporter
 */

Results in context.d.ts with a full reporter type definition:

type Reporter = {
  report: (event: Event) => Promise<void>
  flush: () => Promise<void>
  close: () => Promise<void>
}

Where I'd expect:

type Reporter = import("./report").Reporter

Once again, really impressed with how far along this is already.

sandersn · 2019-07-26T16:18:22Z

@weswigham I think it might be worthwhile to first make constructor functions into real classes. Assuming that they fit into the class mold neatly enough, it should cut the amount of work for this PR.

matthewmueller · 2019-07-26T16:39:05Z

Agreed if that's possible. I'm finding declare class X to be the cleanest way to handwrite definitions for @class.

daKmoR · 2019-08-01T20:27:43Z

@typescript-bot the above-mentioned tgz seem to be no longer available? Could you provide a new version? pretty please 🤗

weswigham · 2019-08-01T20:55:15Z

I can @typescript-bot pack this for your testing yes, but this is currently blocked on #32584 to fix the reported issue with ES5-y class/functions.

typescript-bot · 2019-08-01T20:55:19Z

Heya @weswigham, I've started to run the tarball bundle task on this PR at eb4a036. You can monitor the build here. It should now contribute to this PR's status checks.

typescript-bot · 2019-08-01T21:01:38Z

Hey @weswigham, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/38490/artifacts?artifactName=tgz&fileId=B44B1B9D6D3E1494E9E4D61354B7758F8C6EE6DD0CEA7A4304ABDBD5DF6F7FF002&fileName=/typescript-3.6.0-insiders.20190801.tgz"
    }
}

and then running npm install.

daKmoR · 2019-08-01T23:35:09Z

@weswigham awesome thx 🤗

needed to try it immediately 💪
you can see the result here open-wc/open-wc#679 👍

really promising 🥇 and almost good enough for us... if not be for 1 thing

exports are invalid 😭 (missing the from '../path/to/file.js' part)

a nice to have:

jsDoc gets lost 😢 (e.g. docu would be nicer with it)

weswigham · 2019-08-02T00:59:06Z

@daKmoR just 4 u: @typescript-bot pack this again 😉

weswigham · 2019-09-26T01:53:48Z

For posterity: The reason getAccessibleSymbolChainFromSymbolTable is so bad is because its' fail case is searching every externally visible symbol to see if any refer to the target symbol. Every. Single. One. This is then repeated for every [symbol, enclosingDeclaration] pair (and potentially repeated for the same pair - it's uncached). TBH, the whole algorithm for that process needs to be redesigned for efficiency and hierarchical cachability - I'm thinking something along the lines of maps of which symbols are reachable from which scopes which can just be union'd as scopes nest. A kind of transitiveExports map that isn't keyed by name, but by symbol id, while also storing a bit for merged symbols and alias targets (so a set, really).

weswigham · 2019-09-26T02:01:16Z

I'd like to not fix that in this PR, as that's also a few hundred lines of change that don't directly relate to this feature~

sandersn · 2019-09-26T20:36:59Z

Thanks, the perf analysis sounds like a good starting point for post-beta work. I'll look over your recent changes+replies and sign off.

ahocevar · 2019-09-28T07:48:22Z

Thanks for the great work on this!

Regarding output feedback: with https://github.com/openlayers/openlayers (master), I do not get any output, and this exception is thrown:

/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:77718
                throw e;
                ^

TypeError: Cannot read property 'escapedText' of undefined
    at getIsContextSensitiveAssignmentOrContextType (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:43163:123)
    at getContextualTypeForBinaryOperand (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:43119:44)
    at getContextualType (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:43401:28)
    at getApparentTypeOfContextualType (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:43328:17)
    at checkObjectLiteral (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:43746:34)
    at checkExpressionWorker (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:48054:28)
    at checkExpression (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:47983:38)
    at checkBinaryLikeExpression (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:47366:29)
    at checkBinaryExpression (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:47352:20)
    at checkExpressionWorker (/Users/ahocevar/projects/openlayers/node_modules/typescript/lib/tsc.js:48096:28)

typescript package version: typescript@3.7.0-dev.20190928

Let me know if you need more details to investigate this.

weswigham · 2019-09-28T11:00:30Z

Could you open a new issue with some details (ideally some exact repro steps)? That callstack indicates it's probably unrelated to this PR, and, like 10 things got merged in Friday. :D

Bnaya · 2019-10-03T00:08:08Z

possible related issue:
#33735

vipcxj · 2019-10-15T02:30:18Z

which version this pull affect？ it seems that the latest version still not allow allowJs and declaration to be used together

weswigham · 2019-10-15T06:58:57Z

The nightly and 3.7 beta.

Kinrany · 2019-10-25T00:03:17Z

Are existing .d.ts files taken into account when compiling .js? Can I use JSDoc and declaration files at the same time?

Basically .js + .d.ts -> .js + .d.ts, but the .js file is transpiled, and the resulting .d.ts has full types, not just explicitly specified?

weswigham · 2019-10-25T00:19:35Z

The compiler doesn't merge declaration files with js files, if that's what you're asking. One js file in makes one .d.ts file out that represents it.

Kinrany · 2019-10-25T01:09:25Z

I wasn't sure if we were talking about the same thing, so I wrote an example: https://github.com/Kinrany/ts-3.7-allowjs-test

It has four files: sources src/index.js and src/index.d.ts, and build outputs dist/index.js and dist/index.d.ts.

I was wondering whether src/index.d.ts would affect the output. Looks like it doesn't: the declared type of hello() is string? -> void, but the result is any -> void. I guess that's what you meant by "the compiler doesn't merge [them]".

(I may have made a mistake somewhere in src/index.d.ts, but I hope the idea is clear enough.)

weswigham · 2019-10-25T01:51:32Z

Yah. One input produces one output - the .d.ts and .js input files aren't implicitly related in any way.

golopot · 2019-10-30T21:45:52Z

How do I configure it to emit declaration files without compiled js files?

weswigham · 2019-10-30T23:48:50Z

emitDeclarationOnly compiler option

frags51 · 2022-01-04T12:26:50Z

Does someone know which milestone (TS release) it was added to? @sheetalkamat

sandersn self-assigned this Jul 12, 2019

weswigham force-pushed the symbolic-declaration-files branch from c003de7 to be85d75 Compare July 12, 2019 19:50

sandersn reviewed Jul 12, 2019

View reviewed changes

sandersn requested changes Jul 15, 2019

View reviewed changes

weswigham mentioned this pull request Jul 15, 2019

Use SymbolFlags.TypeAlias when binding a jsdoc enum instead of SymbolFlags.Enum #32413

Closed

weswigham mentioned this pull request Jul 26, 2019

Allow functions and ambient classes to merge #32584

Merged

weswigham force-pushed the symbolic-declaration-files branch 2 times, most recently from 3898322 to cb1e433 Compare August 2, 2019 00:55

Merge branch 'master' into symbolic-declaration-files

d7ef75b

sandersn approved these changes Sep 26, 2019

View reviewed changes

Use x instead of index

007b3d4

weswigham merged commit 61cb06c into microsoft:master Sep 26, 2019

This was referenced Sep 26, 2019

Investigate altering extension priorities for wildcard loading #33623

Open

JS declarations for an export assigned class expression can conflict with local names #33626

Closed

voxpelli mentioned this pull request Oct 7, 2019

Release 3.0.1 when TypeScript 3.7.0 has been released voxpelli/node-bunyan-adaptor#4

Closed

KaiVolland mentioned this pull request Oct 7, 2019

TypeScript (Help Wanted) openlayers/openlayers#8120

Closed

fbartho mentioned this pull request Oct 10, 2019

Use JSDoc with tsc for strong static typing apache/cordova-node-xcode#65

Open

Shinigami92 mentioned this pull request Oct 24, 2019

Provide internal TypeScript support to improve code quality prettier/prettier#6313

Closed

4 tasks

sheetalkamat mentioned this pull request Oct 28, 2019

JSON files generate declarations #34761

Closed

sheetalkamat mentioned this pull request Oct 30, 2019

tsc --watch emits all files even when they have not changed #29927

Closed

sheetalkamat mentioned this pull request Nov 6, 2019

TS5056 when using JSON and TS files with the same basename in composite project #34943

Closed

eric-simonton-sama mentioned this pull request Nov 30, 2020

How do I add a raw JavaScript file to be processed by ng-packagr ng-packagr/ng-packagr#650

Closed

bennypowers mentioned this pull request Jan 18, 2021

Provide Documentation bpmn-io/bpmn-js#332

Open

8 tasks

Allow allowJs and declaration to be used together #32372

Allow allowJs and declaration to be used together #32372

Conversation

weswigham commented Jul 12, 2019 • edited Loading

brendankenny commented Jul 12, 2019

sandersn left a comment

Choose a reason for hiding this comment

weswigham commented Jul 12, 2019

DanielRosenwasser commented Jul 12, 2019

weswigham commented Jul 13, 2019

sandersn left a comment

Choose a reason for hiding this comment

DanielRosenwasser commented Jul 17, 2019

weswigham commented Jul 18, 2019

typescript-bot commented Jul 18, 2019 • edited Loading

typescript-bot commented Jul 18, 2019

matthewmueller commented Jul 24, 2019 • edited Loading

weswigham commented Jul 24, 2019

weswigham commented Jul 24, 2019

matthewmueller commented Jul 24, 2019 • edited Loading

weswigham commented Jul 24, 2019

matthewmueller commented Jul 24, 2019 • edited Loading

sandersn commented Jul 26, 2019

matthewmueller commented Jul 26, 2019 • edited Loading

daKmoR commented Aug 1, 2019

weswigham commented Aug 1, 2019

typescript-bot commented Aug 1, 2019 • edited Loading

typescript-bot commented Aug 1, 2019

daKmoR commented Aug 1, 2019

weswigham commented Aug 2, 2019

weswigham commented Sep 26, 2019 • edited Loading

weswigham commented Sep 26, 2019

sandersn commented Sep 26, 2019

ahocevar commented Sep 28, 2019

weswigham commented Sep 28, 2019

Bnaya commented Oct 3, 2019

vipcxj commented Oct 15, 2019

weswigham commented Oct 15, 2019

Kinrany commented Oct 25, 2019 • edited Loading

weswigham commented Oct 25, 2019

Kinrany commented Oct 25, 2019 • edited Loading

weswigham commented Oct 25, 2019

golopot commented Oct 30, 2019

weswigham commented Oct 30, 2019 • edited by orta Loading

frags51 commented Jan 4, 2022

Allow `allowJs` and `declaration` to be used together #32372

Allow `allowJs` and `declaration` to be used together #32372

weswigham commented Jul 12, 2019 •

edited

Loading

typescript-bot commented Jul 18, 2019 •

edited

Loading

matthewmueller commented Jul 24, 2019 •

edited

Loading

matthewmueller commented Jul 24, 2019 •

edited

Loading

matthewmueller commented Jul 24, 2019 •

edited

Loading

matthewmueller commented Jul 26, 2019 •

edited

Loading

typescript-bot commented Aug 1, 2019 •

edited

Loading

weswigham commented Sep 26, 2019 •

edited

Loading

Kinrany commented Oct 25, 2019 •

edited

Loading

Kinrany commented Oct 25, 2019 •

edited

Loading

weswigham commented Oct 30, 2019 •

edited by orta

Loading