tools: produce JSON documentation using unified/remark/rehype #21697

rubys · 2018-07-07T02:01:28Z

work in progress. tools/doc/json2.js is a first pass attempt to convert tools/doc/json.js to a new markdown toolchain. The original intent was to produce functionally identical JSON, but some oddities have been found along the way. Examples: (highest priority first):

No events have parms in JSON with the original code. The new code picks up the parms.
The JSON produced for fs as a number with a method signature from the next heading.
Various textRaw and default properties seemingly randomly have an extra trailing space in their values.

The current (soon to be previous?) processing is based on a stream of tokens and a state machine. The new process is based on a pipeline of processors that have access to a tree of tokens. Care has been taken to not modify the stream so that this processor can be added to the pipeline that produces HTML.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

rubys · 2018-07-09T11:21:44Z

This pull request will require a lot of discussion. Other puzzles:

Why do methods have an inconsistent number of redundant and incomplete (only names and optional indicator) signatures? Examples:
- asyncResource.emitDestroy has 0.
- asyncResource.asyncId has 1.
- buf.readDoubleBE has 2.
Where did all of the constructors go?

$ grep -l '"ctors"' out/doc/api/*.json
out/doc/api/all.json
out/doc/api/crypto.json
out/doc/api/stream.json
$ git grep -l '## Constructor:' doc/api/*.md
doc/api/inspector.md
doc/api/stream.md
doc/api/url.md
doc/api/vm.md
$ git grep -l '## new ' doc/api/*.md
doc/api/assert.md
doc/api/async_hooks.md
doc/api/buffer.md
doc/api/console.md
doc/api/crypto.md
doc/api/errors.md
doc/api/http.md
doc/api/net.md
doc/api/perf_hooks.md
doc/api/stream.md
doc/api/string_decoder.md
doc/api/tls.md
doc/api/util.md
doc/api/v8.md
doc/api/vm.md
doc/api/worker_threads.md

My current code is rejecting the constructor for console as the first signature defines parameters that are included in the list list below (at least not as first level items):

### new Console(stdout[, stderr][, ignoreErrors])
### new Console(options)
<!-- YAML
...
-->

* `options` {Object}
  * `stdout` {stream.Writable}
  * `stderr` {stream.Writable}
  * `ignoreErrors` {boolean} Ignore errors when writing to the underlying
    streams. **Default:** `true`.
  * `colorMode` {boolean|string} Set color support for this `Console` instance.
    Setting to `true` enables coloring while inspecting values, setting to
    `'auto'` will make color support depend on the value of the `isTTY` property
    and the value returned by `getColorDepth()` on the respective stream.
    **Default:** `'auto'`.

I can fix this up, but not seeing the Console constructor in the current output means that I'm flying blind.

jasnell · 2018-07-10T19:57:20Z

tools/doc/json2.js

@@ -0,0 +1,491 @@
+// Copyright Joyent, Inc. and other Node contributors.


Is this a brand new file or is it pulled from another source? brand new files do not require the copyright header to be added.

Ultimately, it will be renamed to tools/doc/json2.js, and contains significant content from the original source.

Currently separate so that side by side comparisons of the output can be produced, via jsondiff.js, which will be removed when this effort is complete.

Ok, thanks for clarification :-)

rubys · 2018-07-11T02:28:24Z

Latest issue: files like buffer.json sometimes parses stability correctly, e.g.:

"stability": 2

And other times misses it entirely:

"desc": "<p>Stability: 0 - Deprecated: Use\n..."

The issue here is the embedded newline. The bug here is that stabilityExpr needs dotAll or equivalent.

rubys · 2018-07-12T20:06:23Z

This pull request is no longer in progress, nor is it ready to land. It depends on #21782 and #21780. In addition to providing fixes to the bugs mentioned above and listed in jsondiff.js, it differs from the output currently produced in that backslashes won't show up in names (for an example, look for "name": "\_\_filename" in modules.json.

Work remaining to be done before this lands: rename json2.js to json.js (overwriting the original source) and deleting jsondiff.js. For the moment, I'm leaving both sources and the diff tool with the thought that it would help reviewers.

Once #21490 lands, further refactoring and cleanup is possible.
html.js and json.js can be changed to export unified plugins and generate.js can implement the main pipeline that produces both. This will make it easy for other tools to be added to the pipeline, for example, see #21723. Also at that time, remark can be removed from package.json and the node_modules directory could be cleaned up.

Trott · 2018-07-12T23:24:59Z

This pull request is no longer in progress, nor is it ready to land. It depends on #21782 and #21780.

Removed in progress and applied blocked.

Trott · 2018-07-20T23:12:40Z

Removed the blocked label.

vsemozhetbyt · 2018-07-21T23:53:20Z

CI: https://ci.nodejs.org/job/node-test-pull-request/15967/

vsemozhetbyt · 2018-07-22T10:44:36Z

CI is all green.

vsemozhetbyt · 2018-07-22T11:40:20Z

I've compared the diff in *.html results. It seems we have only this significant difference: some comments disappear in the new HTML docs, such as:

<!--introduced_in=v...-->

<!-- type=misc -->
<!--type=class-->
<!--type=module-->
<!-- type=global -->
<!-- type=var -->
<!--type=event-->
<!--type=example-->

<!-- name=dgram -->
<!--name=esm-->
<!--name=fs-->
<!--name=module-->
<!--name=SIGINT, SIGHUP, etc.-->
<!--name=querystring-->
<!--name=vm-->

Not sure if this is breaking in any aspect. Maybe we can ignore this change as these comments seem needed at the generation stage only.

vsemozhetbyt · 2018-07-22T15:41:16Z

Ignorable note: json.js uses many iterations over elements using their properties with repetitive retrieving. Maybe it is worth to cache the properties once, especially if the element itself is not used. Say:

      nodes.forEach((node, i) => {

->

      nodes.forEach(({ type, value, children }, i) => {

or:

  return nodes.map((node) => {

->

  return nodes.map(({ type, position, value, children }) => {

etc.
Maybe this will speed up the process a little.

vsemozhetbyt

Just some comment nits.

vsemozhetbyt · 2018-07-22T15:02:18Z

tools/doc/json.js

+    const metaExpr = /<!--([^=]+)=([^-]+)-->\n*/g;
+    const stabilityExpr = /^Stability: ([0-5])(?:\s*-\s*)?(.*)$/s;
+
+    // extract definitions


Nit: // Extract definitions.

vsemozhetbyt · 2018-07-22T15:17:35Z

tools/doc/json.js

+      // Extract (and remove) metadata that is not directly inferable
+      // from the markdown itself.
+      nodes.forEach((node, i) => {
+        // Input: <!-- name=module -->; output: {name: module}


Nit: output: { name: module }.

vsemozhetbyt · 2018-07-22T15:52:39Z

tools/doc/json.js

-        values.push(item);
-        current = item;
+      // Pluralize type to determine which 'bucket' to put this section
+      // in.


Nit: it seems this can be unwrapped.

vsemozhetbyt · 2018-07-22T15:59:33Z

tools/doc/json.js


-  delete section.list;
+      // Add this section to the parent.  Sometimes we have two headings with a


Nit: extra space in parent. Sometimes.

vsemozhetbyt · 2018-07-22T16:00:07Z

tools/doc/json.js


-  delete section.list;
+      // Add this section to the parent.  Sometimes we have two headings with a
+      // single blob of description.  If the preceding entry at this level


Ditto in description. If.

vsemozhetbyt · 2018-07-22T16:03:30Z

tools/doc/json.js

-        ` > ${text}`
-      );
+
+    // At this point, the name should match.  If it doesn't find one that does.


Ditto in match. If.

vsemozhetbyt · 2018-07-22T16:08:30Z

tools/doc/json.js

-      }
-      return;
-    }
+        // Stablility marker: > Stability: ...


Stablility -> Stability

vsemozhetbyt · 2018-07-22T16:09:43Z

tools/doc/json.js

-  return src;
-}
-
-
 // This section parse out the contents of an H# tag.

 // To reduse escape slashes in RegExp string components.


Sorry, old debt: reduse -> reduce.

vsemozhetbyt · 2018-07-22T16:18:06Z

tools/doc/json.js

-  return src;
-}
-
-
 // This section parse out the contents of an H# tag.


Sorry, old debt: parse out -> parses out.

vsemozhetbyt · 2018-07-22T16:25:03Z

CC @nodejs/documentation @nodejs/build @nodejs/build-files: please, let us know if you plan to review this PR in the near future. Otherwise, I will land it, say, on Wednesday.

vsemozhetbyt · 2018-07-23T05:56:42Z

CI-lite: https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/339/

Trott

Rubber-stamp LGTM

trivikr

RSLGTM

vsemozhetbyt · 2018-07-24T06:14:45Z

@rubys This need rebasing (probably after b38b8d3).

... but we can update to the latest :-)

rubys · 2018-07-24T14:13:28Z

rebased

vsemozhetbyt · 2018-07-24T14:25:05Z

CI: https://ci.nodejs.org/job/node-test-pull-request/15996/

PR-URL: #21697 Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>

vsemozhetbyt · 2018-07-25T18:34:47Z

Landed in f41dd55
Thank you!

targos · 2018-07-26T07:19:52Z

Depends on #21616 to land on v10.x-staging

PR-URL: #21697 Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>

Refs: nodejs#21697

PR-URL: nodejs#22749 Refs: nodejs#21697 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: #22749 Refs: #21697 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

nodejs-github-bot added doc Issues and PRs related to the documentations. tools Issues and PRs related to the tools directory. labels Jul 7, 2018

Trott added the wip Issues and PRs that are still a work in progress. label Jul 7, 2018

rubys mentioned this pull request Jul 9, 2018

Creating a branch of the Docs using RunKit #21723

Open

jasnell reviewed Jul 10, 2018

View reviewed changes

rubys mentioned this pull request Jul 12, 2018

tools: update marked to 0.4.0 #21081

Closed

2 tasks

Trott added blocked PRs that are blocked by other issues or PRs. and removed wip Issues and PRs that are still a work in progress. labels Jul 12, 2018

rubys mentioned this pull request Jul 20, 2018

Produce HTML documentation using unified/remark/rehype #21490

Closed

3 tasks

Trott removed the blocked PRs that are blocked by other issues or PRs. label Jul 20, 2018

rubys force-pushed the remark-json branch from edb8cc6 to b80719b Compare July 21, 2018 01:40

This comment has been minimized.

Sign in to view

vsemozhetbyt approved these changes Jul 22, 2018

View reviewed changes

Trott approved these changes Jul 23, 2018

View reviewed changes

trivikr approved these changes Jul 24, 2018

View reviewed changes

rubys added 2 commits July 24, 2018 06:19

tools: produce JSON documentation using unified/remark/rehype

8a74e42

Further progress trying to produce JSON with remark

0db6681

rubys added 3 commits July 24, 2018 07:10

premature to remove marked entirely

3a4489a

... but we can update to the latest :-)

get test-ci passing again

81d3985

address comment nits

93d2044

rubys force-pushed the remark-json branch from a5ee58c to 93d2044 Compare July 24, 2018 14:11

vsemozhetbyt closed this Jul 25, 2018

targos added the backport-requested-v10.x label Jul 26, 2018

targos mentioned this pull request Jul 31, 2018

tools: convert addon-verify to remark #21978

Closed

2 tasks

targos removed the backport-requested-v10.x label Aug 7, 2018

rvagg mentioned this pull request Aug 13, 2018

Release proposal: v10.9.0 #22295

Merged

tniessen added a commit to tniessen/node that referenced this pull request Sep 7, 2018

build: make doc generation work on Windows

3f0e696

Refs: nodejs#21697

tniessen mentioned this pull request Sep 7, 2018

build: make doc generation work on Windows #22749

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: produce JSON documentation using unified/remark/rehype #21697

tools: produce JSON documentation using unified/remark/rehype #21697

rubys commented Jul 7, 2018

rubys commented Jul 9, 2018

jasnell Jul 10, 2018

rubys Jul 10, 2018

jasnell Jul 10, 2018

rubys commented Jul 11, 2018

rubys commented Jul 12, 2018

Trott commented Jul 12, 2018

Trott commented Jul 20, 2018

This comment has been minimized.

This comment has been minimized.

vsemozhetbyt commented Jul 21, 2018

vsemozhetbyt commented Jul 22, 2018

vsemozhetbyt commented Jul 22, 2018 •

edited

vsemozhetbyt commented Jul 22, 2018

vsemozhetbyt left a comment

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt Jul 22, 2018

vsemozhetbyt commented Jul 22, 2018

vsemozhetbyt commented Jul 23, 2018

Trott left a comment

trivikr left a comment

vsemozhetbyt commented Jul 24, 2018 •

edited

rubys commented Jul 24, 2018

vsemozhetbyt commented Jul 24, 2018

vsemozhetbyt commented Jul 25, 2018

targos commented Jul 26, 2018

		@@ -0,0 +1,491 @@
		// Copyright Joyent, Inc. and other Node contributors.


		delete section.list;
		// Add this section to the parent. Sometimes we have two headings with a

tools: produce JSON documentation using unified/remark/rehype #21697

tools: produce JSON documentation using unified/remark/rehype #21697

Conversation

rubys commented Jul 7, 2018

Checklist

rubys commented Jul 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rubys commented Jul 11, 2018

rubys commented Jul 12, 2018

Trott commented Jul 12, 2018

Trott commented Jul 20, 2018

This comment has been minimized.

This comment has been minimized.

vsemozhetbyt commented Jul 21, 2018

vsemozhetbyt commented Jul 22, 2018

vsemozhetbyt commented Jul 22, 2018 • edited

vsemozhetbyt commented Jul 22, 2018

vsemozhetbyt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vsemozhetbyt commented Jul 22, 2018

vsemozhetbyt commented Jul 23, 2018

Trott left a comment

Choose a reason for hiding this comment

trivikr left a comment

Choose a reason for hiding this comment

vsemozhetbyt commented Jul 24, 2018 • edited

rubys commented Jul 24, 2018

vsemozhetbyt commented Jul 24, 2018

vsemozhetbyt commented Jul 25, 2018

targos commented Jul 26, 2018

vsemozhetbyt commented Jul 22, 2018 •

edited

vsemozhetbyt commented Jul 24, 2018 •

edited