Contribute to the thought process for 3rd party language packaging if you'd like #2

joshgoebel · 2019-12-20T23:05:48Z

As a maintainer of a Highlight.js language grammar you might be interested in the discussion of an official packaging format. I just created an issue to track the discussion and I've been working on this along with the new build system.

highlightjs/highlight.js#2328

Sorry for the spam, but I couldn't think of an easier way to ping the people who might be most interested in weighing in on the subject. Feel free to simply close this issue or leave it open (whatever works best for you!).

gusbemacbe · 2019-12-24T06:14:24Z

@yyyc514

I tried to minify the file cyber-original.js, but it gave an error, due to function(hljs), but I did not see any error.

joshgoebel · 2019-12-24T06:17:28Z

You don't want to minify it yourself - it's not JUST a minified file - it's a packaged CDN ready file, that's what the new build system is for. If you're using my branch that has the new build system and your language is in:

[highlightjs checkout]/extra/highlightjs-cyber

# from the hightlight-js parent folder
node  --stack-size=65500  ./tools/build.js -t cdn

Will build the dist file for you when it builds the CDN assets.

joshgoebel · 2019-12-24T06:18:19Z

You should be exporting a module though:

module.exports = function (hljs) {

gusbemacbe · 2019-12-24T06:22:07Z

The command node --stack-size=65500 ./tools/build.js -t cdn seems not to work.

gusbemacbe · 2019-12-24T06:25:40Z

In spite of it, do the minified files seem OK to you?

joshgoebel · 2019-12-24T06:30:57Z

How does it not work? Are you using the branch with the new build system?

joshgoebel · 2019-12-24T06:32:26Z

In spite of it, do the minified files seem OK to you?

It's not a CDN module. CDN modules include the registration code so that all you have to do is include the JS tag... and NOTHING else.

joshgoebel · 2019-12-24T06:33:24Z

What is cypher vs cypher original? If you're publishing two entirely different languages you need a slightly different structure, more like how the core library does it.

joshgoebel · 2019-12-24T06:36:25Z

Here is what a CDN module looks like:

hljs.registerLanguage("bnf",(function(n){return{contains:[{className:"attribute",begin:/</,end:/>/},{begin:/::=/,starts:{end:/$/,contains:[{begin:/</,end:/>/},n.C_LINE_COMMENT_MODE,n.C_BLOCK_COMMENT_MODE,n.APOS_STRING_MODE,n.QUOTE_STRING_MODE]}}]}}));

gusbemacbe · 2019-12-24T06:36:52Z

The cypher-original was not implemented by me, and does not require you to use the extra script of JavaScript code.

The cypher was implemented by me, and needs the extra script of JavaScript code.

joshgoebel · 2019-12-24T06:38:38Z

You only need the original file, you don't need special code to make it work in a browser... that's what the new build process does for you. :-) And it will generate the dist file for you.

The dist file is what you'd use in the web browser.
The src file is what you'd use with Node/rollup/etc.

joshgoebel · 2019-12-24T06:40:54Z

You need to be on the squash_build_pipeline branch which has the new build system. I'm hoping to get it merged in the next few weeks, but holidays and all...

gusbemacbe · 2019-12-24T06:41:23Z

OK, I created a folder tools and a file build.js (empty), and tried to run node --stack-size=65500 ./tools/build.js -t cdn, but it did not anything or minify the files.

joshgoebel · 2019-12-24T06:43:02Z

./tools/build.js is a part of highlight.js.

You run that command from inside highlight.js. Your repository is checked out UNDER the highlight.js folder. Your repo goes in the extra folder.

joshgoebel · 2019-12-24T06:47:14Z

Of course your package.json also needs to point to the correct location of your src file. :-)

gusbemacbe · 2019-12-24T06:47:41Z

Now I understood. But I do not like to use git@git. I prefer the HTTPS:// Is it possible?

joshgoebel · 2019-12-24T06:48:05Z

You can check them out however you like... what's important is where the folders are.

joshgoebel · 2019-12-24T06:48:59Z

I just fixed package.json locally and changed your main file back to doing module.exports and it builds fine with my branch and creates the dist file as expected.

joshgoebel · 2019-12-24T06:49:45Z

I can make PR if you ant, but you'll still need the correct branch to run the build script.

gusbemacbe · 2019-12-24T06:51:17Z

I can make PR if you ant, but you'll still need the correct branch to run the build script.

You can make a PR now. But firstly will I need to correct the branch?

joshgoebel · 2019-12-24T06:51:54Z

But firstly will I need to correct the branch?

You can't build it (the file in dist) yourself unless you're using the new branch.

gusbemacbe · 2019-12-24T06:53:57Z

You can't build it (the file in dist) yourself unless you're using the new branch.

In the new branch squash_build_pipeline that I have already just created, there is not the folder dist yet.

joshgoebel · 2019-12-24T06:55:52Z

squash_build_pipeline is a branch of highlight.js. It's not a branch you have to create.

highlightjs/highlight.js#2312

joshgoebel · 2019-12-24T06:57:51Z

Inside highlight.js folder this might help:

git checkout squash_build_pipeline
npm install
node ./tools/build.js -t cdn

gusbemacbe · 2019-12-24T07:06:00Z

░▒▓     ~/GitHub/highlight.js     squash_build_pipeline ?1 ▓▒░·················
❯ node ./tools/build.js -t cdn
internal/modules/cjs/loader.js:775
    throw err;
    ^

Error: Cannot find module './roll_browser'
Require stack:
- /home/gusbemacbe/GitHub/highlight.js/tools/rollup_cdn.js
- /home/gusbemacbe/GitHub/highlight.js/tools/build.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:772:15)
    at Function.Module._load (internal/modules/cjs/loader.js:677:27)
    at Module.require (internal/modules/cjs/loader.js:830:19)
    at require (internal/modules/cjs/helpers.js:68:18)
    at Object.<anonymous> (/home/gusbemacbe/GitHub/highlight.js/tools/rollup_cdn.js:11:37)
    at Module._compile (internal/modules/cjs/loader.js:936:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:947:10)
    at Module.load (internal/modules/cjs/loader.js:790:32)
    at Function.Module._load (internal/modules/cjs/loader.js:703:12)
    at Module.require (internal/modules/cjs/loader.js:830:19)
    at require (internal/modules/cjs/helpers.js:68:18)
    at Object.<anonymous> (/home/gusbemacbe/GitHub/highlight.js/tools/build.js:87:13)
    at Module._compile (internal/modules/cjs/loader.js:936:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:947:10)
    at Module.load (internal/modules/cjs/loader.js:790:32)
    at Function.Module._load (internal/modules/cjs/loader.js:703:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/gusbemacbe/GitHub/highlight.js/tools/rollup_cdn.js',
    '/home/gusbemacbe/GitHub/highlight.js/tools/build.js'
  ]
}
zsh: exit 1     node ./tools/build.js -t cdn

joshgoebel · 2019-12-24T07:10:45Z

That's an older copy of the branch... the latest is:

commit 3ffc2bbf2fe34d501d125c908c2d1d7014ad893a (HEAD -> squash_build_pipeline, yyyc514/squash_build_pipeline)
Author: Josh Goebel <me@joshgoebel.com>
Date:   Mon Dec 23 16:56:17 2019 -0500

    wip: package CDN distributable for 3rd party languages

joshgoebel · 2019-12-24T07:11:36Z

Maybe you need to git fetch --all ?

gusbemacbe · 2019-12-24T07:13:14Z

I did, but the error is still insistant. Maybe I will git-reset to your latest communit 3ffc2bbf2fe34d501d125c908c2d1d7014ad893a.

joshgoebel · 2019-12-24T07:14:17Z

Yeah you might have your local branches pointed to the wrong revision. git is just instinctive to me but without being in front of your computer it's hard to tell you what to do, fetch, pull, etc... plus I don't know what remotes you have, etc. :-)

gusbemacbe · 2019-12-24T07:17:29Z

But not successful yet...

git reset --hard 3ffc2bbf2fe34d501d125c908c2d1d7014ad893a
fatal: Could not parse object '3ffc2bbf2fe34d501d125c908c2d1d7014ad893a'.

I used HTTPS instead of SSH. See how I did:

git clone https://github.com/highlightjs/highlight.js
git checkout squash_build_pipeline
mkdir extra && cd extra
git clone https://github.com/highlightjs/highlight-cypher
cd ../..
node ./tools/build.js -t cdn

Did I do something wrong?

gusbemacbe · 2019-12-24T07:20:42Z

The log:

░▒▓     ~/GitHub/highlight.js     squash_build_pipeline ?1 ▓▒░·················································································································································································································
❯ git log

commit 73f0e0bdbae234b35526cd4b0d62f30981166963 (HEAD -> squash_build_pipeline, origin/squash_build_pipeline)
Author: Josh Goebel <me@joshgoebel.com>
Date:   Fri Dec 6 18:51:03 2019 -0500

    rename rollup build files

I switched to master, and the log:

░▒▓     ~/GitHub/highlight.js     master ?1 ▓▒░································································································································································································································
❯ git log

commit bd545651b8bf0e6c4f8b8d37ca4af2ce311127ec (HEAD -> master, origin/master, origin/HEAD)
Author: Youssef Victor <31348972+Youssef1313@users.noreply.github.com>
Date:   Fri Dec 20 18:22:52 2019 +0200

    (vbnet) Add missing `nameof` keyword (#2329)

joshgoebel · 2019-12-24T07:22:16Z

What are your remotes? The latest commit for both my repo and highlightjs is 3ffc2bbf2fe34d501d125c908c2d1d7014ad893a on that branch.

joshgoebel · 2019-12-24T07:23:15Z

Ah I have force pushed it IIRC, so you might need to reset --hard to that specific commit.

gusbemacbe · 2019-12-24T07:26:53Z

AH, I got, the official repository does not have your latest commit in both branches. Your latest commit is in your repository: https://github.com/yyyc514/highlight.js/tree/squash_build_pipeline.

joshgoebel · 2019-12-24T07:30:23Z

Ah. Fixed. :) Too many repos. I did finally rename my remotes so now I'm my own origin. This should help in the future. :-)

gusbemacbe · 2019-12-24T07:43:10Z

Ah, and it worked. I saw cypher files were minified. Should I do anything else?

joshgoebel · 2019-12-24T07:44:01Z

You don't need the two files in root anymore.

joshgoebel · 2019-12-24T07:44:26Z

Your README probably need to be updated to follow the new conventions. See robots-txt repo for an example.

gusbemacbe · 2019-12-24T08:29:36Z

@yyyc514

Ready.

joshgoebel · 2019-12-24T08:43:23Z

You shouldn't include this unless you're actually going to publish your package to NPM.

<script type="text/javascript"
  src="https://unpkg.com/highlightjs-cypher@master/dist/cypher.min.js"></script>

You should change the RobotsTxt variable name:

var hljsRobotsTxt = require('highlightjs-cypher');

gusbemacbe · 2019-12-24T08:44:31Z

You shouldn't include this unless you're actually going to publish your package to NPM.

Actually I published to NPM.

gusbemacbe · 2019-12-24T08:46:20Z

Ready.

joshgoebel · 2019-12-24T08:48:53Z

The link isn't working:

https://unpkg.com/highlightjs-cypher@master/dist/cypher.min.js

gusbemacbe · 2019-12-24T08:52:09Z

Fixed, @yyyc514.

joshgoebel · 2019-12-24T08:55:57Z

Looking pretty good. Still not sure where your markup expect test files are. Did you just not commit them?

gusbemacbe · 2019-12-24T08:58:21Z

Ah, you have removed the expect test files. I will check the old commits to download the old files.

gusbemacbe · 2019-12-24T09:02:45Z

I have just added the missed file sample.expect.txt, and if you need more expect files, let me know.

gusbemacbe · 2019-12-24T09:42:23Z

Anything else, @yyyc514 ?

joshgoebel · 2019-12-24T15:49:52Z

I'm not sure you're understanding how it works. For markup tests you need TWO files for each test:

keywords.txt (the raw input to Highlight.js)
keywords.expect.txt (the correct HTML output that SHOULD be generated if your code is working properly)

You should be able to run the test suite from inside the highlight.js directory and see the failing tests because it will yell at you that it can't find the files.

The custom test scaffold you write LOOKs like you understood the system, but then the files aren't there - so your own tests would never have passed (that I can see).

gusbemacbe · 2020-01-08T21:32:21Z

Hello @yyyc514 !

I am very sorry for my long absence. The company overcharged me with a lot of things.

keywords.txt (the raw input to Highlight.js)

keywords.expect.txt (the correct HTML output that SHOULD be generated if your code is working properly)

The neo4j/Cypher is not inside HTML. It is inside PHP/JavaScript/JSON on a HTML file.

joshgoebel · 2020-01-11T19:37:29Z

Pretend I said "the correct output"... the test files are just:

expected intput
expect output

gusbemacbe · 2020-01-11T21:19:45Z

@yyyc514 , I think @Mogztter has already done it. See the latest commit and if those test files are what you want exactly.

joshgoebel · 2020-01-11T21:30:04Z

Yeah that looks like the idea.

gusbemacbe · 2020-01-12T00:47:57Z

@yyyc514

The repository is ready.

AdamRaichu · 2023-01-22T00:57:01Z

Suggest closing?

gusbemacbe added a commit that referenced this issue Dec 24, 2019

3rd party packaging (#2)

147555e

gusbemacbe added a commit that referenced this issue Dec 24, 2019

3rd party packaging (#2) and fixed cyber-original

f1aae20

ggrossetie mentioned this issue Jan 11, 2020

Document how to generate the dist file #9

Closed

Contribute to the thought process for 3rd party language packaging if you'd like #2

Contribute to the thought process for 3rd party language packaging if you'd like #2

Comments

joshgoebel commented Dec 20, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019 • edited

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019 • edited

gusbemacbe commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019 • edited

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019 • edited

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019 • edited

joshgoebel commented Dec 24, 2019 • edited

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019 • edited

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

gusbemacbe commented Dec 24, 2019

joshgoebel commented Dec 24, 2019 • edited

gusbemacbe commented Jan 8, 2020

joshgoebel commented Jan 11, 2020

gusbemacbe commented Jan 11, 2020

joshgoebel commented Jan 11, 2020

gusbemacbe commented Jan 12, 2020

AdamRaichu commented Jan 22, 2023

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

gusbemacbe commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

gusbemacbe commented Dec 24, 2019 •

edited

gusbemacbe commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

gusbemacbe commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited

joshgoebel commented Dec 24, 2019 •

edited