Emojis

[hut8]

Thanks very much for moving this out into its own repository.

I saw your goal is to make this equivalent to the output of api.github.com/markdown/raw, which is awesome. The emojis are missing for now, however:

liam@hut8 ~ % curl -H"Content-Type:text/plain"  --data ":+1:" https://api.github.com/markdown/raw
<p><img class="emoji" title=":+1:" alt=":+1:" src="https://assets-cdn.github.com/images/icons/emoji/unicode/1f44d.png" height="20" width="20" align="absmiddle"></p>

If I implemented this, would you accept a PR?

[dmitshur]

Hi, I'm glad this package is helpful!

You are right, the goal is to produce output equivalent to https://developer.github.com/v3/markdown/#render-a-markdown-document-in-raw-mode, which includes Emoji, therefore github_flavored_markdown should support Emoji as well.

A PR would be very appreciated! Before you spend too much time on it, can you please outline how you're planning to approach it, so I can review or give some suggestions.

Thanks a lot!

[hut8]

Thank you very much. My blog is powered by Hugo and I would say rendering these things is an absolutely critical feature.

I'm really surprised how complete the GitHub API is. Here's a list of all their emojis:
curl -L https://api.github.com/emojis

It depends on how close you really want the output to be to GitHub's. If you actually want it to be identical, i.e., linking to GitHub's CDN, then generating the output is straightforward. If you want the emojis to be part of the document so that it's usable without hot-linking or Internet access, I'd probably generate data URLs.

I haven't actually looked at the internals of your project much, but I have written a lexer and parser and I'll get back to you on that tomorrow. Thanks for the quick reply.

[dmitshur]

The output should be functionally equivalent, it doesn't need to be byte-for-byte the same. As long as the output produces same or similar looking images.

It depends on how close you really want the output to be to GitHub's. If you actually want it to be identical, i.e., linking to GitHub's CDN, then generating the output is straightforward. If you want the emojis to be part of the document so that it's usable without hot-linking or Internet access, I'd probably generate data URLs.

This is a good question. I'm not sure what's best. I think linking to the "https://assets-cdn.github.com/images/icons/emoji/unicode/..." urls is probably simplest, and it'll ensure the result emoji images match those generated by the GitHub Markdown API endpoint. But if you're making a heavy use of GFM on your site, you may want to host the assets yourself.

The current output already expects you to "bring your own" css for styles and it the example suggests using "//cdnjs.cloudflare.com/ajax/libs/octicons/2.1.2/octicons.css" for octicons (which you can host yourself, of course). Perhaps a similar approach for emoji would be best, if possible?

I haven't actually looked at the internals of your project much, but I have written a lexer and parser and I'll get back to you on that tomorrow. Thanks for the quick reply.

The package code is pretty short. It is a little messy in parts, sorry in advance about that. I made it in limited spare time and I was focusing on functionality first, that's my excuse. :P

[dmitshur]

By the way,

My blog is powered by Hugo

I'm not closely familiar with Hugo, but I thought it doesn't use this package. Or can you use any Markdown processor with it that you want, including github_flavored_markdown?

[hut8]

Hugo doesn't use this package, but it does use blackfriday to render the site and I don't think it would be too hard to modify it. After all, the blackfriday README references this project.

[dmitshur]

github_flavored_markdown is a Markdown HTML renderer that is largely based on blackfriday Markdown HTML renderer. It reuses 90% of its functionality, and adds some tweaks (like fenced code block highlighting, and clickable header anchor links).

If you add support for Emoji in github_flavored_markdown directly, it will only be here. If you add it to blackfriday under a configuration option, then github_flavored_markdown can simply reuse it.

After all, the blackfriday README references this project.

It does because I added it there. :) Thanks for reminding me, as I need to update its import path! (Done in russross/blackfriday@10880f6.)

[thewhitetulip]

Are we going to have emoji support? I'd love to help if required 👍 I am writing a simple todo list manager github.com/thewhitetulip/Tasks and it'd be cool to have emojis directly in our awesome markdown library!

[dmitshur]

Hey @thewhitetulip,

Help would be very appreciated! If you can make a PR that adds this functionality, I'll be happy to review it.

I suggest showing me an idea of what approach you're taking as soon as possible, so I can let you know if it looks on track or not, to avoid wasting time.

[thewhitetulip]

Hey @shurcooL Well I have no idea how to add emojis 😄 But I'd be happy to work on the idea if shown the way :-)

[dmitshur]

Have you read the rest of the comments in this thread? Specifically #1 (comment), and the one above it? They go over in quite a bit of detail about how to approach this.

[thewhitetulip]

Well, to be honest I had gone through it, but not in the depth which the topic deserved. If I worked on a PR I'd like to not link blindly the the github CDN I'd like to build something which gives emoji support even without an Internet connection, the above comment thread mentions this data URLs., what are they?

Correct me if I am wrong in this approach

1. We'll have to download the image files and host it along with the rest of the package
2. It is a matter of replacing the emojis with their respective images then

Is it this simple?

[dmitshur]

I'm not sure if we should be generating output that uses image URLs from the internet, or serving the emojis ourselves so they work even when offline.

GitHub's API obviously produces Markdown output that uses online emoji links.

However, I can see that being able to generate Markdown that works when offline can be nice too.

I hate to say it, but perhaps this should be an option? Which of the two you use depends on your desires/needs.

For serving emojis, perhaps another Go package can be employed. Something similar to https://github.com/shurcooL/octicons, except for emojis.

I think it might be better/simpler to start with relying on online links, and then consider serving emojis locally as an enhancement.

[thewhitetulip]

Yes, I was meaning that we'll download the images offline and then replace them as we replace the parts of `` into code.

My need is that it should work offline, nothing else. I want to use it on my internal network :-)

[dmitshur]

My need is that it should work offline, nothing else. I want to use it on my internal network :-)

I'm okay with going with the offline-capable solution, if you're interested in working on that.

[thewhitetulip]

@shurcooL I started working on a PR for this issue and I have few doubts

my approach is this
search for :[a-z]:, check it in the list I'll have with me, for particular one matching in my list, I'll replace it with the image I will have in my local library.

I am not understanding how the code is being called, as per the main.go file, the TaskList function is literally not called anywhere, so is the highlightCode function which does github specific highlighting, there is something I am missing out, can you please point me in the right direction?

Also if we are going to give offline support, then we must decide on the strategy which we'll use, the URLs that we'll generate will be like /static/image/smile.png, so this means that for this library to work properly, we'd need to somehow embed the images too, this will add a point to deploying the library, we'll need to provide the users with a zip file which they need to keep in their static folder, or devise a mechanism where they don't need to copy it, but our code does it for them

[dmitshur]

From an email,

This is @thewhitetulip, I started working on adding emojis to your excellent library, but when I actually started reading the code, I couldn't understand how it is being executed, in my mind I have this notion that if I am to add an emoji enabled feature it should work like this

1. There will be a TranslateTextToEmoji() which will take the input text like :smile:
2. a map will store the image's name and the text name
3. find the result and <img href="" them to return

There are two questions,

1. I didn't understand how that functions are being used in your library, do i need to look at blackfriday for that?
2. How do we propose to package the images? I want this to work offline, so we'd need to package the images with the library? How do we endup doing that?

I'll answer the second question first.

How do we propose to package the images? I want this to work offline, so we'd need to package the images with the library? How do we endup doing that?

I suggest breaking up this change into two parts. First part will implement support assuming internet connection is available.

This will match the output of the GitHub API endpoint for rendering markdown. They substitute things like :smile: with s like <img class="emoji" title=":smile:" alt=":smile:" src="https://assets-cdn.github.com/images/icons/emoji/unicode/1f604.png" height="20" width="20" align="absmiddle">:

$ curl https://api.github.com/markdown/raw -d 'Hello! :smile:' --header 'Content-Type: text/x-markdown'
<p>Hello! <img class="emoji" title=":smile:" alt=":smile:" src="https://assets-cdn.github.com/images/icons/emoji/unicode/1f604.png" height="20" width="20" align="absmiddle"></p>

Once that's done, we can think about adding support for an option to serve images locally.

I have some ideas for how to achieve that. I suggest we create a generated package similar to https://github.com/shurcooL/octicons, except one that contains emoji available from https://developer.github.com/v3/emojis/. This can be discussed and figured out later.

The other question was:

I didn't understand how that functions are being used in your library, do i need to look at blackfriday for that?

Package github_flavored_markdown defines a renderer type which implements blackfriday.Renderer interface.

It does so by reusing blackfriday's Html renderer for most things, but it has custom implementations for a few methods, currently Header, BlockCode and ListItem only.

You'll probably want to add a custom implementation for NormalText and perform the substitution of :smile: and other supported emoji into <img ...> html tags there.

The list of supported emojis should come from https://developer.github.com/v3/emojis/. It can be a static Go package that is generated from that API. See https://github.com/shurcooL/reactions/blob/8077c757f3cad0c7bcb9b0c72c94b12f5e8fcf7d/generate.go#L13-L17 for some relevant code.

Does that help, @thewhitetulip? Let me know if I should clarify some parts of my answers.