[gatsby-source-wordpress] Implementing gatsby-image for WYSIWYG fields

[lightstrike]

The core feature this PR aims to add is the ability to replace <img> elements in WordPress post and page WYSIWYG fields with Gatsby <Img> components. This feature has been requested in different contexts, such as #1493, #3733 and #6799.

What I really want is to map elements to React components, exactly along the lines of what is in the pipeline for gatsby-mdx and in the using-remark custom components example. I believe enabling this will really open the floodgates for creativity in migrating from legacy CMS frameworks to Gatsby. This gets especially interesting with WordPress Gutenberg blocks, as #7465 recently brought up.

To enable the core feature, these steps were taken:

• building a HAST (Hypertext Abstract Syntax Tree) for WYSIWYG content fields of WordPress Posts and Pages using parse5
• identifying images within the WYSIWYG content using selectAll from hast-util-select
• matching WYSIWYG images with those in WordPress Media nodes through getNodes
• creating remote file nodes for images not found within Wordpress Media nodes with createRemoteFileNode
• creating optimized image files through the fluid function from gatsby-plugin-sharp
• rendering Gatsby Image-like DOM referencing sharp image files by adapting logic from gatsby-remark-images
• replacing the original img WYSIWYG DOM with the Gatsby Image markup with astToHtml from hast-util-to-html

I'm leveraging onCreateNode here with a hastOptions option added to the plugin in gatsby-config.js, creating the possibility to do any kind of transformations on any HTML field:

const { inlineImagesToGatsby } = require(`gatsby-source-wordpress/hast-functions`);
...

plugins: [
  {
    resolve: `gatsby-source-wordpress`,
    options: {
      ...
      hastOptions: [
        {
          nodeTypes: ['wordpress__POST', 'wordpress__PAGE'],
          htmlField: 'content',
          selector: 'img',
          transformFunction: inlineImagesToGatsby,
        }
      ]
      ...
    }
  }
]

While I believe (and hope!) many others will find this kind of functionality useful for gatsby-source-wordpress, I don't believe this PR should be merged as-is or even with minor modifications. It adds too much complexity and too many dependencies to the core WordPress plugin. I see the intersection of three big opportunities here:

• creating a first class gatsby-transformer-hast ecosystem to complement gatsby-transformer-remark (alluded to here)
• creating a powerful, flexible and easy pathway to extend the gatsby-source-wordpress plugin, as explored in Create RFC for extending Gatsby's WordPress plugin #5796
• more closely aligning the gatsby-source-wordpress data pipeline with the Node API to better separate source functionality from transform functionality (which is effectively done through the normalize logic currently)

With all that context in mind, my core goal is to start a discussion about how we can move the Gatsby ecosystem forward to ensure it is the absolute best tool to modernize legacy CMS frameworks like WordPress. 😄

Excited for all your thoughts.

P.S. Documenting Node API helpers more would be so very, very helpful to future endeavors like this 😉

[sebastienfi]

Very interesting feature @lightstrike , I have been willing to do that from day one of the plugin. I am fully booked on a project until the end of this year, but I am planning to come back to GatsbyJS as soon as January 2nd 2019 10am. Therefore, I won't provide you with the review you requested but I am definitely looking forward to see this feature happily merged ;-)

[ChristopherBiscardi]

Very cool

creating a first class gatsby-transformer-hast ecosystem to complement gatsby-transformer-remark (alluded to here)

From the MDX perspective this isn't needed because we already provide hooks to apply mdast/hast/rehype/etc plugins both directly, and soon wrapped in gatsby logic as well (that is, support for the gatsby-remark-* style plugin ecosystem).

---

I should test out wordpress as a source plugin sometime. Maybe it would be good to include a way to say "this MDX content is coming in as html, convert it to mdast please" and then when using the gatsby-mdx transformer option (like we do for contentful) all the mdPlugins spec'd in gatsby-mdx's options would run.

transformers: {
  ContentfulBlogPost: ({ node, getNode }) => {
    const { title } = node
    const mdxContent = getNode(node.mdxContent___NODE)
    return { meta: { title }, content: mdxContent.mdxContent }
  },
},

[roderickhsiao]

Potentially tag like iframe (similar to gatsby-remark-) could also benefits by the html AST parsing

[wardpeet]

Hey @lightstrike! Thanks for doing this work and sorry to be so late to the party. I'm a bit hesitant of putting this in the WordPress plugin as this adds some complexity to it. I love the idea and it would be great for our WordPress users!

We have a wordpress RFC running where this functionality falls right into. It would be great if you could chime in the discussion :D

[pieh]

I'll close this PR as we have much better implementation options (as in not cramming this functionality in wordpress plugin). With schema customization that landed recently, we can create generic html transformer and declare that some field have html as content, so transformer would take care of them - here's proof of concept of this approach that does similar thing - https://github.com/pieh/html-transformer-wp

[lightstrike]

@pieh This looks super promising! Is gastby-transformer-html published as its own plugin? I have a few thoughts on implementation I'll share in that repo. Also where's the best place to review how schema customization works?

Great stuff 🙌

[pieh]

It's not published, but I might do it today with 0.0.1 versions (so at very least we have dibs on gatsby-transformer-html / gatsby-html-images package names)

[pieh]

As for schema customization - we don't have guides yet - so available resources are

API reference:

• https://www.gatsbyjs.org/docs/actions/#createTypes
• https://www.gatsbyjs.org/docs/node-apis/#createResolvers

And couple of blog posts:

• https://www.gatsbyjs.org/blog/tags/schema

We also continuously work on it (i.e. #13028 ), as right now we have basic APIs that allow to adjust schema, but we don't yet have convenience functions or access to some utilities from Gatsby core that will make working with schema much nicer.

[lightstrike]

As for schema customization

Is there any potential to apply this approach toward gatsby-source-graphql? I've been experimenting with WPGraphQL and it's fun to see @jasonbahl is getting into Gatsby. Ultimately I think making transformations on stitched GraphQL APIs that would be the best long-term approach with WordPress and Gatsby.

[jasonbahl]

I’m always open to doing what I can on the WPGrapHQL side to make it easier to work with Gatsby. If there’s anything I can contribute on this end, I’m open for ideas. 😀

[pieh]

Is there any potential to apply this approach toward gatsby-source-graphql? I've been experimenting with WPGraphQL and it's fun to see @jasonbahl is getting into Gatsby. Ultimately I think making transformations on stitched GraphQL APIs that would be the best long-term approach with WordPress and Gatsby.

That's the goal. We were using mostly gatsby-transformer-remark as example for the need for this (mostly because currently there is no html transformer) - there was some discussion about this in gatsbyjs/rfcs#11

It should be possible right now (using createResolvers hook), but I didn't try it yet. I will try to find some time to check it out