Errors with external content.sources ?

[mathieunicolas]

Hi Dave ! I come from here nuxt/content#1837

Describe the bug

I use content.sources in nuxt.config.ts to fetch a repo where there's MD files and some media files.
I tried your module, and I have this error repeatedly, and a 404 error when I go to :3000 :

[nuxt] [request error] [unhandled] [500] The "path" argument must be of type string. Received undefined                                   23:02:35
  at new NodeError (node:internal/errors:399:5)  
  at validateString (node:internal/validators:163:11)  
  at Object.join (node:path:1176:7)  
  at getDocPath (./.nuxt/dev/index.mjs:598:15)  
  at ./.nuxt/dev/index.mjs:607:22  
  at ./node_modules/hookable/dist/index.mjs:48:66  
  at ./node_modules/hookable/dist/index.mjs:48:56  
  at async parseContent (./.nuxt/dev/index.mjs:3574:3)  
  at async getContent (./.nuxt/dev/index.mjs:3550:18)  
  at async Promise.all (index 1)

To Reproduce

add this to a docus template and install nuxt-content-assets :

  content: {
    sources: {
      gh: {
        driver: 'github',
        repo: 'mathieunicolas/alloe',
        prefix: '/'
      }
    },
    navigation: {
      fields: ['auteur']
    }
  },

Additional context

Add any other context about the problem here

Software versions:

Run the code below and replace with the output:

Node v19.6.0

docus-starter@0.1.0 /Users/mathieu/WebstormProjects/alloe-site
├── nuxt-content-assets@0.6.1
└── nuxt@3.3.3

Does your module handle external sources ? Everything works fine if I manually copy the content of the repo in my content folder.

Cheers from France !

[davestewart]

Hello!

Thanks for reporting your use case and the bug.

Right now, no, it doesn't support external sources, but only because I haven't used them, so don't know anything about them (and had assumed they wouldn't work).

If I point the demo at your repo I'll be able to run it?

If so I can dig in and see if it's possible to properly integrate.

I'll report back!

[mathieunicolas]

Okay thanks, I thought it would work since you posted your module in an issue where I was talking about external sources ^^

I will continue developing my project by manually pasting the external repo in the content folder.
When I posted the issue, my code worked. But now it doesn't work anymore, so I'm afraid I won't be able to help in the next 10 days. We can look after that if you want !

I paste here the nitro plugin that worked few months ago :

import fs from "fs";

export default defineNitroPlugin((nitroApp) => {
    nitroApp.hooks.hook('content:file:beforeParse', async (file) => {
        const extension = file._id.split('.')
        const statics = ['png','jpg','pdf','epub', 'excalidraw']
        if (statics.includes(extension.pop())) {
            const filepath_arr = file._id.split(':') // array of driver name, then every subfolder, then filename

            const filepath = './public/'+filepath_arr.slice(1).join('/') // contains the full path to the filename
            const filedir = './public/'+filepath_arr.slice(1, -1).join('/') // the folder to create because it contains the file

            fs.mkdirSync(filedir, { recursive: true })

            if(file.body.constructor.name === 'Blob') {
                const imageBuffer = await file.body.arrayBuffer();
                fs.createWriteStream(filepath).write(Buffer.from(imageBuffer))
            } else if (true) { // some space to be more precise if needed, idk how 
                fs.createWriteStream(filepath).write(JSON.stringify(file.body, undefined, 2))
            }
        }
    })
})

PS : the code was a dirty poc, waiting for refining !

[mathieunicolas]

Hi, for the external source ?

…

On 7 Apr 2023, at 22:26, Dave Stewart ***@***.***> wrote: Do you have a live project you can share, with a sample content configuration? — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APTIBMGC2EBDFUI46UAYUVDXABZ75ANCNFSM6AAAAAAWV4LNYM>. You are receiving this because you authored the thread.

[davestewart]

Don't worry, I figured it!

No freaking idea how to intercept the paths generated by Nuxt Content though.

I know it gets them using unstorage, but no idea how or where I am supposed to hook into this functionality.

[davestewart]

Hehey... a little progress in setting up the storage and grabbing the images from the repo...

[
  'gh:alloe18:3arR5cP.jpg',
  'gh:alloe18:machine.png',
  'gh:alloe18:pommier.png',
  'gh:alloe18:puit.png',
  'gh:alloe18:reparation.png',
  'gh:alloe18:rouages.png',
  'gh:alloe18:tonneaux.png',
  'gh:alloe19:JzU6Iyo.jpg',
  'gh:alloe20:Zx13oLOe.png',
  'gh:alloe21:468zwUH1.png',
  'gh:logo-alloe.excalidraw',
  'gh:public:alloe2018.epub',
  'gh:public:alloe2018.pdf',
  'gh:public:alloe2019.epub',
  'gh:public:alloe2019.pdf',
  'gh:public:alloe2020.epub',
  'gh:public:alloe2020.pdf',
  'gh:public:alloe2021.epub',
  'gh:public:alloe2021.pdf'
]

And it looks like I can get these as Blobs, so maybe can use your code above...

[
  Blob {},
  Blob {},
  Blob {},
  ...

I don't have a lot of time to spare at the moment, but 🤞 for a quick win.

[davestewart]

OK! Some good news; I've got the images saving to disk!

So I think I have all the pieces now to complete the puzzle.

Will report back...

[mathieunicolas]

Woaaaaw you're so quick !!
Btw, I used your module with local content files, and it's amazing, thanks for sharing such a useful tool !!

[davestewart]

So I got it working!

Local content folder:

Remote GitHub repo:

What a relief 😅

I just need to:

• check over the code
• commit the changes
• update the docs
• publish the package

And then you can test 💪

[davestewart]

Rats; thought I had it all working, but no.

[mathieunicolas]

I won't be able to test until monday, congrats on what you've accomplished ! What's not working ?

[davestewart]

It works perfectly in dev, but in static generation the images aren't fetched so the copy step breaks.

I was out last night but I'm going to look at it first thing today.

[davestewart]

OK! As of version 0.7.0 Nuxt Content Assets now supports GitHub external sources 🚀

[mathieunicolas]

Congratulations @davestewart for this ultra quick feature add ! It's amazing.

I have a problem with SRC rewriting on IMG tags :
https://64331db4db504a69ee36d552--kaleidoscopic-brioche-dc7d6b.netlify.app/alloe18/partie1/chapitre1/

you can find here the generated page for this file

my nuxt.config.ts:

  'content-assets': {
    output: '/[folder]/[file]',
    additionalExtensions: 'epub',
  },

The file machine.png is inserted in the /alloe18/partie1/chapitre1.md file with the code ![image](../machine.png),
BUT it is copied in the path /alloe18/machine.png
AND is referenced as alloe18/partie1/alloe18/machine.png in the generated

I guess alloe18/partie1 is the ../machine.png from the MD, and alloe18/machine.png is the correct path of the file, so maybe there's a concatenation where there should be a replacement of the image URL string ?

Again, thanks and congratz !

The issue is closed so I don't know if you will be notified for this message, let me know !

[davestewart]

Actually @mathieunicolas – can you create this as a new issue please?

🙏

PS. The 0.8.0 version has done away with the extensions configuration. Anything now that is not a markdown file is presumed to be an assset!

[shareable-vision]

Question: Does the external Github source work dynamically? In other words, is it a requirement of the assets module to re-build the application in order to serve static assets from an external source?

Ideally what I want is for the application to build once and automatically reflect changes to external sources. I just wrote a webdav driver for unstorage, and my goal is for authors to be able to simply write content on their own desktop or online via NextCloud. This currently works with Nuxt Content module, although I needed to submit a PR for Nuxt Content to support building custom drivers.

I am interested in helping out with Nuxt Content Assets module to make this possible.

[davestewart]

Hey @shareable-vision,

Sorry, only just saw this.

Short answer I think is "no".

Caveat: I'm not really a Nuxt expert, more of a "build what I think I need" kind of plugin developer.

As I understand it, Nuxt Content is a build-time operation; it generates cached files and these are served by Content's API.

Nuxt Content Assets runs immediately after this initial caching, detects relative images, then copies images from content to public, and finally rewrites paths as absolute paths, as outlined here.

I actually don't know enough about Nuxt's other rendering modes to know what the options would be.

TBH, it took a fair bit of investigation and probably a bit of luck to get NCA working and I'm slightly hesitant to unpick it all but it's OSS of course, so more than happy for you to have a look into it 🙂

[shareable-vision]

Thanks for the response, @davestewart.

After running the gamut, I am making the conclusion that this kind of behavior is ubiquitous across all front-end frameworks (take dynamic sources and incessantly re-build them to static bundles).

I am focusing my work on making publishing simple for authors and not for developers per se with as little re-building and deployment operations as possible.

Thanks for your awesome project! There's no way for Nuxt Content to be viable without being smart enough to bundle the assets . . The Nuxt team should take notice of your work and attempt to build a similar solution into their main module.

As for me, I am bowing out of front-end framework universe to get closer to the DOM and the client runtime.

[davestewart]

@shareable-vision - I don't blame you!

TBF I think even projects like Astro don't seem to intuitively support relative paths which is a surprise to me.

It just seems something obvious that it should be supported at the expense of more background gymnastics.

Anyway. Good luck with your endeavours!

:D