How to get working with mergeSchemas

[rohit-ravikoti]

Hello,
I am trying to get this working with graphql-tool's mergeSchemas feature. Here is a contrived example:

import { GraphQLServer } from 'graphql-yoga'
import {
  mergeSchemas,
  makeExecutableSchema,
} from 'graphql-tools';

const UploadScalar = makeExecutableSchema({
  typeDefs: `
    scalar Upload
    type Query {
      _uploadTest: Boolean
    }
    type Mutation {
      _uploadTest: Boolean
      updateProfilePicture(
        picture: Upload!
      ): Boolean
    }
  `,
  resolvers: {
    Mutation: {
      updateProfilePicture(_, args) {
        console.log(args)
        return {
          success: true
        }
      }
    }
  }
})
  const server = new GraphQLServer({
    schema: mergeSchemas({
      schemas: [UploadScalar]
    })
  })
  server.start({
    port: 5444
  }, () => console.log('Server is running on http://localhost:5444'))

Basically, the Upload does not get forwarded to the schema and resolvers. I just get an empty picture object from the args.

If I use UploadScalar directly for the schema (const server = new GraphQLServer({ schema: UploadScalar })), it works. Of course, I don't do this in a real-world scenario. I would be merging multiple remote schemas which may use apollo-upload-server.

Does anyone know of a way I could work around this?

[jaydenseric]

I can't see an Upload scalar resolver defined in that example.

[rohit-ravikoti]

@jaydenseric, you're right. Sorry, I used a bad example. Here is a better one:

import { GraphQLServer } from 'graphql-yoga'
import {
  makeRemoteExecutableSchema,
  introspectSchema,
  mergeSchemas,
  makeExecutableSchema,
} from 'graphql-tools';
import fetch from 'node-fetch';
import { HttpLink } from 'apollo-link-http';
import 'apollo-link';
import { GraphQLUpload } from 'apollo-upload-server'

async function makeMergedSchema() {
  const result = await fetch('http://localhost:4000', {
    method: 'POST',
    headers: { accept: '*/*', 'content-type': 'application/json' },
    body: JSON.stringify({ query: '{viewer {me {id}}}' })
  }
  );
  console.log(await result.text());

  const RemoteLink = new HttpLink({
    uri: 'http://localhost:4000',
    fetch,
  })
  const RemoteSchema = makeRemoteExecutableSchema({
    schema: await introspectSchema(RemoteLink),
    link: RemoteLink,
  });
  // merge actual schema
  const mergedSchema = mergeSchemas({
    schemas: [RemoteSchema],
  });
  return mergedSchema;
};

makeMergedSchema().then((schema) => {
  const server = new GraphQLServer({ schema })

  server.start({
    port: 5444
  }, () => console.log('Server is running on http://localhost:5444'))
});

The remote schema is graphql-yoga (with the same mutation) and I'm simply trying to forward any requests through. If I call that service directly, I correctly get this as the args in the remote schema:

{ picture:
   Promise {
     { stream: [Object],
     filename: 'car.png',
     mimetype: 'image/png',
     encoding: '7bit' } } }

However, if I call the gateway which forwards the mutation to the remote schema, I get this:
{ picture: {} }

[danielmeneses]

Hi all,

the same here. It doesn't forward multipart requests from the gateway (where the schemas were merged) to the microservices.
Seems like I'll have to implement on my own :). I tried a stand-alone approach using apollo-client-upload to upload from the gateway to the microservice, but it also doesn't support because it uses "extract-files" package that only supports File, FileList (browser), Blob and ReactNativeFile. It would have to support FileStream format, maybe I'll see if can change that and contribute pushing some code.

Well this is all that I'm able to tell, correct me if I'm wrong!

[jaydenseric]

Interesting, I have not worked with remote merged schemas yet so I'm not in a good position to chip in. I am working day and night getting ready to announce graphql-react (which BTW happens to make it easy to query multiple GraphQL APIs on the client, without stitching) so won't be able to focus on server stuff for a while, but happy to answer any questions if you take a stab at a solution 👍

[jaydenseric]

This issue was mentioned in the Apollo Slack. Perhaps it relates? Sorry I have not immersed much.

[danielmeneses]

Yeah, it seems to be related, but I see that it's also an open issue here. From my understanding, with a ReadStream returned from a fs.createReadStream you can get the size of the file in sync fashion because the file is on your machine and therefore it's possible and a lot of core tools and packages actually get the size that way. To forward a multipart-request to another server we need to set content-length based on every piece of data we need to send.

For instance, the way apollo-upload-server parses and serves the files is by wrapping each one in a Promise. So you can call .then on it to get the file stream, but then you'll have to get the size of the file, so you need to drain the stream getting all chunks/Buffers, sum the length, then Buffer.concat(buffers) and the first problem starts here, because by this time all the data it's in memory so we are now losing some RAM space.

The other issue is that these Promises will only be resolved one by one, so if we get multiple files and you have to forward them to another server the first Promise is already resolved, but the second will only be after you have drained the first stream, so this makes even harder to get the content-length and even more RAM consuming.

Any suggestion?

[terion-name]

@danielmeneses did you handle this?

In my case with gateway on graphql-express using upload server's middleware I get an error:

import {HttpLink} from 'apollo-link-http';
import fetch from 'node-fetch';

import express from 'express'
import * as bodyParser from 'body-parser-graphql'
import { apolloUploadExpress, GraphQLUpload } from 'apollo-upload-server'
import {graphiqlExpress, graphqlExpress} from 'apollo-server-express'
import {
  introspectSchema,
  makeRemoteExecutableSchema,
} from 'graphql-tools'
import {weaveSchemas} from "graphql-weaver";
import jwt from 'jsonwebtoken';
import createLocaleMiddleware from 'express-locale';
import fetchServices from "./utils/fetchServices";

require('dotenv').config();

const cors = require('cors');

const start = async function () {

  const app = express();

  const PORT = process.env.PORT || 3000;

  app.use(cors());
  app.post('/graphql', bodyParser.graphql());
  app.post('/graphql', apolloUploadExpress());

  app.use('/graphql', async function (req, res, next) {
    // should to fetch this dynamically
    const services = await fetchServices();

    const serviceContainers = await Promise.all(services.map(async (service) => {
      const dosvitMeta = req._dosvit;
      const headers = {};
      // ... some magic with headers
      const link = new HttpLink({uri: service.url, fetch, headers});

      const remoteSchema = makeRemoteExecutableSchema({
        schema: await introspectSchema(link),
        link
      });
      return {service, remoteSchema};
    }));

    const transformedSchema = await weaveSchemas({
      endpoints: serviceContainers.map(s => {
        return {
          typePrefix: s.service.prefix,
          namespace: s.service.namespace,
          schema: s.remoteSchema
        }
      })
    });

    return graphqlExpress({schema: transformedSchema})(req, res, next);
  });

  app.use('/graphiql',
      graphiqlExpress({
        endpointURL: '/graphql',
      }),
  );

  app.listen(PORT);
};

start();

It just fails entirely:

FileStreamDisconnectUploadError: Request disconnected during file upload stream parsing.
    at IncomingMessage.request.on (.../local-gateway/node_modules/apollo-upload-server/lib/middleware.js:151:15)
    at IncomingMessage.emit (events.js:182:13)
    at resOnFinish (_http_server.js:564:7)
    at ServerResponse.emit (events.js:182:13)
    at onFinish (_http_outgoing.js:683:10)
    at onCorkedFinish (_stream_writable.js:666:5)
    at afterWrite (_stream_writable.js:480:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)
Emitted 'error' event at:
    at IncomingMessage.request.on (.../local-gateway/node_modules/apollo-upload-server/lib/middleware.js:149:32)
    at IncomingMessage.emit (events.js:182:13)
    [... lines matching original stack trace ...]
    at process._tickCallback (internal/process/next_tick.js:63:19)

[danielmeneses]

Hey @terion-name ,

I've forked the repo and I did my own implementation supporting file stream (server-side) but I didn't create any npm package. Have a look here.

[terion-name]

@danielmeneses thank you! I'll investigate it

[tlgimenes]

Nice work @danielmeneses ! Why don't you try making a PR so we can use it as an npm package ? I'm using your code and it's working perfectly fine. Thanks !

[danielmeneses]

Hi @tlgimenes, thanks. In fact, I've tried a PR already, but I guess that the repo owner got a bit apprehensive with the number of changes there were to make it work so I never got any definitive answer and I decided to cancel the PR. But I definitely understand your request, I'd also prefer to have it at npm.

[jaydenseric]

Here is the PR @danielmeneses is referring to: jaydenseric/apollo-upload-client#79.

[medeeiros]

I've been fighting with this for a couple of days now.
Thanks @danielmeneses for your PR. It was useful to get it working, but I find it annoying having to add 6 libraries on my server implementation just to workaround this issue.
Is this the right place to discuss this, or should we move the discussion to graphql-tools?

[danielmeneses]

Hi @GuilhermeMedeiros, I got you!! yes, it would be great to have that in place where we'd only need to use mergeSchemas from graphql-tools.

thanks

[jaydenseric]

The right place for this question to be answered is ardatan/graphql-tools#671, since schema merging is a graphql-tools feature; it's really up to Apollo to work out how to support standard file uploads in their products, not the other way around.

Whatever the solution ends up being, it probably won't involve graphql-upload code changes. If there is something actionable in this repo that needs to happen, we can reopen.

Also, a reminder that I monitor the #uploads chanel of the Apollo Slack team. Fee free to spitball ideas there anytime 🤙

[MeiyazhaganSelvam]

@jaydenseric - I was trying the file upload post schema stitching.
Initially, I have two graphql endppoints which are get stitched and exposed as a single graphql endpoint.
and making the graphql server using graphqlExpress.
The multipart request was working fine(tested from postman having variables operation, map and file) incase of each separate graphql servers but post stitching it was failing with the message 'Missing multipart field ‘operations’ even from postman/rest-client. But I was able to get these variables(operations, map and file) in the request body of the stitched graphql server.
I was using multer and graphqlUploadExpress for file uploading.
SERVER:
const express = require("express"),
bodyParser = require("body-parser"),
cors = require("cors"),
{ createWriteStream } = require("fs"),
fetch = require("isomorphic-fetch"),
formidable = require("formidable"),
{ graphqlExpress } = require("apollo-server-express"),
{
introspectSchema,
makeRemoteExecutableSchema,
mergeSchemas,
makeExecutableSchema,
} = require("graphql-tools"),
{ HttpLink } = require("apollo-link-http"),
{ GraphQLUpload, graphqlUploadExpress } = require("graphql-upload"),
{
GraphQLSchema,
GraphQLObjectType,
GraphQLBoolean,
GraphQLString,
} = require("graphql"),
multer = require("multer"),
https = require("https");

const PORT = 9000;

/----------------First End Point----------------------------------/

async function createRemoteSchematest1() {
const link = new HttpLink({
uri: "https://localhost:8443/graphql",
headers: {
Accept: "application/json",
Authorization: "Basic ......",
},
fetchOptions: {
agent: new https.Agent({ rejectUnauthorized: false }),
},
fetch,
});

const schema = await introspectSchema(link);

return makeRemoteExecutableSchema({
schema,
link,
});
}

const storeUpload = ({ stream, filename }) =>
new Promise((resolve, reject) =>
stream
.pipe(createWriteStream(filename))
.on("finish", () => resolve())
.on("error", reject)
);

/----------------Second End Point----------------------------------/
async function createRemoteSchematest2() {
const link = new HttpLink({
uri: "http://localhost:8080/test2",
headers: {
Accept: "application/json",
},
fetchOptions: {
agent: new https.Agent({ rejectUnauthorized: false }),
},
fetch,
});

const schema = await introspectSchema(link);

return makeRemoteExecutableSchema({
schema,
link,
});
}

function searchOriginalError(error) {
if (error.originalError) {
return searchOriginalError(error.originalError);
}
if (error.errors) {
return error.errors.map(searchOriginalError)[0];
}
return error;
}

function createStitcherSchema() {
const bootTime = Date.now();
const stitcherSchema = makeExecutableSchema({
typeDefs: `
type StitcherStatus {
name: String!
nodeJsVersion: String!
uptime: String!
graphiQL: String
}

  type Query {
    stitcherStatus: StitcherStatus!
  }

  scalar Upload
  type File {
    id: ID!
    path: String!
    filename: String!
    mimetype: String!
    encoding: String!
  }
  type Mutation {
    operations: String,
    map: String,
    file: File,
    UploadFile(file: Upload!): File
  }
`,
resolvers: {
  Mutation: {
    async UploadFile(parent, { file }) {
      const { stream, filename, mimetype, encoding } = await storeUpload({
        stream,
        filename,
      });

      return { filename, mimetype, encoding };
    },
  },
  Query: {
    stitcherStatus: () => ({
      name: "schema-stitcher",
      nodeJsVersion: process.versions.node,
      uptime: `${(Date.now() - bootTime) / 1000}s`,
      graphiQL: `http://localhost:${PORT}/graphiql`,
    }),
  },
},

});
return stitcherSchema;
}

async function createCombinedSchema() {
// STEP 1: Create the Remote Schemas
const test1Schema = await createRemoteSchematest1();
const testSchema = await createRemoteSchematest2();

// STEP 4: Merge Schemas
return mergeSchemas({
schemas: [
test1Schema,
test2Schema,
createStitcherSchema(),
],
});
}

(async () => {
const combinedSchema = await createCombinedSchema();

const app = express();

app.use(cors());
app.options("", cors());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
const storage = multer.memoryStorage();
app.use(multer({ storage }).single("sample.pdf"));
app.use(function(req, res, next) {
res.setHeader("Access-Control-Allow-Origin", "");
res.header("Access-Control-Allow-Methods", "GET, POST, OPTIONS");
next();
});
app.use(
"/graphql",
graphqlUploadExpress({ maxFileSize: 10000000, maxFiles: 10 }),
graphqlExpress({
schema: combinedSchema,
})
);
app.use(function(req, res, next) {
console.log(req);
next();
});
app.use(
"/graphiql",
express.static(${__dirname}/graphiql)
);

app.listen(PORT, () => {
console.log(Schema Stitcher running on ${PORT});
console.log(
Schema Stitcher GraphiQL http://localhost:${PORT}/graphiql
);
});
})();

CLIENT:

formData.set("operation",JSON.stringify({query: 'mutation.........', variables: {}'}));
formData.set("map", JSON.stringify({ "sample.pdf": ["variables.file"] }));
formData.set('sample.pdf',file);
return fetch("http://localhost:9000/graphql", {
method: "post",
headers: {
Accept: "application/json",
},
body: JSON.stringify(formdata),
//credentials: 'include',
})
.then(function(response) {
return response.text();
})
.then(function(responseBody) {
try {
return JSON.parse(responseBody);
} catch (error) {
return responseBody;
}
});

I was getting the parameters on request body on merged graphql server but the error says 'Missing multipart field ‘operations'.
Could you please look in to it did I miss anything? or did I need to use processRequest or uploadOptions?

[JMA12]

@tlgimenes, How did you implement @danielmeneses's sollution??

[yaacovCR]

graphql-tools-fork v8.1.0 provides support for proxying remote file uploads for those who want to proxy all the things.

The new createServerHttpLink method should be used to set up a terminating link to a subschema; it will resolve all promises within the variables, and then, if appropriate, append the File or stream to the FormData.

The GraphQLUpload scalar export from graphql-tools-fork is a one-line code change from the original from the graphql-upload package that allows use on the gateway.

Feedback/critiques/suggestions would be much appreciated. I am happy to submit PRs where appropriate, this is more POC.

Please join in further discussion at: jaydenseric/apollo-upload-client#172 (comment)