Azure ServiceBus Webpack Issue

[AdrianMcGrathFlailsoft]

We have a fairly basic consumer for Azure ServiceBus, which we are building with webpack. When we deployed it a month ago, we noticed it was not able to consume messages. It would just sit there and hang after establishing a connection. On the ServiceBus side, it appears that it was attempting to send a message to the consumer, but the consumer was just never getting it or not reacting to it or something like that. If we ran it locally, it would work fine. If we ran it in Azure, it wouldn't consume. We opened a support ticket with Azure where we investigated a handful of different things.

Eventually, we realized that when we ran it locally unbuilt (typescript), it was fine. But after we built it with webpack and tried to run the built code, we would see the same issues as we do when we deploy it. This leaves us believing that there is some kind of issue in the webpack build where it is silently breaking the functionality of the Azure ServiceBus library.

Azure support at this point has been extremely unhelpful. They insist their library works fine with webpack, even though we were able to send them a very basic example of the issue being replicated. Clearly, we aren't going to get much help from them on this issue.

Can anyone provide any good suggestions or tips for how we can properly investigate this on our side? We don't know if some critical dependency is getting tree-shaked out or something like that, and we aren't really sure how to figure that out either.

[alexander-akait]

Can you provide more details - errors/messages/warnings? What is broken and doesn't work? How do you use the library? What is the configuration? Do you have externals? Can you provide a webpack configuration?

[AdrianMcGrathFlailsoft]

What's making it difficult is that we aren't getting error or warnings or whatever. It's just... hanging... I've attached a zip folder which replicates the problem. Anyone who tries it would need an Azure ServiceBus queue set up with Azure Managed Identity and all that nonsense, so asking anyone to actually pull it down and run it might be asking too much, lol.

The way the library is used in this example is quite basic. We are just creating a client and receiver and subscribing.

const client = new ServiceBusClient(serviceBusHostname, new DefaultAzureCredential());
const receiver = client.createReceiver(serviceBusQueueName, {
    identifier: serviceBusIdentifier,
    receiveMode: "peekLock"
});

receiver.subscribe({
    processMessage: async (message: ServiceBusReceivedMessage) => {
        ...
    },
    processError: async (args: ProcessErrorArgs) => {
        ...
    }
}, { autoCompleteMessages: true });

It hangs somewhere deep that subscribe. Every 10 minutes, we get an error from ServiceBus indicating that there has been 10 minutes of inactivity and it's shutting down the connection and recreating it. We shouldn't be getting that message in general, let alone in this situation, because the queues we are listening on have nearly a million records backlogged. Other consumers we have, which are not using webpack, never see that error even after they've sat around for hours waiting for a message.

I believe that error indicates that the handshake between the consumer and ServiceBus has been initialized, but the consumer is never getting the message handoff, so it times out after 10 minutes and reestablishes that connection in an attempt to get a different message.

Below is the webpack file we are using:

/// <reference types="webpack/module" />

import * as path from "node:path";
import type { RequestHandler } from "express";
import { globSync } from "glob";
import type { Configuration } from "webpack";

/** CLI args that can be passed to the config */
export interface Args {
    /**
     * Indicate if this build is a production build or not.
     * @default true
     */
    prod: boolean;

    /**
     * Glob pattern for selecting the entrypoint files for the build.
     * @default ./src/apps/*.ts
     */
    entrypoints: string;

    /**
     * Which part of the filepath of the entrypoint should be the name of the artifact.
     * "dirname" or "filename"
     * @default filename
     */
    artifactPart: "dirname" | "filename";

    /**
     * Comma-separated list of modules to require at the top of the app.
     * @default source-map-support/register, reflect-metadata
     */
    requires: string;

    /**
     * Comma-separated list of modules to not import into the build.
     * @default @koa/cors, @koa/multer, long, pino-pretty
     */
    ignore: string;
}

function baseConfig(cliArgs: Partial<Args>): Configuration {
    // Parse CLI args
    const args: Args = Object.assign(
        {
            prod: false,
            entrypoints: "./src/apps/*.ts",
            artifactPart: "filename",
            requires: "source-map-support/register, reflect-metadata",
            ignore: "@koa/cors, @koa/multer, long, pino-pretty",
        },
        cliArgs);

    // Calculate entrypoints
    const entries = globSync(args.entrypoints).map((p) => [
        pathToArtifactName(p)[args.artifactPart],
        args.requires.split(/\s*,\s*/).concat("./" + p),
    ]);

    // Define base configuration
    const config: Configuration = {
        entry: Object.fromEntries(entries),
        mode: args.prod ? "production" : "development",
        target: "node",
        devtool: "inline-source-map",
        output: {
            filename: "[name]/index.js",
            path: path.resolve(process.cwd(), "dist"),
            clean: true,
            devtoolModuleFilenameTemplate: "[absolute-resource-path]",
            library: {
                type: "commonjs2",
            },
        },
        module: {
            rules: [
                {
                    test: /\.(ts|tsx)$/i,
                    loader: "ts-loader",
                    options: {
                        compilerOptions: {
                            declaration: false,
                        },
                    },
                },
            ],
        },
        plugins: [],
        externals: {
            "@azure/functions-core": true,
        },
        resolve: {
            extensions: [".ts", ".js", ".json"],
            alias: Object.fromEntries(
                args.ignore.split(/\s*,\s*/).map((r) => [r, false]),
            ),
        },
        optimization: {
            splitChunks: false,
        },
        node: {
            __filename: true,
            __dirname: true
        },
        watchOptions: {
            aggregateTimeout: 1000,
            ignored: /node_modules/,
        }
    };

    // *** Load font for the banner text
    config.module!.rules!.push(
        {
            test: /figlet[/\\]lib[/\\]node-figlet\.js$/,
            loader: "string-replace-loader",
            options: {
                // This snippet of code in `figlet` reads font files from disk
                search: /var fontData = fs\.readFileSync\(path\.join\(fontDir, name \+ "\.flf"\), {.+?}\);/s,
                // We replace it with a require so webpack can bundle the files
                replace: `var fontData = require.context("figlet/fonts/", false, /Standard/)("./" + name + ".flf");`,
            },
        }, {
            // Treat all files in figlet/fonts as plain text to be inlined
            test: /figlet[/\\]fonts/,
            type: "asset/source",
        }
    );

    // *** Serve Swagger UI assets
    config.module!.rules!.push(
        {
            test: /swagger-ui-express[/\\]index\.js$/,
            loader: "string-replace-loader",
            options: {
                // This snippet of code in swagger-ui-express configures an express static middleware that reads from disk
                search: /var swaggerAssetMiddleware = options => \{.+?return.+?}/s,
                // We replace it with an express middleware that 'requires' the files instead of reading from disk
                replace: swaggerAssetMiddleware.toString(),
            },
        }, {
            // Treat all files in swagger-ui-dist as plain text to be inlined
            test: /swagger-ui-dist/,
            type: "asset/source",
        }
    );

    return config;
}

/**
 * Utility function to pull an artifact name out of the entrypoint file path
 */
function pathToArtifactName(path: string): Record<"dirname" | "filename", string> {
    const regex = /[/\\]([\w-.]+)[/\\]([\w-.]+)(\.\w+)/;

    const result = regex.exec(path);
    if (!result) {
        throw new Error(`Path '${path}' does not match against Regex ${regex}.`);
    }
    const [_, dirname, filename, ext] = result;
    return {dirname, filename: filename.replace(".", "-")}
}

/**
 * Replacement function for `swagger-ui-express.swaggerAssetMiddleware`.
 *
 * Express middleware that serves static content from the `swagger-ui-dist` package.
 */
function swaggerAssetMiddleware(options: Record<string, unknown> = {}): RequestHandler {
    options.index = false;
    const staticFiles = require.context("swagger-ui-dist", false);
    return (req, res, next) => {

        // Redirect `/explorer` => `/explorer/`
        if (req.path === "/" && !req.originalUrl.endsWith("/")) {
            const redirect = req.originalUrl + "/";

            res.statusCode = 301;
            res.setHeader('Location', redirect);
            res.end();
            return;
        }

        let filename = "." + req.url;

        // Handle root `/` request
        if (filename === "./") {
            if (options.index) {
                filename = "./index.html";
            } else {
                next();
                return;
            }
        }

        // Serve files
        if (staticFiles.keys().includes(filename)) {
            // If file exists...
            const filetype = filename.split(".").pop();
            if (filetype) {
                res.type(filetype);
            }
            res.send(staticFiles(filename));
        } else {
            // If file doesn't exist...
            next();
        }
    };
}

const config = baseConfig({});
export default config;

It really feels like webpack is removing something (like a dependency or whatever) critical to that message handoff, but I have no clue what that could be. We're also very confused because we don't get errors about missing files or missing dependencies, which we usually do get if dependencies are accidently removed. It just... silently does nothing... If webpack is removing something, it would probably need to be something that one of these libraries doesn't freak out about not having. Perhaps like some really important optional dependency or something.

service-bus-issue-example-public.zip

[alexander-akait]

Firstly let's try to check it out again using the development mode, This will allow you to understand whether it breaks any of the optimizations, or is it actually an error in the bundling

[alexander-akait]

Also please run npm ls webpack and npm ls typescript

[AdrianMcGrathFlailsoft]

For webpack, we have:

+-- @types/webpack@5.28.5                                            
| `-- webpack@5.89.0 deduped                                         
+-- string-replace-loader@3.1.0                                      
| `-- webpack@5.89.0 deduped                                         
+-- ts-loader@9.5.1                                                  
| `-- webpack@5.89.0 deduped                                         
+-- webpack-cli@5.1.4                                                
| +-- @webpack-cli/configtest@2.1.1                                  
| | `-- webpack@5.89.0 deduped                                       
| +-- @webpack-cli/info@2.0.2                                        
| | `-- webpack@5.89.0 deduped
| +-- @webpack-cli/serve@2.0.5
| | `-- webpack@5.89.0 deduped
| `-- webpack@5.89.0 deduped
`-- webpack@5.89.0
  `-- terser-webpack-plugin@5.3.10
    `-- webpack@5.89.0 deduped

For typescript, we have:

+-- ts-loader@9.5.1
| `-- typescript@5.4.2 deduped
+-- ts-node-dev@2.0.0
| `-- typescript@5.4.2 deduped
+-- ts-node@10.9.2
| `-- typescript@5.4.2 deduped
`-- typescript@5.4.2

When we run the build with development mode, this the output:

asset app/index.js 6.68 MiB [emitted] (name: app)
orphan modules 61.8 KiB [orphan] 42 modules
runtime modules 1.04 KiB 5 modules
modules by path ./node_modules/ 2.3 MiB 449 modules

• 24 modules
  webpack 5.89.0 compiled successfully in 3463 ms

[AdrianMcGrathFlailsoft]

We had another person looking into this, and it sounds like it could have been the long package we are ignoring that is causing the issue. I will update this tomorrow after further testing.