Feature request: Provide ability to refernce and modify Log Level Numerical Thresholds

[OffensiveBias-08-145]

Use case

Currently our logging standard has both the numerical and string representations of the Log Level. The usage of both allow us to have increased efficiency when retrieving or searching logs from storage.

Indexing the numerical value is more efficient that its string counterpart.

{
  level: 'INFO',
  level_index: 12,
}

Currently it is difficult to directly access the prescribed log levels when extending the Logger class or utilizing a custom log format.

Some of the workarounds include:

• Extending UnformattedAttributes to include the LogLevel Numerical Value (Works but adds another custom interface I then need to manage)
• Manually mapping LogLevel (str) to its corresponding numerical value (Works but extra code is required)
  • With TS, this also means that we also have to force uppercase matching (Less LOC than a switch) since LogLevel used in UnformattedAttributes allows for upper and lowercase values.'
  • Since one cannot directly access the perscribed LogLevel thresholds, one must copy it manually and later update it should it change.

  class CustomLogFormatter extends LogFormatter {
    private getLogLevelNumberFromName(level: LogLevel): number {
      let found = 0;
      for (const [key, value] of Object.entries({
        DEBUG: 8,
        INFO: 12,
        WARN: 16,
        ERROR: 20,
        CRITICAL: 24,
        SILENT: 28,
      })) {
        if (key.toUpperCase() === level) {
          found = value;
          break;
        }
      }
      return found;
    }

    override formatAttributes(
      attributes: UnformattedAttributes,
      additionalLogAttributes: LogAttributes
    ): LogItem {
      const baseAttributes = {
        timestamp: this.formatTimestamp(new Date()),
        level: attributes.logLevel.toUpperCase(),
        level_index: this.getLogLevelNumberFromName(attributes.logLevel),
  
     //........

If I am missing something evident please let me know.
I would be happy to open an PR for this.

Solution/User Experience

Ideal Solutions:

• logLevelThresholds changed from private to protected OR utilize a "getter"
• Add an additional property to UnformattedAttributes or PowertoolsLogData that represents the numerical LogLevel.

This way the numerical log level is easily passed to any Custom log Formatter instead of having to map it using LogAttributes or some other workaround.

Adding this in the other libraries would be beneficial as well!

//Open to feedback on the prop name

type PowertoolsLogData = LogAttributes & {
    environment?: Environment;
    serviceName: string;
    sampleRateValue: number;
    lambdaContext?: LambdaFunctionContext;
    xRayTraceId?: string;
    awsRegion: string;
    logLevelIndex?: number; //OPTION #1 
};
type UnformattedAttributes = PowertoolsLogData & {
    error?: Error;
    logLevel: LogLevel;
    logLevelIndex: number; //OPTION #2
    timestamp: Date;
    message: string;
};

Alternative solutions

Creating a helper function in a Custom Log Formatter to map the string LogLevel to its numerical counterpart.

  class CustomLogFormatter extends LogFormatter {
    private getLogLevelNumberFromName(level: LogLevel): number {
      let found;
      for (const [key, value] of Object.entries({
        DEBUG: 8,
        INFO: 12,
        WARN: 16,
        ERROR: 20,
        CRITICAL: 24,
        SILENT: 28,
      })) {
        if (key.toUpperCase() === level) {
          found = value;
          break;
        }
      }
      return found ?? 0;
    }

    override formatAttributes(
      attributes: UnformattedAttributes,
      additionalLogAttributes: LogAttributes
    ): LogItem {
      const baseAttributes = {
        timestamp: this.formatTimestamp(new Date()),
        level: attributes.logLevel.toUpperCase(),
        level_index: this.getLogLevelNumberFromName(attributes.logLevel),
  
     //........

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

Hi @OffensiveBias-08-145, thank you for taking the time to open this issue and for writing this thoughtful proposal.

I see the value in supporting this use case and I fully agree on the direction of making this an opt-in feature, however I am still not sure on what would be the best way to do this in a backwards compatible way.

To help me understand better your proposal, could you please clarify if the ideal solutions are meant to be taken together or only one of them?

I'm asking because changing the logLevelThresholds to protected is a very low hanging fruit, so if that's all you'd need to enable this I'd be open to review & merge a PR fairly quickly.

If this is the case, could you share an example of how you'd use it in practice, if it were protected? I guess you're thinking of extending the Logger, but I'm not sure how you'd propagate it to the LogFormatter.

Regarding the other option of adding logLevelIndex to the attributes passed to the formatAttribute function, I think it's a sensible proposal, but one that requires a bit more consideration. It could make sense to pass the value, however I'm not sure if it'll impact existing customers since we don't know how they're handing the attributes in the formatAttribute.

[OffensiveBias-08-145]

@dreamorosi I can see how the issue is slightly confusing.
The issue template made more sense in my head when I was doing it.

Hopefully the below provides clarity??

Potential ideal solutions:

In an ideal world I would love to see both 1 and 2 added

1. Change logLevelThresholds to protected and correctly type it to allow the record to be indigestible.

This would not be propgated to LogFormatter. It would only be used for ingestion elsewhere or in other libraries. (Ex: Aligning OpenTelemetry and this logging lib)

EXAMPLE:

//=== UPDATED CLASS PROPERTY ===
class Logger extends Utility implements LoggerInterface {
  //...
  protected readonly logLevelThresholds: Record<Uppercase<LogLevel>, number>;
  //...
}




//=== EXAMPLE USAGE WITH AN EXTENDED LOGGER ===

  export class CustomLogger extends PowertoolsLogger {

    constructor(options: ConstructorOptions = {}) {
      //Override any accidentally passed custom log formatter
      const {logFormatter, rest} = options;
      super({logFormatter: new CustomLogFormatter(), ...rest});
    }

    //getter could also be addeded to the lib itself. 
    public static get logLevelThresholds(): Record<Uppercase<LogLevel>, number> {
      return this.logLevelThresholds;
    }
  }
 //...
}

// === EXAMPLE INGESTION INTO EXTERNAL METHODS ===

    private getLogLevelNumberFromName(level: LogLevel): number {
      let found = 0;
      for (const [key, value] of Object.entries(
        Logger.logLevelThresholds)) {
        if (key.toUpperCase() === level) {
          found = value;
          break;
        }
      }
      return found;
    }

2. Add an optional logLevelIndex to the UnformattedAttributes OR PowertoolsLogData types.

As an optional, this would allow devs the discretion to add the numerical level value to their logs.

By default it can go unused, and if devs want to have it, they can access it by using a custom formatter and using the property itself.

//AN EXAMPLE USAGE WOULD BE AS FOLLOWS:



// === UPDATED TYPES ===
type PowertoolsLogData = LogAttributes & {
    environment?: Environment;
    serviceName: string;
    sampleRateValue: number;
    lambdaContext?: LambdaFunctionContext;
    xRayTraceId?: string;
    awsRegion: string;
};
type UnformattedAttributes = PowertoolsLogData & {
    error?: Error;
    logLevel: LogLevel;
    logLevelIndex?: number; //<-- Could also be moved to PowertoolsLogData
    timestamp: Date;
    message: string;
};


// === USAGE IN A CUSTOM FORMATTER ===

    override formatAttributes(
      attributes: UnformattedAttributes,
      additionalLogAttributes: LogAttributes
    ): LogItem {
      const baseAttributes = {
        timestamp: this.formatTimestamp(attributes.timestamp),
        time: attributes.timestamp.getTime(),
        level: attributes.logLevel.toUpperCase(),
        level_index: attributes.logLevelIndex, //<-- index passed in custom formatter
        message: attributes.message,
        error: attributes.error ? this.formatError(attributes.error) : null,
        //....
        },
      };

      const logItem = new LogItem({attributes: baseAttributes});
      logItem.addAttributes({_data: additionalLogAttributes});
      return logItem;
    }