Hasura Lexer

Hasura is a basic lexer implemented in Go. Designed for tokenizing source code.

Overview

Hasura provides functionality to break down source code into tokens, identifying syntax elements such as parentheses, integers, and identifiers.

Usage

Installation

go get github.com/ezrantn/hasura

Basic Usage

package main

import (
	"fmt"
	"github.com/ezrantn/hasura"
)

func main() {
	// Create a new LexingContent
	lc := hasura.NewLexingContent("example_source_file")

	// Get the tokens from the source
	tokens := lc.Lex()

	// Print the tokens
	for _, token := range tokens {
		fmt.Println(token.Value)
	}
}

Token Types

Hasura recognizes the following types of tokens:

• Syntax Tokens: Such as '(' and ')'
• Integer Tokens: Numeric values, e.g '123'
• Identifier Tokens: Alphanumeric strings, e.g 'foo', 'define''

API

Hasura provides two main types that are relevant to tokenization:

1. LexingContent:

   • This represents the content to be lexed (tokenized).
   • It contains the source content and filename.
   • The main method associated with this type is lex(), which tokenizes the source content and returns a slice of tokens.

2. Token:

   • This represents a token generated by the lexer.
   • It has several fields:
     • Value: The actual string value of the token.
     • Kind: The type of token (syntax, integer, or identifier).
     • Location: The position of the token in the source content.
     • LC: The LexingContent instance associated with the token.
   • The main method associated with this type is debug(), which provides debugging information about the token's location in the source content.

In summary, the LexingContent type is responsible for managing the source content and initiating the tokenization process, while the Token type represents individual tokens generated during tokenization, providing information about their type, value, and location in the source content.