The new architecture of RSpack

[hyf0]

We want to make a bundler that has the following principles.

• extensible
• productive
• performant

Productive means the bundler needs to consider real-world cases in any circumstances.

To achieve the goal of being productive, the bundler needs to be extensible to implement any features easily without huge changing in the codebase. Thus, the bundler will have a robust plugin system.

Performant means the bundler could handle the real-world cases scalably.

General

The bundler has two parts. The core and plugin system.

The core

The responsibilities of the core are

• Scheduling the parsing process
• Providing plugin API to customize the behaviors without changing the logic of the core.
• Make a module graph that describes the relationships between modules.
  • The core won't know whether the module is a javascript module, CSS module, or JSON module. For the core, a module is just an object that has dependencies. The dependency only describes which modules are connected.
  • The details of the module are only cared and stored by plugins.
• Make a chunk graph that describes the relationships between chunks and which modules are in it.
  • generate basic chunks if enable code splitting

That is it. These are the critical points of the core in bundler.

Plugin system

The plugin system is still in the design, but the basic idea is that the bundler will provide a Plugin trait that has many methods. The methods will be called in some special timing.

• version of the main branch
• version of the new architecture

Process

Create Compiler

Compiler

First, we will create a Compiler and invoke Compiler#run() to start the build process.

Compiler organized the whole process of bundling.

• build module graph
• build chunk graph
• manage assets/plugins
• write assets to disk

Compiler will create a Compilation as the context of bundling to share some data.

Build ModuleGraph

Then, we will create Dependencys by the entry options user input.

Dependency

Dependency is a struct that describes a relation between modules(which module imports which module in what way).

code

The Dependency will be transformed into the ResolvingModuleJob struct. By running these ResolvingModuleJobs we start the process of building a module graph.

ResolvingModuleJob

ResolvingModuleJob is a struct that represents the process of resolving Module.

In the ResolvingModuleJob

1. Resolve the URI of the module through the Dependency.

URI

URI is the location of the source code. In most cases, the URI is the disk path of the javascript file.

2. Load the source code by URI
3. Call the transform hook of plugins to transform the source code.
4. Call the parse_module hook to parse the source code
5. Get dependencies by the parsed result and resolve these Dependencys.
6. Wrap the parsed result in ModuleGraphModule and store it in ModuleGraph.

ModuleGraphModule

ModuleGraphModule is a wrapper of the parsed result, the goal is to store some extra information the bundler cared about.

ModuleGraph

ModuleGraphModule stored modules and their relations. We could get a ModuleGraphModule from ModuleGraph in multiple ways.

ResolvingModuleJobs are running in different threads, so we need to deal with concurrent programming. The bundler has a global counter active_task_count to memorize how many ResolvingModuleJobs were created. When the active_task_count is 0, we think the building of the ModuleGraph is finished.

Build ChunkGraph

We will create basic Chunks by entry module and dynamic import. The details are complicated. But the basic rules are quite intuitive.

Chunk

Chunk is a set that contains a series of modules. The type of modules could be different. For example, A chunk could also include javascript modules and CSS modules.

• Each entry and `import(...) will create a chunk

In the end, we will expose the ChunkGraph to plugins to do further optimization.

Make Assets

After building ChunkGraph, we will expose each Chunk to plugins by the render_manifest hook to let plugins decide how and what Asset to generate.

Asset

Asset is a struct that represents a file in RSpack that going to be emitted.

In the end, the bundler will emit these assets to dist. And the whole process is finished.

[hyf0]

(WIP)Data Structures

Compiler

pub struct Compiler {
  pub options: Arc<CompilerOptions>,
  pub compilation: Compilation,
  pub plugin_driver: Arc<PluginDriver>,
}

Compilation

pub struct Compilation {
  pub options: Arc<CompilerOptions>,
  entries: HashMap<String, EntryItem>,
  pub(crate) visited_module_id: VisitedModuleIdentity,
  pub module_graph: ModuleGraph,
  pub chunk_graph: ChunkGraph,
}

ResolvingModuleJob and JobContext

pub struct ResolvingModuleJob {
  context: JobContext,
  dependency: Dependency,
  tx: UnboundedSender<Msg>,
  plugin_driver: Arc<PluginDriver>,
}

#[derive(Debug, Clone)]
pub struct JobContext {
  pub module_name: Option<String>,
  pub(crate) active_task_count: Arc<AtomicUsize>,
  pub(crate) visited_module_identity: VisitedModuleIdentity,
  pub source_type: Option<SourceType>,
}

Module

pub trait Module: Debug + Send + Sync {
  fn render(&self, module: &ModuleGraphModule, compilation: &Compilation) -> String;

  fn dependencies(&mut self) -> Vec<ModuleDependency> {
    vec![]
  }
}

ModuleGraphModule

pub struct ModuleGraphModule {
  // Only user defined entry module has name for now.
  pub name: Option<String>,
  pub id: String,
  // pub exec_order: usize,
  pub uri: String,
  pub module: BoxModule,
  pub source_type: SourceType,
  all_dependencies: Vec<Dependency>,
}

impl ModuleGraphModule {
  pub fn new(
    name: Option<String>,
    id: String,
    uri: String,
    module: BoxModule,
    dependencies: Vec<Dependency>,
    source_type: SourceType,
  ) -> Self {
    Self {
      name,
      id,
      // exec_order: usize::MAX,
      uri,
      module,
      all_dependencies: dependencies,
      source_type,
    }
  }

  pub fn depended_modules<'a>(&self, module_graph: &'a ModuleGraph) -> Vec<&'a ModuleGraphModule> {
    self
      .all_dependencies
      .iter()
      .filter(|dep| !matches!(dep.detail.kind, ResolveKind::DynamicImport))
      .filter_map(|dep| module_graph.module_by_dependency(dep))
      .collect()
  }

  pub fn dynamic_depended_modules<'a>(
    &self,
    module_graph: &'a ModuleGraph,
  ) -> Vec<&'a ModuleGraphModule> {
    self
      .all_dependencies
      .iter()
      .filter(|dep| matches!(dep.detail.kind, ResolveKind::DynamicImport))
      .filter_map(|dep| module_graph.module_by_dependency(dep))
      .collect()
  }
}

pub trait Module: Debug + Send + Sync {
  fn render(&self, module: &ModuleGraphModule, compilation: &Compilation) -> String;

  fn dependencies(&mut self) -> Vec<ModuleDependency> {
    vec![]
  }
}

Chunk

#[derive(Debug)]
pub struct Chunk {
  pub id: String,
  pub(crate) module_uris: HashSet<String>,
  pub(crate) entry_uri: String,
  pub kind: ChunkKind,
  pub module_index: HashMap<String, usize>,
}


impl Chunk {
  pub fn new(id: String, entry_uri: String, kind: ChunkKind) -> Self {
    Self {
      id,
      module_uris: Default::default(),
      entry_uri,
      kind,
      module_index: Default::default(),
    }
  }

  pub fn calc_exec_order(&mut self, module_graph: &ModuleGraph) {
    let entries = [self.entry_uri.clone()];
    let mut visited = HashSet::new();

    let mut next_exec_order = 0;
    for entry in entries {
      let mut stack_visited: HashSet<String> = HashSet::new();
      let mut stack = vec![entry];
      while let Some(module_uri) = stack.pop() {
        if !visited.contains(&module_uri) {
          if stack_visited.contains(module_uri.as_str()) {
            self
              .module_index
              .insert(module_uri.clone(), next_exec_order);
            // tracing::debug!(
            //   "module: {:?},next_exec_order {:?}",
            //   module_uri,
            //   next_exec_order
            // );
            next_exec_order += 1;
            visited.insert(module_uri);
          } else {
            stack.push(module_uri.to_string());
            stack_visited.insert(module_uri.to_string());
            stack.append(
              &mut module_graph
                .module_by_uri(&module_uri)
                .unwrap()
                .depended_modules(module_graph)
                .into_iter()
                .rev()
                .map(|dep_mod| &dep_mod.uri)
                .cloned()
                .collect(),
            )
          }
        }
      }
    }
  }

  #[instrument]
  pub fn ordered_modules<'a>(&self, module_graph: &'a ModuleGraph) -> Vec<&'a ModuleGraphModule> {
    let mut ordered = self
      .module_uris
      .iter()
      .filter_map(|uri| module_graph.module_by_uri(uri))
      .collect::<Vec<_>>();
    ordered.sort_by_key(|m| self.module_index[&m.uri]);
    ordered
  }
}

ModuleGraph

#[derive(Debug, Default)]
pub struct ModuleGraph {
  uri_to_module: HashMap<String, ModuleGraphModule>,
  dependency_to_module_uri: HashMap<Dependency, String>,
  // id_to_uri: hashbrown::HashMap<String, String>,
}


impl ModuleGraph {
  pub fn add_module(&mut self, module: ModuleGraphModule) {
    // let id = module.id().to_owned();
    self.uri_to_module.insert(module.uri.clone(), module);
    // self.id_to_uri.insert(id, uri);
  }

  pub fn add_dependency(&mut self, dep: Dependency, resolved_uri: String) {
    self.dependency_to_module_uri.insert(dep, resolved_uri);
  }

  pub fn module_by_dependency(&self, dep: &Dependency) -> Option<&ModuleGraphModule> {
    let uri = self.dependency_to_module_uri.get(dep)?;
    self.uri_to_module.get(uri)
  }

  pub fn uri_by_dependency(&self, dep: &Dependency) -> Option<&str> {
    let uri = self.dependency_to_module_uri.get(dep)?;
    Some(uri.as_str())
  }

  pub fn module_by_dependency_mut(&mut self, dep: &Dependency) -> Option<&mut ModuleGraphModule> {
    let uri = self.dependency_to_module_uri.get(dep)?;
    self.uri_to_module.get_mut(uri)
  }

  pub fn modules(&self) -> impl Iterator<Item = &ModuleGraphModule> {
    self.uri_to_module.values()
  }

  #[inline]
  pub fn module_by_uri(&self, uri: &str) -> Option<&ModuleGraphModule> {
    self.uri_to_module.get(uri)
    // .unwrap_or_else(|| panic!("fail to find module by uri: {:?}", uri))
  }

  #[inline]
  pub fn module_by_uri_mut(&mut self, uri: &str) -> Option<&mut ModuleGraphModule> {
    self.uri_to_module.get_mut(uri)
  }
}

ChunkGraph

#[derive(Debug, Default)]
pub struct ChunkGraph {
  id_to_chunk: hashbrown::HashMap<String, Chunk>,
  split_point_module_uri_to_chunk_id: hashbrown::HashMap<String, String>,
}

Plugin

pub trait Plugin: Debug + Send + Sync {
  fn build_start(&self) -> PluginBuildStartHookOutput {
    Ok(())
  }

  fn build_end(&self) -> PluginBuildEndHookOutput {
    Ok(())
  }

  fn register_parse_module(&self, _ctx: PluginContext) -> Option<Vec<SourceType>> {
    None
  }

  async fn resolve(
    &self,
    _ctx: PluginContext<&mut JobContext>,
    _agrs: ResolveArgs<'_>,
  ) -> PluginResolveHookOutput {
    Ok(None)
  }

  async fn load(
    &self,
    _ctx: PluginContext<&mut JobContext>,
    _args: LoadArgs<'_>,
  ) -> PluginLoadHookOutput {
    Ok(None)
  }

  fn parse_module(
    &self,
    _ctx: PluginContext<&mut JobContext>,
    _args: ParseModuleArgs,
  ) -> PluginParseModuleHookOutput {
    unreachable!()
  }

  fn render_manifest(
    &self,
    _ctx: PluginContext,
    _args: RenderManifestArgs,
  ) -> PluginRenderManifestHookOutput {
    Ok(vec![])
  }
}

JsModule

pub struct JsModule {
  pub uri: String,
  pub source_type: SourceType,
  pub ast: swc_ecma_ast::Program,
}

impl Debug for JsModule {
  fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    f.debug_struct("JsModule")
      .field("uri", &self.uri)
      .field("source_type", &self.source_type)
      .field("ast", &"{..}")
      .finish()
  }
}

impl Module for JsModule {
  #[instrument]
  fn render(&self, module: &ModuleGraphModule, compilation: &Compilation) -> String {
    use swc::config::{self as swc_config, SourceMapsConfig};
    let compiler = get_swc_compiler();
    let output = compiler.run(|| {
      HELPERS.set(&JS_HELPERS, || {
        swc::try_with_handler(compiler.cm.clone(), Default::default(), |handler| {
          let fm = compiler
            .cm
            .new_source_file(FileName::Custom(self.uri.to_string()), self.uri.to_string());

          let source_map = false;
          compiler.process_js_with_custom_pass(
            fm,
            // TODO: It should have a better way rather than clone.
            Some(self.ast.clone()),
            handler,
            &swc_config::Options {
              config: swc_config::Config {
                jsc: swc_config::JscConfig {
                  target: Some(EsVersion::Es2022),
                  syntax: Some(syntax_by_source_type(self.uri.as_str(), &self.source_type)),
                  transform: Some(swc_config::TransformConfig {
                    react: react::Options {
                      runtime: Some(react::Runtime::Automatic),
                      ..Default::default()
                    },
                    ..Default::default()
                  })
                  .into(),
                  ..Default::default()
                },
                inline_sources_content: true.into(),
                // emit_source_map_columns: (!matches!(options.mode, BundleMode::Dev)).into(),
                source_maps: Some(SourceMapsConfig::Bool(source_map)),
                ..Default::default()
              },
              // top_level_mark: Some(bundle_ctx.top_level_mark),
              ..Default::default()
            },
            |_, _| noop(),
            |_, _| {
              // noop()
              finalize(module, compilation)
            },
          )
        })
        .unwrap()
      })
    });
    output.code
  }

  fn dependencies(&mut self) -> Vec<rspack_core::ModuleDependency> {
    let mut dep_scanner = DependencyScanner::default();
    self.ast.visit_mut_with(&mut dep_scanner);
    dep_scanner.dependencies.into_iter().collect()
  }
}

[hardfist]

we may need to support chunkGroup for future supporting bundle splitting

[hyf0]

Yes. It's still in progress.

[hardfist]

ResolvingModuleJob is nearly the same as NormalModuleFactory, I think we may stick to webpack's name

[hyf0]

Agree. I will change it in the next PR.

[hardfist]

the finalize process is also important, whether we should take webpack's way or rollup's way?

[hardfist]

we may need an architecture overview like https://github.com/evanw/esbuild/blob/master/docs/architecture.md#overview