Skip to content

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

License

Notifications You must be signed in to change notification settings

archivesunleashed/aut

Repository files navigation

The Archives Unleashed Toolkit

codecov Maven Central Scaladoc UserDocs LICENSE Contribution Guidelines

The Archives Unleashed Toolkit is an open-source platform for analyzing web archives using Apache Spark, and makes use of Sparkling for parsing W/ARC records. The toolkit provides powerful tools for analytics and data processing. It is part of the Archives Unleashed Project.

To learn more about the Toolkit and how to use, please see our comprehensive documentation.

If you would like a more in-depth look at the project, please check out the following two articles:

Dependencies

  • Java 11
  • Python 3.7.3+ (PySpark)
  • Scala 2.12+
  • Apache Spark (Hadoop 2.7) 3.0.3+

More information on setting up dependencies can be found here.

Building

Clone the repo:

git clone http://github.com/archivesunleashed/aut.git

You can then build The Archives Unleashed Toolkit.

mvn clean install

Usage

The Toolkit can be used to submit a variety of extraction jobs with spark-submit, as well used as a library via spark-submit, pyspark, or in your own application. More information on using the Toolkit can be found here.

Citing Archives Unleashed

How to cite the Archives Unleashed Toolkit or Cloud in your research:

Nick Ruest, Jimmy Lin, Ian Milligan, and Samantha Fritz. 2020. The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (JCDL '20). Association for Computing Machinery, New York, NY, USA, 157–166. DOI:https://doi.org/10.1145/3383583.3398513

Your citations help to further the recognition of using open-source tools for scientific inquiry, assists in growing the web archiving community, and acknowledges the efforts of contributors to this project.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work is primarily supported by the Andrew W. Mellon Foundation. Other financial and in-kind support comes from the Social Sciences and Humanities Research Council, Compute Canada, the Ontario Ministry of Research, Innovation, and Science, York University Libraries, Start Smart Labs, and the Faculty of Arts and David R. Cheriton School of Computer Science at the University of Waterloo.

Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.