Repositories list

• spruce

  Public
  SPRUCE is an open-source enrichment platform for GreenOps which helps measure and reduce the environmental impact of cloud computing.

  open-sourceawscloud+ 6

  open-sourceawscloudsustainabilityapache-sparkclimatecarbon-emissionsgreenopsgreensoftware

  Java

  •

  Apache License 2.0

  •44 forks•2121 stars•1313 issues•00 pull requests•Updated Apr 14, 2026Apr 14, 2026

• storm

  Public
  Mirror of Apache Storm

  Java

  •

  Apache License 2.0

  •4k4k forks•00 stars•00 issues•00 pull requests•Updated Mar 30, 2026Mar 30, 2026

• digitalpebble.github.io

  Public
  Resources for the DigitalPebble website

  SCSS

  •00 forks•00 stars•00 issues•00 pull requests•Updated Mar 2, 2026Mar 2, 2026

• ginkgo

  Public
  Estimate the environmental impact of GitHub Actions for your entire organization.

  open-sourcesustainabilitycarbon-emissions+ 2

  open-sourcesustainabilitycarbon-emissionsgithub-actionsgreenops

  Rust

  •

  Other

  •00 forks•11 star•55 issues•00 pull requests•Updated Feb 24, 2026Feb 24, 2026

• carbonara

  Public archive
  Enrichment pipeline for CUR / FOCUS reports which adds energy and carbon data allowing to report and reduce the impact of the your cloud usage.

  awscloudsustainability+ 6

  awscloudsustainabilityclimatefocuscarbon-emissionsapachesparkgreenopsgreensoftware

  Java

  •

  Apache License 2.0

  •00 forks•55 stars•00 issues•00 pull requests•Updated Jul 18, 2025Jul 18, 2025

• benchmark

  Public
  StormCrawler topology to evaluate the performance of different backends and configurations

  elasticsearchbenchmarkopensearch+ 1

  elasticsearchbenchmarkopensearchstormcrawler

  Shell

  •00 forks•00 stars•00 issues•00 pull requests•Updated Jul 1, 2025Jul 1, 2025

• stormcrawler-docker

  Public
  Resources for running StormCrawler with Docker services

  dockerapache-stormstormcrawler

  dockerapache-stormstormcrawler

  Dockerfile

  •

  Apache License 2.0

  •44 forks•1010 stars•00 issues•00 pull requests•Updated Nov 10, 2024Nov 10, 2024

• crawlurlfrontier

  Public archive
  Crawl config used to test URL Frontier on a large scale and produce WARCs for CommonCrawl.

  FLUX

  •00 forks•11 star•00 issues•00 pull requests•Updated May 16, 2024May 16, 2024

• tika-detector-stormcrawler

  Public
  Wraps the charset detection logic from StormCrawler as a Tika module

  Java

  •

  Apache License 2.0

  •11 fork•00 stars•00 issues•00 pull requests•Updated Feb 2, 2024Feb 2, 2024

• tika

  Public
  The Apache Tika toolkit detects and extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF).

  Java

  •

  Apache License 2.0

  •922922 forks•00 stars•00 issues•00 pull requests•Updated Jan 25, 2024Jan 25, 2024

• docs

  Public
  Documentation for Docker Official Images in docker-library

  Shell

  •

  MIT License

  •2.2k2.2k forks•00 stars•00 issues•00 pull requests•Updated Jan 16, 2024Jan 16, 2024

• ansible-storm

  Public
  Ansible playbook for deploying a Storm cluster

  ansiblestormplaybook+ 1

  ansiblestormplaybookstormcrawler

  11 fork•77 stars•00 issues•00 pull requests•Updated Dec 7, 2023Dec 7, 2023

• nutch

  Public
  Apache Nutch is an extensible and scalable web crawler

  Java

  •

  Apache License 2.0

  •1.3k1.3k forks•11 star•00 issues•00 pull requests•Updated Nov 8, 2023Nov 8, 2023

• urlfrontier-client

  Public
  URLFrontier client written in Rust (mostly as a way of learning Rust)

  rustgrpcwebcrawler+ 2

  rustgrpcwebcrawlerurl-frontierurlfrontier

  Rust

  •

  Apache License 2.0

  •00 forks•11 star•00 issues•00 pull requests•Updated Dec 5, 2022Dec 5, 2022

• crawler4j-frontier-battle

  Public
  Java

  •11 fork•00 stars•00 issues•00 pull requests•Updated Apr 6, 2022Apr 6, 2022

• TextClassification

  Public
  A Text Classification API in Java originally developed by DigitalPebble Ltd. The API is independent from the ML implementations used and can be used as a front …

  Java

  •

  Apache License 2.0

  •2222 forks•4747 stars•11 issue•00 pull requests•Updated Sep 24, 2021Sep 24, 2021

• stormcrawlerfight

  Public
  Crawl configurations for benchmarking / testing StormCrawler

  Shell

  •

  Apache License 2.0

  •55 forks•1010 stars•00 issues•00 pull requests•Updated Sep 19, 2019Sep 19, 2019

• behemoth

  Public archive
  Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.

  javanlphadoop+ 1

  javanlphadoopmapreduce

  Java

  •

  Other

  •5959 forks•284284 stars•1212 issues•11 pull request•Updated Apr 25, 2018Apr 25, 2018

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •9090 forks•44 stars•00 issues•00 pull requests•Updated Apr 4, 2017Apr 4, 2017

• sc-warc

  Public
  WARC resources for StormCrawler

  11 fork•22 stars•22 issues•00 pull requests•Updated Oct 20, 2016Oct 20, 2016

• tescobank

  Public archive
  Setup for crawling tescobank with SC

  Java

  •

  Apache License 2.0

  •22 forks•44 stars•00 issues•00 pull requests•Updated Sep 23, 2015Sep 23, 2015

• textclassification-examples

  Public
  Use cases for DigitalPebble's TextClassification API

  Java

  •

  Apache License 2.0

  •33 forks•1010 stars•00 issues•00 pull requests•Updated Sep 1, 2015Sep 1, 2015

• behemoth-commoncrawl

  Public archive
  Support for old (pre 2013) CommonCrawl dataset in Behemoth

  Java

  •00 forks•44 stars•00 issues•00 pull requests•Updated Apr 20, 2015Apr 20, 2015

• tika-cc

  Public
  resources for generating a corpus of docs from CC for Tika

  Shell

  •00 forks•00 stars•00 issues•00 pull requests•Updated Nov 28, 2014Nov 28, 2014

• NutchFight

  Public
  Resources for comparison between 1.8 and 2.x of Apache Nutch

  Java

  •

  Apache License 2.0

  •00 forks•44 stars•00 issues•00 pull requests•Updated Jun 4, 2014Jun 4, 2014

• behemoth-elasticsearch

  Public archive
  ElasticSearch module for Behemoth

  Java

  •00 forks•11 star•00 issues•00 pull requests•Updated Feb 12, 2014Feb 12, 2014

• behemoth-textclassification

  Public archive
  Module for classifying Behemoth documents with a model from our Text Classification API

  Java

  •00 forks•11 star•00 issues•00 pull requests•Updated Nov 22, 2012Nov 22, 2012

• TextClassificationPlugin

  Public archive
  GATE Processing Resource wrapping DigitalPebble's TextClassification API

  Java

  •33 forks•55 stars•11 issue•11 pull request•Updated Jul 12, 2012Jul 12, 2012

• ngrams-api

  Public archive
  Java API for querying a N-Grams corpus. Uses Lucene for searching and indexing from the Google Web-1T format

  Java

  •

  Other

  •22 forks•55 stars•00 issues•00 pull requests•Updated Apr 27, 2012Apr 27, 2012