Common Crawl is a nonprofit501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public.[1][2] Common Crawl's web archive consists of petabytes of data collected since 2011.[3] It completes crawls generally every month.[4]
Common Crawl was founded by Gil Elbaz.[5] Advisors to the non-profit include Peter Norvig and Joi Ito.[6] The organization's crawlers respect nofollow and robots.txt policies. Open source code for processing Common Crawl's data set is publicly available.
The Common Crawl dataset includes copyrighted work and is distributed from the US under fair use claims. Researchers in other countries have made use of techniques such as shuffling sentences or referencing the common crawl dataset to work around copyright law in other legal jurisdictions.[7]
Amazon Web Services began hosting Common Crawl's archive through its Public Data Sets program in 2012.[8]
The organization began releasing metadata files and the text output of the crawlers alongside .arc files in July of that year.[9] Common Crawl's archives had only included .arc files previously.[9]
In December 2012, blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012.[10] The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."[10]
In 2013, Common Crawl began using Apache Software Foundation'sNutch webcrawler instead of a custom crawler.[11] Common Crawl switched from using .arc files to .warc files with its November 2013 crawl.[12]
A filtered version of Common Crawl was used to train OpenAI's GPT-3 language model, announced in 2020.[13] One challenge of using Common Crawl data is that despite the vast amount of documented web data, individual pieces of crawled websites could be better documented. This can create challenges when trying to diagnose problems in projects that use the Common Crawl data. A solution proposed by Timnit Gebru, et al., in 2020 to an industry-wide documentation shortfall is that every dataset should be accompanied with a datasheet that documents its motivation, composition, collection process, and recommended uses.[14]
History of Common Crawl data
The following data have been collected from the official Common Crawl Blog.[15]
Crawl date
Size in TiB
Billions of pages
Comments
October 2022
380
3.15
Crawl conducted in September and October 2022
April 2021
320
3.1
November 2018
220
2.6
October 2018
240
3.0
September 2018
220
2.8
August 2018
—
—
July 2018
255
3.25
June 2018
235
3.05
May 2018
215
2.75
April 2018
230
3.1
March 2018
250
3.2
February 2018
270
3.4
January 2018
270
3.4
December 2017
240
2.9
November 2017
260
3.2
October 2017
300
3.65
September 2017
250
3.01
August 2017
280
3.28
July 2017
240
2.89
June 2017
260
3.16
May 2017
250
2.96
April 2017
250
2.94
March 2017
250
3.07
February 2017
250
3.08
January 2017
250
3.14
December 2016
—
2.85
October 2016
—
3.25
September 2016
—
1.72
August 2016
—
1.61
July 2016
—
1.73
June 2016
—
1.23
May 2016
—
1.46
April 2016
—
1.33
February 2016
—
1.73
November 2015
151
1.82
September 2015
106
1.32
August 2015
149
1.84
July 2015
145
1.81
June 2015
131
1.67
May 2015
159
2.05
April 2015
168
2.11
March 2015
124
1.64
February 2015
145
1.9
January 2015
139
1.82
December 2014
160
2.08
November 2014
135
1.95
October 2014
254
3.7
September 2014
220
2.8
August 2014
200
2.8
July 2014
266
3.6
April 2014
183
2.6
March 2014
223
2.8
First Nutch crawl
January 2014
148
2.3
Crawls performed monthly
November 2013
102
2
Data in Warc file format
July 2012
—
—
Data in Arc file format
January 2012
—
—
Public Data Set of Amazon Web Services
November 2011
40
5
First availability on Amazon
Norvig Web Data Science Award
In corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux.[16][17] The award is named for Peter Norvig who also chairs the judging committee for the award.[16]
References
^Rosanna Xia (February 5, 2012). "Tech entrepreneur Gil Elbaz made it big in L.A." Los Angeles Times. Retrieved July 31, 2014.
^"Gil Elbaz and Common Crawl". NBC News. April 4, 2013. Retrieved July 31, 2014.
^"So you're ready to get started". Retrieved 2018-06-02.
^Lisa Green (January 8, 2014). "Winter 2013 Crawl Data Now Available". Retrieved June 2, 2018.
^"Startups - Gil Elbaz and Nova Spivack of Common Crawl - TWiST #222". This Week In Startups. January 10, 2012.
^Tom Simonite (January 23, 2013). "A Free Database of the Entire Web May Spawn the Next Google". MIT Technology Review. Retrieved July 31, 2014.
^Schäfer, Roland (May 2016). "CommonCOW: Massively Huge Web Corpora from CommonCrawl Data and a Method to Distribute them Freely under Restrictive EU Copyright Laws". Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16). Portorož, Slovenia: European Language Resources Association (ELRA): 4501.
^Jennifer Zaino (March 13, 2012). . Semantic Web. Archived from the original on July 1, 2014. Retrieved July 31, 2014.
^ abJennifer Zaino (July 16, 2012). . Semantic Web. Archived from the original on August 12, 2014. Retrieved July 31, 2014.
^ abJennifer Zaino (December 18, 2012). . Semantic Web. Archived from the original on August 12, 2014. Retrieved July 31, 2014.
^Jordan Mendelson (February 20, 2014). "Common Crawl's Move to Nutch". Common Crawl. Retrieved July 31, 2014.
^Jordan Mendelson (November 27, 2013). "New Crawl Data Available!". Common Crawl. Retrieved July 31, 2014.
^Brown, Tom; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini (2020-06-01). "Language Models are Few-Shot Learners". p. 14. arXiv:2005.14165 [cs.CL]. the majority of our data is derived from raw Common Crawl with only quality-based filtering.
^Gebru, Timnit; Morgenstern, Jamie; Vecchione, Briana; Wortman Vaughan, Jennifer; Wallach, Hanna; Daumé III, Hal; Crawford, Kate (March 19, 2020). "Datasheets for Datasets". arXiv:1803.09010 [cs.DB].
^ abLisa Green (November 15, 2012). "The Norvig Web Data Science Award". Common Crawl. Retrieved July 31, 2014.
^. Dutch Techcentre for Life Sciences. Archived from the original on August 15, 2014. Retrieved July 31, 2014.
External links
Common Crawl in California, United States
Common Crawl GitHub Repository with the crawler, libraries and example code
Common Crawl Discussion Group
Common Crawl Blog
January 20, 2023
common, crawl, nonprofit, organization, that, crawls, freely, provides, archives, datasets, public, archive, consists, petabytes, data, collected, since, 2011, completes, crawls, generally, every, month, type, business501, profitavailable, inenglishheadquarter. Common Crawl is a nonprofit 501 c 3 organization that crawls the web and freely provides its archives and datasets to the public 1 2 Common Crawl s web archive consists of petabytes of data collected since 2011 3 It completes crawls generally every month 4 Common CrawlType of business501 c 3 non profitAvailable inEnglishHeadquartersSan Francisco California Los Angeles California United StatesFounder s Gil ElbazKey peoplePeter Norvig Nova Spivack Carl Malamud Kurt Bollacker Joi ItoURLcommoncrawl wbr orgCommon Crawl was founded by Gil Elbaz 5 Advisors to the non profit include Peter Norvig and Joi Ito 6 The organization s crawlers respect nofollow and robots txt policies Open source code for processing Common Crawl s data set is publicly available The Common Crawl dataset includes copyrighted work and is distributed from the US under fair use claims Researchers in other countries have made use of techniques such as shuffling sentences or referencing the common crawl dataset to work around copyright law in other legal jurisdictions 7 Contents 1 History 2 History of Common Crawl data 3 Norvig Web Data Science Award 4 References 5 External linksHistory EditAmazon Web Services began hosting Common Crawl s archive through its Public Data Sets program in 2012 8 The organization began releasing metadata files and the text output of the crawlers alongside arc files in July of that year 9 Common Crawl s archives had only included arc files previously 9 In December 2012 blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012 10 The donated data helped Common Crawl improve its crawl while avoiding spam porn and the influence of excessive SEO 10 In 2013 Common Crawl began using Apache Software Foundation s Nutch webcrawler instead of a custom crawler 11 Common Crawl switched from using arc files to warc files with its November 2013 crawl 12 A filtered version of Common Crawl was used to train OpenAI s GPT 3 language model announced in 2020 13 One challenge of using Common Crawl data is that despite the vast amount of documented web data individual pieces of crawled websites could be better documented This can create challenges when trying to diagnose problems in projects that use the Common Crawl data A solution proposed by Timnit Gebru et al in 2020 to an industry wide documentation shortfall is that every dataset should be accompanied with a datasheet that documents its motivation composition collection process and recommended uses 14 History of Common Crawl data EditThe following data have been collected from the official Common Crawl Blog 15 Crawl date Size in TiB Billions of pages CommentsOctober 2022 380 3 15 Crawl conducted in September and October 2022April 2021 320 3 1November 2018 220 2 6October 2018 240 3 0September 2018 220 2 8August 2018 July 2018 255 3 25June 2018 235 3 05May 2018 215 2 75April 2018 230 3 1March 2018 250 3 2February 2018 270 3 4January 2018 270 3 4December 2017 240 2 9November 2017 260 3 2October 2017 300 3 65September 2017 250 3 01August 2017 280 3 28July 2017 240 2 89June 2017 260 3 16May 2017 250 2 96April 2017 250 2 94March 2017 250 3 07February 2017 250 3 08January 2017 250 3 14December 2016 2 85October 2016 3 25September 2016 1 72August 2016 1 61July 2016 1 73June 2016 1 23May 2016 1 46April 2016 1 33February 2016 1 73November 2015 151 1 82September 2015 106 1 32August 2015 149 1 84July 2015 145 1 81June 2015 131 1 67May 2015 159 2 05April 2015 168 2 11March 2015 124 1 64February 2015 145 1 9January 2015 139 1 82December 2014 160 2 08November 2014 135 1 95October 2014 254 3 7September 2014 220 2 8August 2014 200 2 8July 2014 266 3 6April 2014 183 2 6March 2014 223 2 8 First Nutch crawlJanuary 2014 148 2 3 Crawls performed monthlyNovember 2013 102 2 Data in Warc file formatJuly 2012 Data in Arc file formatJanuary 2012 Public Data Set of Amazon Web ServicesNovember 2011 40 5 First availability on AmazonNorvig Web Data Science Award EditIn corroboration with SURFsara Common Crawl sponsors the Norvig Web Data Science Award a competition open to students and researchers in Benelux 16 17 The award is named for Peter Norvig who also chairs the judging committee for the award 16 References Edit Rosanna Xia February 5 2012 Tech entrepreneur Gil Elbaz made it big in L A Los Angeles Times Retrieved July 31 2014 Gil Elbaz and Common Crawl NBC News April 4 2013 Retrieved July 31 2014 So you re ready to get started Retrieved 2018 06 02 Lisa Green January 8 2014 Winter 2013 Crawl Data Now Available Retrieved June 2 2018 Startups Gil Elbaz and Nova Spivack of Common Crawl TWiST 222 This Week In Startups January 10 2012 Tom Simonite January 23 2013 A Free Database of the Entire Web May Spawn the Next Google MIT Technology Review Retrieved July 31 2014 Schafer Roland May 2016 CommonCOW Massively Huge Web Corpora from CommonCrawl Data and a Method to Distribute them Freely under Restrictive EU Copyright Laws Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 16 Portoroz Slovenia European Language Resources Association ELRA 4501 Jennifer Zaino March 13 2012 Common Crawl To Add New Data In Amazon Web Services Bucket Semantic Web Archived from the original on July 1 2014 Retrieved July 31 2014 a b Jennifer Zaino July 16 2012 Common Crawl Corpus Update Makes Web Crawl Data More Efficient Approachable For Users To Explore Semantic Web Archived from the original on August 12 2014 Retrieved July 31 2014 a b Jennifer Zaino December 18 2012 Blekko Data Donation Is A Big Benefit To Common Crawl Semantic Web Archived from the original on August 12 2014 Retrieved July 31 2014 Jordan Mendelson February 20 2014 Common Crawl s Move to Nutch Common Crawl Retrieved July 31 2014 Jordan Mendelson November 27 2013 New Crawl Data Available Common Crawl Retrieved July 31 2014 Brown Tom Mann Benjamin Ryder Nick Subbiah Melanie Kaplan Jared Dhariwal Prafulla Neelakantan Arvind Shyam Pranav Sastry Girish Askell Amanda Agarwal Sandhini 2020 06 01 Language Models are Few Shot Learners p 14 arXiv 2005 14165 cs CL the majority of our data is derived from raw Common Crawl with only quality based filtering Gebru Timnit Morgenstern Jamie Vecchione Briana Wortman Vaughan Jennifer Wallach Hanna Daume III Hal Crawford Kate March 19 2020 Datasheets for Datasets arXiv 1803 09010 cs DB Blog Common Crawl a b Lisa Green November 15 2012 The Norvig Web Data Science Award Common Crawl Retrieved July 31 2014 Norvig Web Data Science Award 2014 Dutch Techcentre for Life Sciences Archived from the original on August 15 2014 Retrieved July 31 2014 External links EditCommon Crawl in California United States Common Crawl GitHub Repository with the crawler libraries and example code Common Crawl Discussion Group Common Crawl Blog Retrieved from https en wikipedia org w index php title Common Crawl amp oldid 1130089184, wikipedia, wiki, book, books, library,
