Where can I find large datasets open to the public?
Answer Wiki
Here are many of the links mentioned so far:
Cross-disciplinary data repositories, data collections and data search engines:
- https://www.kaggle.com/da
tasets - http://usgovxml.com
- http://aws.amazon.com/dat
asets - http://databib.org
- http://datacite.org
- http://figshare.com
- http://linkeddata.org
- http://reddit.com/r/datas
ets - http://thewebminer.com/
- http://thedatahub.org alias http://ckan.net
- http://quandl.com
- Social Network Analysis Interactive Dataset Library (Social Network Datasets)
- Datasets for Data Mining
- http://enigma.io
- http://www.ufindthem.com/
- http://NetworkRepository.
com - The First Interactive Network Data Repository - http://MLvis.com
Single datasets and data repositories
- http://archive.ics.uci.ed
u/ml/ - http://crawdad.org/
- http://data.austintexas.g
ov - http://data.cityofchicago
.org - http://data.govloop.com
- http://data.gov.uk/
- data.gov.in
- http://data.medicare.gov
- http://data.seattle.gov
- http://data.sfgov.org
- http://data.sunlightlabs.
com - https://datamarket.azure.
com/ - http://developer.yahoo.co
m/geo/g... - http://econ.worldbank.org
/datasets - http://en.wikipedia.org/w
iki/Wik... - http://factfinder.census.
gov/ser... - http://ftp.ncbi.nih.gov/
- http://gettingpastgo.socr
ata.com - http://googleresearch.blo
gspot.c... - http://books.google.com/n
grams/ - http://medihal.archives-o
uvertes.fr - http://public.resource.or
g/ - http://rechercheisidore.f
r - http://snap.stanford.edu/
data/in... - http://timetric.com/publi
c-data/ - https://wist.echo.nasa.go
v/~wist... - http://www2.jpl.nasa.gov/
srtm - http://www.archives.gov/r
esearch... - http://www.bls.gov/
- http://www.crunchbase.com
/ - http://www.dartmouthatlas
.org/ - http://www.data.gov/
- http://www.datakc.org
- http://dbpedia.org
- http://www.delicious.com/
jbaldwi... - http://www.faa.gov/data_r
esearch/ - http://www.factual.com/
- http://research.stlouisfe
d.org/f... - http://www.freebase.com/
- http://www.google.com/pub
licdata... - http://www.guardian.co.uk
/news/d... - http://www.infochimps.com
- http://www.kaggle.com/
- http://build.kiva.org/
- http://www.nationalarchiv
es.gov.... - http://www.nyc.gov/html/d
atamine... - http://www.ordnancesurvey
.co.uk/... - http://www.philwhln.com/h
ow-to-g... - http://www.imdb.com/inter
faces - http://imat-relpred.yande
x.ru/en... - http://www.dados.gov.pt/p
t/catal... - http://knoema.com
- http://daten.berlin.de/
- http://www.qunb.com
- http://databib.org/
- http://datacite.org/
- http://data.reegle.info/
- http://data.wien.gv.at/
- http://data.gov.bc.ca
- https://pslcdatashop.web.
cmu.edu/ (interaction data in learning environments) - http://www.icpsr.umich.ed
u/icpsrweb/CPES/ - Collaborative Psychiatric Epidemiology Surveys: (A collection of three national surveys focused on each of the major ethnic groups to study psychiatric illnesses and health services use) - http://www.dati.gov.it
- http://dati.trentino.it
- http://www.databagg.com/
- http://networkrepository.
com - Network/ML data repository w/ visual interactive analytics - Home (United Nations Environment Programme Grid Genava a lot of GIS datasets)
100+ Answers
I did a blog post about open data a long time ago (http://bret.appspot.com/
Since that post, there have been a lot more comments on the blog (105 and counting), so you may want to comb the comments for any ones the RWW post missed.
Since that post, there have been a lot more comments on the blog (105 and counting), so you may want to comb the comments for any ones the RWW post missed.
- A database of open databases? (also see most-upvoted questions on the Open Data Stack Exchange at Highest Voted Questions)
- http://www.reddit.com/r/d
atasets - https://d396qusza40orc.cl
oudfron... (large collection from Coursera's Data Analysis course) - Where is it possible to find raw climate data? (also NCAR - Climate Data Guide)
- | Ecological Data Wiki
- PhysioNet - largest repository of free, open-access databases and open-source computational tools devoted to complex signals informatics
- Page on sdss.org - SDSS Astronomy datasets. For more on astronomy, see Astronomy: What are some astronomy datasets open to the public?
- http://berkeleyearth.org/
dataset... - Berkeley Earth dataset - http://static.reddit.com/
RedditS... - massive survey of Redditors and their preferences - see http://blog.reddit.com/20 11/09/w... for some analysis - Welcome to the CRCNS data sharing website - for neuroscience
- http://archiveteam.org/in
dex.php... - Old archives of websites that no longer exist. Includes data on the affinities of 60,000+ Reddit users - http://www.r-bloggers.com
/datase... - Datasets to practice your data mining - discussed at http://www.reddit.com/r/M achineL... - http://www.ers.usda.gov/D
ata/ - USDA Economic Research Service datasets - http://www.mortality.org/ - human mortality datasets
- http://www.fda.gov/Food/F
oodSafe... - FDA pesticide datasets - http://www.ams.usda.gov/A
MSv1.0/pdp - USDA pesticide datasets - Climatology: What are some historical weather databases?
- http://www.epa.gov/data/ - EPA data
- http://data.giss.nasa.gov
/ - NASA GISS data - http://jimwatsonsequence.
cshl.edu/ - James Watson's DNA sequence - http://evidence.personalg
enomes.... - public genomes of people enrolled in the personal genome project - includes genomes of Steven Pinker and Esther Dyson. http://evidence.personalg enomes.... for their genomes - http://voteview.org/downl
oads.asp - Congressional Voting datasets (probably contains *everything* about what any politician voted for) - http://www.norc.uchicago.
edu/GSS... - General Social Survey. For tutorial, see http://blogs.discovermaga zine.co... - http://www.cfa.harvard.ed
u/hitran/ - high-resolution transmission molecular absorption database. HITRAN on the web: http://hitran.iao.ru/mole cule - http://sarahsinbox.com/ - Sarah Palin emails - analyzed by Edwin Chen using Latent Dirichlet Allocation (LDA) - see http://blog.echen.me/2011
/06/27/topic-modeling-the -sarah-palin-emails/
Some others:
- http://www.cdc.gov/nchs/n
hanes/n... - National Health and Nutrition Examination Survey - http://www.nlsinfo.org/or
dering/... - NSLY data (sociology) [1] - http://road.hmdc.harvard.
edu/ - election datasets (only 1984-1990 though)
[1] The NLSY79 Geocode data can only be made available to users who have successfully completed a geocode application and signed a confidentiality agreement with the U.S. Bureau of Labor Statistics. If interested in gaining access to the NLSY79 Geocode data, please review the information at http://stats.bls.gov/nls/
I'll try to restrict my answers to datasets greater than 1 GB in size, and order my answers by the size of the dataset.
More than 1 TB
- The 1000 Genomes project makes 260 TB of human genome data available [13]
- The Internet Archive is making an 80 TB web crawl available for research [17]
- The TREC conference made the ClueWeb09 [3] dataset available a few years back. You'll have to sign an agreement and pay a nontrivial fee (up to $610) to cover the sneakernet data transfer. The data is about 5 TB compressed.
- ClueWeb12 [21] is now available, as are the Freebase annotations, FACC1 [22]
- CNetS at Indiana University makes a 2.5 TB click dataset available [19]
- ICWSM made a large corpus of blog posts available for their 2011 conference [2]. You'll have to register (an actual form, not an online form), but it's free. It's about 2.1 TB compressed.
- The Yahoo News Feed dataset is 1.5 TB compressed, 13.5 TB uncompressed
- The Proteome Commons makes several large datasets available. The largest, the Personal Genome Project [11], is 1.1 TB in size. There are several others over 100 GB in size.
More than 1 GB
- The Reference Energy Disaggregation Data Set [12] has data on home energy use; it's about 500 GB compressed.
- The Tiny Images dataset [10] has 227 GB of image data and 57 GB of metadata.
- The ImageNet dataset [18] is pretty big.
- The MOBIO dataset [14] is about 135 GB of video and audio data
- The Yahoo! Webscope program [7] makes several 1 GB+ datasets available to academic researchers, including an 83 GB data set of Flickr image features and the dataset used for the 2011 KDD Cup [9], from Yahoo! Music, which is a bit over 1 GB.
- Google made a dataset mapping words to Wikipedia URLs (i.e., concepts) [15]. The dataset is about 10 GB compressed.
- Yandex has recently made a very large web search click dataset available [1]. You'll have to register online for the contest to download. It's about 5.6 GB compressed.
- Freebase makes regular data dumps available [5]. The largest is their Quad dump [4], which is about 3.6 GB compressed.
- The Open American National Corpus [8] is about 4.8 GB uncompressed.
- Wikipedia made a dataset containing information about edits available for a recent Kaggle competition [6]. The training dataset is about 2.0 GB uncompressed.
- The Research and Innovative Technology Administration (RITA) has made available a dataset about the on-time performance of domestic flights operated by large carriers. The ASA compressed this dataset and makes it available for download [16].
- The wiki-links data made available by Google is about 1.75 GB total [20].
Here is a useful link.
Finding Data on the Internet
Finding Data on the InternetBy RevoJoe
on October 6, 2011
The following list of data sources has been modified as of 8/19/13. Most of the data sets listed below are free, however, some are not.
If an (R) appears after source this means that the data are already in R format or there exist R commands for directly importing the data from R. (Seeexamples :: intro for some code.) Otherwise, I have limited the list to data sources for which there is a reasonably simple process for importing csv files. What follows is a list of data sources organized into categories that are not mutually exclusive but which reflect what's out there.
Economics
Finding Data on the Internet
Finding Data on the InternetBy RevoJoe
on October 6, 2011
The following list of data sources has been modified as of 8/19/13. Most of the data sets listed below are free, however, some are not.
If an (R) appears after source this means that the data are already in R format or there exist R commands for directly importing the data from R. (Seeexamples :: intro for some code.) Otherwise, I have limited the list to data sources for which there is a reasonably simple process for importing csv files. What follows is a list of data sources organized into categories that are not mutually exclusive but which reflect what's out there.
Economics
- American Economic Ass. (AEA): AEAweb: RFE
- UMD:: Inforum - EconData
- World bank: Indicators | Data
- CBOE Futures Exchange: CFE | Market Data
- Google Finance: Stock market quotes, news, currency conversions & more(R)
- Google Trends: Google Trends - Web Search interest - Worldwide, 2004 - present
- St Louis Fed: Federal Reserve Economic Data (R)
- NASDAQ: NASDAQ - Datastore
- OANDA: Forex Trading | Trade Currency Online | Forex Broker | OANDA(R)
- Quandl: Find, Use and Share Numerical Data
- Yahoo Finance: Yahoo Finance - Business Finance, Stock Market, Quotes, News (R)
- Archived national government statistics: Web Archiving Services for Libraries and Archives
- Australia: 3301.0 - Births, Australia, 2009
- Canada: Home | data.gc.ca
- DataMarket: DataMarket - Find, Understand and Share Data - DataMarket
- Fed Stats: FedStats: Subjects A to Z
- Guardian world governments: Page on guardian.co.uk
- London, U.K. data: Catalogue | London DataStore
- NewZealand: http://www.stats.govt.nz/
tools_and_services/tools/ TableBuilder/tables-by... - NYC data: NYC Open Data
- OECD: Page on oecd.org
- RITA: RITA | BTS | Title from h2
- San Francisco Data sets: Data | San Francisco
- U.K. Government Data: Data Search | data.gov.uk
- United Nations: UNdata
- U.S. Federal Government Agencies: Federal Agency Participation - Data.gov
- US CDC Public Health datasets: Public-Use Data Files and Documentation
- The World Bank: World Development Report
- UK 2011 Census Open Atlas Project: Page on alex-singleton.com
- Gapminder: Data
- Airlines Data (2009 ASA Challenge): The data. Data expo 09. ASA Statistics Computing and Graphics
- Airports and their locations: Airports and Their Locations
- AppliedPredictiveModeling (R package): Page on bit.ly
- Australian Weather: Daily Weather Observations
- Causality Workbench: Data - Repository - Causality Workbench
- Edge data for US domestic flights 1990 to 2009: US Domestic Flights From 1990 to 2009
- GroupLens Research (movie ratings and more): Datasets
- Kaggle competition data: Go from Big Data to Big Analytics
- KDNuggets competition site: Datasets for Data Mining and Data Science
- The Koblenz Network Collection: The Koblenz Network Collection
- Machine Learning Data Set Repository: mldata :: Welcome
- Medicare Data File: Page on cms.gov
- Microsoft Research: Our research - Microsoft Research
- Million songs: The Million Song Dataset: Giving Back to Music Research
- RDataMining.com: R and Data Mining R and Data Mining ebook data:Data - RDataMining.com: R and Data Mining
- The Revolution Analytics Collection: Index of /datasets/
- Social Networking: Ancestry.com Forum Dataset
- UCI Machine Learning Repository: UCI Machine Learning Repository
- 53.5 billion clicks: Center for Complex Networks and Systems Research
- Data360: Data360 Homepage
- Page on datamob.org: Page on datamob.org
- Factual: Page on factual.com
- Freebase: Freebase
- Google: Google Public Data Explorer
- infochimps: Big Data - Cloud Services
- numbray: Page on numbrary.com
- Sample R data sets: The R Datasets Package (R)
- SourceForge Research Data: Data
- UFO Reports: National UFO Reporting Center Web Reports
- Wikileaks 911 pager intercepts: 9/11 Pager data
- Resources for AP Statistics, Intro to Statistics, and R | STATS4STEM.ORG: R data sets: Statistical Data Sets, Statistics Data Sets, Data Sets For Statistics, R Datasets (R)
- The Washington Post List: Post Databases (washingtonpost.com)
- Agricultural Experiments: agridat {agridat} (R)
- Climate data: Temperature data (HadCRUT4) andftp://ftp.cmdl.noaa.go
v/ - Gene Expression Omnibus: Home - GEO - NCBI
- Geo Spatial Data: Data | GeoDa Center
- Human Microbiome Project: Microbial Reference Genomes
- MIT Cancer Genomics Data: Page on broadinstitute.org
- NASA: Obtaining Data From the NSSDC
- NIH Microarray data: ftp://ftp.ncbi.nih.gov/pu
b/geo/D... (R) - Protein structure: PSP benchmark
- Public Gene Data: Browse literature or sequence neighbours
- Stanford Microarray Data: Page on stanford.edu
- General Social Survey: General Social Survey
- ICPSR: Page on umich.edu
- SNAP: Stanford Large Network Dataset Collection
- UCLA Social Sciences Archive: Data Portals
- UPJOHN INST: Employment Research Data Center
- Time Series data Library: Time Series Data Library
- Carnegie Mellon University Enron email: Enron Email Dataset
- Carnegie Mellon University StatLab: StatLib---Datasets Archive
- Carnegie Mellon University JASA data archive: StatLib---JASA Data Archive
- Ohio State University Financial data: Financial Data Finder
- UC Berkeley: UC DATA :HOME
- UCLA: SOCR Data - Socr
- UC Riverside Time Series: Welcome to the UCR Time Series Classification/Clustering Page
- University of Toronto: Delve Datasets
Here are some big corpora we use in NLP in addition to the ones already mentioned:
- ukWaC: a 2 billion word corpus constructed from the Web limiting the crawl to the .uk domain and using medium-frequency words from the BNC as seeds. The corpus was POS-tagged and lemmatized with the TreeTagger. There's also a parsed version called pukWac. Get both at: http://wacky.sslmit.unibo
.it/dok... - WaCkypedia: a 2009 dump of the English Wikipedia (about 800 million tokens), including part of speech/lemma information, as well as a full syntactic parse. The texts were extracted from the dump and cleaned using the Wikipedia extractor. Get it at the same URL as ukWac: http://wacky.sslmit.unibo
.it/dok... - USENET corpus: A collection of public USENET postings. This corpus was collected between Oct 2005 and Jan 2011, and covers 47860 English language, non-binary-file news groups. Get it at: http://www.psych.ualberta
.ca/~we... [CAVEAT: it's huge!] - The collection of data that comes with the Natural Language Toolkit (NLTK). It's probably not as large as the others but it's a good set. See descriptions at: http://nltk.googlecode.co
m/svn/t... - Europarl: Proceedings of the European Parliament in 13 languages. Cleaned and pre-processed for machine translation research. Get it at: http://www.statmt.org/eur
oparl [FYI, NLTK has a built-in interface to access this corpus.] - The Google Books Ngram corpus: Pretty big. Get it at: http://books.google.com/n
grams/d...
