WIP: Add Ames housing #155
Conversation
| super() | ||
| @metadata.id = "ames-housing" | ||
| @metadata.name = "Ames Housing" | ||
| @metadata.url = "http://jse.amstat.org/v19n3/decock/DataDocumentation.txt" |
There was a problem hiding this comment.
I saw this comment: #43 (comment)
Maybe should we use house_prices datasets in OpenML because sklearn.datasets.load_boston mentioned it here?
If we use this data, there is one problem. This dataset file is .arff. So we have to implement parser function about it.
There was a problem hiding this comment.
Ahhh I'm sorry. there are json and xml formats too. So how do you think we will get data from OpenML?
ref: https://www.openml.org/search?type=data&sort=runs&id=42165&status=active
There was a problem hiding this comment.
Thanks for your comments. The data is at https://api.openml.org/data/v1/download/21754539/house_prices.arff , the json and xml files contain metadata only.
arff is used in Weka, there are readers for R and Python, so such a reader may be helpful whether it is used here or not.
The data use in sklearn has been cleaned, but is in csv form.
As explained in https://www.tmwr.org/ames.html the data in the model data package has been transformed more than the data used in sklearn, with latitude and longitude information that is helpful for visualization. It seems higher quality, but is available in rda format https://github.com/topepo/AmesHousing/tree/master/data though it is possible to export to csv format as explained at https://stackoverflow.com/questions/4487065/convert-rda-to-csv An rda reader would likely also be helpful.
At the moment, all data is downloaded from external repositories which may disappear. It may be good to host some of the data at a location that has better control, but probably this can be a separate issue.
Similarly plugin readers for rda and arff similar to https://github.com/red-data-tools/red-datasets-pandas should be separate issues.
There was a problem hiding this comment.
Thanks for your comments. The data is at https://api.openml.org/data/v1/download/21754539/house_prices.arff , the json and xml files contain metadata only.
Sorry I thought there were Ames housing data too. But it wasn't.
Thank you for explaining the situation in details.
Now I agree with you to use this data resource. And it looks better way.
There was a problem hiding this comment.
Thanks for the explanation.
I agree with using http://jse.amstat.org/v19n3/decock/AmesHousing.txt (the original data) is suitable. I confirmed https://www.tandfonline.com/doi/citedby/10.1080/10691898.2011.11889627 too because http://jse.amstat.org/ was archived and move to https://tandfonline.com/toc/ujse20/current . It seems that new site https://www.tandfonline.com/doi/citedby/10.1080/10691898.2011.11889627 doesn't provide the original data. Other article such as https://www.tandfonline.com/doi/full/10.1080/26939169.2022.2074923 has the "Supplemental" tab but https://www.tandfonline.com/doi/citedby/10.1080/10691898.2011.11889627 doesn't have the tab.
| super() | ||
| @metadata.id = "ames-housing" | ||
| @metadata.name = "Ames Housing" | ||
| @metadata.url = "http://jse.amstat.org/v19n3/decock/DataDocumentation.txt" |
There was a problem hiding this comment.
Thanks for the explanation.
I agree with using http://jse.amstat.org/v19n3/decock/AmesHousing.txt (the original data) is suitable. I confirmed https://www.tandfonline.com/doi/citedby/10.1080/10691898.2011.11889627 too because http://jse.amstat.org/ was archived and move to https://tandfonline.com/toc/ujse20/current . It seems that new site https://www.tandfonline.com/doi/citedby/10.1080/10691898.2011.11889627 doesn't provide the original data. Other article such as https://www.tandfonline.com/doi/full/10.1080/26939169.2022.2074923 has the "Supplemental" tab but https://www.tandfonline.com/doi/citedby/10.1080/10691898.2011.11889627 doesn't have the tab.
| @metadata.id = "ames-housing" | ||
| @metadata.name = "Ames Housing" | ||
| @metadata.url = "http://jse.amstat.org/v19n3/decock/DataDocumentation.txt" | ||
| @metadata.licenses = ["Unknown"] |
There was a problem hiding this comment.
http://jse.amstat.org/jse_users.htm said:
Unlike other American Statistical Association journals, the Journal of Statistics Education (JSE) does not require authors to transfer copyright for the published material to JSE. Authors maintain copyright of published material. Because copyright is not transferred from the author, permission to use materials published by JSE remains with the author. Therefore, to use published material from a JSE article the requesting person must get approval from the author.
Could you ask the author the license of this dataset?
There was a problem hiding this comment.
Still waiting for this. Will remind again beginning of December.
| data_path = cache_dir_path + "AmesHousing.txt" | ||
| data_url = "http://jse.amstat.org/v19n3/decock/AmesHousing.txt" | ||
| download(data_path, data_url) | ||
| CSV.open(data_path, converters: [:numeric]) do |csv| |
There was a problem hiding this comment.
It's better that we preprocess the original data based on http://jse.amstat.org/v19n3/decock/DataDocumentation.txt .
For example, the "MS SubClass" column values can be converted to "1-STORY 1946 & NEWER ALL STYLES" from "020" and so on:
MS SubClass (Nominal): Identifies the type of dwelling involved in the sale.
020 1-STORY 1946 & NEWER ALL STYLES
030 1-STORY 1945 & OLDER
040 1-STORY W/FINISHED ATTIC ALL AGES
045 1-1/2 STORY - UNFINISHED ALL AGES
050 1-1/2 STORY FINISHED ALL AGES
060 2-STORY 1946 & NEWER
070 2-STORY 1945 & OLDER
075 2-1/2 STORY ALL AGES
080 SPLIT OR MULTI-LEVEL
085 SPLIT FOYER
090 DUPLEX - ALL STYLES AND AGES
120 1-STORY PUD (Planned Unit Development) - 1946 & NEWER
150 1-1/2 STORY PUD - ALL AGES
160 2-STORY PUD - 1946 & NEWER
180 PUD - MULTILEVEL - INCL SPLIT LEV/FOYER
190 2 FAMILY CONVERSION - ALL STYLES AND AGES
Addressing #122 using the original dataset with missing values and without latitude and longitude information