initial commit: modified scala etls to accept Fannie Mae data

[SurajAralihalli]

Signed-off-by: Suraj Aralihalli suraj.ara16@gmail.com

[nvliyuan]

please ignore the markdown links checker warnings, I file an issue for this.

[viadea]

@SurajAralihalli @nvliyuan Could you also check why markdown link failed:

[✖] /docs/get-started/xgboost-examples/on-prem-cluster/standalone-scala.md → Status: 400 [Error: ENOENT: no such file or directory, access '/docs/get-started/xgboost-examples/on-prem-cluster/standalone-scala.md'] {
[56](https://github.com/NVIDIA/spark-rapids-examples/runs/7051224566?check_suite_focus=true#step:5:57)
  errno: -2,
[57](https://github.com/NVIDIA/spark-rapids-examples/runs/7051224566?check_suite_focus=true#step:5:58)
  code: 'ENOENT',
[58](https://github.com/NVIDIA/spark-rapids-examples/runs/7051224566?check_suite_focus=true#step:5:59)
  syscall: 'access',
[59](https://github.com/NVIDIA/spark-rapids-examples/runs/7051224566?check_suite_focus=true#step:5:60)
  path: '/docs/get-started/xgboost-examples/on-prem-cluster/standalone-scala.md'
[60](https://github.com/NVIDIA/spark-rapids-examples/runs/7051224566?check_suite_focus=true#step:5:61)
}

[nvliyuan]

Could you also check why markdown link failed:

the link is not dead, I believe it is a bug in the link checker.

[SurajAralihalli]

@viadea the link checker validates only http/https links and marks any references(links) to the files within the repo as failed (Eg: /docs/get-started/xgboost-examples/building-sample-apps/scala.md)

[nvliyuan]

@viadea the link checker validates only http/https links and marks any references(links) to the files within the repo as failed (Eg: /docs/get-started/xgboost-examples/building-sample-apps/scala.md)

let me give another push to see who can help on this issue

[nvliyuan]

since spark.rapids.sql.incompatibleOps.enabled is true by default now, maybe we can remove the config.

[SurajAralihalli]

Sure, I'll update it in the next commit

[viadea]

yes the key is to keep only non-default settings for spark rapids based on current GA version -- say 2206.

[SurajAralihalli]

I have updated the latest commit to resolve the config issues in both scala and python etl notebooks

[nvliyuan]

since the output of mortgage-ETL.ipynb can be the input of mortgage-gpu.ipynb, it would be nice if we update the mortgage-gpu.ipynb about the data loading part. It will be something like:

df = spark.read.parquet("your etl notebook output path")
splits = df.randomSplit([0.8, 0.2])
train_data = splits[0]
trans_data = splits[1]

[nvliyuan]

@viadea shall we keep a sample mortgage dataset and add a declaration for convenience? So that customers do not need to get the dataset from the Fannie Mae website.

[viadea]

I think we should not. And that is the same reason why we have this PR to use raw data.

[SurajAralihalli]

since the output of mortgage-ETL.ipynb can be the input of mortgage-gpu.ipynb, it would be nice if we update the mortgage-gpu.ipynb about the data loading part. It will be something like:

df = spark.read.parquet("your etl notebook output path")
splits = df.randomSplit([0.8, 0.2])
train_data = splits[0]
trans_data = splits[1]

Thanks, I have updated both scala and python etl notebooks to save train and test datasets based on a boolean saveTrainTestDataset parameter.

[viadea]

Some issues:

1. In ETL part, when reading from raw CSV, there is no header in those CSV files.
   But in notebook I found scala/mortgage-ETL.ipynb:

val reader = sparkSession.read.option("header", true).schema(rawSchema)

We need to make sure the reading the raw CSV to use header=false.

2. In the same ETL notebook, when it tries to write to Parquet file, it has a line:

val optionsMap = Map("header" -> "true")

Actually this optionsMap is never used. If so, we should remove this.

3. Assuming the output of the ETL notebook are just parquet files, however in "scala/mortgage-gpu.ipynb", it is reading from csv. I hope we can change it to read from parquet instead such as the one in "scala/mortgage_gpu_crossvalidation.ipynb"

[viadea]

Make sure ETL reading is reading CSV without header, and ETL writing is writing to parquet files in all notebooks(scala/python.)

[viadea]

Here we save the rawDF to parquet files, and then calculate rawDF again to do the 20-80 split.
Is this the best logic for performance?
Should we : Save the rawDF to parquet files, and then read it back to do the 20-80 split to save the calculation again?
Could you help test it this new approach's performance to see if this is the best logic?