SCRIPTS
- model3.sh and model4.sh These are the two main scripts. model3.sh downloads geographical data for all IP addresses and then generates data_old.csv file. In model4.sh, all the IP prefixes are first extracted from data_old.csv file and placed in a CIDR.txt file. These IP prefixes in CIDR.txt are compared to new IP prefixes generated by model4.sh and then geographical data is downloaded only for new IP prefixes. This geographical data obtained for new IP prefixes are stored in data_new.csv. The data_new.csv is appended to data_new.csv to form a updated data_old.csv. This updated data_old.csv is further used for next comparison.
the command to run model3.sh and model4.sh is: bash model3.sh and model4.sh
-
timesort.awk This awk script selects IP addresses (external to UMKC) from source and destination fields of NetFlow data. If Ip is picked from source field, it is tagged as source and if Ip is picked from destination field it is tagged as destination. It also selects timestamps for every IP selected.
-
select_time.awk This file selects the IP addresses in a particular range of time (hourly basis) from the new test-data. If commented the entire new test-data file would be selected. It is used in model4.sh.
-
both.sed This sed file tags IP addresses either source or destination or 'both'. Suppose that an IP is source as well as destination, then this IP will be tagged as 'both'.
-
both.awk This awk file tags IP prefixes either source or destination or 'both'. This file is used after performing ARIP lookup (lookup used to find IP prefixes based on IP addresses).
-
filter_latlon.sed This sed file is used to change the format of latitude and longitude of City Location. When downloaded, the format is as follows: City Location:x/y After using this file, the format changes to City Latitude:x and City Longitude:y
-
filter2.sed This sed file is also used to process some of the downloaded data.
-
condition_test.awk This is a file that does the condition check. For example, selecting Country, captital, state, city, lalitude/longitude etc. when the data is scraped using xidel
-
data_old.csv This is a ouput csv file generated by model3.sh that contains the geographical data.
-
data_new.csv This is a ouput csv file generated by model4.sh that contains the geographical data.
-
CIDR.txt This file contains IP prefixes extracted from data_old.csv file and it used for comparison. It is used in model4.sh
-
web_application.png It is the output image for the web apllication.
SQL SCRIPTS
-
create_table.sql This file creates table for our postgres database
-
copy_data.sql This file copies the content of data_old.csv file/data_new.csv file into the table in the database.
The command to run these files is : \i /path/to/the/file/file_name.sql
WEB APPLICATION files
-
working_maps6.html This file is one that displays markers on the google maps. It also has the code for datapicker and time slider.
-
working_maps6.php
This is the php file that runs the SQL query in back-end that is used by working_maps6.html file. keep both these files in /var/www/html folder
Some useful links
-
For displaying markers on google maps. https://developers.google.com/maps/articles/phpsqlajax_v3
-
For date picker https://jqueryui.com/datepicker/#dropdown-month-year
-
For time slider http://marcneuwirth.com/blog/2010/02/21/using-a-jquery-ui-slider-to-select-a-time-range/