<jupyter_start><jupyter_text>OpenFindIt - Find FilesSearches domain for files.Enter `url` to crawl at cell with `%system scrapy ...`Then choose ```Kernel > Restart Kernel and Run All Cells...``` from the menu.  Install prerequisites  ! Ensure scrapy is installed and on your local computer !<jupyter_code>pip install scrapy
import pandas as pd
import sys
import scrapy
!pip install -r ./requirements.txt<jupyter_output>Requirement already satisfied: requests==2.20.0 in c:\users\joel\anaconda3\lib\site-packages (from -r ./requirements.txt (line 1)) (2.20.0)
Requirement already satisfied: scrapy==1.6.0 in c:\users\joel\anaconda3\lib\site-packages (from -r ./requirements.txt (line 2)) (1.6.0)
Requirement already satisfied: scrapy-random-useragent==0.2 in c:\users\joel\anaconda3\lib\site-packages (from -r ./requirements.txt (line 3)) (0.2)
Requirement already satisfied: youtube-dl in c:\users\joel\anaconda3\lib\site-packages (from -r ./requirements.txt (line 4)) (2020.5.8)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\joel\anaconda3\lib\site-packages (from requests==2.20.0->-r ./requirements.txt (line 1)) (2019.11.28)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\users\joel\anaconda3\lib\site-packages (from requests==2.20.0->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in c:\users\joel\anaconda3\lib\site-packages (from requests=[...]<jupyter_text>Force the scraper to provide feedback<jupyter_code>from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"<jupyter_output><empty_output><jupyter_text>Create tmp folder to hold results<jupyter_code>mkdir results<jupyter_output>A subdirectory or file results already exists.<jupyter_text>Run OpenFindIt Scrapy script for filesWe have no way to show progress of the run.  It might take a minute, it might take hours depending on how many pages you crawl. Go get a coffee. You will know it is running if you see the time glass icon in this browser tab.Change the sites to crawl by putting the url(s) you want to crawl in the openfindit/config/sites.txt file.<jupyter_code>%system scrapy crawl findfiles -a urls="http://joelcrawfordsmith.com/openassessit/demo/test-index.html" -s DEPTH_LIMIT=3 -s CLOSESPIDER_PAGECOUNT=500 -t csv -o - > "results/openfindit-findfiles-result.csv"
df = pd.read_csv("results/openfindit-findfiles-result.csv", dtype=str)
df<jupyter_output><empty_output>