There are some great wordlists out there for sure… but a targeted wordlist that fits with the subject of the target site/database can prove to be much more effective. Joshua Dustin posted to his blog recently about this and I thought this was an excellent idea and wanted to take it a little bit further. This script adds some automation to his idea and also adds a word-grab from Bing as well. Since it’s a little more modularized in this script it’ll be easy to add other word sources. I’ll be adding more soon once I have additional time to do so. Please check out his post for further information on why this type of wordlist generation can be so effective. He does a great job explaining it.
I have a few other things in the works I had intended on releasing sooner (rather than another post based on someone else’ idea) but those scripts are getting near-daily updates due to the fact that I’m using them constantly. One just got a roughly 20x speed boost today thanks to some command-line option changes. Don’t worry, though, they’re worth the wait! Now, on to the code;
Running this script:
- Run it using as many keywords as you’d like to scrape off the web:
- # ./wordlistgen.sh your keywords go here
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
#!/bin/bash ################################################# # Internet Wordlist Generator by: # # Nathan V # # Cyber Security Analyst # # http://nathanv.com # # # # For assistance and new versions contact # # nathan.v@gmail.com # # This file updated: 18 July 2012 # ################################################# # This script (c)2012 Nathan V : License: GPLv3 # # This is free software, and you are welcome to # # redistribute it under certain conditions; See # # http://www.gnu.org/licenses/gpl.html # ################################################# # getTweets() is based on twitter.sh by: # # Joshua Dustin # #7habitsofhighlyeffectivehackers.blogspot.com.au# ################################################# #import arguments args="$@" #clear screen and check for input parameter clear if [ -z "$1" ] then echo "Missing input parameters. Please use $0 <keyword> <keyword>" kill -int $$ else echo "" fi # scrape Twitter for tweets containing your keywords getTweets() { local key=$1 echo -n "Grabing for keyword $key..." wget -q "http://search.twitter.com/search.json?q=$key&rpp=500" -O result.json cat result.json | tr "," \\n | grep "^\"text" | cut -d"\"" -f4- | tr " " \\n | sed -e 's~&~~' | sed -e 's~>~~' | sed -e 's~<~~' | sed s/[\"=\|?.\!\(\):\;]//g | sed s/\^\#//g | sed s/\^\@//g | sed '/^$/d' | grep -v "^http:" | grep -v "\\\\" >> wl.temp rm -f result.json echo " complete." sleep .1 } # scrape Bing for search results related to your keywords getBing() { local key=$1 echo -n "Grabing for keyword $key..." wget -q "http://api.search.live.com/rss.aspx?source=web&query=$key" -O result.rss cat result.rss | sed -e 's~&~~' | sed -e 's~>~~' | sed -e 's~<~~' | sed -e :a -e 's/<[^<]*>/ /g;/</{N;s/\n/ /;ba;}' | sed 's/http[^<]*\///' | tr " " \\n | sed s/[,\"=\|?.\!\(\):\;]//g | sed s/\^\#//g | sed s/\^\@//g | grep -Ev "[0-9]{4}-[0-9]{2}" | sed '/^$/d' >> wl.temp rm -f result.rss echo " complete." sleep .1 } # loop through keywords calling the twitter scrape function echo "Starting Twitter grabs..." for word in $args do getTweets $word done echo "" # loop through keywords calling the bing scrape function echo "Starting Bing grabs..." for word in $args do getBing $word done echo "" # sort/unique/clean up results echo "Sorting wordlist..." cat wl.temp | sort -u >> wordlist.list sort -u wordlist.list | uniq -u | sort -o wordlist.list sed -i '/^$/d' wordlist.list rm -f wl.temp echo "" # this while block allows us to re-scrape using the keywords found in the original grabs. while [ -z $quit ] do listLength=`wc -l wordlist.list | awk '{print $1}'` echo "Wordlist contains $listLength words so far. We can re-scan" echo "using the words in this list to find even more or we can quit." echo "" echo "Type q to exit or press [ENTER] to re-scan" read -n1 quit if [ $quit ] then echo "" echo "" break else # loop through current results calling the twitter scrape function echo "Starting twitter grabs..." cat wordlist.list | while read word; do getTweets $word done echo "" # loop through current results calling the bing scrape function echo "Starting Bing grabs..." cat wordlist.list | while read word; do getBing $word done echo "" # sort/unique/clean up results once more echo "Sorting wordlist..." cat wl.temp | sort -u >> wordlist.list sort -u wordlist.list | uniq -u | sort -o wordlist.list rm -f wl.temp echo "" fi done rm -f result.json 2> /dev/null rm -f result.rss 2> /dev/null listLength=`wc -l wordlist.list | awk '{print $1}'` echo "Worlist complete: $listLength words." |