There are some great wordlists out there for sure… but a targeted wordlist that fits with the subject of the target site/database can prove to be much more effective. Joshua Dustin posted to his blog recently about this and I thought this was an excellent idea and wanted to take it a little bit further. This script adds some automation to his idea and also adds a word-grab from Bing as well. Since it’s a little more modularized in this script it’ll be easy to add other word sources. I’ll be adding more soon once I have additional time to do so. Please check out his post for further information on why this type of wordlist generation can be so effective. He does a great job explaining it.
I have a few other things in the works I had intended on releasing sooner (rather than another post based on someone else’ idea) but those scripts are getting near-daily updates due to the fact that I’m using them constantly. One just got a roughly 20x speed boost today thanks to some command-line option changes. Don’t worry, though, they’re worth the wait! Now, on to the code;
Running this script:
- Run it using as many keywords as you’d like to scrape off the web:
-
./wordlistgen.sh your keywords go here
-
#!/bin/bash
#################################################
# Internet Wordlist Generator by: #
# Nathan V #
# Cyber Security Analyst #
# http://nathanv.com #
# #
# For assistance and new versions contact #
# nathan.v@gmail.com #
# This file updated: 18 July 2012 #
#################################################
# This script (c)2012 Nathan V : License: GPLv3 #
# This is free software, and you are welcome to #
# redistribute it under certain conditions; See #
# http://www.gnu.org/licenses/gpl.html #
#################################################
# getTweets() is based on twitter.sh by: #
# Joshua Dustin #
#7habitsofhighlyeffectivehackers.blogspot.com.au#
#################################################
#import arguments
args="$@"
#clear screen and check for input parameter
clear
if [ -z "$1" ]
then
echo "Missing input parameters. Please use $0 "
kill -int $$
else
echo ""
fi
# scrape Twitter for tweets containing your keywords
getTweets() {
local key=$1
echo -n "Grabing for keyword $key..."
wget -q "http://search.twitter.com/search.json?q=$key&rpp=500" -O result.json
cat result.json | tr "," \\n | grep "^\"text" | cut -d"\"" -f4- | tr " " \\n | sed -e 's~&~~' | sed -e 's~>~~' | sed -e 's~<~~' | sed s/[\"=\|?.\!\(\):\;]//g | sed s/\^\#//g | sed s/\^\@//g | sed '/^$/d' | grep -v "^http:" | grep -v "\\\\" >> wl.temp
rm -f result.json
echo " complete."
sleep .1
}
# scrape Bing for search results related to your keywords
getBing() {
local key=$1
echo -n "Grabing for keyword $key..."
wget -q "http://api.search.live.com/rss.aspx?source=web&query=$key" -O result.rss
cat result.rss | sed -e 's~&~~' | sed -e 's~>~~' | sed -e 's~<~~' | sed -e :a -e 's/<[^<]*>/ /g;/> wl.temp
rm -f result.rss
echo " complete."
sleep .1
}
# loop through keywords calling the twitter scrape function
echo "Starting Twitter grabs..."
for word in $args
do
getTweets $word
done
echo ""
# loop through keywords calling the bing scrape function
echo "Starting Bing grabs..."
for word in $args
do
getBing $word
done
echo ""
# sort/unique/clean up results
echo "Sorting wordlist..."
cat wl.temp | sort -u >> wordlist.list
sort -u wordlist.list | uniq -u | sort -o wordlist.list
sed -i '/^$/d' wordlist.list
rm -f wl.temp
echo ""
# this while block allows us to re-scrape using the keywords found in the original grabs.
while [ -z $quit ]
do
listLength=`wc -l wordlist.list | awk '{print $1}'`
echo "Wordlist contains $listLength words so far. We can re-scan"
echo "using the words in this list to find even more or we can quit."
echo ""
echo "Type q to exit or press [ENTER] to re-scan"
read -n1 quit
if [ $quit ]
then
echo ""
echo ""
break
else
# loop through current results calling the twitter scrape function
echo "Starting twitter grabs..."
cat wordlist.list | while read word;
do
getTweets $word
done
echo ""
# loop through current results calling the bing scrape function
echo "Starting Bing grabs..."
cat wordlist.list | while read word;
do
getBing $word
done
echo ""
# sort/unique/clean up results once more
echo "Sorting wordlist..."
cat wl.temp | sort -u >> wordlist.list
sort -u wordlist.list | uniq -u | sort -o wordlist.list
rm -f wl.temp
echo ""
fi
done
rm -f result.json 2> /dev/null
rm -f result.rss 2> /dev/null
listLength=`wc -l wordlist.list | awk '{print $1}'`
echo "Worlist complete: $listLength words."