Shell Script: Use Twitter and Bing to Generate Wordlists

There are some great wordlists out there for sure… but a targeted wordlist that fits with the subject of the target site/database can prove to be much more effective.  Joshua Dustin posted to his blog recently about this and I thought this was an excellent idea and wanted to take it a little bit further.  This script adds some automation to his idea and also adds a word-grab from Bing as well.  Since it’s a little more modularized in this script it’ll be easy to add other word sources.  I’ll be adding more soon once I have additional time to do so.  Please check out his post for further information on why this type of wordlist generation can be so effective.  He does a great job explaining it.

I have a few other things in the works I had intended on releasing sooner (rather than another post based on someone else’ idea) but those scripts are getting near-daily updates due to the fact that I’m using them constantly.  One just got a roughly 20x speed boost today thanks to some command-line option changes.  Don’t worry, though, they’re worth the wait!  Now, on to the code;

Running this script:

  • Run it using as many keywords as you’d like to scrape off the web:
    • ./wordlistgen.sh your keywords go here
#!/bin/bash
#################################################
# Internet Wordlist Generator by:               #
#      Nathan V                                 #
#      Cyber Security Analyst                   #
#      http://nathanv.com                       #
#                                               #
# For assistance and new versions contact       #
#      nathan.v@gmail.com                       #
# This file updated:               18 July 2012 #
#################################################
# This script (c)2012 Nathan V : License: GPLv3 #
# This is free software, and you are welcome to #
# redistribute it under certain conditions; See #
# http://www.gnu.org/licenses/gpl.html          #
#################################################
# getTweets() is based on twitter.sh by:        #
#      Joshua Dustin                            #
#7habitsofhighlyeffectivehackers.blogspot.com.au#
#################################################

#import arguments
args="$@"

#clear screen and check for input parameter
clear
if [ -z "$1" ]
then
    echo "Missing input parameters.  Please use $0  "
    kill -int $$
else
  echo ""
fi

# scrape Twitter for tweets containing your keywords
getTweets() {
	local key=$1
	echo -n "Grabing for keyword $key..."
	wget -q "http://search.twitter.com/search.json?q=$key&rpp=500" -O result.json
	cat result.json | tr "," \\n | grep "^\"text" | cut -d"\"" -f4- | tr " " \\n | sed -e 's~&~~' | sed -e 's~>~~' | sed -e 's~<~~' | sed s/[\"=\|?.\!\(\):\;]//g | sed s/\^\#//g | sed s/\^\@//g | sed '/^$/d' | grep -v "^http:" | grep -v "\\\\" >> wl.temp
	rm -f result.json
	echo " complete."
	sleep .1
}

# scrape Bing for search results related to your keywords
getBing() {
	local key=$1
	echo -n "Grabing for keyword $key..."
	wget -q "http://api.search.live.com/rss.aspx?source=web&query=$key" -O result.rss
	cat result.rss | sed -e 's~&~~' | sed -e 's~>~~' | sed -e 's~<~~' | sed -e :a -e 's/<[^<]*>/ /g;/> wl.temp
	rm -f result.rss
	echo " complete."
	sleep .1
}

# loop through keywords calling the twitter scrape function
echo "Starting Twitter grabs..."
for word in $args
	do
	getTweets $word
	done
echo ""

# loop through keywords calling the bing scrape function
echo "Starting Bing grabs..."
for word in $args
	do
	getBing $word
	done
echo ""

# sort/unique/clean up results
echo "Sorting wordlist..."
cat wl.temp | sort -u >> wordlist.list
sort -u wordlist.list | uniq -u | sort -o wordlist.list
sed -i '/^$/d' wordlist.list
rm -f wl.temp
echo ""

# this while block allows us to re-scrape using the keywords found in the original grabs.
while [ -z $quit ]
	do
	listLength=`wc -l wordlist.list | awk '{print $1}'`
	echo "Wordlist contains $listLength words so far.  We can re-scan"
	echo "using the words in this list to find even more or we can quit."
	echo ""
	echo "Type q to exit or press [ENTER] to re-scan"
	read -n1 quit
	if [ $quit ]
	then 
		echo ""
		echo ""
		break
	else
		# loop through current results calling the twitter scrape function
		echo "Starting twitter grabs..."

		cat wordlist.list | while read word;
			do
			getTweets $word
			done
		echo ""

		# loop through current results calling the bing scrape function
		echo "Starting Bing grabs..."
		cat wordlist.list | while read word;
			do
			getBing $word
			done
		echo ""

		# sort/unique/clean up results once more
		echo "Sorting wordlist..."
		cat wl.temp | sort -u >> wordlist.list
		sort -u wordlist.list | uniq -u | sort -o wordlist.list
		rm -f wl.temp
		echo ""
	fi
	done

rm -f result.json 2> /dev/null
rm -f result.rss 2> /dev/null
listLength=`wc -l wordlist.list | awk '{print $1}'`
echo "Worlist complete:  $listLength words."

More Information: