Merge remote-tracking branch 'upstream/master'

For the initial run, you need to populate the CVE database by running:

    ./db_mgmt.py -p

    ./db_mgmt_cpe_dictionary.py

    ./db_updater.py -c

    ./sbin/db_mgmt.py -p

    ./sbin/db_mgmt_cpe_dictionary.py

    ./sbin/db_updater.py -c

It will fetch all the existing XML files from the Common Vulnerabilities

and Exposures database and the Common Platform Enumeration.

An updater script helps to start the db_mgmt_*  

    ./db_updater.py -v

    ./sbin/db_updater.py -v

You can run it in a crontab, logging is done in syslog by default.

To easily drop and re-populate all the databases

    ./db_updater.py -v -f

    ./sbin/db_updater.py -v -f

This will drop all the existing external sources and reimport everything. This operation can take some time

and it's usually only required when new attributes parsing are added in cve-search.

You can search the database using search.py

    ./search.py -p cisco:ios:12.4

    ./search.py -p cisco:ios:12.4 -o json

    ./search.py -f nagios -n

    ./search.py -p microsoft:windows_7 -o html

    ./bin/search.py -p cisco:ios:12.4

    ./bin/search.py -p cisco:ios:12.4 -o json

    ./bin/search.py -f nagios -n

    ./bin/search.py -p microsoft:windows_7 -o html

If you want to search all the WebEx vulnerabilities and only printing the official

references from the supplier.

    ./search.py -p webex: -o csv  -v "cisco"

    ./bin/search.py -p webex: -o csv  -v "cisco"

You can also dump the JSON for a specific CVE ID.

    ./search.py -c CVE-2010-3333

    ./bin/search.py -c CVE-2010-3333

Or you can use the XMPP bot

    ./search_xmpp.py -j mybot@jabber.org -p strongpassword

    ./bin/search_xmpp.py -j mybot@jabber.org -p strongpassword

Or dump the last 2 CVE entries in RSS or Atom format

    ./dump_last.py -f atom -l 2

    ./bin/dump_last.py -f atom -l 2

Or you can use the webinterface.

As an example, you can add a partial CPE name like "sap:netweaver" which is very

critical for your accounting department.

    ./python3.3 db_ranking.py  -c "sap:netweaver" -g "accounting" -r 3

    ./python3.3 sbin/db_ranking.py  -c "sap:netweaver" -g "accounting" -r 3

and then you can lookup the ranking (-r option) for a specific CVE-ID:

    ./python3.3 search.py -c CVE-2012-4341  -r  -n

    ./python3.3 bin/search.py -c CVE-2012-4341  -r  -n

Advanced usage

--------------

As cve-search is based on a set of tools, it can be used and combined with standard Unix tools. If you ever wonder what are the top vendors using the term "unknown" for their vulnerabilities:

    python3 search_fulltext.py -q unknown -f | jq -c '. | .vulnerable_configuration[0]' | cut -f3 -d: | sort  | uniq -c  | sort -nr | head -10

    python3 bin/search_fulltext.py -q unknown -f | jq -c '. | .vulnerable_configuration[0]' | cut -f3 -d: | sort  | uniq -c  | sort -nr | head -10

    1500 oracle

    381 sun

You can compare CVSS (Common Vulnerability Scoring System ) values of some products based on their CPE name. Like comparing oracle:java versus sun:jre and using R to make some statistics about their CVSS values:

    python3 search.py -p oracle:java -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

    python3 bin/search.py -p oracle:java -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.

    1.800   5.350   9.300   7.832  10.000  10.000

    python3 search.py -p sun:jre -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

    python3 bin/search.py -p sun:jre -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.

    0.000   5.000   7.500   7.333  10.000  10.000

If you want to index all the CVEs from your current MongoDB collection:

    ./python3.3 db_fulltext.py

    ./python3.3 sbin/db_fulltext.py

and you query the fulltext index (to get a list of matching CVE-ID):

    ./python3.3 search_fulltext.py -q NFS -q Linux

    ./python3.3 bin/search_fulltext.py -q NFS -q Linux

or to query the fulltext index and output the JSON object for each CVE-ID:

    ./python3.3 search_fulltext.py -q NFS -q Linux -j

    ./python3.3 bin/search_fulltext.py -q NFS -q Linux -j

Fulltext visualization

----------------------

stopwords and lemmatize the output. [NTLK for Python 3](http://nltk.org/nltk3-alpha/)

exists but you need to use the alpha version of NLTK.

    ./python3.3 search_fulltext.py  -g -s >cve.json

    ./python3.3 bin/search_fulltext.py  -g -s >cve.json

![cve-search visualization](https://farm9.staticflickr.com/8109/8603509755_c7690c2de4_n.jpg "CVE Keywords Visualization Using Data From cve-search")

  return lst

def getFreeText(text):

  try: # Before Mongo 3

    return [x["obj"] for x in db.command("text", "cves", search=text)["results"]]

  except: # As of Mongo 3

    return sanitize(colCVE.find({"$text":{"$search":text}}))

def getSearchResults(search):

  result={'data':[]}