Commit 3f9b98bf authored by Alexandre Dulaunoy's avatar Alexandre Dulaunoy Committed by GitHub
Browse files

Create gh-pages branch via GitHub

parent baecfff6

index.html

+61 −47
Original line number Diff line number Diff line
@@ -20,10 +20,14 @@ 


    <section class="main-content">

      <h1>

<a id="cve-search" class="anchor" href="#cve-search" aria-hidden="true"><span class="octicon octicon-link"></span></a>cve-search</h1>

<a id="cve-search" class="anchor" href="#cve-search" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>cve-search</h1>



<p><a href="https://gitter.im/cve-search/cve-search?utm_source=badge&amp;utm_medium=badge&amp;utm_campaign=pr-badge&amp;utm_content=badge"><img src="https://badges.gitter.im/cve-search/cve-search.svg" alt="Join the chat at https://gitter.im/cve-search/cve-search"></a></p>



<p><img src="https://avatars3.githubusercontent.com/u/15033728?v=3&amp;s=200" alt="cve-search logo"></p>



<p><a href="https://travis-ci.org/cve-search/cve-search"><img src="https://travis-ci.org/cve-search/cve-search.svg?branch=master" alt="Build Status"></a></p>



<p>cve-search is a tool to import CVE (Common Vulnerabilities and Exposures) and

CPE (Common Platform Enumeration) into a MongoDB to facilitate search

and processing of CVEs.</p>
@@ -39,10 +43,10 @@ a series of tools to query the system and a web API interface.</p> 
<p>cve-search is used by many organizations including the <a href="https://cve.circl.lu/">public CVE services of CIRCL</a>.</p>



<h2>

<a id="requirements" class="anchor" href="#requirements" aria-hidden="true"><span class="octicon octicon-link"></span></a>Requirements</h2>

<a id="requirements" class="anchor" href="#requirements" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Requirements</h2>



<ul>

<li>Python3.2 or later</li>

<li>Python3.3 or later</li>

<li>MongoDB 2.2 or later</li>

<li>redis server</li>

<li>Pip3
@@ -55,11 +59,17 @@ a series of tools to query the system and a web API interface.</p> 
<li>Tornado</li>

<li>Whoosh</li>

<li>Redis</li>

<li>Python-dateUtil</li>

<li>Python-dateutil</li>

<li>passlib</li>

<li>feedformater (for RSS and Atom dump_last) <a href="http://code.google.com/p/feedformatter/">http://code.google.com/p/feedformatter/</a>

</li>

<li>Whoosh <a href="http://packages.python.org/Whoosh/">http://packages.python.org/Whoosh/</a> (If you're planning to use the Full-text indexer)</li>

<li>irc</li>

<li>sleekxmpp</li>

<li>Werkzeug</li>

<li>Jinja2</li>

<li>itsdangerous</li>

<li>click</li>

</ul>

</li>

</ul>
@@ -70,9 +80,9 @@ a series of tools to query the system and a web API interface.</p> 
</code></pre>



<h2>

<a id="installation-of-mongodb" class="anchor" href="#installation-of-mongodb" aria-hidden="true"><span class="octicon octicon-link"></span></a>Installation of MongoDB</h2>

<a id="installation-of-mongodb" class="anchor" href="#installation-of-mongodb" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installation of MongoDB</h2>



<p>First, you'll need to have a Python 3 installation (3.2 or 3.3 preferred).

<p>First, you'll need to have a Python 3 installation (3.3 or higher).

Then you need to install MongoDB (2.2) from source (this should also work

with any standard packages from your favorite distribution). Don't forget

to install the headers for development while installing MongoDB.
@@ -81,13 +91,13 @@ packages for your distribution, or <a href="http://www.mongodb.org/downloads">ht 
the source code.</p>



<h2>

<a id="populating-the-database" class="anchor" href="#populating-the-database" aria-hidden="true"><span class="octicon octicon-link"></span></a>Populating the database</h2>

<a id="populating-the-database" class="anchor" href="#populating-the-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Populating the database</h2>



<p>For the initial run, you need to populate the CVE database by running:</p>



<pre><code>./db_mgmt.py -p

./db_mgmt_cpe_dictionary.py

./db_updater.py -c

<pre><code>./sbin/db_mgmt.py -p

./sbin/db_mgmt_cpe_dictionary.py

./sbin/db_updater.py -c

</code></pre>



<p>It will fetch all the existing XML files from the Common Vulnerabilities
@@ -96,7 +106,7 @@ and Exposures database and the Common Platform Enumeration.</p> 
<p>A more detailed documentation can be found in the Documentations folder of the project.</p>



<h2>

<a id="databases-and-collections" class="anchor" href="#databases-and-collections" aria-hidden="true"><span class="octicon octicon-link"></span></a>Databases and collections</h2>

<a id="databases-and-collections" class="anchor" href="#databases-and-collections" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Databases and collections</h2>



<p>The MongoDB database is called cvedb and there are 11 collections:</p>
@@ -109,8 +119,7 @@ and Exposures database and the Common Platform Enumeration.</p> 
<li>ranking (ranking rules per group) - local cve-search</li>

<li>d2sec (Exploitation reference from D2 Elliot Web Exploitation Framework) - source d2sec.com</li>

<li>

<a href="https://github.com/toolswatch/vFeed">vFeed</a> (cross-references to CVE ids (e.g. OVAL, OpenVAS, ...)) - source <a href="https://github.com/toolswatch/vFeed">vFeed</a>

</li>

<a href="https://github.com/toolswatch/vFeed">vFeed</a> (cross-references to CVE ids (e.g. OVAL, OpenVAS, ...)) - source <a href="https://github.com/toolswatch/vFeed">vFeed</a> - proprietary feed - <a href="https://cve.mitre.org/data/refs/">MITRE Reference Key/Maps</a> is preferred</li>

<li>ms - (Microsoft Bulletin (Security Vulnerabilities and Bulletin)) - source <a href="http://www.microsoft.com/en-us/download/details.aspx?id=36982">Microsoft</a>

</li>

<li>exploitdb (Offensive Security - Exploit Database) - source <a href="https://github.com/offensive-security/exploit-database">offensive security</a>
@@ -123,60 +132,60 @@ and Exposures database and the Common Platform Enumeration.</p> 
<ul>

<li>10: The cpe (Common Platform Enumeration) cache - source MongoDB cvedb collection cpe</li>

<li>11: The notification database - source cve-search</li>

<li>12: The NIST reference databased is a cross-reference database to CVE ids against various vendors ID - source NVD NIST</li>

<li>12: The <a href="https://cve.mitre.org/data/refs/">CVE reference database</a> is a cross-reference database to CVE ids against various vendors ID - source NVD NIST/MITRE</li>

</ul>



<h2>

<a id="updating-the-database" class="anchor" href="#updating-the-database" aria-hidden="true"><span class="octicon octicon-link"></span></a>Updating the database</h2>

<a id="updating-the-database" class="anchor" href="#updating-the-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Updating the database</h2>



<p>An updater script helps to start the db_mgmt_*  </p>



<pre><code>./db_updater.py -v

<pre><code>./sbin/db_updater.py -v

</code></pre>



<p>You can run it in a crontab, logging is done in syslog by default.</p>



<h2>

<a id="repopulating-the-database" class="anchor" href="#repopulating-the-database" aria-hidden="true"><span class="octicon octicon-link"></span></a>Repopulating the database</h2>

<a id="repopulating-the-database" class="anchor" href="#repopulating-the-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Repopulating the database</h2>



<p>To easily drop and re-populate all the databases</p>



<pre><code>./db_updater.py -v -f

<pre><code>./sbin/db_updater.py -v -f

</code></pre>



<p>This will drop all the existing external sources and reimport everything. This operation can take some time

and it's usually only required when new attributes parsing are added in cve-search.</p>



<h2>

<a id="usage" class="anchor" href="#usage" aria-hidden="true"><span class="octicon octicon-link"></span></a>Usage</h2>

<a id="usage" class="anchor" href="#usage" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Usage</h2>



<p>You can search the database using search.py</p>



<pre><code>./search.py -p cisco:ios:12.4

./search.py -p cisco:ios:12.4 -o json

./search.py -f nagios -n

./search.py -p microsoft:windows_7 -o html

<pre><code>./bin/search.py -p cisco:ios:12.4

./bin/search.py -p cisco:ios:12.4 -o json

./bin/search.py -f nagios -n

./bin/search.py -p microsoft:windows_7 -o html

</code></pre>



<p>If you want to search all the WebEx vulnerabilities and only printing the official

references from the supplier.</p>



<pre><code>./search.py -p webex: -o csv  -v "cisco"

<pre><code>./bin/search.py -p webex: -o csv  -v "cisco"

</code></pre>



<p>You can also dump the JSON for a specific CVE ID.</p>



<pre><code>./search.py -c CVE-2010-3333

<pre><code>./bin/search.py -c CVE-2010-3333

</code></pre>



<p>Or you can use the XMPP bot</p>



<pre><code>./search_xmpp.py -j mybot@jabber.org -p strongpassword

<pre><code>./bin/search_xmpp.py -j mybot@jabber.org -p strongpassword

</code></pre>



<p>Or dump the last 2 CVE entries in RSS or Atom format</p>



<pre><code>./dump_last.py -f atom -l 2

<pre><code>./bin/dump_last.py -f atom -l 2

</code></pre>



<p>Or you can use the webinterface.</p>
@@ -185,7 +194,7 @@ references from the supplier.</p> 
</code></pre>



<h2>

<a id="usage-of-the-ranking-database" class="anchor" href="#usage-of-the-ranking-database" aria-hidden="true"><span class="octicon octicon-link"></span></a>Usage of the ranking database</h2>

<a id="usage-of-the-ranking-database" class="anchor" href="#usage-of-the-ranking-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Usage of the ranking database</h2>



<p>There is a ranking database allowing to rank software vulnerabilities based on

their common platform enumeration name. The ranking can be done per organization
@@ -194,20 +203,20 @@ or department within your organization or any meaningful name for you.</p> 
<p>As an example, you can add a partial CPE name like "sap:netweaver" which is very

critical for your accounting department.</p>



<pre><code>./python3.3 db_ranking.py  -c "sap:netweaver" -g "accounting" -r 3

<pre><code>./sbin/db_ranking.py  -c "sap:netweaver" -g "accounting" -r 3

</code></pre>



<p>and then you can lookup the ranking (-r option) for a specific CVE-ID:</p>



<pre><code>./python3.3 search.py -c CVE-2012-4341  -r  -n

<pre><code>./bin/search.py -c CVE-2012-4341  -r  -n

</code></pre>



<h2>

<a id="advanced-usage" class="anchor" href="#advanced-usage" aria-hidden="true"><span class="octicon octicon-link"></span></a>Advanced usage</h2>

<a id="advanced-usage" class="anchor" href="#advanced-usage" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Advanced usage</h2>



<p>As cve-search is based on a set of tools, it can be used and combined with standard Unix tools. If you ever wonder what are the top vendors using the term "unknown" for their vulnerabilities:</p>



<pre><code>python3 search_fulltext.py -q unknown -f | jq -r '. | .vulnerable_configuration[0]' | cut -f3 -d: | sort  | uniq -c  | sort -nr | head -10

<pre><code>python3 bin/search_fulltext.py -q unknown -f | jq -c '. | .vulnerable_configuration[0]' | cut -f3 -d: | sort  | uniq -c  | sort -nr | head -10



1500 oracle

381 sun
@@ -223,36 +232,36 @@ critical for your accounting department.</p> 


<p>You can compare CVSS (Common Vulnerability Scoring System ) values of some products based on their CPE name. Like comparing oracle:java versus sun:jre and using R to make some statistics about their CVSS values:</p>



<pre><code>python3 search.py -p oracle:java -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

<pre><code>python3 bin/search.py -p oracle:java -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.

1.800   5.350   9.300   7.832  10.000  10.000





python3 search.py -p sun:jre -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

python3 bin/search.py -p sun:jre -o json  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.

0.000   5.000   7.500   7.333  10.000  10.000

</code></pre>



<h2>

<a id="fulltext-indexing" class="anchor" href="#fulltext-indexing" aria-hidden="true"><span class="octicon octicon-link"></span></a>Fulltext indexing</h2>

<a id="fulltext-indexing" class="anchor" href="#fulltext-indexing" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Fulltext indexing</h2>



<p>If you want to index all the CVEs from your current MongoDB collection:</p>



<pre><code>./python3.3 db_fulltext.py

<pre><code>./sbin/db_fulltext.py

</code></pre>



<p>and you query the fulltext index (to get a list of matching CVE-ID):</p>



<pre><code>./python3.3 search_fulltext.py -q NFS -q Linux

<pre><code>./bin/search_fulltext.py -q NFS -q Linux

</code></pre>



<p>or to query the fulltext index and output the JSON object for each CVE-ID:</p>



<pre><code>./python3.3 search_fulltext.py -q NFS -q Linux -j

<pre><code>./bin/search_fulltext.py -q NFS -q Linux -j

</code></pre>



<h2>

<a id="fulltext-visualization" class="anchor" href="#fulltext-visualization" aria-hidden="true"><span class="octicon octicon-link"></span></a>Fulltext visualization</h2>

<a id="fulltext-visualization" class="anchor" href="#fulltext-visualization" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Fulltext visualization</h2>



<p>The fulltext indexer visualization is using the fulltext indexes to build

a list of the most common keywords used in CVE. <a href="http://nltk.org/">NLTK</a> is
@@ -260,7 +269,7 @@ required to generate the keywords with the most common English 
stopwords and lemmatize the output. <a href="http://nltk.org/nltk3-alpha/">NTLK for Python 3</a>

exists but you need to use the alpha version of NLTK.</p>



<pre><code>./python3.3 search_fulltext.py  -g -s &gt;cve.json

<pre><code>./bin/search_fulltext.py  -g -s &gt;cve.json

</code></pre>



<p><img src="https://farm9.staticflickr.com/8109/8603509755_c7690c2de4_n.jpg" alt="cve-search visualization" title="CVE Keywords Visualization Using Data From cve-search"></p>
@@ -268,20 +277,20 @@ exists but you need to use the alpha version of NLTK.</p> 
<p>You can see a visualization on the <a href="http://www.foo.be/cve/">demo site</a>.</p>



<h2>

<a id="web-interface" class="anchor" href="#web-interface" aria-hidden="true"><span class="octicon octicon-link"></span></a>Web interface</h2>

<a id="web-interface" class="anchor" href="#web-interface" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Web interface</h2>



<p>The web interface is a minimal interface to see the last CVE entries and

query a specific CVE. You'll need flask in order to run the website and <a href="http://flask-pymongo.readthedocs.org/en/latest/">Flask-PyMongo</a>. To start

the web interface:</p>



<pre><code>cd ./web

./python3.3 index.py

./index.py

</code></pre>



<p>Then you can connect on <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a> to browser the last CVE.</p>



<h2>

<a id="web-api-interface" class="anchor" href="#web-api-interface" aria-hidden="true"><span class="octicon octicon-link"></span></a>Web API interface</h2>

<a id="web-api-interface" class="anchor" href="#web-api-interface" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Web API interface</h2>



<p>The web interface includes a minimal JSON API to get CVE by ID, by vendor or product.

A public version of the API is also accessible on <a href="https://cve.circl.lu/">cve.circl.lu</a>.</p>
@@ -321,7 +330,7 @@ A public version of the API is also accessible on <a href="https://cve.circl.lu/ 
</code></pre>



<h2>

<a id="software-using-cve-search" class="anchor" href="#software-using-cve-search" aria-hidden="true"><span class="octicon octicon-link"></span></a>Software using cve-search</h2>

<a id="software-using-cve-search" class="anchor" href="#software-using-cve-search" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Software using cve-search</h2>



<ul>

<li>
@@ -333,13 +342,18 @@ A public version of the API is also accessible on <a href="https://cve.circl.lu/ 
</ul>



<h2>

<a id="license" class="anchor" href="#license" aria-hidden="true"><span class="octicon octicon-link"></span></a>License</h2>

<a id="changelog" class="anchor" href="#changelog" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Changelog</h2>



<p>You can find the changelog <a href="https://github.com/cve-search/UpdateLog">here</a></p>



<h2>

<a id="license" class="anchor" href="#license" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>License</h2>



<p>cve-search is free software released under the "Modified BSD license"</p>



<pre><code>Copyright (c) 2012 Wim Remes - https://github.com/wimremes/

Copyright (c) 2012-2015 Alexandre Dulaunoy - https://github.com/adulau/

Copyright (c) 2015 Pieter-Jan Moreels - https://github.com/pidgeyl/

Copyright (c) 2012-2016 Alexandre Dulaunoy - https://github.com/adulau/

Copyright (c) 2015-2016 Pieter-Jan Moreels - https://github.com/pidgeyl/

</code></pre>



      <footer class="site-footer">

params.json

+7 −1

File changed.

Preview size limit exceeded, changes collapsed.

stylesheets/github-light.css

+38 −30
Original line number Diff line number Diff line 
/*

   Copyright 2014 GitHub Inc.

The MIT License (MIT)



   Licensed under the Apache License, Version 2.0 (the "License");

   you may not use this file except in compliance with the License.

   You may obtain a copy of the License at

Copyright (c) 2016 GitHub, Inc.



       http://www.apache.org/licenses/LICENSE-2.0

Permission is hereby granted, free of charge, to any person obtaining a copy

of this software and associated documentation files (the "Software"), to deal

in the Software without restriction, including without limitation the rights

to use, copy, modify, merge, publish, distribute, sublicense, and/or sell

copies of the Software, and to permit persons to whom the Software is

furnished to do so, subject to the following conditions:



   Unless required by applicable law or agreed to in writing, software

   distributed under the License is distributed on an "AS IS" BASIS,

   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

   See the License for the specific language governing permissions and

   limitations under the License.

The above copyright notice and this permission notice shall be included in all

copies or substantial portions of the Software.



THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE

SOFTWARE.



*/
@@ -19,7 +27,7 @@ 
  color: #969896;

}



.pl-c1      /* constant, markup.raw, meta.diff.header, meta.module-reference, meta.property-name, support, support.constant, support.variable, variable.other.constant */,

.pl-c1 /* constant, variable.other.constant, support, meta.property-name, support.constant, support.variable, meta.module-reference, markup.raw, meta.diff.header */,

.pl-s .pl-v /* string variable */ {

  color: #0086b3;

}
@@ -29,8 +37,8 @@ 
  color: #795da3;

}



.pl-s .pl-s1 /* string source */,

.pl-smi      /* storage.modifier.import, storage.modifier.package, storage.type.java, variable.other, variable.parameter.function */ {

.pl-smi /* variable.parameter.function, storage.modifier.package, storage.modifier.import, storage.type.java, variable.other */,

.pl-s .pl-s1 /* string source */ {

  color: #333;

}
@@ -42,13 +50,13 @@ 
  color: #a71d5d;

}



.pl-pds              /* punctuation.definition.string, string.regexp.character-class */,

.pl-s /* string */,

.pl-pds /* punctuation.definition.string, string.regexp.character-class */,

.pl-s .pl-pse .pl-s1 /* string punctuation.section.embedded source */,

.pl-sr /* string.regexp */,

.pl-sr .pl-cce /* string.regexp constant.character.escape */,

.pl-sr .pl-sra       /* string.regexp string.regexp.arbitrary-repitition */,

.pl-sr .pl-sre       /* string.regexp source.ruby.embedded */ {

.pl-sr .pl-sre /* string.regexp source.ruby.embedded */,

.pl-sr .pl-sra /* string.regexp string.regexp.arbitrary-repitition */ {

  color: #183691;

}
@@ -61,13 +69,13 @@ 
}



.pl-ii /* invalid.illegal */ {

  background-color: #b52a1d;

  color: #f8f8f8;

  background-color: #b52a1d;

}



.pl-sr .pl-cce /* string.regexp constant.character.escape */ {

  color: #63a35c;

  font-weight: bold;

  color: #63a35c;

}



.pl-ml /* markup.list */ {
@@ -77,8 +85,8 @@ 
.pl-mh /* markup.heading */,

.pl-mh .pl-en /* markup.heading entity.name */,

.pl-ms /* meta.separator */ {

  color: #1d3e81;

  font-weight: bold;

  color: #1d3e81;

}



.pl-mq /* markup.quote */ {
@@ -86,28 +94,28 @@ 
}



.pl-mi /* markup.italic */ {

  color: #333;

  font-style: italic;

  color: #333;

}



.pl-mb /* markup.bold */ {

  color: #333;

  font-weight: bold;

  color: #333;

}



.pl-md /* markup.deleted, meta.diff.header.from-file */ {

  background-color: #ffecec;

  color: #bd2c00;

  background-color: #ffecec;

}



.pl-mi1 /* markup.inserted, meta.diff.header.to-file */ {

  background-color: #eaffea;

  color: #55a532;

  background-color: #eaffea;

}



.pl-mdr /* meta.diff.range */ {

  color: #795da3;

  font-weight: bold;

  color: #795da3;

}



.pl-mo /* meta.output */ {

CRA Git | Maintained and supported by SUSTech CRA and CCSE