Merge pull request #80 from adulau/master

A more detailed documentation can be found in the Documentations folder of the project.

Database and collections

------------------------

Databases and collections

-------------------------

The MongoDB database is called cvedb and there are 8 collections:

The MongoDB database is called cvedb and there are 10 collections:

* cves (Common Vulnerabilities and Exposure items) - source NVD NIST

* cpe (Common Platform Enumeration items) - source NVD NIST

* ranking (ranking rules per group) - local cve-search

* d2sec (Exploitation reference from D2 Elliot Web Exploitation Framework) - source d2sec.com

* [vFeed](https://github.com/toolswatch/vFeed) (cross-references to CVE ids (e.g. OVAL, OpenVAS, ...)) - source [vFeed](https://github.com/toolswatch/vFeed)

* Microsoft Bulletin (Security Vulnerabilities and Bulletin) - source [Microsoft](http://www.microsoft.com/en-us/download/details.aspx?id=36982)

* info (metadata of each collection like last-modified) - local cve-search

The Redis database has 3 databases:

* 10: The cpe (Common Platform Enumeration) cache - source MongoDB cvedb collection cpe

* 11: The notification database - source cve-search

* 12: The NIST reference databased is a cross-reference database to CVE ids against various vendors ID - source NVD NIST

Updating the database

---------------------

    To add verbose output, add parameter <span class="code">-v</span><br />

    To let the script run automatically at a regular interval, add parameter <span class="code">-l</span><br />

  </p>

  <h3>Fulltext Search</h3>

  <p>

    If you want to enable fulltext search, you have to enable this in the database. <br />

    To do this, log into the Mongo database (<span class="code">$ mongo</span>) and run the following command: <span class="code">db.adminCommand({"setParameter": 1, "textSearchEnabled":true})</span>. <br />

    <b>Note:</b> when the Mongo database is shut down, fulltext search will be disabled again. Simply run the same command again to activate it.

  </p>

  <h2>Final notes</h2>

  <p>

    After this procedure the database is initialized and up to date. From this point on, to update the database, you only have to repeat the procedure to <a href="#update">update the database</a>.<br />

Port: 6379

VendorsDB: 10

NotificationsDB: 11

RefDB: 12

[Mongo]

Host: localhost

Port: 27017

vFeedStatus: http://www.toolswatch.org/update.dat

Vendor: https://nvd.nist.gov/download/vendorstatements.xml

CAPEC: http://capec.mitre.org/data/xml/capec_v2.6.xml

MSBULLETIN: http://download.microsoft.com/download/6/7/3/673E4349-1CA5-40B9-8879-095C72D5B49D/BulletinSearch.xls

Ref: https://cve.mitre.org/data/refs/refmap/allrefmaps.zip

[Webserver]

Host: 127.0.0.1

    def __init__(self, collection="cves", rankinglookup=False,

                 namelookup=False, vfeedlookup=False,

                 capeclookup=False, subscorelookup=False):

                 capeclookup=False, subscorelookup=False, reflookup=False):

        self.collectionname = collection

        self.rankinglookup = rankinglookup

            self.vfeed = connectdb['vfeed']

        if capeclookup:

            self.capec = connectdb['capec']

        if reflookup:

            self.ref = Configuration.getRedisRefConnection()

    def getcapec(self, cweid=None):

        if cweid is None or not self.capeclookup:

            capec.append(f)

        return capec

    def getref(self, cveid=None):

        if cveid is None or not self.ref:

            return False

        ref = self.ref.smembers(cveid)

        return ref

    def getcpe(self, cpeid=None):

        if not(self.namelookup):

    l = last(rankinglookup=False, vfeedlookup=True, capeclookup=True)

    print (l.getcve("CVE-2015-0597"))

    print (l.getcapec("200"))

    l = last(reflookup=True)

    print(l.getref("CVE-2015-0597"))

if __name__ == "__main__":

    test_last()

    ConfigParser.read(os.path.join(runPath, "../etc/configuration.ini"))

    default = {'redisHost': 'localhost', 'redisPort': 6379,

               'redisVendorDB': 10,      'redisNotificationsDB': 11,

               'redisRefDB': 12,

               'mongoHost': 'localhost', 'mongoPort': 27017,

               'mongoDB': "cvedb",

               'mongoUsername': '', 'mongoPassword': '',

               'd2sec': "http://www.d2sec.com/exploits/elliot.xml",

               'vendor': "https://nvd.nist.gov/download/vendorstatements.xml",

               'capec': "http://capec.mitre.org/data/xml/capec_v2.6.xml",

               'msbulletin': "http://download.microsoft.com/download/6/7/3/673E4349-1CA5-40B9-8879-095C72D5B49D/BulletinSearch.xls",

               'ref': "https://cve.mitre.org/data/refs/refmap/allrefmaps.zip",

               'logging': True,           'logfile': "../log/cve-search.log",

               'maxLogSize': '100MB',     'backlog': 5,

        redisDB = cls.readSetting("Redis", "NotificationsDB", cls.default['redisNotificationsDB'])

        return redis.StrictRedis(host=redisHost, port=redisPort, db=redisDB, charset="utf-8", decode_responses=True)

    @classmethod

    def getRedisRefConnection(cls):

        redisHost = cls.getRedisHost()

        redisPort = cls.getRedisPort()

        redisDB = cls.readSetting("Redis", "RefDB", cls.default['redisRefDB'])

        return redis.StrictRedis(host=redisHost, port=redisPort, db=redisDB, charset="utf-8", decode_responses=True)

    # Flask

    @classmethod

    def getFlaskHost(cls):

    @classmethod

    def getCAPECDict(cls):

        return cls.readSetting("Sources", "CAPEC", cls.default['capec'])

    @classmethod

    def getMSBULLETINDict(cls):

        return cls.readSetting("Sources", "MSBULLETIN", cls.default['msbulletin'])

    # Logging

    @classmethod