kgrams 0.2.0

Breaking changes

• tknz_sent() and preprocess() now have a different implementation on Windows and UNIX OSs, respectively (since the previous C++ implementation has impredictable behaviour on Windows, see #30). This fix also included minor changes in the tknz_sent() output, in some corner cases (e.g. tknz_sent("") now returns character(0), wheareas it used to return "").

New features

• perplexity() gets a new argument exp that allows to return the cross-entropy per word, rather than perplexity (its exponential).
• perplexity.character() gets a new argument detailed that allows to return, alongside with the total perplexity of the input document, also the cross-entropies and word lengths of individual sentences. Closes #28.

Improvements

• Minor documentation improvements.

kgrams 0.1.5

• Removed “Tools for…” at the beginning of package DESCRIPTION, as per CRAN’s request.
• Simplified examples in ?kgram_freqs.

kgrams 0.1.4

• Updated R requirements 3.5 -> 4.0.
• Removed SystemRequirements: C++11 (see this tidyverse blog post)

kgrams 0.1.3

• Remove dependency from external online sources in vignette.

kgrams 0.1.2

Overall Software Improvements

• The package’s test suite has been greatly extended.
• Improved error/warning conditions for wrong arguments.
• Re-enabled compiler diagnostics as per CRAN policy (#19)

API Changes

• verbose arguments now default to FALSE.
• probability(), perplexity() and sample_sentences() are restricted to accept only language_model class objects as their model argument.

New features

• as_dictionary(NULL) now returns an empty dictionary.

Bug Fixes

• Fixed bug causing .preprocess and .tknz_sent arguments to be ignored in process_sentences().
• Fixed previously wrong defaults for max_lines and batch_size arguments in kgram_freqs.connection().
• Added print method for class dictionary.
• Fixed bug causing invalid results in dictionary() with batch processing and non-trivial size constraints on vocabulary size.

Other

• Maintainer’s email updated