* Allow a Py_buffer as data for Rules_match

This makes rules matching compatible with data objects
`PyArg_ParseTuple` does not consider read-only (even though they might
actually be), such a memoryviews. The main change is replacing the `s#`
formatter with `s*` and replacing the `(pointer, length)` pair with a
`Py_buffer` object accordingly. Additional care must be taken to release
the `Py_buffer` on every error path.

* Rules_match: zero-initialize data

PyArg_ParseTupleAndKeywords does not initialize optional fields unless
they are passed, which means we need to zero-initialize the data buffer
to be sure the later NULL checks always work.

This commit also gets rid of the unneeded has_data flag.

* Add test for matching on a memoryview

This is regression in introduced in #140. When a string in the metadata section contains invalid UTF-8 characters the behavior Python 2 is leave the string exactly as it appears in YARA, in Python 3 however the invalid characters are removed because Python 3 strings are not handled as bytes like in Python 2, they most have a valid encoding. PR #140 was an attempt to homogenize the behavior in both versions of Python, but it introduced this other issue.

* Support a "is_global" and "is_private" member on Rules.

When writing linters it is currently impossible to know (via rule introspection)
if a given rule is private or global. We have banned global rules for our use
case and we have to resort to a janky regex against our rules files to know if
anyone is about to commit a global rule. I figure exposing these two flags via
python will be useful for programatically checking those bits.

I'm not very pleased with the name "is_global" - I wanted to go with just
"global" and "private" but "global" is a reserved keyword and rule.global breaks
the python interpreter. I'm open to changing the member names if you have any
suggestions.

* Decrement reference counts on global and private.

* Update global and private checks after API changes.

* Handle invalid unicode in metadata values.

In #135 it was brought up that you can crash the python interpreter if you have
invalid unicode in a metadata value. This is my attempt to fix that by
attempting to create a string, and if that fails falling back to a bytes object.
On the weird chance that the bytes object fails to create I added a safety check
so that we don't add a NULL ptr to the dictionary (this is how the crash was
manifesting).

It's debatable if we want to ONLY add strings as metadata, and NOT fallback to
bytes. If we don't fall back to bytes the only other option I see is to silently
drop that metadata on the floor. The tradeoff here is that now you may end up
with a string or a bytes object in your metadata dictionary, which is less than
ideal IMO.

I'm open to suggestions on this one.

Fixes #135

* Add error handling to conversion to Unicode
Metadata test accepts stripped or original characters

* Remove 'or' clause from tests and add another NULL test check.

Co-authored-by: malvidin <malvidin@gmail.com>

See: https://github.com/VirusTotal/yara/issues/990

Fix regression in "includes" argument in yara.compile. With includes=True includes were being disabled instead of enabled.

Closes #78.

* Fix regression in which_callbacks.

* Add test to detect regression fixed in 367290a586ef7b7f83e5db265bfe39b57847473e.

* Fix typo in previous commit.

* Remove redundant conditional.

* stable on python 2

* Stable on python 2 and 3 (fixed utf-8 and ascii encoding issues)
 * Still needs compatible yara submodule update once pull request accepted

* * Fixed all encoding issues

* Proper error handling

* Updating yara submodule to reference yara patched with include callback support

* Updating submodule's branch

* Updating yara submodule

* Updating yara submodule

* Updating yara submodule

* Updating yara submodule

* Submodule update

* * Fixing memory leaks
* Fixing errors handling

* making error messages order consistent between include_callback and default yara behaviour

* Removing exception printing when callback fails

* Minor re-styling.

* Destroy compiler if PyCallable_Check(include_callback) fails.

* References to Py_None should be increased.

* Use Py_DECREF instead of Py_XDECREF for references that can't be NULL.

* Minor re-styling.

* Fix reference leak.

After calling Py_INCREF(include_callback) some code paths were leading to a return without calling Py_DECREF. Calling Py_INCREF before yr_compiler_set_include_callback is not necessary, as this function doesn't yield control to Python, but it should be called before yr_compiler_add_XX.

* Remove unnecessary calls to Py_INCREF/Py_DECREF.

The references were already incremented in yara_compile.

* Implement test case for include callbacks

* Point yara submodule to official repository.

Add a 'which_callbacks' keyword to match(), which limits when the Python callback will be called.