Added lzmavalid plugin; disabled default intallation of bundled libmagic; updated READMEs

INSTALL.md

@@ -17,12 +17,20 @@ $ make
 $ sudo make install
 ```
 
-Binwalk's core features will work out of the box without any additional dependencies. However, to take advantage of binwalk's more advanced capabilities, multiple supporting utilities/packages need to be installed (see the Dependencies section below).
+Many features will work out of the box without any additional dependencies. However, to take advantage of binwalk's more advanced capabilities, multiple supporting utilities/packages need to be installed (see the Dependencies section below).
 
 Dependencies
 ============
 
-The following run-time dependencies are only required for optional binwalk features, such as file extraction and graphing capabilities. Unless otherwise specified, these dependencies are available from most Linux package managers.
+Binwalk's only required run-time dependencies are libmagic and python-lzma:
+
+```bash
+$ sudo apt-get install libmagic1 python-lzma
+```
+
+Note that the libmagic development package is *not* required, and almost all Linux systems will already have libmagic installed. Additionally, python-lzma is a standard package in Python3, and thus requires no additional installation if running binwalk in Python3.
+
+The remaining run-time dependencies are only required for optional binwalk features, such as file extraction and graphing capabilities. Unless otherwise specified, these dependencies are available from most Linux package managers.
 
 Binwalk uses [pyqtgraph](http://www.pyqtgraph.org) to generate graphs and visualizations, which requires the following: 
 
@@ -54,23 +62,17 @@ $ (cd sasquatch && make && sudo make install)
 Bundled Software
 ================
 
-For convenience, the following libraries are bundled with binwalk and installed so as not to conflict with system-wide libraries:
+For convenience, the following libraries are bundled with the binwalk source:
 
     libmagic
 
-Installation of any individual bundled library can be disabled at build time:
-
-```bash
-$ ./configure --disable-libmagic
-```
-
-Alternatively, installation of all bundled libraries can be disabled at build time:
+By default, libmagic is not built or installed unless explicitly enabled during the build process:
 
 ```bash
-$ ./configure --disable-bundles
+$ ./configure --enable-libmagic
 ```
 
-If a bundled library is disabled, the equivalent library must be installed to a standard system library location (e.g., `/usr/lib`, `/usr/local/lib`, etc) in order for binwalk to find it at run time.
+By default, it is assumed that the libmagic library is already installed to a standard system library location (e.g., `/usr/lib`, `/usr/local/lib`, etc) in order for binwalk to find it at run time. 
 
 **Note:** If the bundled libmagic library is not used, be aware that:
 

configure.ac

@@ -21,13 +21,9 @@ AC_ARG_ENABLE([clibs],
               [AS_HELP_STRING([--disable-clibs], [do not build/install binwalk c libraries])],,
               [BUILD_C_LIBS=yes])
 
-AC_ARG_ENABLE([bundles],
-              [AS_HELP_STRING([--disable-bundles], [do not build/install any bundled software])],,
-              [BUILD_BUNDLES=yes])
-
 AC_ARG_ENABLE([libmagic],
-              [AS_HELP_STRING([--disable-libmagic], [do not build/install the bundled libmagic library])],,
-              [BUILD_MAGIC=yes])
+              [AS_HELP_STRING([--enable-libmagic], [build/install the bundled libmagic library])],,
+              [BUILD_MAGIC=no])
 
 CFLAGS="-Wall -fPIC $CFLAGS"
 INSTALL_OPTIONS="-m644"

src/C/README.md

@@ -2,9 +2,13 @@ About
 -----
 
 The libraries in this directory have been patched, extended, or otherwise modified from their original versions for use with binwalk.
-Some may include third-party modifications not available in the standard library release.
 
-Package mantainers should consult their particular distribution's rules regarding bundled libraries.
+Specifically:
+
+ o `libtinfl` includes several bug patches and wrapper functions not available in the upstream source.
+ o `libcompress42` contains code taken from the ncompress Unix utility and turned into a library. To the author's knowledge, this functionality is not available elsewhere as a standard library.
+
+Package mantainers should consult their particular distribution's rules on bundled code with regards to the above libraries.
 
 Installation
 ------------

src/binwalk/plugins/lzmavalid.py

+import lzma
+import binwalk.core.plugin
+from binwalk.core.common import BlockFile
+
+class LZMAPlugin(binwalk.core.plugin.Plugin):
+    '''
+    Validates lzma signature results.
+    '''
+    MODULES = ['Signature']
+
+    # Some lzma files exclude the file size, so we have to put it back in.
+    # See also the lzmamod.py plugin.
+    FAKE_LZMA_SIZE = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF"
+
+    # Check up to the first 64KB
+    MAX_DATA_SIZE = 64 * 1024
+
+    def is_valid_lzma(self, data):
+        valid = True
+
+        # The only acceptable exception is that of IOError "unknown BUF error",
+        # which indicates that the input data was truncated.
+        try:
+            d = lzma.decompress(data)
+        except IOError as e:
+            if e.message != "unknown BUF error":
+                valid = False
+        except Exception as e:
+            valid = False
+
+        return valid
+
+    def scan(self, result):
+        # If this result is an lzma signature match, try to decompress the data
+        if result.valid and result.file and result.description.lower().startswith('lzma compressed data'):
+
+            # Seek to and read the suspected lzma data
+            fd = self.module.config.open_file(result.file.name, offset=result.offset, length=self.MAX_DATA_SIZE)
+            data = fd.read(self.MAX_DATA_SIZE)
+            fd.close()
+
+            # Validate the original data; if that fails, maybe it is missing the size field,
+            # so try again with a dummy size field in place.
+            if not self.is_valid_lzma(data):
+                data = data[:5] + self.FAKE_LZMA_SIZE + data[5:]
+                if not self.is_valid_lzma(data):
+                    result.valid = False
+

src/bundles/README.md

@@ -8,6 +8,6 @@ Package maintainers can generally replace these libraries with standard librarie
 Installation
 ------------
 
-These libraries are built and installed by default, unless the `--disable-bundles` option is provided to the configure script.
+These libraries are not built or installed by default, unless the `--enable-<libname>` option is provided to the configure script.
 
 They will be installed into the `libs` sub-directory of the binwalk Python module, so as to not conflict with existing libraries on the system.