Commit b406691b by devttys0

Added ability to combine all signature scans together into one scan.

parent 9a47ed82
DESCRIPTION
The binwalk python module can be used by any python script to programatically perform binwalk scans and
obtain the results of those scans.
The binwalk python module can be used by any python script to programatically perform binwalk scans and
obtain the results of those scans.
The classes, methods and objects in the binwalk modules are documented via pydoc, including examples,
so those interested in using the binwalk module are encouraged to look there. However, several common usage
examples are provided here to help jump-start development efforts.
The classes, methods and objects in the binwalk modules are documented via pydoc, including examples,
so those interested in using the binwalk module are encouraged to look there. However, several common usage
examples are provided here to help jump-start development efforts.
BINWALK SCRIPTING
Each of binwalk's features (signature scans, entropy analysis, etc) are implemented as separate modules.
These modules can be invoked via the binwalk.core.module.Modules class, which makes scripting trivial
through its execute method.
Each of binwalk's features (signature scans, entropy analysis, etc) are implemented as separate modules.
These modules can be invoked via the binwalk.core.module.Modules class, which makes scripting trivial
through its execute method.
In fact, the binwalk command line utility can be duplicated nearly entirely with just two lines of code:
In fact, the binwalk command line utility can be duplicated nearly entirely with just two lines of code:
import binwalk
binwalk.Modules().execute()
import binwalk
binwalk.Modules().execute()
The Modules class constructor as well as the execute method accept both Python args and kwargs corresponding
to the normal command line options accepted by the binwalk command line utility, providing a large amount of
freedom in how you choose to specify binwalk options (if none are specified, sys.argv is used by default).
The Modules class constructor as well as the execute method accept both Python args and kwargs corresponding
to the normal command line options accepted by the binwalk command line utility, providing a large amount of
freedom in how you choose to specify binwalk options (if none are specified, sys.argv is used by default).
For example, to execute a signature scan, you at the very least have to specify the --signature command line
option, as well as a list of files to scan. This can be done in a number of ways:
For example, to execute a signature scan, you at the very least have to specify the --signature command line
option, as well as a list of files to scan. This can be done in a number of ways:
binwalk.Modules().execute('firmware1.bin', 'firmware2.bin', signature=True)
binwalk.Modules().execute('firmware1.bin', 'firmware2.bin', signature=True)
binwalk.Modules().execute('firmware1.bin', 'firmware2.bin', **{'signature' : True})
binwalk.Modules().execute(*['firmware1.bin', 'firmware2.bin'], signature=True)
binwalk.Modules().execute(*['--signature', 'firmware1.bin', 'firmware2.bin',])
binwalk.Modules().execute('firmware1.bin', 'firmware2.bin', **{'signature' : True})
binwalk.Modules().execute(*['firmware1.bin', 'firmware2.bin'], signature=True)
binwalk.Modules().execute(*['--signature', 'firmware1.bin', 'firmware2.bin',])
binwalk.Modules().execute('--signature', 'firmware1.bin', 'firmware2.bin')
binwalk.Modules().execute('--signature', 'firmware1.bin', 'firmware2.bin')
All args and kwargs keys/values correspond to binwalk's command line options. Either args or kwargs, or a combination
of the two may be used, with the following caveats:
All args and kwargs keys/values correspond to binwalk's command line options. Either args or kwargs, or a combination
of the two may be used, with the following caveats:
o All command line switches passed via args must be preceeded by hyphens (not required for kwargs)
o All file names must be passed via args, not kwargs
o All command line switches passed via args must be preceeded by hyphens (not required for kwargs)
o All file names must be passed via args, not kwargs
ACCESSING SCAN RESULTS
binwalk.core.module.Modules.execute returns a list of objects. Each object corresponds to a module that was run.
For example, if you specified --signature and --entropy, then both the Signature and Entropy modules would be executed
and you would be returned a list of two objects.
binwalk.core.module.Modules.execute returns a list of objects. Each object corresponds to a module that was run.
For example, if you specified --signature and --entropy, then both the Signature and Entropy modules would be executed
and you would be returned a list of two objects.
The two attributes of interest for each object are the 'results' and 'errors' objects. Each is a list of binwalk.core.module.Result
or binwalk.core.module.Error objects respectively. Each Result or Error object may contain custom attributes set by each module,
but are guarunteed to have at least the following attributes (though modules are not required to populate all attributes):
The two attributes of interest for each object are the 'results' and 'errors' objects. Each is a list of binwalk.core.module.Result
or binwalk.core.module.Error objects respectively. Each Result or Error object may contain custom attributes set by each module,
but are guarunteed to have at least the following attributes (though modules are not required to populate all attributes):
o offset - The file offset of the result/error (usually unused for errors).
o description - The result/error description, as displayed to the user.
o module - Name of the module that generated the result/error.
o file - The file object of the scanned file.
o valid - Set to True if the result if value, False if invalid (usually unused for errors).
o display - Set to True to display the result to the user, False to hide it (usually unused for errors).
o extract - Set to True to flag this result for extraction (not used for errors).
o plot - Set to Flase to exclude this result from entropy plots (not used for errors).
o offset - The file offset of the result/error (usually unused for errors).
o description - The result/error description, as displayed to the user.
o module - Name of the module that generated the result/error.
o file - The file object of the scanned file.
o valid - Set to True if the result if value, False if invalid (usually unused for errors).
o display - Set to True to display the result to the user, False to hide it (usually unused for errors).
o extract - Set to True to flag this result for extraction (not used for errors).
o plot - Set to Flase to exclude this result from entropy plots (not used for errors).
binwalk.core.module.Error has the additional guarunteed attribute:
binwalk.core.module.Error has the additional guarunteed attribute:
o exception - Contains the Python execption object if the encountered error was an exception
o exception - Contains the Python execption object if the encountered error was an exception
Thus, scan results and errors can be programatically accessed rather easily:
Thus, scan results and errors can be programatically accessed rather easily:
for module in binwalk.Modules().execute('firmware1.bin', 'firmware2.bin', signature=True):
print ("%s Results:" % module.name)
for result in module.results:
print ("\t%s 0x%.8X %s" % (result.file.name, result.offset, result.description))
for module in binwalk.Modules().execute('firmware1.bin', 'firmware2.bin', signature=True):
print ("%s Results:" % module.name)
for result in module.results:
print ("\t%s 0x%.8X %s" % (result.file.name, result.offset, result.description))
MODULE EXCEPTIONS
The only expected exception that should be raised by binwalk.Modules is that of binwalk.ModuleException. This exception
is thrown only if a required module encountered a fatal error (e.g., one of the specified target files could not be opened):
The only expected exception that should be raised by binwalk.Modules is that of binwalk.ModuleException. This exception
is thrown only if a required module encountered a fatal error (e.g., one of the specified target files could not be opened):
try:
binwalk.Modules().execute()
except binwalk.ModuleException as e:
print ("Critical failure:", e)
try:
binwalk.Modules().execute()
except binwalk.ModuleException as e:
print ("Critical failure:", e)
BEFORE YOU START
----------------------------------
-----------------------------------------
Binwalk supports Python 2.7 - 3.x. Although binwalk is slightly faster in Python 3, the Python OpenGL bindings
are still experimental for Python 3, so Python 2.7 is recommended.
......@@ -12,71 +12,67 @@ You will also need to have a C compiler installed to build the supporting C libr
INSTALLATION
----------------------------------
-----------------------------------------
Installation follows the typical configure/make process:
$ ./configure
$ make deps # Skip this if you don't want extraction utilities and dependencies auto-installed
$ make
$ sudo make install
$ ./configure
$ make deps # Skip this if you don't want extraction utilities and dependencies auto-installed
$ make
$ sudo make install
You can also install optional dependencies and extraction utilities (works on most Debian/RedHat based systems):
$ make deps
Note that 'make deps' should work on most Debian/RedHat based systems. If this does not work for your system,
Note that 'make deps' should work on most Debian and RedHat based systems. If this does not work for your system,
see below for manaully installing dependencies / extraction utilities.
INSTALLING DEPENDENCIES
----------------------------------
MANUALLY INSTALLING DEPENDENCIES
-----------------------------------------
Most binwalk features will work out of the box without any additional dependencies. However, to take full advantage
of binwalk's capabilities, you need to install:
o libmagic
o libfuzzy
o pyqtgraph
o libmagic
o libfuzzy
o pyqtgraph
Most Unix systems will already have libmagic installed (it is used by the Unix 'file' utility). If not, you can install
it via your system's package manager:
$ sudo apt-get install libmagic1
$ sudo apt-get install libmagic1
Or install it from source:
$ wget ftp://ftp.astron.com/pub/file/file-5.14.tar.gz
$ tar -zxvf file-5.14.tar.gz
$ cd file-5.14 && ./configure && make && sudo make install
$ wget ftp://ftp.astron.com/pub/file/file-5.14.tar.gz
$ tar -zxvf file-5.14.tar.gz
$ cd file-5.14 && ./configure && make && sudo make install
For most Debian based distros, libfuzzy can be installed via apt-get:
$ sudo apt-get install libfuzzy2
$ sudo apt-get install libfuzzy2
To install libfuzzy from source, download the gzipped tarball from http://ssdeep.sourceforge.net/#download, then run:
$ tar -zxvf ssdeep-2.10.tar.gz
$ cd ssdeep-2.10 && ./configure && make && sudo make install
$ tar -zxvf ssdeep-2.10.tar.gz
$ cd ssdeep-2.10 && ./configure && make && sudo make install
You'll need to install the dependencies for pyqtgraph (you may also need to install OpenGL drivers for your video card):
$ sudo apt-get install libqt4-opengl python-opengl python-qt4 python-qt4-gl python-numpy python-scipy
$ sudo apt-get install libqt4-opengl python-opengl python-qt4 python-qt4-gl python-numpy python-scipy
Most distros don't have pyqtgraph in their default repositories, so it's best to install it from source:
$ wget http://www.pyqtgraph.org/downloads/pyqtgraph-0.9.8.tar.gz
$ tar -zxvf pyqtgraph-0.9.8.tar.gz
$ cd pyqtgraph-0.9.8 && sudo ./setup.py install
$ wget http://www.pyqtgraph.org/downloads/pyqtgraph-0.9.8.tar.gz
$ tar -zxvf pyqtgraph-0.9.8.tar.gz
$ cd pyqtgraph-0.9.8 && sudo ./setup.py install
INSTALLING EXTRACTION UTILITIES
----------------------------------
MANUALLY INSTALLING EXTRACTION UTILITIES
-------------------------------------------
Binwalk can automatically invoke external extraction utilities to extract various types of files that it
may find during a scan. These utilities are optional, but recommended if you plan on using binwalk's
......@@ -85,22 +81,22 @@ extraction features.
Most utilities can be installed from your distro's repositories (package names may vary slightly based
on your particular distro):
$ sudo apt-get install git mtd-utils zlib1g-dev liblzma-dev ncompress gzip bzip2 tar arj p7zip p7zip-full openjdk-6-jdk
$ sudo apt-get install git mtd-utils zlib1g-dev liblzma-dev ncompress gzip bzip2 tar arj p7zip p7zip-full openjdk-6-jdk
However, the Firmware-Mod-Kit, which contains many file system extractors, needs to be built from source
and placed in the /opt/firmware-mod-kit directory:
$ sudo mkdir -p /opt/firmware-mod-kit
$ sudo chmod -R a+rwx /opt/firmware-mod-kit
$ git clone https://code.google.com/p/firmware-mod-kit /opt/firmware-mod-kit/
$ cd /opt/firmware-mod-kit/src && ./configure && make
$ sudo mkdir -p /opt/firmware-mod-kit
$ sudo chmod -R a+rwx /opt/firmware-mod-kit
$ git clone https://code.google.com/p/firmware-mod-kit /opt/firmware-mod-kit/
$ cd /opt/firmware-mod-kit/src && ./configure && make
UNINSTALL
----------------------------------
-----------------------------------------
The following command will remove binwalk from your system (note that this does *not* include dependencies installed via 'make deps'):
$ sudo make uninstall
$ sudo make uninstall
......@@ -21,10 +21,10 @@ class FunctionHandler(object):
Class for abstracting function calls via ctypes and handling Python 2/3 compatibility issues.
'''
PY2CTYPES = {
bytes : ctypes.c_char_p,
str : ctypes.c_char_p,
int : ctypes.c_int,
float : ctypes.c_float,
bytes : ctypes.c_char_p,
str : ctypes.c_char_p,
int : ctypes.c_int,
float : ctypes.c_float,
bool : ctypes.c_int,
None : ctypes.c_int,
}
......@@ -32,10 +32,10 @@ class FunctionHandler(object):
RETVAL_CONVERTERS = {
None : int,
int : int,
float : float,
float : float,
bool : bool,
str : bytes2str,
bytes : str2bytes,
bytes : str2bytes,
}
def __init__(self, library, function):
......
......@@ -8,6 +8,7 @@ class Magic(object):
LIBMAGIC_FUNCTIONS = [
binwalk.core.C.Function(name="magic_open", type=int),
binwalk.core.C.Function(name="magic_close", type=None),
binwalk.core.C.Function(name="magic_load", type=int),
binwalk.core.C.Function(name="magic_buffer", type=str),
]
......@@ -31,6 +32,12 @@ class Magic(object):
self.magic_cookie = self.libmagic.magic_open(self.MAGIC_FLAGS | flags)
self.libmagic.magic_load(self.magic_cookie, self.magic_file)
def close(self):
if self.magic_cookie:
self.libmagic.magic_close(self.magic_cookie)
self.magic_cookie = None
def buffer(self, data):
return self.libmagic.magic_buffer(self.magic_cookie, str2bytes(data), len(data))
if self.magic_cookie:
return self.libmagic.magic_buffer(self.magic_cookie, str2bytes(data), len(data))
......@@ -84,8 +84,6 @@ class Entropy(Module):
self.block_size = self.DEFAULT_BLOCK_SIZE
def run(self):
from pyqtgraph.Qt import QtGui
for fp in iter(self.next_file, None):
if self.display_results:
......@@ -97,6 +95,7 @@ class Entropy(Module):
self.footer()
if self.do_plot and not self.save_plot:
from pyqtgraph.Qt import QtGui
QtGui.QApplication.instance().exec_()
def calculate_file_entropy(self, fp):
......
......@@ -9,12 +9,6 @@ from binwalk.core.module import Module, Option, Kwarg
class HexDiff(Module):
ALL_SAME = 0
ALL_DIFF = 1
SOME_DIFF = 2
DEFAULT_DIFF_SIZE = 0x100
DEFAULT_BLOCK_SIZE = 16
COLORS = {
'red' : '31',
......@@ -23,6 +17,7 @@ class HexDiff(Module):
}
SEPERATORS = ['\\', '/']
DEFAULT_BLOCK_SIZE = 16
TITLE = "Binary Diffing"
......
......@@ -11,8 +11,8 @@ class Signature(Module):
CLI = [
Option(short='B',
long='signature',
kwargs={'enabled' : True},
description='Scan target file(s) for file signatures'),
kwargs={'enabled' : True, 'force_default_scan' : True},
description='Scan target file(s) for common file signatures'),
Option(short='R',
long='raw-bytes',
kwargs={'raw_bytes' : None},
......@@ -44,6 +44,7 @@ class Signature(Module):
Kwarg(name='search_for_opcodes', default=False),
Kwarg(name='cast_data_types', default=False),
Kwarg(name='dumb_scan', default=False),
Kwarg(name='force_default_scan', default=False),
Kwarg(name='magic_files', default=[]),
]
......@@ -56,26 +57,27 @@ class Signature(Module):
# If a raw byte sequence was specified, build a magic file from that instead of using the default magic files
if self.raw_bytes is not None:
self.magic_files = [self.parser.file_from_string(self.raw_bytes)]
self.magic_files.append(self.parser.file_from_string(self.raw_bytes))
# Use the system default magic file if no other was specified
# Append the user's magic file first so that those signatures take precedence
if not self.magic_files:
if self.search_for_opcodes:
self.magic_files = [
if self.search_for_opcodes:
self.magic_files += [
self.config.settings.paths['user'][self.config.settings.BINARCH_MAGIC_FILE],
self.config.settings.paths['system'][self.config.settings.BINARCH_MAGIC_FILE],
]
elif self.cast_data_types:
self.magic_files = [
]
if self.cast_data_types:
self.magic_files += [
self.config.settings.paths['user'][self.config.settings.BINCAST_MAGIC_FILE],
self.config.settings.paths['system'][self.config.settings.BINCAST_MAGIC_FILE],
]
else:
self.magic_files = [
]
# Use the system default magic file if no other was specified, or if -B was explicitly specified
if not self.magic_files or self.force_default_scan:
self.magic_files += [
self.config.settings.paths['user'][self.config.settings.BINWALK_MAGIC_FILE],
self.config.settings.paths['system'][self.config.settings.BINWALK_MAGIC_FILE],
]
]
# Parse the magic file(s) and initialize libmagic
self.mfile = self.parser.parse(self.magic_files)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment