Firmware Mod Kit - Modify the Files in Firmware Binaries!

Visit the NEW bitsum.com

Firmware Modification Kit
Giving YOU the power to customize your firmware
documentation version 0.73 beta
all scripts, untrx, wrt_vx_imgtool, crcalc, binwalk and docs (c) 2006-2011
Craig Heffner <heffnercj@gmail.com>, Jeremy Collake <jeremy.collake@gmail.com>, and other contributors (see project page)
other portions (c) their respective holders

Please visit The Firmware Mod Kit on Google Code. These docs may no longer be accurate.

Contents

I. Description
II. Why is this useful?
III. Where to get the kit
IV. Requirements
V. Compatibility
VI. Instructions
    A. extracting firmware
    B. modifying firmware
        1. installing packages
        2. removing packages
    C. re-building firmware
VII. Caveats
    A. binary compatibility issues
    B. maximum firmware size
    C. solutions to file system incompatibilities
VIII. Acknowledgements and more
IX. Links
X. Revision history

I. Description

This kit gives the user the ability to make changes to a firmware image without recompiling the firmware sources. It works by extracting the firmware into its component parts, then extracting the file system image (assuming its in one the supported formats). The user can then make modifications to the extracted file system, and rebuild the firmware image.

Although this kit completely automates the process, it is also useful for those who need a handy collection of utilities to manipulate firmware images. The following utilities are included in this kit, along with scripts to automate their use:

ASUSTRX	An extended version of ASUSTRX that can build both 'normal' TRX files and, optionally, those with an ASUS *addver* style header appended. It can also, uniquely, force segment offsets in the TRX (with -b switch) for compatibility with Marvell ASUS devices like the WL-530g. This tool replaces both 'normal' trx tool and addver. Current versions included are: 0.90 beta.
ADDPATTERN	Utility to pre-pend Linksys style HDR0 header to a TRX.
ADDVER	ASUS utility to append a header to a TRX image that contains version information. ASUSTRX includes this capability. Current version: unversioned.
BINWALK	Firmware analysis tool, used for searching firmware images for known file types. Used by the extract-ng.sh script to identify valid file systems inside firmware images. Current version included is 0.3.9. Tool home page: http://binwalk.googlecode.com.
CRAMFSCK	CRAMFS file system image checker and extractor. Current versions included are: 2.4x.
CRCALC	Re-calculates and patches header checksums for TRX and uImage firmware headers.
MKSQUASHFS	Builds a squashfs file system image. Current versions included are: 2.1-r2, 3.0.
MKCRAMFS	Builds a cramfs file system image. Coming in next version. Current versions included are: 2.4x.
MOTOROLA_BIN	A utility that prepends 8 byte headers to TRX images for Motorola devices WR850G, WA840G, WE800G. Current version: unversioned.
UNCRAMFS	Alternate tool to extracts a cramfs file system image. Use cramfsck instead whenever possible as it seems to be more reliable. Current versions included are: 0.7 (for cramfs v2.x).
UNCRAMFS-LZMA	Extracts an lzma cramfs file system image, specifically those included in OpenRG based firmware. Current version included is 0.7rg.
UNSQUASHFS	Extracts a zlib squashfs file system image. Current versions included are 1.0 for 3.0 images and 1.0 for 2.x images (my own blend).
UNSQUASHFS-LZMA	Extracts an lzma squashfs file system image. Current versions included are 1.0 for 3.0 images and 1.0 for 2.x images (my own blend). Note: Not all squashfs-lzma patches are compatible with one another. I'm working on adding support for all common squashfs-lzma variations.
UNTRX	Splits TRX style firmwares into their component parts. Also supports pre-pended addpattern HDR0 style headers. This was developed exclusively for this kit. Current versions included are: 0.45.
WRT_VX_IMGTOOL	Utility to generate VxWorks compatible firmware images for the WRT54G(S) v5 series.

Tools/changes unique to this kit

This kit doesn't just collect existing tools, it also offers new ones and modifications to existing ones. Custom code currently is:

ASUSTRX - Modified to allow '-b' switch to force segment offsets (needed for some devices, like the WL-530g).
CRCALC - A new tool to re-calculate and patch the checksum fields for common firmware headers. Currently TRX and uImage headers are supported.
UNTRX - A new tool to extract the contents of a TRX image into its component parts.
UNSQUASHFS - Custom blend that supports extraction of 2.x squashfs images. This tool wasn't officially made available until squashfs 3.0, so up until now it wasn't available for 2.x images.
WRT-VX-IMGTOOL - A new tool to view, extract, build, and fix VxWorks compatible firmware images for the WRT54G(S) v5 series.
SPLITTER3 - Custom file splitter for certain types of linux firmware images.

II. Why is this useful?

A list of some of the many reasons this might be desired are:

add initialization scripts
install new packages
extend/change HTTP managed console
remove un-needed packages
mix-and-match packages from various flavors of the firmware image.

III. Where to get the kit

The best way to download and use this kit is to check out the sources from the repository. The automated scripts all automatically build the executables, and the sources. It is easy as downloading something and using it that way. Also, if you are unable to use a simple SVN checkout, then it is likely that this kit is a danger to you because it is for advanced users.

To anonymously check out the firmware-mod-kit:

mkdir firmware_mod_kit
cd firmware_mod_kit
svn checkout http://firmware-mod-kit.googlecode.com/svn/trunk/ firmware-mod-kit-read-only

Kit downloads and links:

Project hosting and repository (new): http://code.google.com/p/firmware-mod-kit/
Guest SVN Checkout: svn checkout http://firmware-mod-kit.googlecode.com/svn/trunk/ firmware-mod-kit-read-only
User forums can be found here.

Packages:

All of the many OpenWrt packages should work with OpenWrt images and many other firmware images.

An old (2009) specific collection of DD-WRT packages for use with the kit: http://www.bitsum.com/files/ddwrt_packages.tar.gz .

IV. Requirements

I've worked to make sure the requirements are low enough for anyone to use this kit. However, to reduce distribution costs and better represent the multi-platform capabilities of this kit, the binary tools in the kit do automatically build themselves the first time you run one of the scripts. Any machine set up to build anything C or C++ on the machine will probably be ready. The few requirements are all basic items that everyone has easy access to:

A compatible platform. The following are specifically supported (or specifically unsupported):
- LINUX (recommended)
- OS X **warning: I don't test every build under OS X. It's up to you to email me if you have problems under it. Current builds may have issues.**
- CYGWIN **alpha stage testing (!! built firmwares may not boot !!)**
GNU C (gcc)
GNU C++ (g++)
GNU make
Standard C runtime library development
Standard C++ runtime library development
Zlib library development (for Ubuntu it is zlib1g-dev)
TAR and GZIP (optional, used by ipkg_install.sh)

OS/X is not supported at this time. The kit is probably not far from compatibility, but easier to just install a linux VM.

V. Compatibility

Any TRX style firmware image format should that uses squashfs-lzma as a root file system work fine. Firmwares that use regular zlib squashfs or unsupported file systems will work if the steps that extract the file system and rebuild the file system are tweaked.

The kit can build firmware images for the following devices, amongst many others:

WRT54G v1 - v6
WRT54GS v1 - v6
WRTSL54G v?
ASUS WL-330g
ASUS WL-500g/p
ASUS WL-520g
ASUS WL-530g
Belkin 7230-4 (some versions)
Belkin 7231-4 (?)
Buffalo WHR-G54S
Buffalo WHR-HP-G54
Siemens SE505
Trendnet TEW-632BRP
D-Link DIR-615 (maybe)
... lots of others* ...

The kit is known (or thought) to be compatible with the following firmwares:

DD-WRT v23	tested - versions v23 SP1 and later are compatible (soon older versions too). WARNING: Builds of DD-WRT later than 08/04/2006 (v23 SP2) have some protections against changing the web UI.
DD-WRT v24	tested
OpenWrt White Russian	tested
OpenWrt Kamikaze	untested (should work) - not really necessary, based on OpenWrt has its Image Builder.
FreeWrt	untested - should work ok
HyperWrt	untested
Ewrt	untested
Sveasoft Alchemy	untested
Sveasoft Talisman	untested
Linksys / other vendor	not supported by scripts yet - haven't added cramfs handling
ASUS WL-330G	untested - should work ok
ASUS WL-520G	untested - should work ok
ASUS WL-530G	supported
ASUS WL-550G	untested - should work ok
Trendnet TEW-632BRP	tested
DLink DIR-615	untested
many others*	untested

* Even if the kit doesn't automatically support the firmware or device you want, you can manually use the tools inside to manipulate many firmware images, or request support be added.

VI. Instructions

I've scripts that simply and abstract the firmware modification process. Basically, it's a one-step process to extract the firmware, and a one-step process to rebuild the firmware. There are also scripts to install or remove individual packages or entire directories of packages.

The procedure of calls is:

extract_firmware.sh ....
(optional) ipkg_install[_all]).sh ...
(optional) ipkg_remove[_all].sh ...
build_firmware.sh

A. Extracting the firmware (extract_firmware.sh, extract-ng.sh)

Simply run extract_firmware.sh with the following parameters. You must run this tool from inside the directory it exists in.

$ ./extract_firmware.sh firmware.bin working_directory/

- firmware.bin is the firmware image you want to extract, i.e. DD-WRT.v23_MICRO_GENERIC.BIN.
- working_directory is the working directory you want to use. This is where the intermediate files and the extracted file system will be stored. Note that files in the working directory may be deleted!

It doesn't matter which firmware image you supply, i.e. for a WRT54G or an ASUS WL-500G. These images are usually all the same and differ only in the header format. The rebuilding process will create images for the various models.

An extraction log is created as extract.log.

If extract_firmware.sh fails, the newer extract-ng.sh can be used. Although newer and less thoroughly tested than extract_firmware.sh, it is more flexible and can often extract file systems that extract_firmware.sh cannot:

$ ./extract-ng.sh firmware.bin working_directory/

B. Making changes to the firmware's file system

Modifying the firmware is simple. The file system is stored in the working directory you supplied to extract_firmware.sh. Here are the subfolders of this directory:

    rootfs/
        This is where the file system is. Make modifications here.
    image_parts/
        This is where intermediate files are stored. If you need to replace the kernel (not at all recommended), you can do so by replacing the appropriate file here (usually segment2).
    installed_packages/
        If you use the ipkg_install.sh tool, this is where packages you've installed after firmware extraction will reside.

One can manually change the files in rootfs and/or use the automated package scripts.

Installing packages (.IPK)

In compliment to manually making changes, one can use pre-built IPKG format package files with the ipkg_install/remove scripts. Although not all will work and a repository for packages verified to work has not yet been produced, many packages work just fine.

Packages are pre-built collections of files pertaining to a set of software. OpenWrt and DD-WRT both use packages, and most are cross-compatible. These packages are stored in a tar/gzip archive of a pre-defined structure that includes some control files. You can extract and copy the files manually, or use the ipkg_install.sh script included with this kit. For a list of some of the available packages (not all may work), see http://downloads.openwrt.org/whiterussian/packages/ and http://ftp.selfnet.de/pub/openwrt/backports/rc5/ . We are currently in the process of compiling a set of packages known to be good for DD-WRT ... to help in this matter please visit here.

For information on how to create IPKs, see the IPK Creation Kit.

Example use:

$ ./ipkg_install.sh some_package-1.2.5.ipk working_directory/

some_package-1.2.5.ipkg would be the filename of the package.
working_directory is the same working directory you supplied to the extract_firmware.sh script.

Notes:

A script, ipkg_install_all.sh is also available to install an entire folder of packages.

The collection of installed packages will be stored in working_directory/installed_packages .

Removing packages (.IPK)

To remove packages there is the ipkg_remove.sh script.

Example use:

$ ./ipkg_remove.sh some_package-1.2.5.ipk working_directory/

some_package-1.2.5.ipkg would be the filename of the package.
working_directory is the same working directory you supplied to the extract_firmware.sh script.

Notes:

A script, ipkg_remove_all.sh is also available to install an entire folder of packages.

The collection of installed packages stored working_directory/installed_packages will be updated to reflect the change if successful.

C. Re-building the firmware (build_firmware.sh, build-ng.sh)

Rebuilding the firmware is as easy as extracting it. Use the build_firmware.sh script to automate the process. You must run this tool from inside the directory it exists in.

Example use:

$ ./build_firmware.sh output_directory/ working_directory/

- output_directory is the path to which the created firmware images should be stored. If images already exist here, they will be over-written. Firmware images for various models will be emitted (these images are all the same but have different header patterns so they are accepted by the target models).
- working_directory is the working directory supplied to extract_firmware.sh.

A building log is created as build.log.

If extract-ng.sh was used to extract the firmware image, build-ng.sh must be used to rebuild it:

$ ./build-ng.sh working_directory/

VIII. Caveats

There are a few things to watch out for when modifying firmware images. If you make reasonable changes these problems will probably never affect you. Be warned, this section may be technical in parts.

A. Binary compatibility issues

If installing a binary package:
- It is possible the uclibc or glibc library was pruned (unused objects removed). In the unlikely event the binary package requires some of these previously unused (and now missing) functions, you may need to install to the firmware file system a new uclibc or glibc library. If you are installing another pruned uclibc or glibc library, then it is remotely possible some pre-existing package requires objects not found in this new library. In such a very rare case you can re-link the library or find a full copy of the library to install.
- Different firmwares may use different kernel versions, or have different kernel modules installed. If you get too wild with your mix-and-matching (i.e. use older versions with newer, or mixtures of different firmwares all-together), problems can result. To install a new kernel (not at all recommended), replace the segment2 (or other) file found found in the intermediate directory with an appropriate one (can be from another firmware as long as lzma compression is used).
- Make sure any dependencies are also installed to the file system, along with initialization scripts.
If making web UI changes:
- WARNING: DD-WRT builds dated later than 08/04/2006 have some protections against modifying the web UI.

B. Maximum firmware size

Currently, the maximum firmware image size supported by this kit is 5,898,240 (0x5A0000) bytes, but can be adjusted through the "-m [size]" parameter of the asustrx tool.

C. Solutions to file system incompatibilities

! This section is very much a work in progress !

The squashfs file system is case sensitive (i.e. teSt.bin ! = test.bin) and adheres to EXT2 naming rules. To support proper extraction of the file system, the working directory should be on a case sensitive file system and support symbolic links. The scripts will warn you if you attempt to use as working directory on a case INsensitive file system. You'll see a number of errors if you extract to a file system that doesn't support symbolic links.

Note: Case insensitivity doesn't always cause problems, especially if the file system preserves case ok. It depends on the firmware and version. Most of the time for DD-WRT as long as a case preserving file system is used as a working directory, all should work fine.

These solutions apply to those using CYGWIN, or some other environment that may not have a case sensitive file system by default. Linux users typically do not have this problem since EXT2 and EXT3 file systems are case sensitive.

WARNING: THESE ARE IN EARLY STAGES OF TESTING AND MAY NOT WORK.

	Applies to	Description
SOLUTION 1	CYGWIN, OTHER	It's reported this does not work under CYGWIN due to the CYGWIN utilities all being case INsensitive (thanks pagedude). Use a network share to a case sensitive file system. This is, by far, the easiest solution. Simply use an appropriate network share for the working directory.
SOLUTION 2	CYGWIN	For NT partitions it's reported you can enable case sensitivity through the following registry key, but I do not know if this works or not. \HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive [REG_DWORD] = 0
SOLUTION 3	OS X (not usually needed, at least for DD-WRT)	One can either Turn on HSFX for the boot disk, or (more easily) create a temporary HSFX case sensitive file system to use as a working directory. Example of how to create a temporary HSFX case sensitive file system: to create: $ hdiutil create -size 20m -fs HFSX -volname db90h -nouuid test-image to mount: $ hdiutil attach test-image.dmg to un-mount: $ hdiutil detach /Volumes/db90h (thanks solarflare for this information)
SOLUTION 4	CYGWIN	Mount an ext2 files system. Eko, of DD-WRT forums, contributes this tip. Make a partition on a hard drive or USB thumb drive, then install "Ext2 Installable File System For Windows". 10MB should generally be plenty of working directory space, but 20MB is suggested for more optimal file system performance.

IV. Acknowledgments and more

The authors and maintainers of this kit are Jeremy Collake (db90h) and Craig Heffner. Special thanks to Solarflare, TexHex <floh@fbesser.de>, and others.

If YOU would like to contribute to this kit, send me an email at jeremy.collake@gmail.com or visit the current project hosting at http://code.google.com/p/firmware-mod-kit/ . Developers are welcome.

X. Links

IPK Creation Kit (alpha testing)

X. Revision history

0.73

New pre-req check and more modern build system (by Craig)

0.72

Fix for recent updates to linux build tools that mandated different command line ordering (by Craig)

0.64-0.71

See Google Code
NG edition created by Craig Heffner (total new generation of kit)

0.63

Added support for a new raw linux type of firmware (completely untested on the rebuild side flash, so beware)
Added new splitter3.cc to split up this linux raw firmware type into its segments
Other misc tweaks

0.60

Fixed issue with non-standard du switches, i.e. under OS X.
Fixed untrx inability to recognize squashfs signatures on big endian systems (patch from Jaifu Gao)

0.58

Removed multi-threaded brute testing of LZMA compression params from the 'damn small' variant DD-WRT v24 uses. This may fix some reported problems of failed thread creation.

0.56

Add support DD-WRT v24 firmware images
New unsquashfs variant written for DD-WRT v24
Cleanup script output indentation, it was quite sloppy
Apply some fixes and other tweaks

0.54

Add support for Trendnet TEW-632BRP images
Add support for OpenWrt White Russian images
Don't force root user, just warn
Other maintenance and tweaks

0.51

Increased maximum supported size of TRX images (now can handle > 4MB images).

0.50

Removed unnecessary -lz parameter (zlib) from gcc.

0.49

unsquashfs: created first ever (afaik) blend of unsquashfs for squashfs 2.x images.
mksquashfs: added 2-1.r2. Script support remains todo.
other misc. changes.

0.48

scripts: now use cramfsck -x to extract cramfs images instead of uncramfs. uncramfs proved to fail to extract some images.
dirs: renamed cramfs-1.1 to cramfs-2.x to reflect the accurate version number.

0.471

motorola_bin: added source code.

0.47

wrt_vx_imgtool: added this utility, which can build, extract, view, and fix firmware images for WRT54G(s) v5/v6 series running VxWorks.
motorola_bin: added this binary only (linux) utility. Build script doesn't invoke it yet, but it's easy to use for those who need it.
untrx: minor memory leak fixed.
docs: added some device compatibility notes and other edits.

0.46:

scripts: added support for cramfs (testing)
scripts: added support for WL-520/530/550G images (only 530g tested).
sources: added cramfs 1.1 tools (mkcramfs, cramfsck)
images: a symlink is now used between the TRX and generic image. Previously, the intermediate TRX image was just renamed to the generic image.
scripts: no longer show untrx output
docs: minor additions

0.45:

untrx: added support for attempted detection of file system and version and emission of appropriately named segment.
untrx: can now handle trx's that have less than 3 segments.
untrx: complete overhauled (needed it). many changes.
sources: moved squashfs 3.0 stuff to a dedicated directory.
packages: added dd-wrt package for Samba. thanks TexHex <floh@fbesser.de>.
asustrx: updated to add -b switch for padding segments (used by ASUS).
asustrx: other misc. enhancements.
scripts: added check_for_upgrade.sh which automatically checks for a newer version of the kit. It's launched when you use extract or build firmware.
scripts: many other fixes and enhancements.

0.43:

scripts: ipkg_remove.sh - rewritten core templated file removal. It actually works now.
scripts: moved build stuff to shared.inc, as appropriate.
packages: added dd-wrt package KAID, mainly for removal of this gargantuan (nearly 1MB uncompressed) package. Note: you can easily create your own packages for easier removal of things you don't want by using the IPK template.

0.42:

ipk_template: added new ipk_template directory and make_ipk.sh script to create IPKG format packages (.IPK).
packages: added my newly created ssmtp package with nvram->conf file startup script. Installs completely into the DD-WRT file system.
packages: added ddwrt_packages subdirectory which will include pre-built packages custom tailored or known to work
scripts: ipkg_install.sh numerous improvements and fixes.
scripts: ipkg_remove.sh numerous improvements and fixes.
scripts: ipkg_install.sh has new untested/alpha stage capabilities to handle IPKs with remote sources.
docs: several updates

0.40 beta (md5: 786472cfa03fb1cd70e1b025d091168b)

scripts: ipkg_remove.sh - new script for basic IPK package removal
scripts: ipkg_install.sh possible OS X fixes
scripts: ipkg_install.sh changed technique of recording installed packages
scripts: ipkg_install.sh outputs control file so user can see dependencies and more
scripts: several improvements
docs: hopefully more readable format in places
docs: many mods and extensions

0.33 beta

structure: moved tool source codes to src subfolder
scripts: added ipkg_install_all.sh to install an entire folder of packages.
scripts: consolidated shared code into an 'include' script
scripts: added forgotten parameter count check to ipkg_install.sh
scripts other minor changes

0.31 beta

new tests indicate OS X with native HFS+ file system results in firmware images that work just fine! This build should be fully compatible with OS X without even necessitating HSFX use for most firmwares.
mksquashfs: fix for big endian machines (specifically ppc / OS X) resulting from change to support dd-wrt 08/10/06+ builds.

0.30 beta

scripts: created symlinks that omit the .sh extension
scripts: expanded build success check
scripts: fixed failure to exit if tools build failed
scripts: other minor cosmetic improvements
untrx: fixed a file handle leak in v0.28 beta
docs: added example procedure to install SSMTP into DD-WRT image
docs: added table of contents
docs: misc. additions and revisions

0.29 beta

scripts: added ipkg_install.sh to install .IPK format packages.

0.28 beta

the v0.26 fix for changed signature of 08/10/06 builds was incomplete (test was flawed), resulting in builds that wouldn't run. fixed properly now.
mksquashfs: added -magic [file] parameter to embed whichever is used in the original firmware
untrx: added extraction of squashfs signature/magic for passing to mksquashfs
docs: added more stuff

0.27 beta

scripts: changed warning about file system.

0.26 beta

Fixed compatibility with DD-WRT builds 08/10/06 and higher. Brainslayer played a joke on me.

0.25 beta

OS X now works. When using case insensitive file system though built firmwares might have errata.
trx: removed this tool due to odd command line parsing bug on (big endian?) OS X systems. asustrx now used in its place.
scripts: file system case insensitive now warning, can be continued in default settings.
untrx: fixed some endian problems.
untrx: fixed a problem with U2ND headers (previous version only worked on trx/generic firmwares).
docs: minor additions.

0.23 beta

package: removed linux executables, download now half size. Surely everyone can install basic build tools..
scripts: added working directory file system test for case insensitivity and write access
scripts: added option to continue on after warning when OS X or CYGWIN present
scripts: removed pre-fixed script name, much cleaner looking output now
scripts: added variable to choose whether or not to abort if file system found incompatible
scripts: added cleanup.sh script to clean tools and logs.
untrx: renamed extfirmware to untrx, much more intuitive name

0.22 beta

scripts: added support for OS X
scripts: added support for CYGWIN
scripts: minor cosmetic work
extfirmware: added endian neutrality
trx: fixed OS X support
asustrx: fixed OS X support

0.20 beta

scripts: added check for Linux and OS X and appropriate compatibility messages
extfirmware: added GNU license header
extfirmware: removed some unnecessary header includes
extfirmware: switched to u_int32_t for wider compatibility
extfirmware: other minor improvements
trx: increased maximum firmware size to 0x5A0000 (OpenWrt) from 0x3A0000 (DD-WRT).
trx: switched to u_int32_t for wider compatibility
makefile: uses gcc instead of cc now
docs: several updates

v0.15 beta

Reworked scripts so they have cleaner, clearer output and use a log file (build.log and extract.log)
Added more sanity checks in the scripts
Documentation additions
Misc. polishing

v0.13 alpha: Documentation updates, re-included pre-built executables
v0.10 alpha: First decent public release.

This document (c)2006-2011 Jeremy Collake and Craig Heffner.
All Rights reserved. This document may be freely republished in its unaltered and whole form only. Alterations or partial publishing requires approval of creator(s).

This kit is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA