Recovering deleted files with SleuthKit

      No Comments on Recovering deleted files with SleuthKit

SleuthKit is probably one of the most comprehensive collections of tools for forensic filesystem analysis. One of the most basic use-cases is the recovery of files that have been deleted. However, SleuthKit can do much, much more. Have a look at the case studies wiki page for an impression.

Let’s assume, there is a FAT volume on our disk (maybe a USB stick or a memory card) and we want to recover all deleted file. The safest way is probably to duplicate the entire volume first and perform an offline analysis. Again, there are quite a few tools for creating (forensic) images, the simpliest probably being dd.

To dump all partitions of your disk use

$ dd if=/dev/sdg of=/tmp/disk.img bs=512

Of course, you could also dump just one partition (e.g. /dev/sdg2).

To get the partition table layout, you can the use mmls on the image file

$ mmls /tmp/disk.img
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

     Slot    Start        End          Length       Description
00:  Meta    0000000000   0000000000   0000000001   Primary Table (#0)
01:  -----   0000000000   0000000134   0000000135   Unallocated
02:  00:00   0000000135   0003858623   0003858489   DOS FAT16 (0x06)
03:  -----   0003858624   0003862527   0000003904   Unallocated

fsstat is used to get information about the filesystem itself. In this case, the target partition starts with an offset of 135, so the imgoffset flag -o is mandatory (if you just dump a single partition, there is of course no offset and -o is not needed).

$ fsstat -o 135 /tmp/disk.img
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT16

OEM Name: MSDOS5.0
Volume ID: 0x34333064
Volume Label (Boot Sector): NO NAME    
Volume Label (Root Directory):
File System Type Label: FAT16   

Sectors before file system: 135
[...]

You can now do all sorts of things with your image file, e.g. recursively list all file and directories (including deleted ones);

$ fls -o 135 -r /tmp/disk.img

To recover a deleted file by inode number, you can use the command line tool icat

icat -o 135 -r /tmp/disk.img 54 > /tmp/DeletedPicture.jpg

For a quick overview and simple examples on other commands have a look at http://rationallyparanoid.com/articles/sleuth-kit.html.

The SleuthKit wiki points to a great script by Dave Henkewick that runs recursively through the image file and uses fls and icat to retrive the inode numbers and restore the files:

#!/usr/bin/perl -w
use strict;

# (C) 2004 dave (at) hoax (dot) ca
# ************* THIS SCRIPT HAS NO WARRANTY! **************
#
# this script works with the output from SleuthKit's fls and icat version: 3.00
# using afflib-3.3.4
#
# dont worry if you do not have the same versions because it should work unless
# the output from the commands have changed
#
# if the script does not work, please email me the debug output and
# the output from manually running fls and icat, thanks!
#
# set the recovery directory 
my $fullpath="/tmp/recover/";

# set the absolute path of fls binary
my $FLS="/usr/bin/fls";

# set the fls options
my @FLS_OPT=("-o","135","-f","fat","-pr","-m $fullpath","-s 0");

# set the path of the device to be recovered
my $FLS_IMG="/tmp/disk.img";

# set the inode of the directory to be recovered
my $FLS_inode="2";

# set the path of the icat STDERR log
my $ICAT_LOG="/tmp/icat.log";

# set the absolute path of the icat binary
my $ICAT="/usr/bin/icat";

# set the icat options
my @ICAT_OPT=("-o","135","-f","fat");

my $ICAT_IMG="$FLS_IMG";

# here we go. hold on tight!
list($FLS_inode);


sub list($) {
	#make the recovery dir
	system("mkdir","-p","$fullpath") && die "Cannot mkdir $fullpath while processing: $_";
	#run a recursive FLS on our chosen inode and regex each line
	foreach $_ (`$FLS @FLS_OPT $FLS_IMG $_[0] 2>&1`) {
		regex($_);
		#print $_;
	}
}

sub regex($) {
	#first, regex for dirs, clean 'em up, and create 'em in recovery dir
	#
	# the following regex will work on output of the format:
	# 0|/directory/file.foo.bar (deleted)|0|r/----------|0|0|0|0|0|0
	# 0|/directory/file.foo.bar|1384462|r/rrw-r--r--|1000|1000|971556|1218136846|1218136846|1225037181|0
	# 0|/directory|1392712|d/drwxr-xr-x|1000|1000|4096|1225309096|1225309096|1226059913|0
	# 0|/directory/file.foo.bar -> /directory2/file2.foo.bar|1384462|l/lrw-r--r--|1000|1000|971556|1218136846|1218136846|1225037181|0
	#
	#
	if (/(\d\|([\S\s]+)\|(\d+)\|\S\/d([\w-]{3})([\w-]{3})([\w-]{3})(\|\d+\|\d+\|\d+\|\d+\|\d+\|\d+\|\d+))/) {
		my $fulldir = $2;
		my $uid = $4; my $gid = $5; my $oid = $6;
		$fulldir =~ s/ (\(deleted(\)|\-realloc\)))$//g;
		$fulldir =~ s/ /_/g;
		$uid =~ s/-//g; $gid =~ s/-//g; $oid =~ s/-//g;
		$uid = lc($uid); $gid = lc($gid); $oid = lc($oid);
		#print "mkdir -p $fulldir\n";
		system("mkdir","-p","$fulldir") && die "Cannot mkdir $fulldir while processing: $_";
		#print "chmod u=$uid,g=$gid,o=$oid $fulldir\n";
		system("chmod","u=$uid,g=$gid,o=$oid","$fulldir") && die "Cannot chmod u=$uid,g=$gid,o=$oid $fulldir while processing: $_";
	#second, regex for files, sockets, fifos then
	#clean and dump them in recovery dir	
	} elsif (/(\d\|([\S\s]+)\|(\d+)\|\S\/(-|s|f|r)([\w-]{3})([\w-]{3})([\w-]{3})((\|\d+\|\d+\|\d+\|\d+\|\d+\|\d+\|\d+)|(\|\d+\|\d+\|\d+\|\d+\|\d+\|\d+)))/) {
		my $inode = $3;
		my $fullfile = $2;
		$fullfile =~ s/ (\(deleted(\)|\-realloc\)))$//g;
		$fullfile =~ s/ /_/g;
		#print "$ICAT @ICAT_OPT $ICAT_IMG $inode > $fullfile\n" if ($inode != 0);
		system("$ICAT @ICAT_OPT $ICAT_IMG $inode > \"$fullfile\" 2>> $ICAT_LOG") if ($inode != 0);
		#cannot use die cuz an invalid inode will kill the script
		#&& die "Cannot icat $inode into \"$fullfile\" while processing: $_"
		
	# thrid, regex for symlink, clean, and create in recovery dir
	} elsif (/(\d\|([\S\s]+)\s\-\>\s([\S\s]+)\|(\d+)\|\S\/(l)([\w-]{3})([\w-]{3})([\w-]{3})(\|\d+\|\d+\|\d+\|\d+\|\d+\|\d+\|\d+))/) {
		#print "$1\n";
		my $fullsym_dst = $2; my $fullsym_src = $3;
		$fullsym_dst =~ s/ /_/g; $fullsym_src =~ s/ /_/g;
		#print "ln -s $fullsym_src $fullsym_dst\n";
		system("ln","-s","$fullsym_src","$fullsym_dst") && die "Cannot ln $fullsym_src $fullsym_dst while processing: $_";
	} else {
		print "Unknown directory listing. File or directory NOT recovered\nDebug:\n$_[0]\n";
	}
} #that's all folks. hope y'all had fun!

Another great tool is the Autopsy Forensic Browser, a graphical frontend to the SleuthKit commands that runs in your browser.

How to find multiple patterns with GNU findutils

Actually, searching for multiple patterns should be a trivial task. Find provides a -o operator (and many others) that lets you combine multiple expressions.

A simple Example: You want to find all files in the current directory whose filename extension are either .c or .h

$ find . \( -name "*.c" -o -name "*.h" \) -print

This is not limited to the -name test but can be combined with any other test (like -perm, -size, -type, etc.)

But Careful! You need to quote patterns that contain metacharacters (such as * in the example above). Singe quotes work as well as double quotes. The braces surrounding the expression have to be escaped, too. And watch those spaces right before and after the braces, they’re essential.

See also find manpage, GNU find documentation

Hack: How to disable recently used items list

      7 Comments on Hack: How to disable recently used items list

Quite a few applications use ~/.local/share/recently-used.xbel to keep track of a user’s most recent files. Unfortunately, not every application offers customization options to disable this list.

One possible solution, granted, a quite hacky one, is to clear recently-used.xbel and revoke a user’s permission to edit it again.

First, remove and re-create the file to clear it

$ rm -f ~/.local/share/recently-used.xbel 
$ touch ~/.local/share/recently-used.xbel 

You can then edit the permission so that the user can’t edit the file any more.

$ chmod -w ~/.local/share/recently-used.xbel 

For KDE, there is quite a similar mechanism. While there is no single file that stores the recently used items, a .desktop file is created in ~/.kde/share/apps/RecentDocuments/ for every item.

If if you revoke a user’s writing permission to that folder, KDE won’t add any item to the ‘recently used’ list.

$ chmod -w ~/.kde/share/apps/RecentDocuments/

Split flac file into tracks using a cuesheet

      1 Comment on Split flac file into tracks using a cuesheet

shnsplit (part of the “multi-purpose WAVE data processing and reporting utility” shntool package) provides a simple method to split flac files into individual tracks specified in a cuesheet.

$ shnsplit -o flac -f CUESHEET.cue -t %n.%t FLACFILE.flac

With the custom output format module, you can even transcode the tracks directly to another format, e.g. mp3, if your mobile music player doesn’t support flac.

snippets.dzone.com provides an exemplary script for this.

#!/bin/sh
set -e
ENCODE="cust ext=mp3 lame -b 192 - %f"
FORMAT="%n.%t"

FLACFILE=$1
CUEFILE=$2
echo $FLACFILE - $CUEFILE
if [ -z "$FLACFILE" ]; then
    echo "usage: flac2mp3 FLAC_FILE [CUE_FILE]"
    exit 1
elif [ -z "$CUEFILE" ]; then
    DIRECTORY=$(dirname "$FLACFILE")
    BASENAME=$(basename "$FLACFILE" ".flac")
    CUEFILE="$DIRECTORY/$BASENAME.cue"
fi

shnsplit -O always -o "$ENCODE" -f "$CUEFILE" -t "$FORMAT" "$FLACFILE"

Convert subversion repository to GIT

      No Comments on Convert subversion repository to GIT

First, you need to create a file that maps the subversion authors to GIT users (let’s say /tmp/svnusers). The syntax is pretty easy:

kmartin = Kirk Martin <marty@localhost.com>
mattaway = Marshal Attaway <marshal@localhost.com>

To get a list of all your SVN authors, run

$ svn log --xml | grep author | sort -u | perl -pe 's/.>(.?)<./$1 = /'

on you subversion working copy.

Next, you have to create a temp directory (which will be cloned later to get rid of all the SVN stuff).

$ mkdir /tmp/MyProject_tmp
$ cd /tmp/MyProject_tmp

Now, you can fetch the SVN files from you subversion server

$ git-svn init svn+ssh://user@SVNHost/MyProject/trunk/ --no-metadata
$ git config svn.authorsfile /tmp/svnusers
$ git-svn fetch

Please note, that you may need to adjust the protocol (svn+ssh, http, https, ftp, etc.), user, host, path to the project files etc.
To get rid of all the SVN remains, simply clone the newly created GIT repo

$ git clone MyProject_tmp MyProject

Resources and further reading:
http://progit.org/book/ch8-2.html
http://www.jonmaddox.com/2008/03/05/cleanly-migrate-your-subversion-repository-to-a-git-repository/

Specifying file encoding when writing dom Documents

Assumed, we got a fully parsed org.w3c.dom.Document:

Document doc;
//parse doc etc...

Just using LSSerializer‘s writeToString method without specifying any encoding will result in (rather impractical) UTF-16 encoded xml file per default

DOMImplementation impl = doc.getImplementation();
DOMImplementationLS implLS = (DOMImplementationLS) impl.getFeature("LS", "3.0");
LSSerializer lsSerializer = implLS.createLSSerializer();
lsSerializer.getDomConfig().setParameter("format-pretty-print", true);
String result = ser.writeToString(doc);

will output

<?xml version="1.0" encoding="UTF-16"?>
...

Unfortunately, specifying an encoding isn’t trivial. Here are two solutions that don’t require any third party libraries:

1. Using org.w3c.dom.ls.LSOutput

DOMImplementation impl = doc.getImplementation();
DOMImplementationLS implLS = (DOMImplementationLS) impl.getFeature("LS", "3.0");
LSSerializer lsSerializer = implLS.createLSSerializer();
lsSerializer.getDomConfig().setParameter("format-pretty-print", true);

LSOutput lsOutput = implLS.createLSOutput();
lsOutput.setEncoding("UTF-8");
Writer stringWriter = new StringWriter();
lsOutput.setCharacterStream(stringWriter);
lsSerializer.write(doc, lsOutput);

String result = stringWriter.toString();

2. Using javax.xml.transform.Transformer

Transformer transformer = TransformerFactory.newInstance().newTransformer();
transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
transformer.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "2");
DOMSource source = new DOMSource(doc);
Writer stringWriter = new StringWriter();
StreamResult streamResult = new StreamResult(stringWriter);
transformer.transform(source, streamResult);        
String result = stringWriter.toString();

Non-interactive ssh with expect

      1 Comment on Non-interactive ssh with expect

If you need a programmatic ssh login, i.e. in a shell script, the best way to make ssh non-interactive usually is a public-key authentication. This requires the public key to be stored on the host machine, which (admitted, in very rare cases) can be hard or impossible. One of those rare cases might be a chrooted environment on a webserver without write access to the ssh config file or the per-user ssh directory.

Unfortunately, most ssh clients don’t allow piping a clear text password from a file or varibale. One solution is to use the SSH_ASKPASS environment variable and a wrapper script but most recommendations suggest the use of expect.

Expect is basically a scripting language to make interactive applications non-interactive. It is often used to automate reactions to certain outputs from programs like ftp, ssh, scp and others.

A simple expect script to rsync a backup from one host to another could look like this

#!/usr/bin/expect
set timeout 600
spawn rsync -avhS user@host:/backups/ /backups/
expect {
   "password" {send "secret\n";}
}
expect {
   eof {exit 0;}
}

Note that expect eof is used here, because the alternative interact command is not compatible with cron scripts.

For a list of commands have a look at the expect man page or expect’s sourceforge page.

Fix veetle audio delay

      1 Comment on Fix veetle audio delay

When you try to output a sound stream from Veetle 0.9.17 through PulseAudio, you will most likely run into an annoying audio delay. Most of the fixes that can be found usually suggest changing the Veetle settings to use OSS for sound output.
However, there is a beta version of the latest Veetle plugin that enables you to continue using PulseAudio and get rid of the audio delay:

http://veetle.com/download.php/veetle-0.9.17plus-linux-install.sh
http://veetle.com/download.php/veetle-0.9.17plus.tgz

Please note that this version is still marked experimental. It seems to work fine on Fedora 15 though.

Resources:
http://www.lziegler.com/wordpress/?p=108

Escaping WordPress shortcodes

      No Comments on Escaping WordPress shortcodes

With WordPress 2.5 came the WordPress shortcodes, a simple set of functions to create macros to be used in WordPress posts. Quite a few plugins were developed subsequently to enable escaping of those shortcodes. This is necessary because the shortcode parser, as it tries to interprete everything between square brackets, makes it impossible to include a non-interpreted shortcode such as [gallery] into continuous text.

However, those plugins are actually obsolete. The issue has been fixed with the release of WordPress 2.8. Now there is an official syntax that allows you to escape shortcodes in your posts without the need for any additional pluging.
Simply double the square brackets to allow any shortcode in your post escape the parsing:

    [[shortcode]]

will be displayed as [shortcode] in your post.

Optional parameters to the shortcode are of course still possible, i.e.

    [[[gallery link="file" columns="2"]]]

will be displayed as [gallery link="file" columns="2"]

Resources:
http://www.wordpresspluginfu.com/2010/04/escaping-shortcodes-in-wordpress/

Typesetting formuals with MathJax

      No Comments on Typesetting formuals with MathJax

Most hosting providers don’t have LaTeX packages installed on their servers and installing LaTeX yourself is usually only possible on rootservers, which are quite a bit more expensive (and therefore much less common) than regular hosting plans.

However, there is a great project called MathJax, that enables you to display formulas directly inside your wordpress posts without the need to install any LaTeX packages on your server. While you can install the javascript that is used to typeset your formula from either LaTeX or MathML code locally, the easiest way to use MathJax is probably to their content delivery network. For even more convenience, there is even a wordpress plugin that makes typesetting formulas easy as pie.

First, enable the CDN option in MathJax-Latex Plugin Options:

MathJax-LatexPluginOptions

Enable "Use MathJax CDN Server" and save the changes

You can now use regular LaTeX code within your post:

[[latex] S = \frac{1}{(1 - P) + \frac{P}{N}}\leq\frac{1}{1-P} [/latex]]

will be typesettet as

\(S = \frac{1}{(1 – P) + \frac{P}{N}}\leq\frac{1}{1-P}\)

For a more detailed explanation, have a look at the plugin’s description.

Of course, there are quite a few other CDNs, that will typeset your LaTeX formulas. But usually, they just render an image that can be embedded in your post or use flash.
The main advantage of MathJax is that you get your formula as scalable instance that will fit nicely even if you resize your text. You can also access a formula’s source and copy and paste it into your math application (like Mathematica). Have you tried right-clicking the formula above?