Conference in Belgrade

Our colleague suggested one conference in Belgrade on 11th and 12th December 2014, about money laundering, fraud, financial forensics and similar issues "Stručni skup - Borba protiv prevara i korupcije u privredi". First day was interesting a professional auditors and experts were talking about their practice and experience in the region which is probably the most corrupted and fraudulent in whole Europe. Nice touch a bit off from forensic but very interesting was lecture about lies and detecting fraud and lies in communication. To summarize  first, technical day was very good one.

Second day was intended to be scientific one, but it turned into something strange, almost supernatural.  Schedule was changed, lectures was not done in language noted in materials but on whatever author was found of, talking mostly without presentations ... horrible. There was even a bit of a scandal when one ambassador left the in the middle of presenting of the papers being probably insluted with topics. This whole days looks like straight from Branislav Nušić play "Gospođa Ministarka".

Also it was fun to watch and listen trying to figure what presenters are talking about on Russian, Bulgarian etc ..

Anyhow I don't think we will attend same conference next year 

Programming and digital forensics

Just again trying to get some courage and inspiration to write about programming and digital forensics, or even wider aspect of why introduction of purposely designed language can be a good idea.

It looks all not very important issue to have programming and specially defined language, but in my opinion and by experience from other computer fields purpose oriented language or dialect can be very beneficial.

Digital forensics is troubled from begining with it different sources and legal overlooking.  As any forensic science it is partly forensic and partly computer science and engineering. Unfortunately tools and practices and not good as it should be and it suffers a lot from not using ideas like scripting, automation, parallelism which are in use for ages in computer engineering and computer science.

Depending on canned solutions which are powerful in its own world but very limiting in cooperation and performance is a mayor problem. This is again coupled with Microsoft Windows as mayor platform and not a lot of not enough educated and trained users. And are in most  scenarios maybe well trained in using specific tool but usually missed understanding how to efficiently use modern computers or better to say missing the whole idea why computer is here and why it is programmable in more complex way than a washing machine.
Very often basic idea of what we are actually doing is missing,  inadequate procedures, huge volume of data which should be processed which again overwhelms capacity of forensic workstation are common problems. Huge bottlenecks in processing is usual state of art, days long processing also.
Maybe most strange is inability to share results among tools (there are few exceptions to be honest). Impossible situation when something should be compared or reused and vendor tool simply does not allow that .. and everyone involved accept such state of affairs as normal, or when you know that data are already here but because of clumsy interface or any other reason you can't reach it without a lot of codding and using other non forensic tools.

So why all this and specific language ?  Because language is a tool to describe tasks and data involved and result produced. When there is something like that tools are not so important since there is specification which gives clear idea what should be done.  It also means using different tools are simple since all have common language and most of the tasks can be automated and parallelised.


create case  caseone
 add evidence evidence1-file with filesystem
 do signature analyses
 do hash analyses
 do search raw  by keyword list list1, bookmark finds
 do image search by hashset hashone, bookmark finds
 report from bookmark
 close case

This looks like one small dedicated program ... and you have to learn additional syntax :)
so where are advantages ? Ok, lets imagine EnCase has module to process this code and FTK has same, or any other tool


I have to elaborate this more ... :)

Some python scripts ideas as examples in digital forensics

Python works perfectly in UNIX like environment, also very well in windows but when we start to tackle low level issues like accessing raw devices some troubles get to surface ...

It is not about python but about MS platform and it very non-posix behavior, worse of all lack of documentation about some system programming issues on the windows. All this issues makes very hard to achieve reliability required for forensic grade application,  basically you can't trust code because it is not defined how it should work or how it is working this is why most of python forensic things and other serious forensic tool exists on UNIX.

Same problem can be defined for commercially available forensic applications, we only can trust their developers that code is right and doing right thing since we don't have access to it and don't have specification what it should do, especially if we are talking about some esoteric situation about complex registry interpretations etc ..

Let talk about simple task of erasing disk, on posix machine it is easy since all steps are defined and standardized.. on windows it is nightmare standard fileio calls are not working because of some obscure security mechanism which is not described well and semantics of operation can be found only trough scattered examples. Worst of it if example works on XP it will surely not work on newer platforms because of security straightening.

Since we are bound to use windows as platform implementation some examples are quite challenge

• Forensically acceptable media erasing
• Hashing files, folders, raw devices
• Keyword search
• File indexing and using index structures for search
• Timeline analyses
• Interacting with other tools and programs
• Recycle bin recovery
• Windows registry file access

In  more details whet we will play with in future

Excersice script task	Description
Version and type of OS	Get the type and vesrion of OS on your living machine, trough python. Can be done trough from command line or from application
OS module in python to retrive this informations	Type command in cmd line of python and than execute script
Interaction with live live program from python interpreter	Call cmdline windows utilty from python and filter out required data
Hashing	Create the script to hash files, folders and raw devices on windows, also entropy is defined too
Hash files, folders and has for raw devices and logical devices	Create the script to hash files, folders and raw devices on windows, dicuss issues in various implementation solutions and storing / reuing results
redirection of output / input	How to conncet two python scripts together in pipe in command line
storing output data in cvs format	Hot to store results into other formats like cvs, and log the events during execution
Keyword search	Do search for keyowrds in various files Show results in hex and string mode
Grep, plain search in binary/ hex formatu	Introuction to regexp usage
storing results of search to list or dictinary	How to store reuslts
Indexing	How to index file content
Indexing set of files and storing data for usage	How to index file content and use reuslts for search, hotw to store results
Using existing index from other python programs / cmd line
Timeline creation
log2timeline as application (?) probably not because of sift WS	Create timeline from set of files based on metadata values and analyze it (?) Log2timeline as tool called from cmdline
Metadata EXIF / GPS handling	How to extract and anylze EXIF / GPS metadata from pitcures by python script
Windows registry analyses access	How to acess data in registry files and analyze it trough python
general access to registry files	How to acess regitry from command line
WIFI locations as example	How to extract WIFI location stored in registry
Recyle bin access	How to access recyle bin windows folder and recover data
Examining Metadata in PDFs and Microsoft Documents
Investigating Skype Artifacts
Enumerating Browser Artifacts from Firefox Databases
Examining Mobile Device Artifacts
drive cleaning	Cleaning the drive by means of external program from pytohon command line
using drive cleanser as module or as external tool	Cleaning the drive by means of external program from pytohon command line
libwef compliation on windows	Compile / or just install libwef library which allows to access images in e01 and other formats
detecting truecrypt volumes in DD or forensic images	Comment script and how to reuse script or change it for your enviroment

Python training for forensics

Python training for forensics  is very interesting topic, taking into account all issues around professional digital forensic training.  I'll put some articles about since it is intriguing and a challenge .

When you start to play to define tasks, scenarios and scripts to be used it becomes even more interesting. Basically most of the forensic work with python is done on UNIX based platforms and not so much on windows, but as usual because of some strange conditions we are bound to do a training which is based on windows platform.

My colleague Jakob Vukalović is developing full scale tools and scripts, some topics will be soon posted on his blog

So first issue is how to calculate hash on different files and evidence ?
For entry in the file system it is relatively easy traverse folders and run hash on it, but what about the E01 files and dd and other formats, some of them are easy, external tools can be used like FTKimager or python libraries to access it.

If we have e01 image and FTKimager simplest way is to do image mount and get the image as windows raw and hard disk in read only access mode.

When we have a raw device available can we use python to get check-sum as for raw devices on UNIX like machines ? Yes it is possible but a bit strange

On  MS support forum you can find how to access raw devices and logical devices trough open/read/close paradigm. If we are going to access physical disk N device path is "\\.\PhysicalDriveN" and for logical drive X path is "\\.\X:" .   So basic idea is to use same semantics as on other platforms, open device path in read-only and binary access mode, read data from file to buffer , update hash with buffer and do all that  till EOF is reached, than print hash and close file. Strangely there is an error on reaching EOF on the reading raw  physical disk "IOError: [Errno 13] Permission denied" which does not make much sense but kills the process, fortunately python has exception capabilities and this paradigm saves the day :)

Still this tweak does not sound right, since handling exception always mean something went wrong and that is a bad sing in forensic process.. but this is MS operating system and its semantics in working environment of scripting language which is not native to the that closed platform whatever we say.

#!/usr/bin/python

###################################################################

#code snippet to show how to calculate md5 on raw disk 2 on windows platform

#and how to handle  IOError: [Errno 13] Permission denied

#'\\\\.\\PhysicalDrive2' how to code path to pyhsical drive 2 

#'\\\\.\\X:'  how to code path to logical drive X

#this is just a skeleton and must be much much improved for real usage :) 

###################################################################

import os

import sys 

import hashlib

#path to raw the device 

p='\\\\.\\PhysicalDrive2'

#flags can be 'r' or 'rb' same behaviour 

#open path p for reading 

f=open(p,'r')

#initialise the hash module

h=hashlib.md5()

#set the size of read from file, performance varies on the size

#if size is not defined f.read try to load whole file and full memory and crash
s=4096

#optimal size for buffer is 32K  32768  found it later in
#http://ojs.pythonpapers.org/index.php/tpp/article/viewFile/243/211

#reading loop, basically read chunk into rd, if exception it is EOF kink break loop

#also rd is than "" 

while True:

 try:

  rd=f.read(s)

 except IOError: 

  rd="" 

 h.update(rd)

 if rd=="" :

  break

#close raw device, good behavior

f.close() 

#print hash 

print h.hexdigest()

###################################################################

9.2.2017
There was a huge peak on this page in February 3, almost 70 visits mostly from Germany, very unusual.

It is worth of mentioning a new very interesting book by Chet Hosmer "Integrating Python with Leading Computer Forensics Platforms"  I have not yet get a copy, but topics listed are very intriguing. I see one thing missing at first glance python and mobile forensic tool, but probably there are some copyright issues.