RFOR is finishing at VISTE

Lecturing at this academic year is just about to  finish, it is exam time coming, time to wrap up and lessons learned not only for students but for teachers too.

As for the the first run of the Digital Forensics Basics (RFOR) on Vsite I have my doubts and fears if I done things as it is supposed to be.  Class was small 18 people, easy to work and enjoyable maybe a bit to much chance for lecturers ego trip sometimes.

Results are OK, but that can be misleading, I still feel we need more practical work with commercial tools. To add some practice work we added some python scripting, based on "Python Forensics" and some elements from "Violent Python", all trough SIFT workstation from SANS.

As we are at computer science and engineering school a lot of things are already known, so we'he been concentrated to general principles and ideas, not so much on the methods and tools.

My colleague  Darijo Puntarić was busy with laboratory exercises, as CCIE he added a lot of good stuff especially to network forensic part.

An article about law enforcement and high end computer skills

This one is posted at one of the sites I've read often, "Information Warfare: Scary Monsters Pursue The FBI"  title is a bit scary, actually shows the problem  if law enforcement has to  be about computer security issues. To be honest most of the organisation of any kind has this problem  if they are not from start based on computing technology.  Unfortunately even such powerful organizations as FBI or DOJ   somehow lack strategic approach and understanding of the problem. At first glance it is OK, found you re in trouble and there is someone who can help you .. cooperate use resources and prepare but for what and how,  results are showing this is missing. 

Digital Forensic Tools and parallelism - initial thoughts

Without going deep into any theoretical discussion it is quit obvious that digital forensic tasks are actually very well suited for parallel processing. The key issue is readonly access to data in most data intensive operations, but also in other parallelism can be applied too taking into account nature of digital forensic process and its dataprocessing steps:

• acquisition, 
• analyses and 
• reporting. 

If we discuss different each step in forensic process we can see how parallelism can be used.
First data processing step in any digital forensics task is acquisition of data from device or media .

Device acquisition is serial task, since  without live access we have only one channel  the devices. Parallelism here is more question of device itself than forensic tool. As for example if we have more than one access channel to  device data  and we are in read only mode acquisition can be parallel too.
Key element is readonly access to data.

In other steps  like during analyses,  situation is just slightly different.  It is usually data extraction and reconstruction which results finally in data size reduction, Data which we are using in this step is under read-only access, while results of processing are written and maybe again read back into process.  This two modes of data access are well separated in analyses process. Each analyses task can go  in parallel with other tasks without corrupting data. In most situation this new result are actually metadata. Such metadata is much smaller than original data, and can be put back into analyses cycle if it is necessary. As it is shown for most of the analyses step parallelism also  can be used.

To illustrate this in more details we can discuss important forensic tasks indexing and  raw search.
Indexing is specific since it can generate almost same volume of data as the original data. It is also highly repetitive task, since it depends new recovered or unlocked documents to be indexed and data added to existing index structure. Operations are very disk intensive  but again can be done effectively in parallel, especially if index structure is stored in database way. It maybe sound strange but raw search is very close to index process, especially in phase of building index structure, in fact it is the same, simply said we have to extract raw data from disk and in that data find words which are indexed. Exactly the same as raw search do.  Conclusion again is the same parallelism can be used too, also parallelism is important for indexing and search tasks since it requires processing  huge amount of data .

Forensic data processing  usually generates metadata which presents a new logical view on the original data. Good examples are bookmarks, so much loved in digital forensics.

As for report creating again the same approach works, from data and bookmarks report is compiled, data is not changed in that process, so it can be parallelised too.

So what is conclusion ?
Parallelism is highly desirable in digital forensics, but we don't have tools which are very effective in using parallelism, This is something what is happening just now with various level of success for different vendors.
My opinion is that vendors are landlocked in their tools and the real advantage of parallelism is in type of forensic tools which can be fully automated and  can freely and easily cooperate, being scripted and capable of working standardised on the highly parallel computing infrastructure:)

I'll elaborate on this later, while talking about what such tools and systems have to be able to do and which already existing knowledge we have.

MS Windows, Python and Digital Forensics woes

Since Vista comes it turns out that it is impossible to compile and add into Python on windows essential libraries like libewf  It is very frustrating and senseless , but this is MS way of things ..
It boils down to undocumented features and behaviour of required low level windows calls and instrumentation. My colleague Jakob spend a lot of time to try and test all available compiling procedures but nothing worked in the end. It all worked once in 2009 and earlier but not today.
This event troubles us a because of planned training propositions, since we have to introduce additional unix topics  where tools work as it is supposed to work. 

Conference in Belgrade

Our colleague suggested one conference in Belgrade on 11th and 12th December 2014, about money laundering, fraud, financial forensics and similar issues "Stručni skup - Borba protiv prevara i korupcije u privredi". First day was interesting a professional auditors and experts were talking about their practice and experience in the region which is probably the most corrupted and fraudulent in whole Europe. Nice touch a bit off from forensic but very interesting was lecture about lies and detecting fraud and lies in communication. To summarize  first, technical day was very good one.

Second day was intended to be scientific one, but it turned into something strange, almost supernatural.  Schedule was changed, lectures was not done in language noted in materials but on whatever author was found of, talking mostly without presentations ... horrible. There was even a bit of a scandal when one ambassador left the in the middle of presenting of the papers being probably insluted with topics. This whole days looks like straight from Branislav Nušić play "Gospođa Ministarka".

Also it was fun to watch and listen trying to figure what presenters are talking about on Russian, Bulgarian etc ..

Anyhow I don't think we will attend same conference next year 

Programming and digital forensics

Just again trying to get some courage and inspiration to write about programming and digital forensics, or even wider aspect of why introduction of purposely designed language can be a good idea.

It looks all not very important issue to have programming and specially defined language, but in my opinion and by experience from other computer fields purpose oriented language or dialect can be very beneficial.

Digital forensics is troubled from begining with it different sources and legal overlooking.  As any forensic science it is partly forensic and partly computer science and engineering. Unfortunately tools and practices and not good as it should be and it suffers a lot from not using ideas like scripting, automation, parallelism which are in use for ages in computer engineering and computer science.

Depending on canned solutions which are powerful in its own world but very limiting in cooperation and performance is a mayor problem. This is again coupled with Microsoft Windows as mayor platform and not a lot of not enough educated and trained users. And are in most  scenarios maybe well trained in using specific tool but usually missed understanding how to efficiently use modern computers or better to say missing the whole idea why computer is here and why it is programmable in more complex way than a washing machine.
Very often basic idea of what we are actually doing is missing,  inadequate procedures, huge volume of data which should be processed which again overwhelms capacity of forensic workstation are common problems. Huge bottlenecks in processing is usual state of art, days long processing also.
Maybe most strange is inability to share results among tools (there are few exceptions to be honest). Impossible situation when something should be compared or reused and vendor tool simply does not allow that .. and everyone involved accept such state of affairs as normal, or when you know that data are already here but because of clumsy interface or any other reason you can't reach it without a lot of codding and using other non forensic tools.

So why all this and specific language ?  Because language is a tool to describe tasks and data involved and result produced. When there is something like that tools are not so important since there is specification which gives clear idea what should be done.  It also means using different tools are simple since all have common language and most of the tasks can be automated and parallelised.


create case  caseone
 add evidence evidence1-file with filesystem
 do signature analyses
 do hash analyses
 do search raw  by keyword list list1, bookmark finds
 do image search by hashset hashone, bookmark finds
 report from bookmark
 close case

This looks like one small dedicated program ... and you have to learn additional syntax :)
so where are advantages ? Ok, lets imagine EnCase has module to process this code and FTK has same, or any other tool


I have to elaborate this more ... :)

Some python scripts ideas as examples in digital forensics

Python works perfectly in UNIX like environment, also very well in windows but when we start to tackle low level issues like accessing raw devices some troubles get to surface ...

It is not about python but about MS platform and it very non-posix behavior, worse of all lack of documentation about some system programming issues on the windows. All this issues makes very hard to achieve reliability required for forensic grade application,  basically you can't trust code because it is not defined how it should work or how it is working this is why most of python forensic things and other serious forensic tool exists on UNIX.

Same problem can be defined for commercially available forensic applications, we only can trust their developers that code is right and doing right thing since we don't have access to it and don't have specification what it should do, especially if we are talking about some esoteric situation about complex registry interpretations etc ..

Let talk about simple task of erasing disk, on posix machine it is easy since all steps are defined and standardized.. on windows it is nightmare standard fileio calls are not working because of some obscure security mechanism which is not described well and semantics of operation can be found only trough scattered examples. Worst of it if example works on XP it will surely not work on newer platforms because of security straightening.

Since we are bound to use windows as platform implementation some examples are quite challenge

• Forensically acceptable media erasing
• Hashing files, folders, raw devices
• Keyword search
• File indexing and using index structures for search
• Timeline analyses
• Interacting with other tools and programs
• Recycle bin recovery
• Windows registry file access

In  more details whet we will play with in future

Excersice script task	Description
Version and type of OS	Get the type and vesrion of OS on your living machine, trough python. Can be done trough from command line or from application
OS module in python to retrive this informations	Type command in cmd line of python and than execute script
Interaction with live live program from python interpreter	Call cmdline windows utilty from python and filter out required data
Hashing	Create the script to hash files, folders and raw devices on windows, also entropy is defined too
Hash files, folders and has for raw devices and logical devices	Create the script to hash files, folders and raw devices on windows, dicuss issues in various implementation solutions and storing / reuing results
redirection of output / input	How to conncet two python scripts together in pipe in command line
storing output data in cvs format	Hot to store results into other formats like cvs, and log the events during execution
Keyword search	Do search for keyowrds in various files Show results in hex and string mode
Grep, plain search in binary/ hex formatu	Introuction to regexp usage
storing results of search to list or dictinary	How to store reuslts
Indexing	How to index file content
Indexing set of files and storing data for usage	How to index file content and use reuslts for search, hotw to store results
Using existing index from other python programs / cmd line
Timeline creation
log2timeline as application (?) probably not because of sift WS	Create timeline from set of files based on metadata values and analyze it (?) Log2timeline as tool called from cmdline
Metadata EXIF / GPS handling	How to extract and anylze EXIF / GPS metadata from pitcures by python script
Windows registry analyses access	How to acess data in registry files and analyze it trough python
general access to registry files	How to acess regitry from command line
WIFI locations as example	How to extract WIFI location stored in registry
Recyle bin access	How to access recyle bin windows folder and recover data
Examining Metadata in PDFs and Microsoft Documents
Investigating Skype Artifacts
Enumerating Browser Artifacts from Firefox Databases
Examining Mobile Device Artifacts
drive cleaning	Cleaning the drive by means of external program from pytohon command line
using drive cleanser as module or as external tool	Cleaning the drive by means of external program from pytohon command line
libwef compliation on windows	Compile / or just install libwef library which allows to access images in e01 and other formats
detecting truecrypt volumes in DD or forensic images	Comment script and how to reuse script or change it for your enviroment