Ideas about forensic tool evaluation matrix for our practical lab

Ideas about forensic tool evaluation matrix for our practical lab 

The starting point for analyses is table 1, it presents an relation among courses, learning outcomes and forensic tools planned to be used.  

Semester	Curriculum	Number of learning  outcomes	No. of planned tools
1	The legal basis for digital forensics	7	0
1	Information Security	7	0
1	Mathematical Information Security Models	6	1
1	Digital Forensics	7	14
1	Web Application Security	7	1
1	Digital Media, Security and Privacy	6	4
2	IT Forensics documents and securities	6	4
2	Integrated information security systems	6	1
2	Computer Forensics	7	14
2	Applied cryptography	6	4
2	Risk management and information security incidents management	7	1
2	Computer Network Security	6	11
2	Organization and management of digital forensic analysis	7	6
3	Ethical hacking	7	6
3	Forensics of Mobile Devices	7	11
3	Business Continuity Management	7	0
3	Malware forensics	7	2
3	Forensics of Computer Networks	6	3
3	Forensics of working memory	6	9
3	Script and object languages ​​for digital forensics	6	10
3	Safe Programming Techniques	7	3
4	Methodology of Professional and Exploratory Work	6	0
4	Graduate thesis	8	0

Table 1: the structure of study courses, number of learning outcomes and planned forensic tools for the courses.

From table 1 it is visible some courses will be not using forensic tools since the course is not involved in forensic directly.  Just to explain one good special situation is Graduation thesis. It is listed as no forensic tool used, since it is not sure which subject will be in the student graduation thesis,  it was decided to count it as 0 tools to keep scores minimal.

Tool	Description
Foreman	Laboratory management tool
Autopsy	General purpose forensic tool
Wireshark	Network sniffer – network forensic tool
EnCase	General purpose forensic tool
Amped Soft	Video and image forensic tool
Karens Hasher	Hashing tool
PhotoME	Video and image forensic tool
Python	Scripting language
FTK Imager	General purpose forensic tool
Mitec	General purpose forensic tools
Volatility	Memory forensic tool
Rekall	Memory forensic tool
Kali	Linux forensic distribution
Encrypted Disk Detector	Detects encrypted disks
Network Miner	Network forensic tool
RAM Capturer	Memory forensic tool
Browser History	Host forensic tool analyze web browser artifacts
InnoD	Database tool
UFED	Mobile forensic tool
Cryptool	Cryptography learning tool
GRR	Enterprise forensic tool
Bulkextractor	General purpose forensic tool
Regripper	Forensic registry analyzer
Stegodetect	Forensic steganography detector
Packettracer	Network simulator
Imunes	Network simulator
SonarQube	Source code inspection tool
Java/Eclipse	Java environment

Table 2: forensic tools planned to be part of the new study

Tools planned to be used in new curriculum are presented in the table 2. Criteria to choose tools are same as criteria described in table 4 as for the classification framework. These criteria are based on experience and primary on the availability of forensic tools, but also research on market needs, employer surveys, projections for qualifications required, and analysis of the same or similar study programs.

Forensic tools classification

At the moment there is no consensus about what exactly digital forensic curriculum should be and as result, there is no consensus about recommended tools and infrastructure.   It is possible to consult different classification ideas and resulting set of requirements, but basically, it is best to use own experience and understanding of the environment to define requirements and attributes used for classification.

Requirements must address:

·         forensic tools capabilities in digital forensic and related process,

·         the capability to be used in existing laboratory environment,

·         support and maintenance,

·         quality of results produced,

·         the simplicity of using, compatibility with other forensic tools,

·         compatibility with learning outcomes and usage in real life environment.   

This set  of requirements is based on  current experience and  on current  available resources and knowledge.  To keep process simple only seven attributes are defined, each attribute has set of tag values, and for each tag value, numerical value is defined to allow simple calculations and comparations. Much more elaborate mechanism can be devised, but this simple one is sufficient for creating staring point.

This  forensic tools attributes used for classification:

·         Community usage;

·         Digital forensic area;

·         Licensing options;

·         Opensource, free or commercial;

·         Operating system;

·         Resource requirements;

·         User skill level;

Community usage: this attribute presents how well tool is used and if it is visible in the forensic community, practically addressing almost all of the mentioned requirements.

To measure this attribute set of values is defined based on our experience and working environment:

·         minimal – tool just being used by experts;

·         maximal - widely used and trusted, the tool which currently defines standard;

·         used -  accept and everyday usage;

·         decline - was maximal or used but now usage is declining, such tool is important because of still available expertise;

Licensing options; this attribute presents the method of product licensing, it is important because of available infrastructure. It defines set of values which present policy and availability of resources:

·         Commercial – must be paid for usage;

·         None – no fees, no license access requirements;

·         Network license – there is commercial license server available;

·         Single license – there is electronic or token license;

Resource requirements: this attribute presents how much resources tool needs for minimally acceptable performance, it is important because of the available infrastructure of shared classroom environments. Two broad categories based on laboratory and cloud environment are defined:

·         low resources – works on minimal OS requirements for Windows 10;

·         high resources – requires more resources than low resources;

Operating system: this attribute show on which operating systems tool can be installed and used, again it is important because of available infrastructure. For example, ftk imager has score 2 since it works on both Windows and Linux.  Possible values are:

·         full – meaning tool works on all listed OS;

·         windows – supported version of windows OS win 7,win 10;

·         Linux – any Linux version;

·         mac  - current version of OSx;

Digital forensic area: this attribute presents the digital forensic area in which tool can be used, it is important because of simplicity of usage in training and preparations. Tools with complete range  is preferred, example are EnCase or Autopsy, both tools are general purpose forensic tools. Values are based on the compilation of available classifications since there are plenty of classifications available, but often not comparable. Chosen values are mostly based on NIST and SANS classification and new TVZ curriculum requirements:

·         Complete – means all forensic areas in the curriculum, numerically it is a sum of available values;

·         Management – special situation for tools like Foreman designed to be a management tool for the digital forensic process, but not forensic tool per se;

·         host forensic – host forensic, traditional forensic;

·         mobile forensic  - mobile devices forensic;

·         network forensic – network/enterprise forensic;

·         image/multimedia  forensics;

Opensource, free or commercial: this attribute presents the type of forensic tool,  what is important attribute because of cost  and support. It is important to notice a difference among free and opensource, as an example there is a FTKimager a free forensic tool commonly used while not being opensource one since it code is not available for analyses. If it is possible opensource is preferred over free, since it covers code access. Possible values for this attribute are:

·         Free – free to use, no access to the code;

·         Opensource – free to use, access to the code ;

·         Commercial – no access to the code, license to be paid or can be used in test and trial period;

User skill level: this attribute presents how much skilled user must be to effectively use a forensic tool on the defined task. This attribute is important since it presents how much work student can do with the forensic tool, and how much of preparations are needed for using the forensic tool in the education.

·         Low – means that student can predefined task without special training and preparation;

·         High – means that student must have done preparational training before it can use forensic tool efficiently.

Forensic tool attribute	Meaning of the attribute	tag value
Community usage	maximal	4,00
Community usage	used	3,00
Community usage	Decline	2,00
Community usage	Minimal	1,00
Digital forensic area	Complete	8,00
Digital forensic area	Management	2,00
Digital forensic area	host forensic	1,00
Digital forensic area	mobile forensic	1,00
Digital forensic area	network forensic	1,00
Digital forensic area	image forensics	1,00
Licensing options	None	4,00
Licensing options	network license	3,00
Licensing options	Commercial	2,00
Licensing options	single license	1,00
Opensource, free or commercial	Free	2,00
Opensource, free or commercial	Opensource	3,00
Opensource, free or commercial	Commercial	1,00
Operating system	Full	4,00
Operating system	Windows	1,00
Operating system	Linux	1,00
Operating system	Mac	1,00
Resource requirements	low resources	2,00
Resource requirements	high resources	1,00
User skill level	Low	2,00
User skill level	High	1,00

Table 3:  relation  among tool attributes, tag values and numerical tag values for the scoring process.

The simple sum is used to create the score for each forensic tool. Such approach is used to keep results simple and easy to maintain while numerical values of tag values are chosen on experience. It was  decided not to use negative values to avoid complicating and making results unintuitive, but it is possible to do if further  refinements will be needed. Tag values are  based on the experience of the curriculum writers, current information from forensic conferences and data gathering in 2017/2018. The idea was to create a high score which will show the most important tools.

Tool	Opensource, free or commercial	Licensing options	Resource requirements	Operating system	Digital forensic area	User skill level	Community usage
GRR	opensource	none	high resources	full	network forensic	high	minimal
bulkextractor	opensource	none	high resources	full	complete	high	used
packettracer	commercial	single license	high resources	full	network forensic	high	used
Wireshark	opensource	none	low resources	full	network forensic	high	maximal
Volatility	opensource	none	high resources	full	memory forensic	high	maximal
Imunes	opensource	none	high resources	Linux	network forensic	high	minimal
EnCase	commercial	network license	high resources	windows	complete	high	decline
Python	opensource	none	low resources	full	complete	high	maximal
Network Miner	commercial	none	low resources	windows	network forensic	high	used
InnoDB	free	none	low resources	full	complete	high	used
Java/Eclipse	free	none	low resources	full	complete	high	used
Autopsy	opensource	none	low resources	full	complete	low	maximal
Amped Soft	commercial	single license	high resources	windows	image forensics	high	maximal
UFED	commercial	single license	low resources	windows	mobile forensic	high	maximal
Cryptool	free	none	low resources	windows	complete	high	used
SonarQube	opensource	none	low resources	windows	complete	high	minimal
Foreman	opensource	none	low resources	full	management	low	minimal
Rekall	opensource	none	low resources	Linux	complete	high	maximal
Kali	opensource	none	low resources	Linux	complete	high	maximal
regripper	opensource	none	low resources	windows	host forensic	high	maximal
stegodetect	opensource	none	low resources	windows	host forensic	low	used
Karens Hasher	opensource	none	low resources	windows	complete	low	minimal
Encrypted Disk Detector	free	none	low resources	windows	complete	low	used
RAM Capturer	free	none	low resources	windows	memory forensic	low	used
Browser History	free	none	low resources	windows	host forensic	low	used
PhotoME	free	none	low resources	windows	image forensics	low	maximal
FTK Imager	free	none	low resources	windows, Linux	host forensic	low	maximal
Mitec	free	none	low resources	windows	host forensic	low	maximal

Table 4:  tool classification based on criteria listed in table 3.

Classification in table 4 is based on practical experience with forensic tools and current situation in the community.  As an example of classification EnCase can be used. It is defined as in “decline”, since tool was recently de facto standard of digital forensics, but its start to lose ground due to some development and managerial issues. Still there is a vast amount of knowledge and resources in community which defines EnCase as still important tool. To add to its importance, it is a commercial tool with network licensing what simplify laboratory virtualization and control.

No formal process was used to fill data in table 4, a group of experienced forensic examiners and forensic trainers who also were curriculum writers and designers did brainstorm to fill  table 4.

The other important value in the evaluation process is the number of in how many learning outcomes in the curriculum each tool will be used.  This value shows most heavily used tools which require the most attention and care. It also reveals some issues about infrastructure and possible correctness of digital forensic tool, since some of mentioned tools like Imunes or packetracer are not common in digital forensics.

Forensic tool	score from table 3	number of usages in learning outcomes
Foreman	17	75
Python	20	53
FTK Imager	13	46
EnCase	20	40
Autopsy	18	40
packettracer	25	34
Imunes	22	34
Wireshark	22	34
Karens Hasher	16	34
GRR	25	33
Volatility	22	29
Amped Soft	18	20
PhotoME	13	20
Kali	17	13
Network Miner	20	12
Rekall	17	12
Mitec	13	8
InnoDB	20	7
Java/Eclipse	20	7
SonarQube	18	7
UFED	18	7
bulkextractor	25	6
Cryptool	18	6
Encrypted Disk Detector	16	6
RAM Capturer	15	6
stegodetect	17	5
regripper	17	4
Browser History	15	1

Table 5: comparation of validation score based on the table 3 and count of  how many learning outcomes forensic tool will be used or will be mentioned.

Conclusion

In the table 5 it is possible to see difference among top tools based on attributes score and top tools based on the learning outcomes usage.  

Top scoring tools are Packettracer, GRR, bulkextractor, Imunes, Wireshark and Volatility if we are looking based on validation score, but if we are looking based on the learning outcomes there are Foreman, Python, FTK Imager EnCase, and Autopsy.  

Meaning of this difference is there is still a discrepancy among tools available and reality of education since top scoring tools are not most often used in learning outcomes. Assuring fact is that general purpose forensic tools (FTKimager, EnCase, Autopsy) are most common with Python as scripting platform next, this tools are also with high score presetting its usability in TVZ environment. 

The next evaluation will probably have some attribute and  tag values changes based on the gathered experience, what will influence current gap.

It is also a high chance that some tools will be replaced and some learning outcomes modified, what will again create a new version of table 5 and new evaluation.