Friday, August 31, 2018

Ideas about forensic tool evaluation matrix for our practical lab


Ideas about forensic tool evaluation matrix for our practical lab 

The starting point for analyses is table 1, it presents an relation among courses, learning outcomes and forensic tools planned to be used.  

Semester
Curriculum
Number of learning  outcomes
No. of planned
tools
1
The legal basis for digital forensics
7
0
1
Information Security
7
0
1
Mathematical Information Security Models
6
1
1
Digital Forensics
7
14
1
Web Application Security
7
1
1
Digital Media, Security and Privacy
6
4
2
IT Forensics documents and securities
6
4
2
 Integrated information security systems
6
1
2
Computer Forensics
7
14
2
Applied cryptography
6
4
2
Risk management and information security incidents management
7
1
2
Computer Network Security
6
11
2
Organization and management of digital forensic analysis
7
6
3
Ethical hacking
7
6
3
Forensics of Mobile Devices
7
11
3
Business Continuity Management
7
0
3
Malware forensics
7
2
3
Forensics of Computer Networks
6
3
3
 Forensics of working memory
6
9
3
Script and object languages ​​for digital forensics
6
10
3
Safe Programming Techniques
7
3
4
Methodology of Professional and Exploratory Work
6
0
4
Graduate thesis
8
0

Table 1: the structure of study courses, number of learning outcomes and planned forensic tools for the courses.
From table 1 it is visible some courses will be not using forensic tools since the course is not involved in forensic directly.  Just to explain one good special situation is Graduation thesis. It is listed as no forensic tool used, since it is not sure which subject will be in the student graduation thesis,  it was decided to count it as 0 tools to keep scores minimal.

Tool
Description
Foreman
Laboratory management tool
Autopsy
General purpose forensic tool
Wireshark
Network sniffer – network forensic tool
EnCase
General purpose forensic tool
Amped Soft
Video and image forensic tool
Karens Hasher
Hashing tool
PhotoME
Video and image forensic tool
Python
Scripting language
FTK Imager
General purpose forensic tool
Mitec
General purpose forensic tools
Volatility
Memory forensic tool
Rekall
Memory forensic tool
Kali
Linux forensic distribution
Encrypted Disk Detector
Detects encrypted disks
Network Miner
Network forensic tool
RAM Capturer
Memory forensic tool
Browser History
Host forensic tool analyze web browser artifacts
InnoD
Database tool
UFED
Mobile forensic tool
Cryptool
Cryptography learning tool
GRR
Enterprise forensic tool
Bulkextractor
General purpose forensic tool
Regripper
Forensic registry analyzer
Stegodetect
Forensic steganography detector
Packettracer
Network simulator
Imunes
Network simulator
SonarQube
Source code inspection tool
Java/Eclipse
Java environment

Table 2: forensic tools planned to be part of the new study
Tools planned to be used in new curriculum are presented in the table 2. Criteria to choose tools are same as criteria described in table 4 as for the classification framework. These criteria are based on experience and primary on the availability of forensic tools, but also research on market needs, employer surveys, projections for qualifications required, and analysis of the same or similar study programs.

Forensic tools classification

At the moment there is no consensus about what exactly digital forensic curriculum should be and as result, there is no consensus about recommended tools and infrastructure.   It is possible to consult different classification ideas and resulting set of requirements, but basically, it is best to use own experience and understanding of the environment to define requirements and attributes used for classification.
Requirements must address:
·         forensic tools capabilities in digital forensic and related process,
·         the capability to be used in existing laboratory environment,
·         support and maintenance,
·         quality of results produced,
·         the simplicity of using, compatibility with other forensic tools,
·         compatibility with learning outcomes and usage in real life environment.   
This set  of requirements is based on  current experience and  on current  available resources and knowledge.  To keep process simple only seven attributes are defined, each attribute has set of tag values, and for each tag value, numerical value is defined to allow simple calculations and comparations. Much more elaborate mechanism can be devised, but this simple one is sufficient for creating staring point.
This  forensic tools attributes used for classification:
·         Community usage;
·         Digital forensic area;
·         Licensing options;
·         Opensource, free or commercial;
·         Operating system;
·         Resource requirements;
·         User skill level;
Community usage: this attribute presents how well tool is used and if it is visible in the forensic community, practically addressing almost all of the mentioned requirements.
To measure this attribute set of values is defined based on our experience and working environment:
·         minimal – tool just being used by experts;
·         maximal - widely used and trusted, the tool which currently defines standard;
·         used -  accept and everyday usage;
·         decline - was maximal or used but now usage is declining, such tool is important because of still available expertise;

Licensing options; this attribute presents the method of product licensing, it is important because of available infrastructure. It defines set of values which present policy and availability of resources:
·         Commercial – must be paid for usage;
·         None – no fees, no license access requirements;
·         Network license – there is commercial license server available;
·         Single license – there is electronic or token license;

Resource requirements: this attribute presents how much resources tool needs for minimally acceptable performance, it is important because of the available infrastructure of shared classroom environments. Two broad categories based on laboratory and cloud environment are defined:
·         low resources – works on minimal OS requirements for Windows 10;
·         high resources – requires more resources than low resources;

Operating system: this attribute show on which operating systems tool can be installed and used, again it is important because of available infrastructure. For example, ftk imager has score 2 since it works on both Windows and Linux.  Possible values are:
·         full – meaning tool works on all listed OS;
·         windows – supported version of windows OS win 7,win 10;
·         Linux – any Linux version;
·         mac  - current version of OSx;

Digital forensic area: this attribute presents the digital forensic area in which tool can be used, it is important because of simplicity of usage in training and preparations. Tools with complete range  is preferred, example are EnCase or Autopsy, both tools are general purpose forensic tools. Values are based on the compilation of available classifications since there are plenty of classifications available, but often not comparable. Chosen values are mostly based on NIST and SANS classification and new TVZ curriculum requirements:
·         Complete – means all forensic areas in the curriculum, numerically it is a sum of available values;
·         Management – special situation for tools like Foreman designed to be a management tool for the digital forensic process, but not forensic tool per se;
·         host forensic – host forensic, traditional forensic;
·         mobile forensic  - mobile devices forensic;
·         network forensic – network/enterprise forensic;
·         image/multimedia  forensics;

Opensource, free or commercial: this attribute presents the type of forensic tool,  what is important attribute because of cost  and support. It is important to notice a difference among free and opensource, as an example there is a FTKimager a free forensic tool commonly used while not being opensource one since it code is not available for analyses. If it is possible opensource is preferred over free, since it covers code access. Possible values for this attribute are:
·         Free – free to use, no access to the code;
·         Opensource – free to use, access to the code ;
·         Commercial – no access to the code, license to be paid or can be used in test and trial period;

User skill level: this attribute presents how much skilled user must be to effectively use a forensic tool on the defined task. This attribute is important since it presents how much work student can do with the forensic tool, and how much of preparations are needed for using the forensic tool in the education.
·         Low – means that student can predefined task without special training and preparation;
·         High – means that student must have done preparational training before it can use forensic tool efficiently.


Forensic tool attribute
Meaning of the attribute
tag value
Community usage
maximal
4,00
Community usage
used
3,00
Community usage
Decline
2,00
Community usage
Minimal
1,00
Digital forensic area
Complete
8,00
Digital forensic area
Management
2,00
Digital forensic area
host forensic
1,00
Digital forensic area
mobile forensic
1,00
Digital forensic area
network forensic
1,00
Digital forensic area
image forensics
1,00
Licensing options
None
4,00
Licensing options
network license
3,00
Licensing options
Commercial
2,00
Licensing options
single license
1,00
Opensource, free or commercial
Free
2,00
Opensource, free or commercial
Opensource
3,00
Opensource, free or commercial
Commercial
1,00
Operating system
Full
4,00
Operating system
Windows
1,00
Operating system
Linux
1,00
Operating system
Mac
1,00
Resource requirements
low resources
2,00
Resource requirements
high resources
1,00
User skill level
Low
2,00
User skill level
High
1,00

Table 3:  relation  among tool attributes, tag values and numerical tag values for the scoring process.

The simple sum is used to create the score for each forensic tool. Such approach is used to keep results simple and easy to maintain while numerical values of tag values are chosen on experience. It was  decided not to use negative values to avoid complicating and making results unintuitive, but it is possible to do if further  refinements will be needed. Tag values are  based on the experience of the curriculum writers, current information from forensic conferences and data gathering in 2017/2018. The idea was to create a high score which will show the most important tools.

Tool
Opensource, free or commercial
Licensing options
Resource requirements
Operating system
Digital forensic area
User skill level
Community usage
GRR
opensource
none
high resources
full
network forensic
high
minimal
bulkextractor
opensource
none
high resources
full
complete
high
used
packettracer
commercial
single license
high resources
full
network forensic
high
used
Wireshark
opensource
none
low resources
full
network forensic
high
maximal
Volatility
opensource
none
high resources
full
memory forensic
high
maximal
Imunes
opensource
none
high resources
Linux
network forensic
high
minimal
EnCase
commercial
network license
high resources
windows
complete
high
decline
Python
opensource
none
low resources
full
complete
high
maximal
Network Miner
commercial
none
low resources
windows
network forensic
high
used
InnoDB
free
none
low resources
full
complete
high
used
Java/Eclipse
free
none
low resources
full
complete
high
used
Autopsy
opensource
none
low resources
full
complete
low
maximal
Amped Soft
commercial
single license
high resources
windows
image forensics
high
maximal
UFED
commercial
single license
low resources
windows
mobile forensic
high
maximal
Cryptool
free
none
low resources
windows
complete
high
used
SonarQube
opensource
none
low resources
windows
complete
high
minimal
Foreman
opensource
none
low resources
full
management
low
minimal
Rekall
opensource
none
low resources
Linux
complete
high
maximal
Kali
opensource
none
low resources
Linux
complete
high
maximal
regripper
opensource
none
low resources
windows
host forensic
high
maximal
stegodetect
opensource
none
low resources
windows
host forensic
low
used
Karens Hasher
opensource
none
low resources
windows
complete
low
minimal
Encrypted Disk Detector
free
none
low resources
windows
complete
low
used
RAM Capturer
free
none
low resources
windows
memory forensic
low
used
Browser History
free
none
low resources
windows
host forensic
low
used
PhotoME
free
none
low resources
windows
image forensics
low
maximal
FTK Imager
free
none
low resources
windows, Linux
host forensic
low
maximal
Mitec
free
none
low resources
windows
host forensic
low
maximal

Table 4:  tool classification based on criteria listed in table 3.

Classification in table 4 is based on practical experience with forensic tools and current situation in the community.  As an example of classification EnCase can be used. It is defined as in “decline”, since tool was recently de facto standard of digital forensics, but its start to lose ground due to some development and managerial issues. Still there is a vast amount of knowledge and resources in community which defines EnCase as still important tool. To add to its importance, it is a commercial tool with network licensing what simplify laboratory virtualization and control.
No formal process was used to fill data in table 4, a group of experienced forensic examiners and forensic trainers who also were curriculum writers and designers did brainstorm to fill  table 4.
The other important value in the evaluation process is the number of in how many learning outcomes in the curriculum each tool will be used.  This value shows most heavily used tools which require the most attention and care. It also reveals some issues about infrastructure and possible correctness of digital forensic tool, since some of mentioned tools like Imunes or packetracer are not common in digital forensics.

Forensic tool
score from table 3
number of usages in learning outcomes
Foreman
17
75
Python
20
53
FTK Imager
13
46
EnCase
20
40
Autopsy
18
40
packettracer
25
34
Imunes
22
34
Wireshark
22
34
Karens Hasher
16
34
GRR
25
33
Volatility
22
29
Amped Soft
18
20
PhotoME
13
20
Kali
17
13
Network Miner
20
12
Rekall
17
12
Mitec
13
8
InnoDB
20
7
Java/Eclipse
20
7
SonarQube
18
7
UFED
18
7
bulkextractor
25
6
Cryptool
18
6
Encrypted Disk Detector
16
6
RAM Capturer
15
6
stegodetect
17
5
regripper
17
4
Browser History
15
1

Table 5: comparation of validation score based on the table 3 and count of  how many learning outcomes forensic tool will be used or will be mentioned.

Conclusion

In the table 5 it is possible to see difference among top tools based on attributes score and top tools based on the learning outcomes usage.  
Top scoring tools are Packettracer, GRR, bulkextractor, Imunes, Wireshark and Volatility if we are looking based on validation score, but if we are looking based on the learning outcomes there are Foreman, Python, FTK Imager EnCase, and Autopsy.  
Meaning of this difference is there is still a discrepancy among tools available and reality of education since top scoring tools are not most often used in learning outcomes. Assuring fact is that general purpose forensic tools (FTKimager, EnCase, Autopsy) are most common with Python as scripting platform next, this tools are also with high score presetting its usability in TVZ environment. 
The next evaluation will probably have some attribute and  tag values changes based on the gathered experience, what will influence current gap.
It is also a high chance that some tools will be replaced and some learning outcomes modified, what will again create a new version of table 5 and new evaluation.