Ideas about forensic tool evaluation matrix for our practical lab
The
starting point for analyses is table 1, it presents an relation among courses,
learning outcomes and forensic tools planned to be used.
Semester
|
Curriculum
|
Number of learning outcomes
|
No. of planned
tools
|
1
|
The
legal basis for digital forensics
|
7
|
0
|
1
|
Information
Security
|
7
|
0
|
1
|
Mathematical
Information Security Models
|
6
|
1
|
1
|
Digital
Forensics
|
7
|
14
|
1
|
Web
Application Security
|
7
|
1
|
1
|
Digital
Media, Security and Privacy
|
6
|
4
|
2
|
IT
Forensics documents and securities
|
6
|
4
|
2
|
Integrated information security systems
|
6
|
1
|
2
|
Computer
Forensics
|
7
|
14
|
2
|
Applied
cryptography
|
6
|
4
|
2
|
Risk
management and information security incidents management
|
7
|
1
|
2
|
Computer
Network Security
|
6
|
11
|
2
|
Organization
and management of digital forensic analysis
|
7
|
6
|
3
|
Ethical
hacking
|
7
|
6
|
3
|
Forensics
of Mobile Devices
|
7
|
11
|
3
|
Business
Continuity Management
|
7
|
0
|
3
|
Malware
forensics
|
7
|
2
|
3
|
Forensics
of Computer Networks
|
6
|
3
|
3
|
Forensics of working memory
|
6
|
9
|
3
|
Script
and object languages for digital forensics
|
6
|
10
|
3
|
Safe
Programming Techniques
|
7
|
3
|
4
|
Methodology
of Professional and Exploratory Work
|
6
|
0
|
4
|
Graduate
thesis
|
8
|
0
|
Table 1: the structure of study courses, number
of learning outcomes and planned forensic tools for the courses.
From table
1 it is visible some courses will be not using forensic tools since the course
is not involved in forensic directly. Just
to explain one good special situation is Graduation thesis. It is listed as no
forensic tool used, since it is not sure which subject will be in the student
graduation thesis, it was decided to
count it as 0 tools to keep scores minimal.
Tool
|
Description
|
Foreman
|
Laboratory
management tool
|
Autopsy
|
General purpose
forensic tool
|
Wireshark
|
Network sniffer –
network forensic tool
|
EnCase
|
General purpose
forensic tool
|
Amped Soft
|
Video and image
forensic tool
|
Karens Hasher
|
Hashing tool
|
PhotoME
|
Video and image
forensic tool
|
Python
|
Scripting language
|
FTK Imager
|
General purpose
forensic tool
|
Mitec
|
General purpose
forensic tools
|
Volatility
|
Memory forensic tool
|
Rekall
|
Memory forensic tool
|
Kali
|
Linux forensic
distribution
|
Encrypted Disk
Detector
|
Detects encrypted
disks
|
Network Miner
|
Network forensic
tool
|
RAM Capturer
|
Memory forensic tool
|
Browser History
|
Host forensic tool
analyze web browser artifacts
|
InnoD
|
Database tool
|
UFED
|
Mobile forensic tool
|
Cryptool
|
Cryptography
learning tool
|
GRR
|
Enterprise forensic
tool
|
Bulkextractor
|
General purpose
forensic tool
|
Regripper
|
Forensic registry analyzer
|
Stegodetect
|
Forensic
steganography detector
|
Packettracer
|
Network simulator
|
Imunes
|
Network simulator
|
SonarQube
|
Source code
inspection tool
|
Java/Eclipse
|
Java environment
|
Table 2: forensic tools planned to be part of
the new study
Tools
planned to be used in new curriculum are presented in the table 2. Criteria to
choose tools are same as criteria described in table 4 as for the
classification framework. These criteria are based on experience and primary on
the availability of forensic tools, but also research on market needs, employer
surveys, projections for qualifications required, and analysis of the same or
similar study programs.
Forensic
tools classification
At the
moment there is no consensus about what exactly digital forensic curriculum
should be and as result, there is no consensus about recommended tools and
infrastructure. It is possible to consult different
classification ideas and resulting set of requirements, but basically, it is
best to use own experience and understanding of the environment to define
requirements and attributes used for classification.
Requirements
must address:
·
forensic
tools capabilities in digital forensic and related process,
·
the
capability to be used in existing laboratory environment,
·
support
and maintenance,
·
quality
of results produced,
·
the
simplicity of using, compatibility with other forensic tools,
·
compatibility
with learning outcomes and usage in real life environment.
This set of requirements is based on current experience and on current available resources and knowledge. To keep process simple only seven attributes
are defined, each attribute has set of tag values, and for each tag value,
numerical value is defined to allow simple calculations and comparations. Much
more elaborate mechanism can be devised, but this simple one is sufficient for creating
staring point.
This forensic tools attributes used for
classification:
·
Community
usage;
·
Digital
forensic area;
·
Licensing
options;
·
Opensource,
free or commercial;
·
Operating
system;
·
Resource
requirements;
·
User
skill level;
Community usage: this attribute presents how well tool is used and if it is visible in the forensic community,
practically addressing almost all of the mentioned requirements.
To
measure this attribute set of values is defined based on our experience and working
environment:
·
minimal – tool just being used by
experts;
·
maximal - widely used and trusted,
the tool which currently defines standard;
·
used - accept and everyday usage;
·
decline - was maximal or used but now
usage is declining, such tool is important because of still available expertise;
Licensing options; this attribute presents the method of product licensing, it is important because of available
infrastructure. It defines set of values which present policy and availability
of resources:
·
Commercial – must be paid for
usage;
·
None – no fees, no license access
requirements;
·
Network license – there is
commercial license server available;
·
Single license – there is electronic
or token license;
Resource requirements: this attribute presents how much resources tool needs for minimally acceptable performance, it
is important because of the available infrastructure of shared classroom environments.
Two broad categories based on laboratory and cloud environment are defined:
·
low resources – works on minimal
OS requirements for Windows 10;
·
high resources – requires more
resources than low resources;
Operating system: this attribute show on which operating systems tool can be installed and used, again it is
important because of available infrastructure. For example, ftk imager has
score 2 since it works on both Windows and Linux. Possible values are:
·
full – meaning tool works on all
listed OS;
·
windows – supported version of
windows OS win 7,win 10;
·
Linux – any Linux version;
·
mac - current version of OSx;
Digital forensic area: this attribute presents the digital forensic area in which tool can be used, it is important because
of simplicity of usage in training and preparations. Tools with complete range is preferred, example are EnCase or Autopsy,
both tools are general purpose forensic tools. Values are based on the compilation
of available classifications since there are plenty of classifications
available, but often not comparable. Chosen values are mostly based on NIST and
SANS classification and new TVZ curriculum requirements:
·
Complete – means all forensic
areas in the curriculum, numerically it is a sum of available values;
·
Management – special situation for
tools like Foreman designed to be a management tool for the digital forensic
process, but not forensic tool per se;
·
host forensic – host forensic,
traditional forensic;
·
mobile forensic - mobile devices forensic;
·
network forensic – network/enterprise
forensic;
·
image/multimedia forensics;
Opensource, free or commercial: this attribute
presents the type of forensic tool, what is important attribute because of cost and support. It is important to notice a difference
among free and opensource, as an example there is a FTKimager a free forensic
tool commonly used while not being opensource one since it code is not
available for analyses. If it is possible opensource is preferred over free,
since it covers code access. Possible values for this attribute are:
·
Free – free to use, no access to the
code;
·
Opensource – free to use, access
to the code ;
·
Commercial – no access to the code,
license to be paid or can be used in test and trial period;
User skill level: this attribute presents how much skilled user must be to effectively use a forensic tool on the
defined task. This attribute is important since it presents how much work
student can do with the forensic tool, and how much of preparations are needed
for using the forensic tool in the education.
·
Low –
means that student can predefined task without special training and preparation;
·
High – means that student must
have done preparational training before it can use forensic tool efficiently.
Forensic tool attribute
|
Meaning of the attribute
|
tag value
|
Community usage
|
maximal
|
4,00
|
Community usage
|
used
|
3,00
|
Community usage
|
Decline
|
2,00
|
Community usage
|
Minimal
|
1,00
|
Digital forensic area
|
Complete
|
8,00
|
Digital forensic area
|
Management
|
2,00
|
Digital forensic area
|
host
forensic
|
1,00
|
Digital forensic area
|
mobile
forensic
|
1,00
|
Digital forensic area
|
network
forensic
|
1,00
|
Digital forensic area
|
image
forensics
|
1,00
|
Licensing options
|
None
|
4,00
|
Licensing options
|
network
license
|
3,00
|
Licensing options
|
Commercial
|
2,00
|
Licensing options
|
single
license
|
1,00
|
Opensource, free or commercial
|
Free
|
2,00
|
Opensource, free or commercial
|
Opensource
|
3,00
|
Opensource, free or commercial
|
Commercial
|
1,00
|
Operating system
|
Full
|
4,00
|
Operating system
|
Windows
|
1,00
|
Operating system
|
Linux
|
1,00
|
Operating system
|
Mac
|
1,00
|
Resource requirements
|
low
resources
|
2,00
|
Resource requirements
|
high
resources
|
1,00
|
User skill level
|
Low
|
2,00
|
User skill level
|
High
|
1,00
|
Table 3: relation among tool attributes, tag values and
numerical tag values for the scoring process.
The simple sum
is used to create the score for each forensic tool. Such approach is used to
keep results simple and easy to maintain while numerical values of tag values
are chosen on experience. It was decided
not to use negative values to avoid complicating and making results unintuitive,
but it is possible to do if further
refinements will be needed. Tag values are based on the experience of the curriculum
writers, current information from forensic conferences and data gathering in 2017/2018.
The idea was to create a high score which will show the most important tools.
Tool
|
Opensource, free or commercial
|
Licensing options
|
Resource requirements
|
Operating system
|
Digital forensic area
|
User skill level
|
Community usage
|
GRR
|
opensource
|
none
|
high
resources
|
full
|
network
forensic
|
high
|
minimal
|
bulkextractor
|
opensource
|
none
|
high
resources
|
full
|
complete
|
high
|
used
|
packettracer
|
commercial
|
single
license
|
high
resources
|
full
|
network
forensic
|
high
|
used
|
Wireshark
|
opensource
|
none
|
low
resources
|
full
|
network
forensic
|
high
|
maximal
|
Volatility
|
opensource
|
none
|
high
resources
|
full
|
memory
forensic
|
high
|
maximal
|
Imunes
|
opensource
|
none
|
high
resources
|
Linux
|
network
forensic
|
high
|
minimal
|
EnCase
|
commercial
|
network
license
|
high
resources
|
windows
|
complete
|
high
|
decline
|
Python
|
opensource
|
none
|
low
resources
|
full
|
complete
|
high
|
maximal
|
Network Miner
|
commercial
|
none
|
low
resources
|
windows
|
network
forensic
|
high
|
used
|
InnoDB
|
free
|
none
|
low
resources
|
full
|
complete
|
high
|
used
|
Java/Eclipse
|
free
|
none
|
low
resources
|
full
|
complete
|
high
|
used
|
Autopsy
|
opensource
|
none
|
low
resources
|
full
|
complete
|
low
|
maximal
|
Amped Soft
|
commercial
|
single
license
|
high
resources
|
windows
|
image
forensics
|
high
|
maximal
|
UFED
|
commercial
|
single
license
|
low
resources
|
windows
|
mobile
forensic
|
high
|
maximal
|
Cryptool
|
free
|
none
|
low
resources
|
windows
|
complete
|
high
|
used
|
SonarQube
|
opensource
|
none
|
low
resources
|
windows
|
complete
|
high
|
minimal
|
Foreman
|
opensource
|
none
|
low
resources
|
full
|
management
|
low
|
minimal
|
Rekall
|
opensource
|
none
|
low
resources
|
Linux
|
complete
|
high
|
maximal
|
Kali
|
opensource
|
none
|
low
resources
|
Linux
|
complete
|
high
|
maximal
|
regripper
|
opensource
|
none
|
low
resources
|
windows
|
host
forensic
|
high
|
maximal
|
stegodetect
|
opensource
|
none
|
low
resources
|
windows
|
host
forensic
|
low
|
used
|
Karens Hasher
|
opensource
|
none
|
low
resources
|
windows
|
complete
|
low
|
minimal
|
Encrypted Disk Detector
|
free
|
none
|
low
resources
|
windows
|
complete
|
low
|
used
|
RAM Capturer
|
free
|
none
|
low
resources
|
windows
|
memory
forensic
|
low
|
used
|
Browser History
|
free
|
none
|
low
resources
|
windows
|
host
forensic
|
low
|
used
|
PhotoME
|
free
|
none
|
low
resources
|
windows
|
image
forensics
|
low
|
maximal
|
FTK Imager
|
free
|
none
|
low
resources
|
windows,
Linux
|
host
forensic
|
low
|
maximal
|
Mitec
|
free
|
none
|
low
resources
|
windows
|
host
forensic
|
low
|
maximal
|
Table 4: tool classification based on criteria listed
in table 3.
Classification
in table 4 is based on practical experience with forensic tools and current
situation in the community. As an example
of classification EnCase can be used. It is defined as in “decline”, since tool
was recently de facto standard of digital forensics, but its start to lose
ground due to some development and managerial issues. Still there is a vast
amount of knowledge and resources in community which defines EnCase as still
important tool. To add to its importance, it is a commercial tool with network licensing
what simplify laboratory virtualization and control.
No formal
process was used to fill data in table 4, a group of experienced forensic
examiners and forensic trainers who also were curriculum writers and designers
did brainstorm to fill table 4.
The other
important value in the evaluation process is the number of in how many learning
outcomes in the curriculum each tool will be used. This value shows most heavily used tools which
require the most attention and care. It also reveals some issues about infrastructure
and possible correctness of digital forensic tool, since some of mentioned tools
like Imunes or packetracer are not common in digital forensics.
Forensic tool
|
score from table 3
|
number of usages in learning outcomes
|
Foreman
|
17
|
75
|
Python
|
20
|
53
|
FTK
Imager
|
13
|
46
|
EnCase
|
20
|
40
|
Autopsy
|
18
|
40
|
packettracer
|
25
|
34
|
Imunes
|
22
|
34
|
Wireshark
|
22
|
34
|
Karens
Hasher
|
16
|
34
|
GRR
|
25
|
33
|
Volatility
|
22
|
29
|
Amped
Soft
|
18
|
20
|
PhotoME
|
13
|
20
|
Kali
|
17
|
13
|
Network
Miner
|
20
|
12
|
Rekall
|
17
|
12
|
Mitec
|
13
|
8
|
InnoDB
|
20
|
7
|
Java/Eclipse
|
20
|
7
|
SonarQube
|
18
|
7
|
UFED
|
18
|
7
|
bulkextractor
|
25
|
6
|
Cryptool
|
18
|
6
|
Encrypted
Disk Detector
|
16
|
6
|
RAM
Capturer
|
15
|
6
|
stegodetect
|
17
|
5
|
regripper
|
17
|
4
|
Browser
History
|
15
|
1
|
Table 5: comparation of validation score based
on the table 3 and count of how many
learning outcomes forensic tool will be used or will be mentioned.
Conclusion
In the table 5 it is possible to see difference
among top tools based on attributes score and top tools based on the learning
outcomes usage.
Top scoring tools are Packettracer, GRR,
bulkextractor, Imunes, Wireshark and Volatility if we are looking
based on validation score, but if we are looking based on the learning outcomes
there are Foreman, Python, FTK Imager EnCase, and Autopsy.
Meaning
of this difference is there is still a discrepancy among tools available and
reality of education since top scoring tools are not most often used in
learning outcomes. Assuring fact is that general purpose forensic tools
(FTKimager, EnCase, Autopsy) are most common with Python as scripting platform
next, this tools are also with high score presetting its usability in TVZ
environment.
The
next evaluation will probably have some attribute and tag values changes based on the gathered
experience, what will influence current gap.
It is
also a high chance that some tools will be replaced and some learning outcomes
modified, what will again create a new version of table 5 and new evaluation.