text: This work presents a method for the measurement of the accuracy of evidential artifact extraction and categorization tasks in digital forensic investigations. Instead of focusing on the measurement of accuracy and errors in the functions of digital forensic tools, this work proposes the application of information retrieval measurement techniques that allow the incorporation of errors introduced by tools and analysis processes. This method uses a ‘gold standard’ that is the collection of evidential objects determined by a digital investigator from suspect data with an unknown ground truth. This work proposes that the accuracy of tools and investigation processes can be evaluated compared to the derived gold standard using common precision and recall values. Two example case studies are presented showing the measurement of the accuracy of automated analysis tools as compared to an in-depth analysis by an expert. It is shown that such measurement can allow investigators to determine changes in accuracy of their processes over time, and determine if such a change is caused by their tools or knowledge.
  @cite_19: Current digital forensic text string search tools use match and or indexing algorithms to search digital evidence at the physical level to locate specific text strings. They are designed to achieve 100 query recall (i.e. find all instances of the text strings). Given the nature of the data set, this leads to an extremely high incidence of hits that are not relevant to investigative objectives. Although Internet search engines suffer similarly, they employ ranking algorithms to present the search results in a more effective and efficient manner from the user's perspective. Current digital forensic text string search tools fail to group and or order search hits in a manner that appreciably improves the investigator's ability to get to the relevant hits first (or at least more quickly). This research proposes and empirically tests the feasibility and utility of post-retrieval clustering of digital forensic text string search results - specifically by using Kohonen Self-Organizing Maps, a self-organizing neural network approach. This paper is presented as a work-in-progress. A working tool has been developed and experimentation has begun. Findings regarding the feasibility and utility of the proposed approach will be presented at DFRWS 2007, as well as suggestions for follow-on research.
  @cite_4: The ability to automatically classify files based on their low-level, short-range structures is of particular importance in computer forensics. We report a study on the automatic learning of file classification using byte sub-stream kernels that capture these low-level structures. We automatically discover byte-level patterns in a file by extracting a byte sequence feature map and use a suffix trie data structure to efficiently store and manipulate the feature map. Using the feature map we compute the spectrum kernel and, together with a support vector machine classifier algorithm, we are able to efficiently categorize a variety of different system and application file types. Experiments have provided good file classification performance results.
  @cite_23: The ability to automatically classify document fragments based on their contents is important in digital forensics. This paper proposes an Enhanced String Kernel (ESK) to classify file header fragments with Support Vector Machine (SVM). ESK can extract a byte sequence feature map about document fragment. The map consists of byte-level patterns of document fragments, and captures the characteristic of document fragments. An extended suffix array (ESA) data structure is presented to efficiently store and manipulate the feature map. We can compute the ESK by using the feature map. This method can efficiently categorize a variety of different systems and application file header fragment types. Experiments have provided good classification performance results about file header fragments.
summary: Information Retrieval (IR) is one area where accuracy measurement is paramount. Much work has been done in the area of IR, and IR accuracy measurement techniques have previously been applied to forensic text sting searching @cite_19 , document classification @cite_4 , and even fragmented document analysis in digital forensics @cite_23 . The focus, however, has been on the accuracy measurement of particular techniques or tools within the digital examination process, and not for the examination process itself.