Comments on: The Reports of Our Professional Deaths Have Been Greatly Exaggerated, Part 2 https://agnosticmaybe.wordpress.com/2010/10/31/the-reports-of-our-professional-deaths-have-been-greatly-exaggerated-part-2/ the neverending reference interview of life Tue, 14 Apr 2015 18:36:05 +0000 hourly 1 http://wordpress.com/ By: Tweets that mention The Reports of Our Professional Deaths Have Been Greatly Exaggerated, Part 2 « Agnostic, Maybe -- Topsy.com https://agnosticmaybe.wordpress.com/2010/10/31/the-reports-of-our-professional-deaths-have-been-greatly-exaggerated-part-2/#comment-2206 Mon, 01 Nov 2010 06:09:40 +0000 http://agnosticmaybe.wordpress.com/2010/10/31/the-reports-of-our-professional-deaths-have-been-greatly-exaggerated-part-2/#comment-2206 […] This post was mentioned on Twitter by Andy Woodworth, Tina Reynolds. Tina Reynolds said: RT sounds good! @wawoodworth: Blogged: The Reports of Our Professional Deaths Have Been Greatly Exaggerated Part 2 http://bit.ly/9jRDtF […]

]]>
By: Dr. Dale https://agnosticmaybe.wordpress.com/2010/10/31/the-reports-of-our-professional-deaths-have-been-greatly-exaggerated-part-2/#comment-2205 Mon, 01 Nov 2010 03:44:45 +0000 http://agnosticmaybe.wordpress.com/2010/10/31/the-reports-of-our-professional-deaths-have-been-greatly-exaggerated-part-2/#comment-2205 I would like to provide some technical perspective on this issue and provide a motivation as to the usefulness of librarians in this problem. I’ve dealt with information classification at my job (although only in passing since we decided not to pursue further research into it at the moment). I would group this problem of information overload / classification / retrieval under the auspices of applied computer science. In particular the area of research is machine learning, natural language processing, textual classification, document classification, etc.

I’m going to limit what I discuss to textual (document) information that is available in an electronic form. The idea is how to automatically classify documents into a taxonomy (and how to define such a taxonomy). Since there is such an abundant quantity of information, it will require farms of computers to do the work of document association and classification. Since we would like a computer to do this, that means we need to develop algorithms for the job, however these algorithms must be based on some mathematical framework.

Some of the issues that must be addressed in this mathematical framework are what features of a document are necessary for its classification, how to quantify the similarity between documents, and how to cluster the resulting data points, etc. Depending on how you want to spin this, this is a supervised/unsupervised machine learning problem. On top of this there is the more subtle (and difficult) problem of training a computer how to learn and detect contextual (semantic) information in documents.

Ultimately this is a nontrivial problem and the focus of current research. Of course practical progress has been made on classification of information,and some have managed to make enormous sums of money from their efforts (see Google…).

Now to provide my small plug for librarians…As librarians, a contribution could definitely be (among other potential contributions) in providing researchers with area expertise to develop better features (metrics to define on documents) and metrics of similarity. Basically, what it is that should be measured in a document that helps it be classified correctly (for example the percentage of words in a document that are “literary terms” vs. other preset classes of words) and how to successfully compare two documents for similarity using those measurements.

]]>
https://agnosticmaybe.wordpress.com/2010/10/31/the-reports-of-our-professional-deaths-have-been-greatly-exaggerated-part-2/feed/
agnosticmaybe.wordpress.com
crawl=CC-MAIN-2015-18
8468
application/http; msgtype=response
sha1:VTIQ2TU2VTEY6XMGZZIVYXZHVQJY2NTN
<urn:uuid:967b9d72-35a1-49cc-b3ce-a00b0fe54d7a>
2015-04-21T11:36:27Z
192.0.78.13
sha1:NKFOG44AYAGI7673FWCJ3PGB6QYTPKIZ
<urn:uuid:8298f3ae-599a-4d0e-bb7c-a7b81abe63f6>
https://agnosticmaybe.wordpress.com/2010/10/31/the-reports-of-our-professional-deaths-have-been-greatly-exaggerated-part-2/feed/
length
response
<urn:uuid:8c356fc1-cdbb-41b5-a604-ca2d6bd741c2>
robots: classic
hostname: ip-10-235-10-82.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2015-18
operator: CommonCrawl Admin
description: Wide crawl of the web for April 2015
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf