A Method for the evaluation of knowledge management systems

.


INTRODUCTION
Knowledge management is the process of creating, storing, managing, using and sharing the knowledge and information of an organization [1]. It refers to a multidisciplinary approach (including the fields of business administration, information systems, management, library, and information sciences) to achieving organizational objectives by making the best use of knowledge [2][3] [4]. It is an important tool for the preservation of the knowledge and information of an organization, and for its transference from older to newer staff. Many large companies, public institutions and non-profit organizations have resources dedicated to internal knowledge management (KM) efforts, often as a part of their business strategy, information technology, or human resource management departments [5].
Several information systems can be employed on an organization to assist on its efforts on KM.
On the Nuclear Engineering Institute (IEN), a Brazilian federal research institute of the Nuclear Energy National Commission (CNEN), three systems are in use on this field: WikiIEN, Nuclear Energy in Magazines and CarpeDIEN. These systems are web applications, a kind of software that runs on a combination of web server (the processing and data storage side) and web browser (the user interface side) over a computer network, such as the internet or the IEN's intranet network [6].
Each of these web applications have different purposes and therefore function differently from each other.
Thus, the purpose of this article is to compare the functionality of each of these web applications using a single methodology, pointing the verified strengths and weaknesses of each one regarding their use on knowledge management. First, each web application will be briefly reviewed.
Then, the methodology will be presented and applied to evaluate the functionality of the KM systems. Finally, a comparison of the results will be shown and commented.

WikiIEN
WikiIEN is a wiki-type web application based on the MediaWiki software. It enables the users to create and modify articles on a WYSIWYG ("What You See Is What You Get") editor with text, links to another articles on the same wiki or to web sites, mathematical formulae, images and The first public version of DSpace was released in November 2002, as a joint effort between developers from MIT and HP Labs. Currently the DSpace software and user community receives leadership and guidance from DuraSpace, a not-for-profit organization.
DSpace is constructed with Java web applications, many programs, and an associated metadata store. The web applications provide interfaces for administration, deposit, ingest, search, and access. The asset store is maintained on a file system or similar storage system. The metadata, including access and configuration information, is stored in a relational database and supports the use of PostgreSQL and Oracle database. DSpace holdings are made available primarily via a web interface. Some most important features of DSpace are as follows: • Completely customizable to fit user needs; • Manage and preserve all format of digital content (PDF, Word, JPEG, MPEG, TIFF files); • Granular group based access control, allowing setting permissions down to the level of individual files; • Optimized for Google Scholar indexing.
Notable DSpace repositories: • The World Bank -Open Knowledge Repository; • Apollo -University of Cambridge Repository; • Digital Access to Scholarship at Harvard; • DSpace@MIT; • Spiral -Imperial College London Repository; • WHO Institutional Repository for Information Sharing;

MATERIALS AND METHODS
In the last two decades, several software packages were developed specifically for use in the KM field. At the same time, software originally developed for other fields, such as process modeling and database management, were eventually branded as KM software. Added to a lack of adequate criteria, the commonly vague understandings of KM concepts make difficult the task of choosing the most appropriate software for a given need in an organization [12].
Several studies presented criteria and information to assist in the evaluation and choice of the most appropriate KM system for a given use [12][13][14] [15]. Evaluation criteria usually focus on software functionalities grouped in categories, such as: usability, search features, user customization, content management, online collaboration and administration. The redundancy, complexities and abstract definitions found in each KM system brings difficulties in properly using a given set of criteria. Therefore, the characteristics and functionalities are considered in the criteria so that the most adequate KM system is chosen to serve the institutional needs. The evaluation of such criteria is subjective, and based on the evaluators' experience and technical knowledge while using such systems.
The methodology in use is the field research, in which phenomena are analyzed as they naturally occur [16]. The research carried out is of participant-observer type, that the researcher acts in relation to the effects of the phenomena studied [16]. The method of evaluation was created and used by the researchers and authors of this work. The evaluation of WikiIEN, Nuclear Energy in Magazines and CarpeDIEN systems was structured in two steps: 1. Selection criteria;

Selection criteria
The criteria used in this work consists of a list of 14 criteria, classified into 7 categories, based on literature [12][14] [15]. The separation of criteria into categories allows perceiving the level of development of each system in relation to a broader set of criteria. The categories were drawn up to reflect functionality, demonstrating positive aspects (strengths) and negative aspects (weaknesses) of each system. The categories, criteria and descriptions are presented in Table 1.
WikiIEN, Nuclear Energy in Magazines and CarpeDIEN are evaluated by experts in KM systems using the criteria presented on table 1. The objective of this evaluation is to verify whether the systems meet the criteria. In this step is performed a comparison between the systems.

Categories
Group content by affinity.

Search Features 3.1 Search engine
Tool for searching data on system.

Scope of search results
Search includes data and metadata.

Presentation of search results
Appropriate presentation of search results.

User Customization 4.1 Layout Customization
Ability to change the system interface.

Content creation and visualization
Create and view data and metadata in-system.

Rules of publication and access
Change of user permissions or or restrictions.

Changes Management
Ability to revert content changes.

Integration
Ability to connect to other systems.
• "Ease of use" criterion: WikiIEN is based on the MediaWiki software, which is developed by Wikimedia Foundation for use on Wikipedia, which is aimed at the general public, so its user interface was designed to be more user-friendly from start, and has received large user feedback over a long time. On the other hand, Nuclear Energy in Magazines and CarpeDIEN are based on the OJS and DSpace software respectively, which are directed toward niche communities. And, in OJS' case, it uses an editorial management workflow, which is uncommon for the general public.
• "Scope of search results" criterion: Content in WikiIEN (e. g. text entered by the enduser) is directly stored into the system's relational database, so its search engine can search for keywords in articles and metadata. Content in Nuclear Energy in Magazines and CarpeDIEN is stored in files generated by external application software. These files cannot be searched for keywords by their search engine, which are restricted to metadata, resulting in a more restricted search scope.
• "Content creation and visualization" criterion: WikiIEN provides means to create and to visualize data and metadata, making it less dependent on external software. Nuclear Energy in Magazines and CarpeDIEN provide means to create and visualize metadata only, thus requiring external application software to create and to visualize data. Some web browser features can supplement this lack of functionality from the systems.
• "Online discussion tools" criterion: WikiIEN is focused at a community of users who collaborate more closely between themselves to aggregate knowledge. It needed better online discussion tools if compared to Nuclear Energy in Magazines and CarpeDIEN. It even enables users to discuss individual articles and comment each article modification.
Nuclear Energy in Magazines and CarpeDIEN, which are aimed at aggregating documents d articles/works from authors or groups of authors, didn't need so detailed discussion tools.

CONCLUSION
We proposed a method that uses criteria, classified into 7 categories, and the judgment of the experts in KM systems. These categories were drawn up to reflect the functionality of the systems.
A pilot study in WikiIEN, Nuclear Energy in Magazines and CarpeDIEN shows that this method based on criteria offers interesting perspectives to compare these systems, to point weaknesses of each system and to choose the most appropriate system for a given need in IEN.
The results point out that WikiIEN is a solution more indicated for a bigger, general public because of its more user-friendly interface and workflow, auto-sufficient set of features not dependent on external software. Nuclear Energy in Magazines and CarpeDIEN are solutions more indicated for smaller, niche public because of a less user-friendly interface, workflow directed at individual or small group of users who uses specialized software.
Using this method we identify problems related to four criteria: "Ease of use", "Scope of search results", "Content creation and visualization" and "Online discussion tools". These specific problems pointed weaknesses in Nuclear Energy in Magazines and CarpeDIEN.
As suggestions for future research, we highlight: (1) the development of a computational system in order to automate the use of the method to evaluate a KM system online; (2) the periodic application of the method to estimate how changes made on these systems improved their performance.