development of a prototype inventory for evaluation of educational

DEVELOPMENT OF A PROTOTYPE INVENTORY FOR
EVALUATION OF EDUCATIONAL SOFTWARE FROM
ETHICAL, SOCIAL AND CULTURAL DIMENSIONS
Hasan TINMAZ
hasan_tinmaz@hotmail.com
ABSTRACT
Evaluation of educational software is still an essential concern of most academicians and practitioners. However, another concern about the
relation between end-users of software and technology has been neglected; ethical, social and cultural dimensions of educational software.
The study is a preliminary attempt to scrutinize these issues. Data were gathered in two must courses of “Computer Education and
Instructional Technology (CEIT)” department; “Instructional Technology and Material Development” and “Instructional Material Preparation”.
The study has been conducted in two steps. In step one, 22 third grade CEIT preservice teachers were asked to write their statements that
they believed to measure ethical, social and cultural dimensions of assessing educational software. Afterwards, researcher combined similar
sentences and restated them as items of an inventory. Subsequent to this first prototype, in the second step, second (N=46) and third (N=22)
grade CEIT preservice teachers were asked to state their ideas on each item. Last inventory included forty-nine items.
Keywords: Ethical, Social and Cultural Perspectives of Software, Educational Software Evaluation, Inventory Development.
INTRODUCTION
or age. In that sense offering well-qualified educational
software is very crucial duty of teachers, families and etc…
To serve evaluation purposes, different inventories and
questionnaires, different evaluation methods and techniques
concerning different aspects of educational software have
been developed. Nonetheless, it seems that still one more
concern about the relation between end-users of software
and technology has been neglected; ethical, social and
cultural dimensions of educational software.
Assessment and evaluation are two concepts dominantly
placed in instructional environment. On the other hand,
Hoise, Schibeci and Backhaus (2005) noted that there is no
consensus about what this evaluation should cover and how
it could be done. They also notify that if there is any type of
“information and communication technology” in instructional
framework, the concept of “evaluation” always sounds by
people.
Technological improvements have inevitably influenced the
every aspects of our daily life. Our educational systems (Lin
& Dwyer, 2005), shopping habits, communication tools (and
ways) and so forth have been altering due to the penetration
of technology into our lives (Deuze, 2006). On the other
hand, the pace of human’s adaptation to those changes is
not as rapid as technological innovations themselves. Our
modern life comprises a relatively constant people factor
and a relatively variant technology issue. Nowadays, the
concept of “technology” is directly linked to computer
technologies where from early childhood to late adulthood,
we frequently interact. In that perspective, all issues related
to “people and technology interaction” are worthwhile for us
to be scrutinized.
One of these issues is the utilization of software for
instructional purposes. Noticeably, “technology-basededucation” phenomena increased the importance of
educational software use. However, the design and
development of such software is not straightforward to
manage. Historically, at the beginning of designing such
educational software, developers resembled people as the
“machines” and believed that the way people learn is just
passing through some predefined steps. Yet, as these
software utilized in original settings, developers realized that
they ignored the personal and social characteristics of
human in the design and development processes.
Afterwards, scholars, researchers and developers have
investigated this process more complicatedly and the
different aspects of educational software have been
evaluated such as “curriculum adequacy”, “visual design”,
“utilized technology” and so forth. Subsequent to the
analyses of educational software with respect to those
distinct properties, several guidelines about educational
software have been created. Now, there is a wellestablished sector of designing software for instructional
aims and the sector still deals with question of “how best to
manage the process” (Krishna, Sahay & Walsham, 2004,
p.63).
As Herring, Notar and Wilson (2005) expressed “the
evaluation of educational software has become a critical
information skill for teachers” (p.100). In their study for
preparing an evaluation form for software, they listed
several items under four major areas; (a) content, (b)
student involvement, (c) ease of use, and (d) design
(esthetic). They offer some statements for the appropriate,
biased-free, and up-to-date content. As a suggestion, they
offer adaptations for evaluative purposes of differentiated
instructional context.
After the five year research on software development for
inter-cultural adaptation, Krishna, Sahay and Walsham
(2004), concludes with a charming remark that creating
software for different cultures is not an easy-going activity.
They explain this issue by addressing some challenges
such as adaptation to cultural norms, linguistics problems
and so on. They also offer to produce culturally nonaligned
software to handle these problems. Globalization, as
reflected on each dimension of our lives, has affected the
process of software development. On the other hand,
globalization doesn’t have to mean that software should be
culture-free (Krishna, Sahay & Walsham).
Within this sector, there are several tracks implying the
entire process like analysis, design, development, utilization
and evaluation. The last step, evaluation of educational
software is still an essential concern of most academicians
and practitioners. Drew and Thorpe (2006) concluded in
their study that the utilization of software or the attitude
toward software are affected from how it is or how it is used
in the classrooms, rather than from other factors like gender
The purpose of this preliminary study is to trigger for
developing such an inventory to assess ethical, social and
cultural issues of educational software. Therefore, the
results of study cannot be generalized to all settings and still
need to spend time for improving.
485
METHOD
Preparation) on designing, developing and evaluating
instructional materials.
Since the second step was conducted in the last week of the
semesters, second grade preservice teachers were also
assumed as knowledgeable in assessing educational
materials including educational software.
Participants
The participants of study were the second grade (N=46) and
the third grade preservice teachers of “Computer Education
and Instructional Technology” (CEIT) Department (N=22) of
Başkent University. This group is purposefully selected due
to three reasons; (a) group is familiar with the evaluation of
software, (b) group knows how to develop such software,
and (c) group is computer preservice teachers that will
interact both with technology and people. Data were
gathered in two must courses of CEIT department;
“Instructional Technology and Material Development” and
“Instructional Material Preparation”. The content of these
courses aim to furnish the preservice teachers with the
knowledge, skills and abilities in regarding to utilization of
different media in instructional settings including the
educational software. For the entire sample, age average is
22 where there are 30 male and 38 female; the second
grade, 19 male and 27 male, whereas the third grade, 11
male and 11 female.
Findings and Discussions
After data collection (N=68), initial statistical analyses were
conducted on forty-nine items in the inventory. Mean scores
(from highest to lowest) and standard deviations were
calculated as in Table 1. The Cronbach alpha for the fortynine items is calculated as 0.88 which is a good reliability
coefficient. However, since the number of participants is
limited (N=68), further studies need to explain and re-test
this reliability coefficient. Moreover, data set has problems
with applying factor analysis. Basically, according to the
results of Kaiser-Meyer-Olkin (KMO) measure of sampling
adequacy test (KMO=0.39), the distribution of values is not
sufficient for conducting a factor analysis, since the value of
KMO is less than 0.60 (George & Mallery, 2001).
Design of the Study and Instrumentation
The study has been conducted in two steps. In step one, 22
third grade CEIT preservice teachers were asked to write
their statements that they believed to measure ethical,
social and cultural dimensions of assessing educational
software. Afterwards, researcher combined similar
sentences and restated them as items of an inventory.
Subsequent to creation of this first prototype, in the second
step, second (N=46) and third (N=22) grade CEIT
preservice teachers were asked to state their ideas on each
item. For each item, they decided on a 5-scale Likert type
from “very important” item to “very unimportant” items. The
reason of including second grade preservice teachers is that
they were attending their first course (Instructional Material
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
As revealed from Table 1, moral (e.g. appropriateness of
pictures and animations) and discrimination (e.g. national,
religious and gender) issues have evaluated as very
important items. The relationships between the social and
ethical values of a culture and the elements of software like
characters, pictures, speeches and so forth are considered
as the important items of the inventory. Local (country-wise)
values have higher mean scores than global values which
mean that preservice teachers want to observe first the
representation of their own cultures issues and then the
universal ones. The extreme ideas like representation of
religious figures or nationalistic point of view have moderate
mean scores.
Table 1. The item descriptions, means and standard deviations
Item Description
Yazılımda kullanılan resimler ahlaki açıdan çocuklar için uygundur.
Eğitim yazılımı, kullanıcı kitlesinin ahlaki gelişimini olumsuz yönde etkiliyor.
Yazılımda ırk ayrımcılığı yapılmaktadır.
Yazılımda din ayrımcılığı yapılmaktadır.
Eğitim yazılımı ayrımcılık yapmadan konuyu kullanıcıya sunuyor.
Eğitim yazılımının içeriği, kullanıcıları kötü alışkınlıklara (sigara, fast-food, vb…) özendirici öğeler
içermiyor.
Yazılım anlayabileceğim bir sadelikle hazırlanmış.
Yazılım, hazırlanmış olan ülkenin inançlarına (sosyal, ahlaki, dini, vb…) ters düşmemektedir.
Yazılım içerisinde verilen örnekler gerçek yaşama uygundur.
Yazılım içerisinde kullanılan animasyonlarda öğrencinin zihninde canlandıramayacağı öğeler
kullanılmamaktadır.
Yazılım, öğrencinin zihnindeki bilgi yapılandırmasını kültürel anlamda zenginleştiriyor.
Eğitim yazılımının fiyatı, toplumun genel alım gücüne uygundur.
Yazılımın kullandığı örnekler günceldir.
Yazılımda cinsel ayrımcılık yapılmaktadır.
Yazılımın içeriği, içinde bulunduğu toplumun bütün sosyal kesimlerine hitap ediyor.
Yazılım, bilmediğim ya da yabancısı olduğum öğeler içeriyor.
Yazılım içindeki karakterler, kıyafet olarak ahlakımıza uymayacak şekilde giyiniyor.
Yazılım içinde geçen karakterlerin isimleri kültürümüze ait değil.
Yazılımın içeriği argo kelimeler içeriyor.
Yazılımdaki karakterler, toplumumu doğru yansıtmaktadır.
Yazılım içerisindeki resimler öğrencinin günlük hayatında karşılaşabileceği türdendir.
Yazılım içinde kullanılan animasyonlar öğrencileri yabancı kültürlere özendiriyor.
Yazılım içerisinde kullanılan örnekler, toplumsal statüler göz önünde bulundurularak hazırlanmıştır.
Yazılım içerisinde kullanılan simgeler kültürümüze uygundur.
Yazılımın içerisinde kullanıcının, sosyal değerlere uyum sağlamasına yardımcı öğeler bulunmaktadır.
Yazılımda işlenen tema beni olumsuz etkiledi.
Yazılım içinde kullanılan markalar yabancı ürünleri içeriyor.
Yazılımda beni rahatsız eden unsurlar bulunmamaktadır.
Yazılımda kullanılan görüntüler günlük yaşamdan alınmıştır.
486
M.
4,63
4,54
4,43
4,38
4,38
S.D.
0,57
0,84
1,07
1,07
0,85
4,28
1,10
4,24
4,24
4,22
0,99
1,14
0,69
4,22
0,93
4,21
4,19
4,19
4,12
4,10
4,07
4,07
4,06
4,04
4,03
4,03
3,99
3,99
3,94
3,94
3,91
3,91
3,87
3,84
0,66
0,94
0,85
1,26
0,92
1,01
1,07
1,05
1,04
0,79
1,01
1,28
0,87
0,99
0,88
1,19
1,24
1,22
0,91
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
Item Description
Yazılım, aile yapımıza uyuyor.
Yazılım içerisindeki kullanılan animasyonlar, halkın örf ve adetlerine uygundur.
Yazılım içerisindeki animasyonlarda yer alan karakterler kültürümüze ait insan profilini yansıtmaktadır.
Yazılım içinde kültürümüzde olmayan varlıklar bulunmaktadır.
Yazılım evrensel değerlere göre hazırlanmıştır.
Yazılım boyunca animasyonlarda toplum içindeki genel ahlak kuralları hakkında örneklemeler
kullanılmıştır.
Yazılım her türlü ekonomik düzeye hitap etmektedir.
Yazılımın içeriği ülkemizin tarihi ile tutarlıdır.
Yazılım içerisinde kullanılan seslendirmelerdeki şive kültürümüzle örtüşüyor.
Yazılım diğer kültürlerden, örnekler ya da müzikler içermektedir.
Yazılım boyunca animasyonlarda aile kavramı hakkında örneklemeler kullanılmıştır.
Yazılım boyunca animasyonlarda toplum içinde nasıl hareket edileceği hakkında örneklemeler
kullanılmıştır.
Bu yazılımı aileme önerebilirim.
Yazılımda kullanılan simgelerden hiçbiri dini sembol içermiyor.
Yazılım içerisinde kullanılan animasyonlarda kullanılan karakterler halktan seçilmiştir.
Yazılımda ülkemizde olmayan bir hayvan kullanılmıştır.
Yazılım içerisinde kullanılacak olan video çekimleri mümkün olduğu kadar dışarıda yaşanılan
mekânlarda çekilmelidir.
Yazılımda ülkemizde olmayan bir bitki kullanılmıştır.
Yazılım çok milliyetçi bir düşünceyle hazırlanmıştır.
Yazılımda yer alan karakterlerden bazıları hep iyi insan olarak değerlendirilirken bazıları ise hep kötü
insan rolündedir.
The differences between genders (Table 2) and classes
(Table 3) in relation to items were statistically checked by
independent samples t-test for each item.
M.
3,82
3,81
3,79
3,77
3,76
S.D.
0,98
1,11
1,00
1,19
1,12
3,75
0,92
3,66
3,60
3,59
3,52
3,50
1,30
1,11
1,30
1,13
0,97
3,46
1,09
3,44
3,41
3,37
3,25
1,19
1,25
1,23
1,14
3,21
1,14
3,15
3,12
1,20
1,26
3,01
1,22
FURTHER STUDIES AND LIMITATIONS
The study is delimited to second and third grade CEIT
preservice teachers. Nevertheless, the sample should be
extended not only to other departments of education
faculties but also to the utilization areas of such software. It
is obvious that the inventory needs modifications and
improvements. Thus, the ideas of teachers, parents, faculty
members and software developers should also gathered in
scientific ways. It is possible to design a qualitative study to
learn why these items are important or unimportant from the
points of stakeholders. Besides, by including enough
number of people as sample, well-planned principal and
confirmatory factor analyses should be conducted to have
more ideas of underlying elements in this inventory.
Table 2. The differences between items in relation to gender
Item No
Gender
N
M
SD
t
p
30
Male
30 4.17 0.69
2.692
0.009
Female
38 3.55 1.08
13
Male
30 3.93 0.91
-2.249 0.029
Female
38 4.39 0.76
39
Male
30 3.20 1.12
-2.087 0.041
Female
38 3.76 1.08
For item 30, males found the item on family-appropriateness
of software more important than females. For both item 13
and 39, females have higher mean scores than males that
females perceived significant for up-to-date and multicultural
examples items to be included in the inventory.
REFERENCES
Deuze, M. (2006). Participation, remediation, bricolage:
considering principal components of a digital
culture. The Information Society, 22, 63–75.
Table 3. The differences between items in relation to
classes
Item No
Class
N
M
SD
t
p
11
Second
46 4.37 0.61
3.098
0.004
Third
22 3.86 0.63
34
Second
46 3.98 1.14
2.791
0.007
Third
22 3.27 0.88
44
Second
46 2.87 1.15
-2.611 0.011
Third
22 3.63 1.36
48
Second
46 3.11 1.30
-2.279 0.029
Third
22 3.91 0.86
Drew, S. & Thorpe, L. (2006). Factors affecting students’
usage and perceptions of a generic intranet
learning resource: models of use. Innovations in
Education and Teaching International, 43(4), 381–
396.
George, D., & Mallery, P. (2001). SPSS for Windows step
by step: A simple guide and reference 10.0
update (3rd ed.). Boston: Allyn and Bacon.
Herring, D. F., Notar, C. E., & Wilson, J.D. (2005).
Multimedia software evaluation form for teachers.
Education, 126(1), 100-111.
Even though there are statistical differences for four items
(Table 3) in regarding to classes, it could be interpreted
sensitively due the large numerical differences between
compared groups (22 versus 46). For the items 11 and 34,
second grades found more significant than third grades to
include items about the cultural enrichment of students
through software and the universality of software elements.
For the items 44 and 48, third grades perceive more
significant to contain items about cultural and nationalistic
elements in software.
Hoise, P., Schibeci, R., & Backhaus, A. (2005). A framework
and checklists for evaluating online learning in
higher education. Assessment & Evaluation in
Higher Education, 30(5), 539-553.
Krishna, S., Sahay, S., & Walsham, G. (2004). Managing
cross-cultural
issues
in
global
software
outsourcing. Communications of the ACM, 47 (4),
62-66.
Lin, H. & Dwyer, F. (2005). The Fingertip effects of
computer-based assessment in education.
TechTrends, 50(6), 27-31.
487