A Research on Visualization of Finger Spelling in Sign Animation
Tomohiro KURODA and Kunihiro CHIHARA
Graduate School of Information Science, Nara Institute of Science and Technology
8916-5, Takayama, Ikoma, Nara, 630-0101, Japan
{tomo,chihara}@is.aist-nara.ac.jp
ABSTRACT
Hearing disabled people have the communication barrier because they have no way to access phonetic languages. In order to get rid of this barrier, media between phonetic languages and sign languages are indispensable. For this purpose, a new research criterion called Sign Linguistic Engineering emerged from the intermediate of Linguistics, Pedagogy, Information Science and so on. Many foregoing studies on sign linguistic engineering utilize CG animation to show signs. A sign animation should present whole upper-body motions for signs and small hand motions for finger spellings clearly at the same time. This paper presents a method to give legible finger characters in sign animations.
Keywords: Sign Animation, Avatar Based Communication, Sign Language, Finger Spelling, Human Interface, Legibility
1. Introduction
Today, many researches on assistive technology, which give methods to overcome the social barriers for the disabled people, are coming. The barrier of the hearing impaired is the communication barrier; the hearing impaired has no way to access phonetic languages and the non-deaf has difficulties to understand the mother tongues of the deaf, that is sign languages. The new research field named sign linguistic engineering aims to develop media between the phonetic and sign languages unifying the knowledge of the linguistics, the information science and so on [1].
One of the important problems of the sign linguistic engineering is the method to visualize sign language. Many foregoing sign linguistic engineering studies utilize computer graphics (CG) animation to show signs. A sign animation should presents whole upper-body motions for signs and small hand motions for finger spelling clearly at the same time. This paper proposes a new method to give legible finger characters in sign animations.
2. Sign Animation
A sign animation is a stream of sign given by the virtual human given by computer graphics called CG-avatar. Most of foregoing researches utilize sign animation to show signs instead of video. Because A sign animation is easy to compose smooth sign conversation from elemental data of sign words, and because a video needs much more storage space than a sign animation.
The sign animation can be classified into two types; the notation-based animation and the motion-based animation. The notation-based animation is a sign animation produced from certain notation code [2]. This type animation is suitable for the systems, which should produce a sign animation based on some certain rules, such as computer aided sign education systems [3] and automatic interpretation system between a sign language and a phonetic language [4]. The motion-based animation is a sign animation produced from captured motion data by a certain motion capture system [5]. This type animation is suitable for the systems, which should visualize real sign motion, such as sign language telephones [6].
Most of CG-avatars for sign animations are in the shape of standard human body. However, finger characters are illegible when whole signing space [7], which is the space where signs are given, is shown on a screen. Because the size of CG-avatar's hand is only a few percent of whole screen. Kurokawa [2] utilize CG-avatar, whose hands are 1.4 times wider to increase the legibility of finger characters (Figure 1). However, signs given by this avatar are less illegible, because the wide hands break the relations between the hands. The notation-based animation can avoid this relationship problem by editing motion parameters to fit them to the avatar. However, this problem is fatal for the motion-based animation, especially for the on-fly animation. Moreover, sign readers feel smashed by the avatar when the avatar stretches his hand to the reader.
Figure 1. Kurokawa's avatar
3. On-demand expansion of the avatar's hand
As discussed in section 2, in order to give legible finger characters, an avatar's hand should be 1.5 times wider. However, the wide hands make signs illegible by destroying the relationship between the hands.
Kamei et al. [8] claims that the view center of reader moves from its default position, face, to the strong hand, by which finger characters are given while a signer spells. This discussion gives a good suggestion for better sign animation. An avatar's strong hand should be expanded only when the avatar spells. The expanded strong hand gives legible finger characters and the relationship problem won't occur as the size of the hand is as same as standard human body when the avatar speaks in signs.
Therefore, this paper proposes the method to expand the avatar's strong hand while the avatar spells. Section 3.1 explains the method to distinguish the period when the avatar spells, and section 3.2 gives the method to expand the avatar's hand.
3.1 Finger character detection
Kimura et al. [9] says that finger characters are shown in the limited space in front of the shoulder of the strong hand side. Therefore, the spelling period can be defined as the period when the strong hand stays in the finger spelling space, where the finger character is shown.
In order to define the finger spelling space, the authors measured finger spelling spaces using the motion capture system for sign language developed by the authors [5] as shown in Figure 2. The result shows the finger spelling space is in the spherical space. The center of the sphere is 40cm from the shoulder of the strong hand side to the front, and the diameter is 40cm.
Figure 2. An example of finger spelling space
The authors also measured how long the strong hand stays in the finger spelling space. The result clears that the strong hand stays the finger spelling space at least one second during spelling. Additionally, the result clears that the strong hand sometimes go out of the finger spelling space while the avatar spells continuously. Therefore, the spelling period is defined when the strong hand stays in the finger spelling space more than 80 percent of the preceding one second.
At last, the accuracy of this detection is experimented by 9 sign sentences. The result is shown in Figure 3. In this figure, the right circle shows the frames of the spelling period detected by the testees eyes, and the left circle shows the ones detected by this method. This result shows that the 90 percent of finger characters are correctly detected. Most of the spelling period, which isn't detected by the method, is the beginning 0.5 seconds of the spelling period. This error could be caused by the detection procedure, which needs, at least, preceding 0.8 seconds. In addition, the result shows 5% of other signs are misunderstood as finger characters. These are the case when the signer stops to move between two topics.
Figure 3. Accuracy of finger spelling detection
3.2 The strong hand expansion
Along the Kurokawa's method, the authors decided to expand the avatar's strong hand 1.5 times wider. Actually, the avatar has the two strong hands in different size, and the avatar switches the strong hand to the big size one during the spelling period. Figure 4 shows the avatar, which shows the Japanese finger character 'TO' with the normal size hand, and Figure 5 shows the avatar, which shows the same character by the big size hand.
Figure 4. A normal avatar showing finger character 'TO'
Figure 5. An avatar showing finger character 'TO' by expanded hand
4. Experiment
In order to evaluate the effectiveness of the proposed method, the produced sign animation is experimented.
A signer spoke following three sentences in signs and the given signs were captured by the motion capture system for sign language. The capitalized characters indicates the finger characters.
Two types of sign animations were produced from these captured data. Type A utilizes a normal size avatar, and type B utilizes the proposed method. One sentence was given four times (two As and two Bs) to testees in random order.
Testees stood 1m away from 14 inches TV display, and testees tried to read the given sign animations. The testees were four deaf, two sign experts, three sign intermediates and two sign beginners.
The experimental results are shown in Table 1 and 2. Here, the improvement is defined as Formula 1 and the units are percent.
Formula 1. The improvement of legibility
Table 1. The improvements on reading whole skit
|
Deaf |
Experts |
Intermediates |
Beginners |
|
|
Skit 1 |
33 |
9 |
11 |
11 |
|
Skit 2 |
31 |
14 |
14 |
7 |
|
Skit 3 |
29 |
16 |
45 |
36 |
Table 2. The improvements on reading finger characters
|
Deaf |
Experts |
Intermediates |
Beginners |
|
|
Skit 1 |
40 |
11 |
4 |
8 |
|
Skit 2 |
37 |
17 |
19 |
8 |
|
Skit 3 |
32 |
18 |
42 |
40 |
This experimental results show that the proposed method realizes 25% improvement of the legibility of finger characters. Especially, the deaf people show the high improvement. Ordinary, the deaf has some difficulties to read finger characters, because the deaf doesn't use finger characters in their sign conversation so often. These results show the effectiveness of the proposed method.
Moreover, most of testees didn't notice the fact that the strong hand is expanded while the avatar spells. This fact shows that the sign animation produced by the proposed method gives no unpleasant feelings or visual mismatch for readers.
5. Conclusion
This paper presents a new method to give legible finger characters in sign animations. This method expands the avatar's strong hand while the avatar spells. The experimental results show that the proposed method increase the legibility of the sign animation without visual mismatch.
Acknowledgement
This research is partly funded by the Telecommunications Advancement Foundation. We thank the Kamigyo branch of Kyoto City Sign Language Club "MIMIZUKU", and the Kamigyo branch of Kyoto City Deaf Association for their comments and support on the experiments.
Bibliography