The Integrated Multimedia Terminal

Georg Strøm
Lejre, Denmark
The terminal is already here, but it is not integrated.


Users are often forced to operate communication equipment at home or at work, while keeping their attention on the message or on their surroundings. When that is the case, they have to be inattentive to the terminal. Such inattentive operation will become increasingly more difficult when traditional voice only phones are replaced by more complex multimedia terminals.

This paper analyses the requirements for an integrated multimedia terminal for inattentive use.
Inattentive use is only possible, when a number of specific requirements are fulfilled. These are described, together with one function set covering all message formats employed in a multimedia terminal.
Three existing terminals are compared with the requirements for inattentive use: An electronic mail system and two combinations of telephone/voice communication and other media's.
Finally, a description of an integrated multimedia terminal is presented. It is a generalised terminal, where the principles may be adapted for different realisations, e.g. a hand portable device or as part of a work station. A structure for the states and functions in the terminal and a suggested dialogue style for inattentive use is described.


Most users require that the multimedia terminal is as easy to use and understand as an ordinary voice only telephone. The users may use the equipment while consulting notes or while keeping their eyes on "Calvin and Hobbes" [Watterson, 88] or in other situations where their attention cannot be on the operation of the equipment. Even though the users want new media's and to some extent new functions adding to the complexity of the terminal, the operation of the terminal must still appear simple. The terminal shall be used by the entire household, it must not be reduced to a challenge for the mechanically inclined, as the home computer to a large extent is today.

In addition, an increasing amount of the communication equipment will be hand portable, and the users will expect the same level of facilities in pocket size portable equipment as in equipment connected to a fixed telephone network [Stroem 94], even though the user interface for portable terminals cannot employ the same standards as used for computers, e.g. it is not possible to equip a hand portable device with a full size VDU.

Finally, the market for communication equipment is changing. Formerly all equipment was for professional use, during the last few years a market for private use has started to grow [Berlingske a]:

* The prices have steadily declined [Berlingske b & c]
* The full extent of the communication systems are being utilised, making it possible for ordinary telephone networks or cellular systems to offer most of the services and message types available in the ISDN digital telephone system. The consumers will therefore be able to get the ISDN facilities without paying the higher subscription rates for an ISDN connection [Acerfax 93a & Nokia 94]
* The increasing number of people who feel a need to be able to work from home or during holidays [observation].

The consumer market may end up being at least as large as the professional market.

Inattentive use

In the first part of my project the consequences of inattentive use has been investigated, through experiments, observations and by the use of available literature [Eyseneck 90, James 83, Norman 90, Suchman 91 and other].

The user may be more or less inattentive to the actual operation of the terminal, i.e. some users may concentrate on the terminal while occasionally looking on the surroundings, other users may perform an automatic process while being unaware of the individual actions they are performing. When the user has acquired some experience, operating the terminal as an automatic inattentive process may be the rule rather than the exception.

The mental range of the user will be significantly reduced when the user cannot allocate his or her full attention to the operation of the terminal:

* It is not always possible for the user to remember the state of the terminal or the last action that has been performed. Even in the best case such information does not stay more than app. 5 sec. in the consciousness, and it is normally lost if the consciousness is focused elsewhere even for a brief moment.
* It shall be possible for the user to operate the terminal by recognising the appropriate procedures on the user interface, since it is more difficult to recall the procedures than to recognise them.
* The user is incapable of planning the total operation. The steps in the operation will be decided one at a time, i.e. one step will be decided and then carried out. In some cases the user will even employ a trial and error principle, testing the different possibilities and accepting a high percentage of errors. The user may find it easier to try a number of possible operations and recognise the results, than to consider which of the possible operations that may lead to the desired result.
* It is likely that the user only will take note of part of the information on the user interface, a large proportion of the information will not be noticed.
* It is not possible to perform precise operations requiring visual feedback, these are only possible with a fairly high degree of attention.

The remaining part of this paper describes the tentative results of the second part of my project.

Requirements for the terminal

The user interface for inattentive use shall fulfil the following requirements:

* Hidden states are not allowed, i.e., the state of the terminal shall always be visible for the user.
* A user interface operated by entering "codes" is not applicable, since the user has to remember the appropriate codes.
* The operation of the terminal shall be fault tolerant, i.e. in most cases it shall be possible for the user to undo the last step. When that is not possible, proper safeguards against any damaging operations must be taken. It shall be possible for the user to feel confident that any errors made during the operation easily can be corrected.
* The highest possible percentage of available operations shall lead towards the results that the user wants to accomplish. The terminal shall maximise the number of "right" choices, while minimising the number of "wrong" ones.
* The number of steps taken to achieve a given result shall be minimised, more steps means a slower operation and a higher number of errors.
* The different states shall be operated in a uniform manner, in order to make it easier for the user to decide the next step in the operation.
* The user interface shall be as simple as possible, i.e. superfluous information shall be removed.
* It is not possible to use a pen based user interface, and a mouse or track ball interface can only be used for table mounted equipment and it shall operate with simplified movements and positioning and with a fairly large margin of error.

The specification of the user interface shall finally be reviewed and debugged in the same manner as a computer program. The operation of the terminal will be significantly more difficult, even when only a small fraction of the user interface does not fulfil the requirements. Similar to other system design, the devil rests in the details.

Necessary functions

The formats available for the transmission of messages are defined as bearer services providing a transparent channel [CCITT 89b] or message transfer services with special format and control functions [CCITT 89b]. The following message formats are employed today:

* Voice messages with a bandwidth on 3.1 kHz. Limitations in the speech coding may prohibit the transmission of arbitrary sound in the voice frequency range even if the bandwidth on 3.1 kHz is not exceeded [CCITT 89a].
* Letters, i.e. alphanumeric and some formatting characters, as for instance teletex [CCITT 89a].
* Fax: digital group 3 fax [Acerfax 93a & b] or group 4 fax [CCITT 89a] for transmission of copies of documents.
* Forms, containing information to be processed automatically when received, e.g. ticket ordering.
* Files, i.e. a general data format, with a header describing the format.
* Vidoetex [CCITT 88] or other picture transmission capabilities.

The message may be encoded and send either in real time or with the use of one or more storage's. The "plain old telephone" or a teletype terminal is an example of the former, fax or an electronic mail system is an example of the latter.

The following is based on the function sets available in a number of communication systems [CCITT 89a & Stroem 94]. In order to use a uniform nomenclature, the transfer of a message will throughout be termed a "Call", as common in telephone technology. A call consists of a main message, and may in addition include one or more appendices, i.e. additional stored messages that may be in the same or a different message format than the main message. Each call may have one or more receivers, it may even be envisaged to have "cc's" on a call, these may receive the message but cannot reply during the call [after CCITT 89b].

Apart from the actual content of the messages, the following signals may be received from other terminals:

* Incoming call.
* Termination of call.
* Call waiting, while another one is being attended to.

The functions fall into 3 groups:

* Communication functions to be used when setting up or receiving a call.
* Routing functions for regulating the transfer of the messages.
* Support functions that may be specific for either the terminal or the system, i.e. encoding of short numbers or abbreviations, control of power supply, printer and connections.

Communication functions:

* Selection of message format, including selection between real time or stored message, for encoding a main message or an appendix.
* Encoding of a message when it is not done in some application program.
* Selection of main message when the message already is encoded.
* Selection of an appendix when the appendix already is encoded.
* Finding the address of the receiver, or receivers.
* Selecting the address of the receiver or receivers.
* Terminating a call.
* Reply to real time message.
* Reply to stored message.
* Call waiting, responding to a new incoming call while being in the middle of another call, and maybe doing a transfer to the waiting call.
* Call transfer, transferring an active call to another receiver.
Routing functions:
* Store and forward, i.e. store the message and send it on a pre-coded time.
* Call forwarding, re-routing all calls to another address or an automatic answering service.
* Call barring, blocking for calls to or from certain other addresses.

Support functions:

* Power on and off.
* Password or access control
* Encoding of new receivers.
* Power supply control and adjustment.
* Printout function.
* Connection control and adjustment.
* User interface settings.

The use of different function sets for different message formats would make the terminal significantly more difficult to operate, in particular if the user makes a mixed mode call [CCITT 89a] combining teletex and group 4 fax or combines other messages with different formats in the same call or switches from one message format to another during the call. In both the existing systems and in some of the new emerging systems [CCITT 89a] a number of restrictions are made on the allowed combinations of functions and message formats. However, a function set similar to the one above may be used for all message formats.

Attempted solutions

The following types of equipment has been investigated:

* Electronic mail: DSI-MAIL operating on a PC [DSI 94].
* Cellular (GSM) telephone: Mobira Cityman 2000, a pocket size phone, app. weight 500 gr. [Mobira 93].
* Telephone combined with fax: Acerfax F-22, a desk top unit app. 40 x 20 x 20 cm [Acerfax 93.b].

A sample of each type of equipment has been available for analyses, and each may be considered state of the art within its own area.

In addition to various tones used as alerts or warnings, the user interfaces has the following facilities for presenting information:

* Electronic mail: Full size colour screen of 400 x 640 points.
* Cellular (GSM) telephone: 2 lines each of 8 dot-matrix characters.
* Telephone combined with fax: 1 line of 16 dot matrix characters.

The differences are due to the physical restrictions posed on each type of equipment.

Several dialogue styles or means of entering of information are used. They are listed here with some remarks on their operation [after Mayhew 92]:

* Menu: one or more items may be selected in a list, i.e. the menu. Unless visible "shortcut" buttons are available, the user will have to perform app. twice the number of steps needed by any other dialogue style before the result is reached.
* Speech: explanation superfluous. This mean of entering information is probably the best for free style information. However, it is not suited for entering commands for the system, since a restricted artificial language has to be recalled and used, and since it often will be more error prone and slower than the other means of entering information.
* Free text that may be entered as a string without any restrictions on format.
* Fill in forms: one or more fields are filled in a form with text or marks. This dialogue style is one of the best for inattentive use, as long as the move from one field to the next is done in a fault tolerant and immediately understandable manner.
* Command language: one or more preselected commands are entered combined with numerical codes or other parameters. This dialogue style is not suited for inattentive use, since the user has to recall the appropriate codes and in some cases also to remember the already performed steps in the operation.
* Function buttons: one button activates one function. This dialogue style is well suited for inattentive use, but due to physical restrictions only a limited number of functions can be allocated their own button.

					Electronic mail		Cellular (GSM) 		Telephone 
								telephone 		comb. w. fax 
Communication functions:
Selection of message format.		NA			Menu			Menu

Encoding of a message. Fill in forms Speech/free text Speech

Selection of main message. Fill in forms Menu NA

Selection of an appendix. Fill in forms NA Func. buttons

Finding receivers addressees. Fill in forms Menu HW labels **

Selecting receivers addressees. Menu Command language Comm.lang.

Reply to real time message. NA Command language Comm.lang.

Reply to stored message. Function buttons Function buttons Func. buttons

Call waiting. Menu Menu/Comm.lang. Comm.lang.

Call transfer. - - -

Routing functions:

Store and forward. Menu Menu/Comm.lang. Comm.lang.

Call forwarding. Fill in forms Menu/Comm.lang. Comm.lang.

Call barring. Fill in forms Menu/Comm.lang. Comm.lang.

Support functions:

Power on/off Function buttons Function buttons Func. buttons

Password or access control Fill in forms Command language NA

Encoding of new receivers. Fill in forms Command language Comm.lang.

User interface control Function buttons & Menu NA

and adjustment. Menu

Power supply control NA Menu NA

and adjustment.

Printout function. Fill in forms NA Comm.lang.

Connection control and adjustment. Fill in forms Menu NA

When the Not Applicable or not implemented functions are excluded, 12 of the 19 types of functions use a different dialogue style for the same function type. These differences cannot be explained solely as due to different requirements or physical restrictions for the equipment.

The following characteristics of each type of equipment can be identified in relation to the requirements for inattentive use:

* Electronic mail: this system demands a confirmation for almost every time a window in the display is closed, adding app. 25 % to the number of steps needed for each operation. This may be due to the electronic mail system being developed from a word processing program. In that program it may be necessary to be confirmed that the work on a particular document has been completed and the document is ready for storage. In addition this system employs 2 parallel systems for electronic mail, one for letters and one for messages or notes, with almost similar functions in the 2 systems. This concept is probably taken directly from a paper-based office where letters are used for official writings and small "stick-it" messages for internal reminders or notes.

* Cellular (GSM) telephone: This telephone uses a special command language of the type "* digits * digits #" for part of the operations, probably because the numbers 0-9, * and # were the only buttons available on older phones. App. 6 % of the states are hidden; this may seem a low value, but since most operations involve a number of steps, a significant part of the operations are affected. At the same time it is only possible to undo 40 % of the possible operations. For that reason the user may feel a need to concentrate deeply on the operation of the terminal. Both the occurrence of hidden states and the many steps that cannot be undone are common in telephone systems where the functions are implemented in the exchange, making a detailed feedback to the user or an undo function impossible.

* Telephone combined w. fax: This terminal uses a command language without any prompts for most of the functions, making it necessary for the user to refer frequently to the manual and to focus the attention on the terminal and manual when one of the functions are operated. This mode of operation may be due to limitations posed by older phones or faxes. An overview over the states in the terminal is shown in fig. 1. The overview shows 24 different states. In comparison, a simple voice only telephone may be implemented with as little as 5 states. The overview may also give the impression that the states and functions has grown gradually and rather freely in a number of different directions.

The use of different dialogue styles for the same functions cannot be explained as due to different functional requirements or different limitations on the equipment. In addition, each type of integrated terminal contains a number of less functional details that clearly are relics from earlier types of equipment, see fig. 2.

A truly integrated terminal

A truly integrated terminal can be designed with a more uniform mode of operation:

* No matter the media of communication, the functions and their operation shall be the same
* The operations shall be the same whether or not the telephone has an active call or not. That will make the inclusion of appendices the same before and in the call, and it will make it possible to attend to a waiting call in precisely the same manner as when an incoming call is attended to or a new call made. In addition, the introduction of package switched communication [CCITT 89a] will make it difficult for the user to decide whether the terminal is conducting a call or standing idle.
* In order to reduce the number of steps needed for making a call, there shall be no separate "un-hook" or Start-Of-Call function. Even though a suitable "Enter" function may be needed after entering names of messages or receivers where the length of the name may differ, any additional confirmation needed when the message and receivers has been selected shall be optional for the user and not a requirement for the system.
* It shall be possible to select receivers and messages in any order and to select additional appendices while conducting a call, in order to optimise the number of valid operations and reducing the number of invalid ones.
* It shall be possible to select the message and appendices and the list of receivers by filling them into a stack. It is thus not necessary to use one function for selecting the message, and one for selecting the appendices, and it may even be possible to have similar functions for selecting messages/appendices and receivers.
* The selection of message formats and the choice between real time and stored messages shall either be determined by the format of the selected message or selected automatically based on information about the capabilities of the receiver.
If the simplifications above are made, a terminal with the full set of media's and functions only needs the main states shown in fig. 3. The number of states are reduced from 23 to 10 compared with the terminal in fig. 1. At the same time, the number of connections between the states are raised from 1.5 connections/state to 2.1 connections/state, and the number of choices that cannot be undone in one operation is reduced from 9 to 3. These changes makes it easier for the user to browse in the functions, progressing step by step while circumventing or undoing mistakes. The data flows in the integrated terminal may be as complex as in the terminal in fig. 1 [after Hatley 88]. However, the control functions and states in the integrated terminal is comparatively more simple than in the terminal shown in fig. 1. The integrated terminal may fulfil the requirements for inattentive use, my own experience shows that the terminal in fig. 1 definitely does not.
The terminals for inattentive use may be implemented in desk top computers, cellular telephones or telephones for fixed installation. The terminals may in some cases employ only a subset of the functions for communication and routing, and the set of support functions may differ from terminal to terminal. Even then, it is still be possible to give the different terminals a uniform structure and mode of operation.

It is likely that the best dialogue style for inattentive use may be based on "soft keys" with keys implemented in hardware and an electronically displayed text at each key showing its function in the actual moment. Although soft keys are not suited for entering written information, they may be the best choice for fulfilling the requirements for inattentive use:

* The actual state is at all times visible for the user.
* The user does not have to recall the operation, only to recognise the texts at the softkeys.
* It is easy to group the steps in a 2-dimensional structure, making both backstepping and side-stepping possible after an error, while minimising the number of steps needed for achieving a given result, i.e. the number of steps may be below 50 % of the ones needed in a menu based dialogue style.
* The softkeys may operate in a uniform manner, no matter the state of the terminal.
* Each state only has to show the information needed for the next step, reducing the amount of superfluous information.
A number of specific problems still remain to be solved:
* Finding a method for selection of messages and receivers that fulfils the requirements for inattentive use.
* Finding a method for entering written information that fulfils the requirements for inattentive use of hand portable equipment.
* Defining the level of performance actually expected by the users. Since the ordinary voice-only telephone may be expected to represent that level, the level may be defined by investigating the degree of attention, speed of operation and percentage of errors while operating the telephone.
* Field tests of prototype terminals. The human mind is extremely complex, for that reason it is necessary to do an extensive testing in a realistic setting employing a wide range of potential users.


It is likely that the telephone in the home will be replaced by multimedia communication terminals, integrating voice messages, letters, fax, and the transmission of files or filled out forms.

The multimedia terminal must be as easy to use as an ordinary telephone, in particular it shall be possible for the user to operate it while focusing his or her attention elsewhere. It is possible, that sort of inattentive use is the rule rather than the exception.

Three already available types of multimedia terminals are examined: an electronic mail system, a cellular GSM phone and a combine telephone and telefax. The terminals are more products of a gradual development than of an analysis of the user requirements, and they are difficult to use when the user has to focus the attention on other tasks than the actual operation of the terminal.

It is shown, that an integrated terminal with a more uniform operation of all functions, may be better suited to inattentive use. In particular the number of necessary states of the terminal may be reduced to app. 50 % while the average number of connections to or from a state may be increased 33 %. It is finally suggested that a dialogue style with so-called "soft keys" may be better suited for a terminal for inattentive use.

A number of areas still require further studies, e.g. elegant solutions for a number of details in the implementation, a definition of the ease-of-use level expected by the users, and testing of prototype terminals.


Acerfax F-26 PC link fax og scanner, Multinet Danmark 1993 [Acerfax 93a]
Acerfax F-22 facsimile machine
- users manual, Acer peripherals inc., Taiwan 1993 [Acerfax 93b].
Berlingske Tidende:
19.4.94 [Berlingske a], Interview Keld Balle Mortensen, Tele Danmark Mobil, September 93 [Berlingske b], 3.11.93 [Berlingske c]
Brugervejledning til DSI-MAIL ver. 2.2
, Dansk system Industri, 1994 [DSI 94]
Blue book volume III facile III.7,
Integrated services digital network (ISDN) General structure and service capabilities, CCITT 1989 [CCITT 89a]
Blue book volume VII facicle VII.5,
Terminal equipment and protocols for telematic services, CCITT 1988 [CCITT 88]
Blue book volume VIII facicle VIII.7,
Data communication networks message handling systems, CCITT 1989 [CCITT 89b]
, Michael W.: Cognitive Psychology, Lawrence Erlbaum associates, 1990 [Eyseneck 90].
, Derek J.: Strategies for real time system specification, Dorset House Publishing, 1988 [Hatley 88]
, William: The principles of Psychology, Harvard University Press 1983 [James 83].
, Deborah J.: Principles and guidelines in software user interface design, Prentice Hall, 1992 [Mayhew 92].
Mobira cityman 2000,
Nokia Mobile Phones, 1993 [Mobira 93].
Nokia cellular data card
, Nokia Mobile Phones, 1994 [Nokia 94].
, Donald A.: Design of everyday things, Doubleday, 1990 [Norman 90].
, Georg: Mobilguiden, Nokia Mobile Phones, 1994 [Stroem 94]
, Lucy A.: Plans and situated actions, Cambridge University Press, 1991 [Suchman 91].
, Bill: Calvin and Hobbes, Sphere Books, 1988 [Watterson 88] (Popular cartoon about a creative and difficult to control kid)
Based on an informal statistic from the area where I live, app. 40 % has during the last 1 or 2 years acquired fax, modems or other communication equipment. [observation]

Return to HOIT 94 Proceedings