Several approaches for context-aware public display systems exist but none has been able to bridge the gap between the myriad of possible interactive features of a display and adaptation rules for its content. In this paper, we propose a framework of digital footprints generated by the interaction with public displays that can be used as a means to dynamically characterise a place. We describe these footprints, how they can be generated and how they can be used by context-aware display systems to adapt to the social environment of a place.
Keywords: situated displays, digital footprints, context-aware
The overall idea of a context-aware public display that is able to deliver "the right information at the right time" has been pursued for some time, but remains to be realised. Most public displays are not even sensitive to their context. Content is entirely defined and fully controlled by the display owner, who, at best, uses some knowledge about the local place and the intended audience to define what might be interesting content. This, however, is a limited approach because public places are inherently very dynamic and diverse, supporting a broad range of situated practices. If all the decisions must be made a priori, they will not take into account the fluidity and heterogeneity of the social context around the display. The absence of interactive or sensing features also means that there will be no meaningful traces of user activity: their intended use is simply to be seen by people, so it will normally be used without generating any information about how it was used.
Enriching public displays with interaction capabilities provide the obvious path for addressing these two issues. Displays that offer people the possibility to interact can lead to stronger user engagement and possibly user-generated content. They will also be able to produce traces of user activity upon which multiple adaptation processes can be implemented. Multiple variants of this approach have been tried to explore the obvious potential of interactive features in generating activity traces that may support context-aware adaptation. However, success has also been very limited, especially in obtaining results that could be generalized to multiple adaptation processes. Part of the problem may originate from the clear gap between the information generated from interaction events in public displays and adaptation processes. The key problem is that given the broad diversity of interaction modalities and adaptation rules, there is nothing as obvious as a user click that we can immediately take as a concept for linking these two sides.
In this work, we propose a framework for designing context-aware public displays. Our goal is to create a design space that can serve as tool for informing designers of situated displays about the relation between the supported interaction modes, the type of digital footprints they can generate and the type of adaptation processes they may support. We started by analyzing multiple interaction alternatives from the perspective of the information they generate. Rather than considering the specific affordances or semantics of the interactive features offered by the display, we focused on the type of digital trace they generate. We use the concept of digital footprint to refer to the digital traces generated as a side-effect of implicit or explicit interactions with the display, which can be of many different types e.g. keywords, content, presence, indication of external content, feedback on presented content, external personal profiles, or others. Based on their key properties, we aggregated those digital footprints according to 4 main categories: presence, presence self-exposure, content suggestion and actionables, providing a mapping between multiple interaction alternatives and their contribution to the generation of local digital footprints. We then analyse the types of adaptation processes that can be associated with each of those digital footprints, thus providing a mapping from footprints into context-aware adaptation processes. Overall, these mappings provide the framework for reflecting on context-aware behaviours without being caught by the specificities of any particular interaction or sensing mechanisms, thus providing a path for generic context-aware mechanisms.
The potential existence of a very broad range of sensing and interaction mechanisms, with very diverse properties in terms of the digital footprints they can generate represents a major challenge towards a generic model of context-aware displays. To address this issue, we will now analyze the various types of footprint from the perspective of their key properties. This classification is clearly display-centred, in that the categories were defined according to the type of footprint that gets generated at the display, without any consideration for the particular interaction model provided to the user. We have divided the digital footprints into four categories: presence, presence self-exposure, content suggestion and actionables.
Presence corresponds to the ability of the display to collect information about nearby people. There are several levels of presence information that may generate very different digital footprints, more specifically, we consider the following levels: presence detection, presence characterisation and presence identification.
Presence detection is the most basic level of presence information in which the system is simply able to detect whether or not there is someone nearby. Knowing that someone is near a display, even without knowing who or how many, may be used as a simple way to characterise a place, but is most likely to serve as a trigger for specific content on the display to get people's attention and attract them to interact.
Commercial motion or distance sensors can be used for this purpose. In Virtual Kitchen [7], for example, a passive infrared sensor to detect presence in the Kitchen. Presence was used to disable the outside "Off" button that stopped the video streaming if someone wanted more privacy (the button was only available if the no one was in the kitchen already). Distance can also be used by the display. There is usually a strong correlation between distance and awareness level towards the display. In [9], for example, an infrared distance sensor was used to determine the distance of people using a whiteboard application and trigger different interaction modes. Computer vision techniques such as frame differencing to determine movement can also be used for this purpose. In the Aware Community Portals [21] frame differencing was used to detect passers-by and triggered the display to cycle through images of recent stories. Pressure mats [5], usually designed for security applications, can also be used as a presence detection mechanism, for very well-defined and small areas.
The digital footprint generated by these presence detection mechanisms is a presence/absence pattern that may help to characterise the nature of the place in terms of people flow.
The second level of presence information is the ability to characterise presence. This may involve determining how many people are near the display or inferring some type of characteristic about viewers, such as age or gender. Periods of high activity or low activity in a place, or the presence of people with specific characteristics, can all be used to trigger specific content in the display.
Commercial people counters [25] that count the number of people entering/exiting a room can be used by a display system to estimate the number of people nearby. Computer vision techniques such as face detection, gender classification [23] and age classification [10], used by some audience metering products [18], can also be used to characterise and estimate the number of people in front of a display. These audience metering products can deliver reports about the number, attention span, gender and age of the viewers of a particular display.
Presence characterisation generates a richer description of people flow. The display system is able not only to determine periods of presence/absence, it also becomes able to characterise the changes in the number and type of viewers.
Presence identification corresponds to the ability to detect unique identities in the presences. Determining who is present, in the sense that the display system is able to determine that the same person is present in different occasions, gives the display system, not only the possibility to determine how many people are present, but also to establish a correlation between different people or groups of people. This may be achieved through face recognition techniques, but the most common approach is by far the use of some personal device (with Bluetooth or RFID capabilities, for example) as a proxy for the person.
Bluetooth has been used extensively as presence detection mechanism since many people already own a Bluetooth enabled mobile phone. The BluScreen system [20] uses Bluetooth detection to avoid showing advertisements to users more than once. The Cityware project [12] explored several ways in which to analyse Bluetooth mobility traces, including a set of in situ visualizations about Bluetooth presences [11]. These visualisations provide people with information about current or recent Bluetooth presences.
Radio Frequency Identification (RFID) tags can also be used for presence identification. In the IntelliBadge project [4], users participating in a conference were given RFID augmented badges that were used to track them through the conference rooms. A display at the conference cycled through several visualizations of the resulting data. RFID tags have the advantage that they are small and can be incorporated into many existing artifacts. In situations such as a conference, as in the IntelliBadge system, where people are already obliged to wear a badge, this may be a good choice. Bluetooth, on the other hand, is a very widely deployed technology and many mobile-phones are already Bluetooth enabled. This means that it is possible to use the Bluetooth discovery features to detect presence without requiring the user to carry any additional object (as with the RFID tags), as most people already carry a mobile phone regularly. Also, Bluetooth allows the user to manage his presence by turning it on or off at will.
Self-exposure corresponds to the ability of the display to obtain information about the interests, preferences or activities of nearby people. This type of knowledge about the people that use a place may enable the display to adapt itself to their expectations and preferences. For this to happen, users must be willing to let the display system know something about them. This personal information can take many forms: it may be a reference to a user's personal webpage, a set of user associated tags, the username for some social sharing website, a set of interest categories or even personal information, such as age and gender.
The most common approach for supporting presence self-exposure combines presence identification with the a priori definition of a user profile that becomes associated with the identity. This approach was used in the Proactive Displays [13], were users attending a conference registered their affiliation, interests and personal webpage before the conference day and were given RFID augmented conference badges at the conference site. In this system, the user does not have easy access to their information in order to update it which means that they have less control over what information the display system uses in a given moment.
Another way to achieve this self-exposure is to use an information device (e.g. mobile phone) with a custom application that allows users to register a profile. This application can connect automatically, or on demand, to the display system and communicate users' preferences. One example of this is Camera-Phone [22], where a custom mobile application is used to interact with public displays. This application may be configured with personal information that is made automatically available to the display system when a user interacts with the display. One advantage of this approach is that the information is always available to be updated by its owner.
Bluetooth naming, as described in [8], is yet another alternative for managing self-exposure. Naming is explored to allow users to enter predefined commands in their mobile phone Bluetooth name. Since these names can be read by any other Bluetooth device, this can be used to provide an opportunistic interaction mechanism to any user since there is no need to install an application. This approach, however, is less suited for private information since anybody can read the Bluetooth name.
Personal information can also be sent explicitly by the user, using OBEX over Bluetooth, for example, to push a vCard or other structured text file to the display.
The display may offer user the possibility to upload content or references to content for presentations. By suggesting content, users are implicitly saying that such content belongs to that place. This is thus a way for the display system to sense the kind of adequate content for a place. Content may be specified directly or indirectly by the user: by sending the content itself, e.g., a picture, video, text or audio; or by providing a reference to the content (e.g. an URL); whatever the means used to suggest content, the display system will receive or access the content itself and possible meta-data associated with it.
Many display system provide various alternatives for users to send content in order to facilitate content submission. WebWall [6], for example, allowed users to suggest content using SMS, email or a web interface. Plasma Poster [2] is another example of a display system that allows content (photos, text, web pages) submission through two interfaces: email and web form. Web Glance [17], a group web browsing system, also allows several input interfaces to be used: email and instant messaging.
Bluetooth can be used in two ways to send content to a display system: using the standard OBEX protocol or a custom mobile application. Both Hermes [3] and Snap and Grab [15] use the OBEX feature to enable users to send pictures (in the case of Hermes) or any other media type to a display. In both cases, the user just selects the content on his mobile phone, selects the "send via Bluetooth" command and selects a particularly named device.
Bluetooth can also be used by mobile applications to communicate with a display system. The advantage over using just OBEX to transfer files is that a custom application can be built to interact specifically with a given display thus allowing a more rich interaction. OBEX has an obvious advantage over a custom application: it does not need the user to install any additional software on his mobile device and so allow a slightly more opportunistic interaction.
Content suggestion can be used by display systems in many ways, depending on the type of content. However, in most cases the display system will be able to associate, at least, keywords with the content the user submitted (either by gathering them from the content itself or from meta-data).
Actionables detection corresponds to the ability of the display to detect the user reactions to any suggested action. A considerable part of the information shown on public displays is intended to cause people to act [14]. In many cases, the action is completely unrelated with the interaction supported by the display, and there is no way to perceive the efficiency of actionables. However, it is also possible to conceive actionables that are intrinsically linked to the interaction modes supported by the public display, and thus obtain feedback on how they are used. This enables the system to infer interest on the content or services that are being offered.
The concept of actionable is very broad and can take many and very diverse forms. We will explore in more detail the following approaches: content download, content control, rating, voting and classification.
Content download is a way to get a personal record of something that is currently displayed. A user may wish to download an item for various reasons: to keep a permanent record of an item or as a way to inspect an item in more detail if the display only shows a summary, for example.
Content can be downloaded to the user's mobile device if a custom mobile application is provided by the display system that allows browsing and selecting content to download as in Content Cascade [19] or a user can browse for his Bluetooth device to send the selected content item in a touch-screen as in the Hermes Photo Display [3]. A different approach is taken by the Snap and Grab technique [15] where a user can select an item on the public display by taking a picture of it with a camera phone and then send it via Bluetooth (OBEX) to the display. The display system then searches the picture for embedded visual tags that identify the item; if a tag if found, the associated content is sent back (also via Bluetooth) to the users' mobile phone.
By downloading an item the user is implicitly saying that he finds that particular item of some interest, or at least of potential interest.
Content control gives users some type of control over the information being displayed. In a touch-sensitive screen this may result in something very similar to a browsing experience, where the user can navigate through content and fully control the display. Other interaction techniques may offer lighter forms of control such as selecting which video should be presented next from a set of possible alternatives. Other alternatives may include asking for more details or extending the presentation time of an item being displayed, or asking for an item currently not being displayed. If the user asks to skip the current item, the display system can infer that the user does not find that item particularly interesting and instead wants to see what is next. If the display shows a list of scheduled items to appear next and the user is able to skip to a particular item the display system can infer interest on that particular item. Content control can be achieved by any selection mechanism. A touch screen is an obvious choice for this. Both Hermes Photo Display and Plasma Poster [2] use a touch screen interface to let users navigate their content. Jukola [16] also uses a touch screen, but in this case content is indirectly controlled through voting: users of a bar have the possiblity to vote on the next music to be played by selecting a music, among a list. Other selection mechanisms such as the one used by Snap and Grab or Content Cascade could be used for this purpose. Text commands sent by SMS, email or IM, as in the In WebGlance [17] system where users send an email IM message to the display system with a number corresponding to an option on the could also be used.
Content control is also a way to collect users' implicit interest on an item, similarly to what happens with content download.
By rating an item, the user is explicitly saying the he likes or dislikes that item, depending on the value of the rating. This is a way for the display system to allow a user to explicitly indicate his preferences. Rating is found on many websites such as Youtube, Lastfm, Amazon, etc. On public displays, rating can be implemented using any selection mechanism or through text commands.
Displays can also collect users' preferences by crafting polls which allow it to extract information directly or indirectly from an individual. For example, sports preferences of a user can be estimated by asking him to vote on his preferred athlete from a list of athletes from different sports.
As with rating, voting can be accomplished through many different interaction mechanisms. As an example, Bluevote [1] uses images push via Bluetooth. In this case the selection command is a picture sent previously by the display system (by pushing the images to all discoverable Bluetooth devices). Users send back to the system the picture that corresponds to their vote. Bluevote was used in a conference setting to allow participants to vote on the best paper award.
Classification is of a different nature than the previous categories because the result is not a preference but the association of a description or keywords, for example, with a given content item. This can be a less natural action for a user, especially for public items, but it can be provided by displays in a more ludic perspective following the approach of Games With a Purpose [24]. Classification requires that the user is able to send free text to the display system and so requires a text submission mechanism such as SMS, email, IM, Bluetooth names, etc.
The previous sections have highlighted the types of interaction mechanisms that we may need if we want to gather a particular type of footprint. This section will now analyse how those multiple footprints can be used to support various types of context-aware adaptation processes.
Table 1 presents a mapping between the digital footprints and the most widely used interaction or presence mechanisms that generate those footprints. This mapping can be used by situated display designers to help choose the interaction mechanisms provided by the display in order to be able to collect a given set of footprints.
Overall, the entire set of digital footprints constitutes a collection of data which can be used to characterise the place profile, enabling the display system to adapt its behaviour to that particular social setting. Regardless of their generic contribution to this broad adaptation, specific types of footprint can support specific types of adaptive behaviour. Figure 1 summarizes the relationships that can be established between the different footprints and possible adaptation processes.
Table 1 - Mapping between interaction mechanisms and digital footprints.
|
Footprint |
Interaction Mechanism |
| Presence Detection |
. Movement detection (proximity sensor; computer-vision) |
|
Presence Characterisation |
. Face detection with age or gender classification, people counters |
|
Presence Identification |
. Bluetooth |
| Presence Self-exposure |
. Bluetooth (profile on device name; a priori profile definition) . RFID (a priori profile definition) |
|
Suggest Content |
. Email/IM . SMS/MMS . Bluetooth (OBEX; BT Name) |
|
Actionables
|
. Touch screen (Standard GUI controls) . Email/IM (Text commands) . SMS/MMS (Text commands) . Bluetooth (Text commands, e.g. BT naming; Standard GUI mobile application) . RFID (Proximity activation, e.g. Touch & Interact) |
Situated displays cannot rely solely on a static pre-characterisation of the place they were designed to. They must adapt themselves to their changing environment by collecting digital footprints that will help in characterising the social context in which the display is embedded.
In order to be efficient, digital displays need to target their audience's needs, expectations and tastes. By collecting digital footprints of people's interactions, displays can take a step in this direction.
We have presented an interaction design space that defines a mapping between interaction mechanisms and their contribution to the generation of digital footprints with relevance for the characterisation of a place. Each footprint may be used in isolation or in conjunction with other footprints by digital displays to target specific aspects of their audience.
Figure
1 - Possible adaptive processes associated with
different digital footprints.
This work has been supported by "Fundação para a Ciência e Tecnologia" and "Programa Operacional Ciência e Inovação 2010" (POCI 2010), co-funded by the Portuguese Government and European Union by FEDER Program and by "Fundação para a Ciência e Tecnologia" training grant SFRH/BD/47354/2008.
1. Bortenschlager, M. & Rehrl, K. (2007), BlueVote - A Ubiquitous Audience Voting Service, in Adjunct Proc. of the 9th Intl Conference on Ubiquitous Computing UbiComp07.
2. Churchill, E. F.; Nelson, L.; Denoue, L.; Helfman, J. & Murphy, P. (2004), Sharing multimedia content with interactive public displays: a case study, in DIS04: Proc. of the 5th conference on Designing interactive systems, ACM, New York, NY, USA, pp. 7-16.
3. Cheverst, K.; Dix, A.; Fitton, D.; Kray, C.; Rouncefield, M.; Sas, C.; Saslis-Lagoudakis, G. & Sheridan, J. G. (2005), Exploring bluetooth based mobile phone interaction with the hermes photo display, in MobileHCI05: Proceedings of the 7th international conference on Human computer interaction with mobile devices & services, ACM, NY, USA, pp. 47-54.
4. Cox, D.; Kindratenko, V. & Pointer, D. (2003), IntelliBadge : Towards Providing Location-Aware Value-Added Services at Academic Conferences, pp. 264-280.
5. Electronics, A. (2009), Pressure Mats, Website, [Visited April 2009] http://www.arun-electronics.co.uk/pressure_mat.htm.
6. Ferscha, A.; Kathan, G. & Vogl, S. (2002), WebWall - An Architecture for Public Display WWW Services, in The Eleventh International World Wide Web Conference.
7. Jancke, G.; Venolia, G. D.; Grudin, J.; Cadiz, J. J. & Gupta, A. (2001), Linking public spaces: technical and social issues, in CHI01: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, pp. 530-537.
8. Jose, R.; Otero, N.; Izadi, S. & Harper, R. (2008), Instant Places: Using Bluetooth for Situated Interaction in Public Displays, Pervasive Computing, IEEE 7(4), 52-57.
9. Ju, W.; Lee, B. A. & Klemmer, S. R. (2008), Range: exploring implicit interaction through electronic whiteboard design, in CSCW08: Proceedings of the ACM 2008 conference on Computer supported cooperative work, ACM, New York, NY, USA, pp. 17-26.
10. Kwon, Y. H. & Vitoria Lobo, N. d. (1999), Age classification from facial images, Comput. Vis. Image Underst. 74(1), 1-21.
11. Kostakos, V. & O'Neill, E. (2008), Capturing and visualising Bluetooth encounters, in adjunct proceedings of the conference on Human factors in computing systems (CHI 2008).
12. Kostakos, V. & O'Neill, E. (2008), Cityware: Urban Computing to Bridge Online and Real-world Social Networks, in M. Foth, ed., Handbook of Research on Urban Informatics: The Practice and Promise of the Real-Time City, Inf. Science Ref., IGI Global, pp. 195-204.
13. McDonald, D. W.; McCarthy, J. F.; Soroczak, S.; Nguyen, D. H. & Rashid, A. M. (2008), Proactive displays: Supporting awareness in fluid social environments, ACM Trans. Comput.-Hum. Interact. 14(4), 1-31.
14. Müller, J.; Krüger, A. & Kuflik, T. (2007), Maximizing the Utility of Situated Public Displays, in UM 07: Proc. of the 11th Intl Conf. on User Modeling, Springer-Verlag, Berlin, Heidelberg, pp. 395-399.
15. Maunder, A.; Marsden, G. & Harper, R. Creating and sharing multimedia packages using large situated public displays and mobile phones MobileHCI '07: Proc. of the 9th Intl. Conf. on Human computer interaction with mobile devices and services, ACM, 2007, 222-225
16. O'Hara, K.; Lipson, M.; Jansen, M.; Unger, A.; Jeffries, H. & Macer, P. (2004), Jukola: democratic music choice in a public space, in DIS '04: Proceedings of the 5th conference on Designing interactive systems, ACM, New York, NY, USA, pp. 145-154.
17. Paek, T.; Agrawala, M.; Basu, S.; Drucker, S.; Kristjansson, T.; Logan, R.; Toyama, K. & Wilson, A. (2004), Toward universal mobile interaction for shared displays, in CSCW '04: Proc. of the 2004 ACM Conf. on Computer supported cooperative work, ACM, New York, NY, USA, pp. 266-269.
18. Quividi (2009), Quividi - Automated Audience Measurement of Billboards and Out Of Home Digital Media, Website, [Visited April 2009] http://www.quividi.com.
19. Raj, H.; Gossweiler, R. & Milojicic, D. (2004), ContentCascade incremental content exchange between public displays and personal devices, The First Annual Intl Conf. on Mobile and Ubiquitous Systems: Networking and Services, 2004, 374-381.
20. Sharifi, M.; Payne, T. & David, E. (2006), Public Display Advertising Based on Bluetooth Device Presence. Mobile Interaction with the Real World (MIRW) in conjunction with the 8th Intl Conf. on Human-Comp. Interaction with Mobile Devices and Services, 2006
21. Sawhney, N.; Wheeler, S. & Schmandt, C. (2001), Aware Community Portals: Shared Information Appliances for Transitional Spaces, Personal Ubiquitous Comp. 5(1), 66-70.
22. Toye, E.; Madhavapeddy, A.; Sharp, R.; Scott, D.; Blackwell, A. & Upton, E. (2004), Using camera-phones to interact with context-aware mobile services, Technical report, University of Cambridge, Computer Laboratory.
23. Verschae, R.; Ruiz-del-Solar, J. & Correa, M. (2008), A unified learning framework for object detection and classification using nested cascades of boosted classifiers, Mach. Vision Appl. 19(2), 85-103.
24. von Ahn, L.; Dabbish, L. Designing games with a purpose, Comm. ACM, 2008, 51, 58-67
25. Wikipedia (2009), 'People counter - Wikipedia, The Free Encyclopedia', [Online; accessed 6-April-2009].
Published originally in the 5th International Workshop on Context Aware Mobile Systems (CAMS2009):
bibtex
Send comments to: 
December 4, 2009