paper: The Personal Server: changing the way we think about ubiquitous computing

roy want et al

improve the mobile digital experience by annexing large-screen interfaces in environment

Tech: bluetooth for wireless commination. UPnP for discovery

REFERENCES:

Bluetooth, UPnP, Personal Area Networks

paper: XICE Windowing Toolkit: Seamless Display Annexation

Richard ARTHUR and Dan R. OLSEN

framework implementation that uses wireless networks to connect portable devices to display servers

focus on sparing mobile device for CPU usage and power consumption, as well as data protection

alloes user to annex screen only, or screen and input

allows multiple users to annex same display

based on scene-graphs (like WPF)

comparison of existing UI distribution technologies (incl. VNC)

does not implement discovery protocol (manual configuration)

VERY GOOD REFERENCES!!!

Implementation status

1) Setup

VNC

Done. With problems. Veency runs on iPhone but has serious lag. VncSharp is used as VNC client integrated to MS application.

DEVICE DETECTION

Done. Camera-based using OpenCV. Black iPhone body is invisible, so I use available features (apple logo and camera point) to detect the iPhone. Detection is obviously device specific, but I implemented a generic framework that can accomodate other types of devices.

DISCOVERY PROTOCOL

Cancelled. Require implementing a daemon on device that can detect when it is laid upon a table, and launch a discovery process. That implementation revealed to be too complicated on the iPhone. But solutions already exist.

CONNECTION

Done. Wireless on local network. As discovery is not available, server IP can either be hard-coded or manually entered by user.

2) Device remote interaction

INPUT

In Progress. Mapping is fixed, but having troubles detecting all types of touch event on the surface user control.

OUTPUT

Done. Though with serious delay due to VNC third part implementation and/or other unknown issues.

3) Application window interaction

I still need to finalize the analysis of the preliminary user study results, to find out which interaction techniques I will implement. Below are the ones that are already implemented.

DRAG

Done. With iphone body/active border. Feature is inherited from MS ScatterViewItem.

ROTATE

Done. With iphone body/active border. Feature is inherited from MS ScatterViewItem.

RESIZE

Done. With iphone body/active border. Feature is inherited from MS ScatterViewItem.

MINIMIZE

To do. Wish to implement active border on MS where window can be dragged to minimize.

HIDE

To do.

EXIT

Done. With specific button. Wish to implement active corners on MS where window can be dragged to stop application.

HOME

Done. Home button on iPhone body/active border is active.

Interaction strategies and primitives

We identify 6 interaction strategies for manipulating the application window:

1. Action tabs: menu tabs along the lower edge of the window.
2. Action bar: an action bar (including a manipulation area and buttons) along the lower edge of the window.
3. Active/Inactive window: the window has a switch and can be made inactive and thus manipulable as a common digital picture.
4. Active borders: the window has active borders (including the corners) that can be used for manipulation.
5. Active borders and corners: the window has both active borders and corners (different effect) that can be used for manipulation.
6. Other suggestions

We identify 6 interaction primitives:

1. Dragging
2. Rotating
3. Resizing
4. Minimizing
5. Hiding
6. Exiting

Concerns:

• PAIRING:  security (dialog on phone) & obtrusiveness (dialog on phone first)
• RESTORING has been removed from original primitives, because restoring can only be performed when the application is minimized as an icon, and the icon will not offer the same interaction possibilities than the full sized window, so I see only the obvious solution of tapping the icon to restore the window to its previous size.
• In some cases it is necessary to choose from several design suggestions for the same action/strategy (see Minimizing)
• For Hiding and Exiting, the design suggestions seem ‘forced’. The obvious solution (such as placing full hand on window to hide it) do not relate to any particular strategy. Some strategies do not offer any design suggestions (such as Exiting using active borders).

For each interaction primitive we define design possibilities following each of the strategies:

Dragging

1. By performing a one finger dragging gesture on a specific action tab.
2. By performing a one finger dragging gesture on the action bar.
3. By tapping a tab to render the window inactive, then performing a one finger dragging gesture anywhere on the window.
4. By performing a one finger dragging gesture on the active border (incl. corners) of the window.
5. By performing a one finger dragging gesture on the active border (excl. corners) of the window.
6. By holding a finger on a specific tab, and using another finger to tap a destination target to move the window to.

Rotating

1. By performing a two finger touch rotating gesture with one finger placed on a specific tab, and the other anywhere on the window.
2. By performing a two finger touch rotating gesture on the action bar.
3. By tapping a tab to render the window inactive, then performing a two finger touch rotating gesture anywhere on the window.
4. By performing a two finger touch rotating gesture on the active border (incl. corners).
5. By performing a two finger touch rotating gesture on the active border (excl. corners).
6. By performing a one finger dragging gesture on a corner of the window.

Resizing

1. By performing a one finger dragging gesture on a specific tab.
2. By performing a two finger pinching gesture on the action bar.
3. By tapping a tab to render the window inactive, then performing a two finger pinching gesture anywhere on the window.
4. By performing a two finger pinching gesture on the active border (incl. corners).
5. By performing a one finger dragging gesture on an active corner.
6. By pulling the window apart with both whole hands.

Minimizing

1. By tapping a specific tab.
2. By using Resizing on the action bar to reduce the window until it snaps to icon shape OR double tapping the action bar.
3. By tapping a tab to render the window inactive, then performing a specific gesture anywhere on the window.
4. By using Resizing on the active border to reduce the window until it snaps to icon shape OR double tapping the active border.
5. By using Resizing on an active corner to reduce the window until it snaps to icon shape OR double tapping the active corner.
6. By dragging the window to the bottom of the surface.

Restoring the window will be achieved by simply tapping the minimized icon.

Hiding

1. By tapping a specific tab.
2. By performing a specific gesture on the action bar.
3. By tapping a tab to render the window inactive.
4. By double tapping the active border.
5. By double tapping an active corner.
6. By placing and holding a full hand on the window.

Exiting

1. By tapping a specific tab.
2. By performing a specific gesture on the action bar.
3. By tapping a tab to render the window inactive, then performing a specific gesture on the window.
4. By using Minimizing to minimize the window, then tapping a specific tab.
5. By using Minimizing to minimize the window, then tapping a specific tab.
6. By dragging the window to a specific location on the surface.

procedure for early user study with low fidelity prototypes

Setup

1 person at a time, the volunteer sits in front of tabletop, with phone and paper representation of virtual UI + a set of abstract paper controls + paper/scissors/pens to fabricate controls on-the-go if necessary.

Next to the tabletop is another computer screen for the purpose of showing animated content to the user. Instructions to the user are read from a fixed document, so as to minimize bias.

The interaction on the tabletop is recorded by a top mounted camera, and the vocal exchange is recorded on audio.

Procedure

For each interaction primitive (ex: resizing, pairing):

1. present the user with an interaction primitive, by showing the effect of it in an animation on the accessory screen.
2. Ask the user how he would interact with the device in order to achieve the wanted effect.
3. Show the user 3 different ways of achieving the wanted effect
4. Ask the user to classify the techniques in the order of his preference.

General reflections about project

What are the concrete applications for this system?

Can/should we do without the physical phone on the table?

partition the UI/applications that are projected on the table?

privacy issues!!

do a user study with paper prototypes to decide which interaction techniques and to find out the relevance of the system, and which context suits it best..

Future directions (post-master) for the integration between the tabletop and the phone:

• Having a UI (iPhone’s dashboard) specific when the UI is on the table
• Having the apps be aware that they are on the table (TabletopAPI for mobile apps)
• Widgets

Design choices: interaction primitives

This is an overview of the application features that we aim at implementing. For each feature, we list possible design options, and argument for our choices.

Pairing

Intention: the application is launched and the connection is established.

1. confirm dialog on table only
2. confirm dialog on phone only
3. confirm dialog first on table, then on phone

We choose option 3. When placing an enabled device on the tabletop, we do not wish to force the TableShare application upon the user. Therefore, we start by showing a discrete popup on the table, allowing the user to actively launch TableShare if he wishes. However, such a dialog can be activated by mistake or intentionally hijacked. Therefore, we add a dialog confirm on the mobile device, before pairing the devices.

Pinning

Intention: a window mirroring the phone’s screen (1:1 scale) is attached on the table independently of the device, allowing the phone to be removed.

1. when removing phone, popup asking if screen should be pinned or not.
2. by tapping a simple button
3. when launching the application, pin by default.

we choose option 2, because we want to be able to implicitly close the connection by removing the phone. Furthermore, we want to avoid pinning the screen on the table without an explicit trigger from the user.

Attaching to device

Intention: the mirrored screen window is displayed on the table, but virtually attached to the side (one of four possible sides) of the physical device.

1. tapping a button
2. flicking/dragging the window
3. flicking/dragging the window by using a tab
4. dragging the window by using active corners

we choose option 4. The window space is meant to allow for interaction with the  phone’s applications and can therefore not be used for manipulating the actual window. The active corners, meant to resize the window, can be used initially to allow the user to flick/drag the window to the side of the device. We wish to implement it with the same feel that the swapping between home windows on iPhone iOS. If the swap gesture is not big enough, the window will fall back into place.

Dragging (when pinned)

Intention: to move the window on the table space. When window is attached to device, this will be done by moving the phone.

1. with one finger drag on window
2. with one finger drag on tab
3. with one finger drag on ‘active borders’ of window

we choose option 2, we want to be consistent with the use of tabs, but reduce the amount of tabs. We choose to use the same tab to pin and drag the window around

Resizing

Intention: to resize the window.

1. active corners on the window
2. by using tabs

we choose option 1. Active corners on windows are known to users, they can be implemented in an intuitive way, making their use easy and straightforward.

Rotating

Intention: rotating the window when pinned.

1. double touch on window
2. double touch with one finger on tab, one finger on window
3. touch on active border
4. touch on two tabs
5. double touch on one big tab

we choose option 2. we see as straightforward that the ‘grab’ tab should be used to rotate the window, by simple adding a second finger on the actual window space. Touching the tab will have the effect of disabling the window for input to the mobile device

Tilting (Portrait/Landscape)

Intention: to tilt the orientation of the display between portrait and landscape views

1. with a tab
2. with active corners

we choose option 2. A resizing gesture in the correct direction can be interpreted to signify a change of the orientation. This way, we avoid more buttons.

Minimizing

Intention: when window is in attached mode, to make it retract to under the phone (starting position). When pinned, to iconize the window.

1. with a tab
2. with active corners

We choose option 2 because we think that it will be intuitive and allow us to avoid more buttons. A resize gesture to reduce the window, when carried on past the 1:1 scale, will trigger a minimizing of the window. Additionally, we wish to implement the double tap on an active corner, with the same effect.

Restoring

Intention: after a minimize action, to restore the window to its previous state within a session.

1. with a tab
2. with a double tap on active corners

we choose option 2. this action is not indispensable, but attractive for learned users, who will have spent time with the application. Therefore we do not feel the need for an explicit button, but prefer the use of the double tap on the active corners.

Backing (iPhone Home button)

Intention: replicates the action of the iPhone physical Home button.

1. with a tab

We see only this solution. The actual button on the phone will still be active, but we need a tab to handle the situation where the phone is not on the table.

Hiding

Intention: easily hide the currently viewed screen.

1. with a tab
2. by obstructing the window with an object/hand, thus making it blurred
3. by removing the phone /closing the application

we choose option 3. We see a ‘Close’ button as necessary, and believe that this would be the best way to interrupt the current displaying. We assume that the protocol to pair the devices will be light and fast enough to allow the users to quickly resume their activity by simply connecting again.

Closing

Intention: close the application, interrupt the connection

1. by removing the phone / hitting the close button

We see only this solution. We want the user to be able to intuitively interrupt the connection by simply lifting his phone off the table.

FURTHER SYSTEM FEATURES:

Handling window obstruction

Intention: make it clear to the user that the window is obstructed, and offer him possibilities of remedying to the situation.

1. make window inactive only + make it manipulable as a picture if pinned
2. make window inactive and blurred + make it manipulable as a picture if pinned
3. only disable the window in locations where it is obstructed, keep it otherwise active

we choose option 3. If we can detect objects, we can disable touch input on those locations. No reason not to allow the user to keep interacting with the rest of the window.

Notifying the user of connectivity status

???

Technical Features Overview

Capital letters A,B,C denote importance of feature. Numbers in parentheses refer to storyboard squares.

Coffee Shop

• A (1) enable phone: app, setting or daemon?
• A (2, 3) connection protocol (see details on page bottom)
• A (4b) UI transfer to surface, inactive, with menu tabs.
• A (11) make transfered display active = integrate phone applications
• A (6,12,13) ‘Expand/Retract/Restore’ tab: when UI attached to phone, allow to show/hide display. Window size is mirror on first use, afterwards size should be remembered.
• A (7) ‘Resize’ corners: only on relevant side when UI attached
• A (8) UI attached to phone, draggable
• C (9,10) Handle obstruction: make window inactive (except resize & menu tabs) until obstructing object removed. When pinned, make inactive window resizable and draggable like an ordinary picture. Make window active again when removing obstruction. Obstructions can be objects, hands (other application windows?).
• A (14) disconnect TableShare when removing phone.

Meeting

• B (3) ‘Grab’ tab used to expand window to custom position beside phone
• B (7) single tap on ‘Grab’ tab used to pin window to table
• B (9) ‘Resize’ corners used to flip display orientation, simply by resizing to horizontal shape instead of vertical shape
• B (10) ‘Grab’ tab used to move window, and rotate window by secondary touch
• C (11,12) ‘Retract’ tab used to iconize window when in pin mode

Office

• remember devices, profile database
• C (2,3,4) move apps from iPhone to tabletop via widget (bridge) on phone

———————————————————————-

???

• NEED BACK BUTTON!
• tab placement?

———————————————————————-

Connection Protocol

Option 1:

1. enabled mobile device broadcasts service
2. MS app is triggered when potential device detected
3. MS discovers service. If positive,
4. MS requests connection
5. User Input on device to allow connection
6. device sends relevant information
7. MS connects to device and takes over

Option 2:

1. enabled mobile device broadcasts service
2. MS app is triggered when potential device detected
3. tab appears next to potential device to launch TableShare
4. User Input to launch TableShare
5. MS discovers service, requests connection
6. Optional: User Input on device to confirm connection
7. device sends relevant information
8. MS connects to device and takes over

OpenTable – Scenarios

1. The Coffee Shop

Alice is in a Coffee Shop, waiting for her friend Bob. She is sitting at an interactive table, which allows her to order her drink via a digital interface, as well as supports TableShare technology.

She takes her phone and notebook out of her purse and places them on the table. A dialog pops up on her smartphone, asking her to confirm the UI transfer, which Alice does by a simple touch. A simple menu appears on the table beside her phone. Alice taps the ‘Expand’ tab, and her phone’s screen appears on the table beside her phone, as a same size mirrored display.

She resizes the window to her convenience, and moves it closer to her by sliding her phone on the surface. The screen goes gray to notify Alice that an object (the noteboook) is in the way. Alice removes the notebook, and the window becomes active again. She accesses her phone’s applications, and starts typing an email.

When Bob arrives, Alice minimizes her display, keeping her phone in place. Bob orders a drink and they start catching up. After a while, Bob leaves. Alice restores her phone’s screen, finishes up her email, and disconnects TableShare by simply lifting the phone off the table.

2. The Meeting

Jim, Jack and Jill are having a meeting about the development of a product. They are sitting around an interactive table, with different artefacts, including paper, pens, computing devices and coffee cups. Jill is responsible for the meeting’s agenda, which is stored on her smartphone.

Jill places her phone on the top right corner of the table and establishes a UI transfer via TableShare. She uses the ‘Grab’ tap to drag the display window to the left of the phone where there is space. She opens the document containing the agenda, and uses the ‘Resize’ corner of the window to enlarge the display, so as to allow convenient visual reference for all present.

It is now time for Jack to present a diagram of the development process. He switches on his tablet computer, opens said diagram, and places the tablet on the table for the others to see. The screen is however too small, so Jack decides instead to use the TableShare application. With a single tap on the ‘Grab’ tab, Jack pins the UI to the table, allowing him to remove the physical tablet while keeping the mirrored display active. By using the ‘Resize’ corner, he enlarges and flips the orientation of the window to a landscape display. By the use of a double touch on the ‘Grab’ tab and the active window, Jack rotates the display to a convenient position, and presents the diagram to his colleagues. When done, Jack minimize the window by tapping the ‘Retract’ tab. The window takes the shape of an active icon, ready to be restored or closed as convenient. When the meeting is over, Jack taps the ‘Close’ tab on the icon to interrupt TableShare.

3. The Office

It is monday morning and Bill arrives at his office. His working desk is made up of an interactive table, extended with a vertical screen, mouse and keyboard. On it are various physical objects, including a stack of papers, some books, pens, an empty cup and a lamp.

Bill wakes the tabletop up from its standby state by simply placing his smartphone on it. The devices know each other, so TableShare is automatically launched. Bill uses the TablePush widget on his phone to push other widgets to the table space. Bill places his calender up in one corner, together with his Skype widget.

After reading through his mail on the vertical screen, Bill starts typing an answer using the keyboard. He needs to refer to a document that is stored on his phone. Bill and uses the ‘Grab’ tab beside his phone to attach its display beside the device. He enlarges the window and moves the display to a convenient location by sliding the phone. Bill types on..

Suddenly the phone rings. Bill taps the ‘Grab’ tab to effectively pin all applications and UI display to the tabletop, allowing him to pick up the phone without interrupting the UI transfer.

papers: Smart Rooms – workspaces of the future

Augmented surfaces: a spatially continuous work space for hybrid computing environments, 1999, Rekimoto, Jun and Saitoh, Masanori.

smart room, device composition, augmented environment, spatially continuous extended workspace, integration physical/digital, hyperdragging, focal/peripheral information space, co-located collaboration

TECH: overhead camera-based interactive surface, visual markers, Java (object serialization, RMI)

i-LAND: an interactive landscape for creativity and innovation, 1999, Streitz, Norbert A. and Geißler, Jörg and Holmer, Torsten and Konomi, Shin’ichi and Müller-Tomfelde, Christian and Reischl, Wolfgang and Rexroth, Petra and Seitz, Peter and Steinmetz, Ralf.

integration of information and architectural spaces, smart room, augmented reality, ubiquitous and collaborative workspace, roomware: InteracTable, Passage: tangible integration, HCI

TECH: LAN and WIFI, BEACH software (SmallTalk)

Software Infrastructure for Ubiquitous Computing Environments: Supporting Synchronous Collaboration with Heterogeneous Devices, 2001, Tandler, Peter.

requirements: mobile/dynamic/flexible, software architecture based on i-LAND infrastructure, co-located collaboration, roomware device integration, smart room HCI.

TECH: BEACH software (shared object space: documents accessible via multiple interaction devices concurrently)

BlueSpace: Creating a Personalized and Context-Aware Workspace, 2001, Paul Chou and Marco Gruteser and Jennifer Lai and Anthony Levas and Scott McFaddin and Claudio Pinhanez and Marisa Viveros and Danny Wong and Sachiko Yoshihama.

smart room environment, context awareness, augmented reality, flexible workspace (individual/collaborative)

TECH: Bluespace prototype = smart cubicle (sensors, displays, Personal Environment Module), XML for exchange of contextual data.

focus on workplace productivity