Hypergeusia

iOS 8 Wishlist

This is not a prediction, just a personal list of what I’d like to see added to iOS to enable users and developers to do more. Of course a challenge with a system this widely used is to empower the more advanced user without complicating the experience for the novice.

Some Simple Updates

  • Remove the stupid textures from notes and reminders and the bubbles from Game Center.
  • Spotlight: Access from within apps, perhaps from a gesture, multitasking mode, control center etc. I’d also like it to better search content in third party apps.
  • Reminders has the opportunity to be one of the most useful apps on a device that’s always with you. But it’s kind of terrible. It could use a few new features (Geofencing an entire list, pasting several items at once, sharing, more repeat options) but most importantly it needs a much simpler user interface.
  • iCloud Clipboard: When enabled, you simply have a single clipboard between all devices. Copy here, paste there.
  • Airdrop between iOS and Macs.
  • In-app access to autofill passwords: Currently it’s only available in Safari. I’d like to see it not just in competing browsers, but in all apps.
  • Favor FaceTime audio over voice calls (just like iMessage is favored over SMS).
  • Multiple user accounts. It’s absurd to think a family can’t easily share an iPad.

Modes

  • Battery-saver Mode: A one-step battery saver that reduces brightness, network activity, push notifications, background processes etc.
  • More Do Not Disturb: A new setting where notifications can be hidden even while using the device. Perhaps activated automatically when using specified apps (for example, while reading a book, or watching a movie).
  • Night Mode: Dark interfaces for everything, a developer API, and a setting to let the ambient light sensor activate it automatically.

Homescreen

  • Live App Icons. Restrictions such as updating only once per hour, for apps that were used recently, etc. would keep battery life in check. Obviously there’s not a ton you can show in such a small area, but I think simple things could make a big difference: see clock and calendar. I think this could likely replace the red badge count (which many developers have hacked into something else) for more interesting solutions.
  • Hide apps, just like on Apple TV. Tons of apps I want to keep but don’t need to look at every day. Including system apps.
  • Set default apps, for common functions: Browser, Mail, RSS, Twitter, Messaging, Notes, Maps, Music and Reminders would be a good start. Ideally it’s a manual setup so every app won’t be bugging me to make it the default.
  • More advanced home screen editing. Moving an app over several screens, moving entire screens, moving several apps from one folder to another. Just a few examples of things that are much harder than they could be.
  • Genius App Layout: Moves, groups, and hides your apps based on what you use, and what other people use.

Big Features

  • Customizable Status Bar: A settings panel allowing users to show, hide and rearrange the status bar icons. A few new options could include upcoming calendar events, today’s date, unread iMessages, VIP emails, more specific location alerts, twitter mentions, etc. I wouldn’t mind third party support here, as long as it’s off by default.
  • Notifications: I’d like to see some actions made available in the notification like “reply” or “archive” to emails for example. Android is handling it quite well. I’d like apps that use Auto-layout to shrink vertically when a notification shows up, so that nothing is obstructed by the notification.
  • Better inter-app communication: To be able to easily move documents, files, text, images, audio etc between apps. This is a limiting factor for taking iOS productivity to the next level. The important part is for an app to say “I’m sending out X filetype, which apps can receive it? Windows and android handle this well.
  • Siri For Everyone: third party developers could make Siri so much more useful. The biggest challenge with this is competing apps. If she doesn’t know which app to use, she could simply ask, and after a while she’ll learn how you want things done.
  • Today Tab Widgets: in notification center for third party widgets with a corresponding section for them on the store.
  • Official Safari Extensions: bookmarklets are obviously a hack. We’ll also need a place in the app store for downloading them. I’m sure for security reasons there would be plenty of limitations, which is fine. We probably need an updated interface for accessing them, but can also serve to organize Safari’s more advanced features like iCloud Tabs, reading list, etc.
  • Improved keyboard shortcuts for hardware keyboards. I’d like shortcuts for Home, back, app switching, type to select (like on Apple TV), navigational shortcuts etc. I’d like to see a global shortcut (command-?) to display shortcuts for the current app.
  • Dual-app Multitasking: a way to see and use two apps at once, likely only on the bigger iPad (and definitely not on a phone). Pages and Safari, Reminders and Calendar, FaceTime and Notes etc. Others have tried to pull this off with varying levels of success. I think Apple should give it a shot.

iPad Notebook

Is it the physical form factor or the simplicity of the operating system that has made the iPad successful? Tablets running desktop software have never done well, so I suggest that It may be more about the software. If that is the case, I think there is room for an exciting new product category from Apple: the iPad Notebook. (A better name escapes me.)

What is it

I envision this product as a hybrid between MacBook and iPad. At first glance it may resemble a MacBook, but internally it’s much more an iPad. The most noticeable feature is a permanent hardware keyboard, so it must be big enough to accommodate one of full size. So let’s say 12-inch multi-touch retina display. As I’ve mentioned before, it could run iPad apps scaled up, and also allow for a new class of apps that take advantage of the extra screen space.

ipadnotebook

Many users are already enjoying an iPad combined with an external keyboard or keyboard case. iPad Notebook would easily provide a better experience for users who prefer that setup. Build quality would be unquestionably better. The device would be better balanced with more weight under the keys. And the device would likely have room for a bigger battery than an iPad.

As I see it, this is Apple’s ultra-portable touch screen notebook. Most importantly it runs an OS designed for touch. Apple strives to create thoughtful and simple products with clear intended uses. iPad Notebook is for customers who prefer a notebook form-factor, but don’t need the complexity of a desktop operating system.

Gesturepad

iOS does not and should never support a mouse/cursor. If developers come to expect precision pointing, the touch experience could be ruined. Therefore, a traditional trackpad should not be present on this device. However reaching up to the screen for almost every interaction is not the best experience either.

So, in the trackpad’s place I imagine something similar: a multi-touch gesture area. I see it spanning the entire width of the device and being effectively invisible. The goal would be to allow the most common actions—such as scrolling, swiping, and basic navigation—to be performed without requiring users to reach up to touch the screen. APIs would let developers easily build these gestures into their own apps.

System defaults are already defined by current iPads: single-finger swipe to scroll, bezel-swipes to access notification center and control center, and four-finger pinch and swipe for Home and Multitasking.

This leaves two and three finger swipes, double taps, multi-finger taps, and anything else open to developers to create shortcuts for their apps. I can imagine many creative uses: musical instruments, steering wheels, page-turning, map-zooming, text manipulation, etc.

No Portrait or Tablet Mode

I doubt even Apple’s Industrial Design team could create a twisting hinge that isn’t stupid. A portrait-only app could run column-boxed on a landscape screen without being too small, but that’s terribly inelegant. What I prefer: landscape only. How many Apps require portrait orientation? I can only think of a few games. Similarly the keyboard should not flip back behind the screen or otherwise “transform” in an effort to feel more like an iPad. Doing so would detract from the simplicity and purpose of the device.

Keys and Buttons

I envision the keyboard being almost the same as a MacBook. I’d like to see a few changes to the Function keys to include Home, Sleep, Multitasking and a way to show the on-screen keyboard.

functionkeys

My concern here is that putting the Home button off to the side may not be prominent enough, as it is so central to the iOS experience. My thought was that replacing escape would make sense as you “escape” from an app. An alternate solution could put the Home button below the display. Fn and Control keys are probably unnecessary.



iPad Notebook will not be for everyone. If you are a happy iPad or Mac User, you will likely find little use for this product. But I do think users who work mostly at a desk or on their lap, and do a lot of typing would find such a device quite useful. Diversification of hardware will help iOS and it’s ecosystem enter and thrive in new markets.

Launchpad as a Desktop Replacement

I’d like to have the desktop removed from my Mac. I’ve known the Macintosh Desktop for 20 years now, and I’m afraid it might be time to move on. I can’t think of any feature of the Desktop that can’t be accomplished as well or better through a Finder window. I’d still like to keep a wallpaper behind my windows, but no icons. The last few months I’ve spent ignoring the Desktop: I won’t put a single thing on it, and life has only been better. I’ve been more organized because I’ve filed everything away immediately instead of hoping to organize it later (i.e. never).

And that got me thinking about Launchpad, as a sort of Desktop replacement.

Everyone seems to hate Launchpad, and I think I’ve realized why: few take the time to organize it. You scrutinize over the arrangement of your iPhone screen, yet neglect your poor Mac. (And before you even say Spotlight, my well organized and memorized Launchpad screen puts any App two clicks away—which is faster than moving my hands to the keyboard.) I challenge you Launchpad haters to organize the apps that you use and put anything you don’t

Launchpad is an obvious desktop replacement, but it’s not quite that simple. You can’t just drop it in. The primary goal of disabling the desktop was to reduce clutter, so the last thing I want is more icons. What I need is a smarter and more accessible launchpad.

Fixing Launchpad

Launchpad is far from perfect. Here’s what I’d do to fix it:

  • Animations and other UX choices could better position Launchpad as a default bottom layer, a true “home screen” rather than a secondary layer laid on top of everything else.
  • Kill the background blur.
  • Rather than Finder, Launchpad could be made the new default app. It could appear automatically when all apps have been quit, hidden, or had their windows closed. You would now be able to quit Finder.
  • Menubar apps and other non-dock apps shouldn’t be shown in Launchpad, at least by default. These often don’t need to be launched and only add clutter.
  • The search bar should simply be Spotlight, not just an app search.
  • Easier to access. I suggest a hardware Home Button on laptops and keyboards.
  • Add a software Home button in the top left corner next to the Apple Menu. Fitz Law makes it incredibly easy to click, just throw your mouse into the corner.
  • Apps in the dock should not also appear in the Launchpad grid.
  • iOS icon shapes. They look better on a grid, and it helps with the app ecosystem brand image.
  • Customizable grid spacing and icon size, this would coincide with the dock size.

iPad Pro

The name iPad Air all but implies there is some sort of (as I’ll call it) iPad Pro in the works. How would Apple design an iPad for Professionals?

Specs

Of course it’s got a faster CPU, more storage and all that, but that’s expected and boring.

Screen Size

It’s bigger. I’d guess 12.1 inches diagonal at 2560 x 1920 (similar pixel density as the iPad Air). The existing library of iPad apps should be able to scale up to fit without making the interface unbearably large. However I think the App Store should expand to hold a new class of “Pro” apps designed to really take advantage of the new size. AutoLayout and other techniques will help minimize development effort for building these new apps, and the Universal app model will still apply. The beauty of this system is that a 10 inch iPad Air could also run these new pro-sized apps at a scaled down size resulting in a virtual density similar to what we’re used to with the iPad mini.

At these scaled sizes (from iPad up to iPad Pro and iPad Pro down to iPad Air) pixels would not map 1:1, but I would expect similar results as a retina Mac running at a scaled resolution: practically perfect.

Real Multitasking

According to my calculations, a 12 inch iPad Pro has plenty of screen real estate to simultaneously show two iPad-mini-sized screens side by side. Notes and Browser, Code and Browser, Messages and Twitter, etc. There are sure to be some interaction issues (keyboard, focus, entering this mode) to work out in an iOS update but it’s a seemingly simple solution compared to what Microsoft is doing.

IO

A single Thunderbolt 2 port. Give developers and hardware manufacturers whatever they need to build really interesting Thunderbolt accessories for iPad. Storage, audio, video, cameras, networking, anything. And then there’s external displays. A lot like existing HDMI output, but higher resolution. The iPad would act as a controller for content on the external display. It’s a new interaction model that would need some work to get right, but I don’t think it’s out of the question. Games, photo and video editing, and general content consumption come to mind as good uses.

iPad Keyboards and Shortcuts

Forward

This was in my drafts folder for the last month until I learned today that Apple has added (bluetooth) keyboard shortcut support for developers in iOS 7, in what seems to be exactly how I hoped. I figured I’d post this anyway:

The iPad wants to replace the laptop. And for many types of users it already has. There are still many advanced users who need more precision. I think there is a small change that would be quite easy to implement. It would bring much added benefit to those who want it and would bring no added complexity for those who don’t.

The iPad has always supported an external keyboard, and there are a few shortcuts supported by the system, mostly cursor movement and basic text formatting. But beyond that it’s mostly a bummer. I’d love to see some some APIs and other tools open up for developers to build in complex navigation, controls, and anything else we can think of into our apps.

When using the iPad propped up (with a Smart Cover, dock, stand, etc.) continuously reaching up to touch the screen is far from the perfect interaction. The laptop design solves this problem by putting keyboards and trackpads flush with the surface of the table so your hands are always at rest. The Microsoft Surface’s keyboard cover was a nice try, but I think including a trackpad (an mouse support) only encourages the development of apps with small touch targets, designed for a pointer rather than a finger. It just gets awkward.

Designing an interface to be controlled with a keyboard does not alter the visual interface of an application the way designing for a mouse pointer would. For example converting a pointer-optimized interface to a touch-optimized one, a button must grow significantly larger to be easier to tap with a finger. Keyboard shortcuts on the other hand are invisible to the average user.

One counter argument is exactly that: the shortcuts will not be discoverable. But I have a solution. A global keyboard shortcut for “Keyboard Help” supported by the system and therefore available in every app. I propose Command-? the Mac equivalent for help. Keyboard users must commit only this single shortcut to memory and they can easily reference all available shortcuts in any app.

A few system shortcuts would be a good start. I’d like to see additions for pressing the home button, navigating pages of the home screen, type the name of an item to jump to it (just like on Apple TV), control center, notification center, etc. The built in apps could follow suit, and my suggestion makes it easy: copy from the Mac.

I understand that we are introducing a layer of abstraction here. Typically the iPad benefits from direct manipulation of onscreen elements. If you want a thing to happen, you simply touch it, a concept so simple a toddler can understand. And yet, this proposition takes nothing away from that. Only the power users who’ve paired up their bluetooth keyboard will ever see this complexity. And they’ll be more productive because of it.

Missing Mac Apps

Mac OS X Mavericks introduced two new native Mac apps: iBooks and Maps. Despite being basic clones of their iOS counterparts, they do bring useful functionality to the Mac. However there are still a few more apps that Apple has yet to bring back to the Mac. (Listed in order of importance)

Siri isn’t really an app, but I could see her living happily in the dock. And with a full keyboard, I wouldn’t mind typing to her either: less awkward and more accurate. I have no idea why this hasn’t happened yet.

Newsstand. If we’re going to be reading books, why not magazines and newspapers too? They may need to be displayed at iPad size, until publishers begin supporting fluid or multiple layouts.

Find my Friends, Find my iPhone. The map is already there, dropping some pins on it seems simple enough.

Weather. Simple enough, with room for even more data. Full screen mode could be gorgeous too. While we’re at it, let’s do the same for iPad.

Clock. Alarm, stopwatch, timer, world clock. No reason a Mac shouldn’t have these functions right out of the box.

The Mac may not be the ideal device for many of these apps. But the best device for the job is the one you have on you. No reason to make me get off the couch, or worse, open a web browser, just to check the weather.

The case for a Google Glass competitor from Apple

If Apple wanted to compete with Google’s Glass (and surely they do), I think an interesting and appropriate strategy would be an updated Siri designed specifically for use via headphones plugged into an iPhone. Just as glass users get constant notifications and interaction in the corner of their vision, Siri could periodically deliver similar services to the ears. She could also be called upon with a likely optional, Glass-like “okay Siri” command.

Interaction would be would be as simple as carrying on a conversation, which could even be initiated by Siri herself. For example:

Siri: A new email just arrived from your sister
User: About what?
S: Vacation Plans. Would you like me to read it?
U: Yes
S: The email reads: ‘Hey, I’d like to go over some of our plans for the vacation. What days do you have available in October?’ Would you like to reply now?
U: No, but remind me when I get home
S: OK, I’ll remind you to email your sister when you get home.

In this example, Siri is set to announce the arrival of any email from your VIP list. She understands that your question “about what?” could be answered by reading the subject of the email. She then offers you additional assistance: to read the contents of the email and to reply to it. She also understands that “remind me when I get home” refers to the email you were just talking about, so she sets up a geofence and creates the reminder.

Siri is already quite close to doing many of these things. She just needs to have a better grasp on the idea of a conversation, the ability to initiate one herself, and to recognize that your headphones are plugged in. (A mode could exist for her to converse over the speaker, but in many situations that might be too awkward or reveal private information, so I think headphones are best.) And of course she would certainly benefit from being faster.

Here are a few ways Siri for headphones could be better than Google Glass:

  • The hardware, specifically headphones, are common, cheap, included with iPhones, comfortable, and socially acceptable
  • Headphones do not need to be charged
  • Users are already very comfortable and accustomed to wearing headphones in public
  • Bluetooth headsets would also be compatible
  • Spoken notifications are potentially less distracting than visual notifications
  • Headphone-Siri could be made available to millions of existing iPhone users at no additional cost

Of course, there is no camera, but some might see that as a benefit. There’s no display for video, photos or menus. There’s also no touch-control, but could a rudimentary interface be made from the playback remote? Click once to accept, click twice to decline… And the phone itself is always close by for any advanced functionality.

I think we’d all love an API for developers, but that’s a different story.

In general I am still skeptical about conversation and voice as a user interface. I don’t think users are quite ready to talk to their devices for everything. But in the right situation (Driving, walking, or otherwise preoccupied) it could be a reasonable way to get something done.

Wipe the windows

The purpose of a windowed user interface, simply put, is for multitasking. For seeing several things at once. For dragging or pasting data between documents and applications. A browser window and a page of notes. Email and a calendar. Perhaps a music player or a twitter stream in the background.

Yet managing these windows is not perfect. They are flexible, but they are really quite inelegant. And much of the management must be tediously done by the user. Windows can get pushed offscreen, overlap and obscure content, or be lost entirely. And everything covers the desktop. We’ve come up with clever ways to deal with these issues: Mission Control, the “window” menu, “Expose”, “hide others”, “bring all to front” and so on. But these are just proof that dealing with windows is far from perfect.

image

I propose a new system that divides the screen into sections. For example you might have a section on the left half for a browser window, and two sections stacked on the right half for twitter and a to-do list. Making the browser section wider would simultaneously shrink the other two sections. The content in each section would adjust to best fit its new size, not unlike current windows, but now you’re essentially resizing everything at once, without any overlap or unused space.

image

Developers might be encouraged to support more sizes and create more responsive layouts, but for the most part, we’re not far off from supporting this today. Tabs will be more useful than ever, allowing users to keep several windows in one section.

Mac OS has lately been following the trends set by iOS. One such trend has been the full screen experience. And with smaller screened devices this is often an appropriate use of space. However, the more screen real-estate you have, the better you can take advantage of seeing more at once. On 15, 21, 27 inch screens, using an application in full screen is often a terribly inefficient use of space. With my proposed system, the full screen experience need not be limited to a single task.

A mission-control-like system could be used for moving, resizing, adding, and removing sections. Spaces would work well to group various tasks together (and I’d welcome return of the 2-dimensional grid). Common window groupings and sizes could be maintained across sessions.

image

Known Issues

Menus. If several windows are now spanning across the screen, where do you put the menu bar? This is about to get a bit blasphemous. Mac users (myself included) have long argued that a reliable menu bar at the top of the screen is easier to find, easier to point to, and just generally better. However, I foresee a change of times. More and more the menus hold only advanced features for advanced users—users who might often be using keyboard shortcuts anyway. My thought is to add a button to every app that produces a menu of menus where it can stay out of the way until you need it.

No window. A mac app can run with no windows open. It’s a nice feature, when you want a download to finish or a song to keep playing. And to get something going again you go to the menu bar or click on the dock. How does this work with sections? I’d say clicking an icon in the dock would still open a new window, and a click-hold on the dock icon could give access to the menus as well.

Active window. If two windows are guaranteed not to overlap, must there still be an active window? I would love to say no, but what to do with conflicting keyboard shortcuts? I think there must be an active window for that reason, but click-through rules can be greatly simplified.

New windows. Oftentimes an app will create a new window, clicking compose in Mail, for example. A few options come to mind: A new section is created for the new window, the app presents the windows as a tab, the new window is presented as a modal above the first window. Ideally developers would be able to decide which actions produce which results.

The Desktop. Perhaps it’s time to lose that too. A full screen finder window would be a fine replacement. Or perhaps a new app called desktop? LaunchPad might finally feel like it fits. And the dock can certainly stay as a quick way to jump to open apps.

The New Living Room Experience™

A set-top box is the type of device that can progress quickly. Like smartphones and tablets, this little computer’s processor, RAM, GPU, networking, storage etc. are all getting better and cheaper every year. Now throw in an ecosystem of apps and games designed for the TV with developers pushing the hardware to its limit. The technology progresses quickly and a relatively low price will encourage users to upgrade frequently.

A big display is quite the opposite. It is expensive and would not find itself feeling obsolete nearly as soon. Many five-year-old TVs still look great today. With a modern, capable input, such a screen could easily stay relevant for over half a decade.

And that is my proposed New Living-Room Strategy™ for any company looking to develop a living room strategy: An affordable little computer paired with a simple and beautiful display. With this combination, consumers can feel comfortable upgrading their little computers often to get the latest features while plugging each upgrade into the same display for years.

Big Display Hardware

For my plan to make any sense I must impose a few constraints on the display hardware. 1: The big display must stand the test of time such that a consumer won’t hesitate to purchase for fear of it being made obsolete prematurely. Thus, hardware used for anything other than audio and video should reside in the little computer. 2: The big display must not contain any hardware that would cause the little computer to be incompatible with any other television or display. To create a powerful ecosystem, the little computer must be accessible (affordable) to the mass market by being compatible with existing and cheaper displays. (Certainly, the proprietary display will provide a better, more integrated experience. More on that in a minute.) 3: Inversely, the big display must play nicely with other companies devices, as no device is seemingly ready to tackle all the needs of the living room just yet, and it would be thought of as a waste to purchase such a nice display if it only works with the one device. Here are some more details:

  • Inputs: I’d say two Thunderbolt and two HDMI ports would be plenty. The little computer would connect via Thunderbolt (and I bet that could even provide power). The other Thunderbolt could be used for any additional expansion.
  • Size: Big.
  • Resolution: 2K is fine, then 4K whenever it’s affordable.
  • Speakers: good enough and loud enough for the people who don’t care too much about sound and the right audio outputs for the people who do.
  • Smart, automatic input selection, automatic volume, and other stuff like that to just work whenever possible.

Little Computer Hardware

I think the little computer should exist in the form of a hockey-puck style device intended to sit right in front of the TV (Not unlike AppleTV). This is for two reasons. 1: Miniaturization is limiting and expensive especially for a non-mobile device that’s meant to be affordable yet powerful. And 2: this placement allows for the inclusion of more interesting hardware such as:

  • Multiple cameras positioned across the front of the device in such a way so as to capture a “panorama” of an entire room. Ideal for video conferencing from any seat or with the whole family in the room.
  • Microphones for video and audio calls, voice control, games, etc. I don’t think voice will be the primary interaction, but I think it’s really going to catch on for some things.
  • Infrared Motion Sensors. I see one pointed out across the room like Kinect, and another pointed up in front of the TV like LeapMotion. The latter would essentially turn the TV into a touchscreen with hover and gesture support. Gestures should by no means be the primary method of interaction, but it does open some incredible opportunities to developers.
  • Smartphone-caliber CPU and GPU, storage, etc.
  • A controller (and os) capable of controlling real games and applications. Something that’s good at actually getting things done. I’d also keep a bluetooth keyboard on the coffee table for when I need to quickly reply to an email, although that could be optional.

A Better, More Integrated Experience

Although it’s important to let the little computer work with any TV, the experience could be greatly improved with seriously integrated hardware.

If I pick up a controller or keyboard, everything turns on. It knows who I am, and can provide customized content. If I broadcast a video from another device, the display turns on and the video plays. Perhaps it could turn itself on for approved notifications: There are new photos from friends, a game you were downloading is ready to play, a show you’ve been watching has a new episode, new emails, reminders, upcoming events, alarms, etc. An incoming video call rings, and your friend’s face is on the screen, just say “answer that”.

Let’s keep going. Say the display will only turn on and display information pertaining to you if you’re in the room. The display could turn off automatically too, when a movie is done playing, when an album has ended, or when everyone has left the room. There’s no harm in turning it off if a quick gesture, grab of the remote, or notification can quickly and automatically turn it back on. (Advanced settings could tailor behavior for specific rooms and use cases: for example you may wish for the bedroom setup not to turn itself on after 10PM, and so on.)

The TV has potential to become the central hub for all shared digital information in the home. Content is automatically presented to whomever is nearby, specific to that users, their habits and preferences, date and time, etc. With the right methods of interaction and the right software and developer support, the TV could once again become the most exciting device in the house. The refrigerator might just lose its magnets.

Moving your hands to control a piece of technology and keyboard shortcuts.

There’s the Kinect. There’s the LeapMotion. A few from Samsung, of course. “With the wave of a hand…” such a natural interface sounds great in theory, but it doesn’t seem like we’re quite there yet.

Problems with gestural interfaces:

  • Fatigue
  • Interference and accidental input
  • Discoverability
  • Memorability
  • Awkwardness

I recently heard of researchers who sampled a few hundred tech-savvy individuals with various cultural backgrounds, to see if they could agree on a gesture system for controlling a television. The consensus was that for a few simple tasks, gestures were typically agreed upon. But beyond basic controls, participants weren’t coming to any real consensus on what would be a natural mapping.

This got me thinking, can you really count on the average user to design a complex gesture-system? Can you democratize these decisions? And I have to say no.

I think some of the gestures can and should be intuitive to a degree, but I think they must be designed and crafted by a team that has considered (and of course tested) the system as a whole. The most common tasks can be quite natural, and the rest should build off that system in such a way that is both easy learn and easy remember. If the system as a whole makes sense, then it is less important that each gesture is immediately intuitive on it’s own.

For example there will be conflicting gestures. A user might think to perform the same gesture to raise the volume, increase the channel, or scroll down a list of options. Can the user make a clear decision about which is the best action for that gesture? Does the user know which of these actions is the most common? Perhaps this gesture is done so commonly that the action could be triggered accidentally.

Allow me to compare such gestures to something we are more familiar with: the keyboard shortcut. Not easily discovered, learned, or remembered by the user. However a shortcut system can be designed in such a way to improve these shortcomings, through key placement and relationships, mnemonic devices, character shapes, modifier consistency, consistency, etc.

Look at Cut, Copy, Paste for example. Let’s start by using keys closest to the modifiers. We’ve already got a conflict with cut and copy for the C. Clearly one of them should be on the C. Copy is more useful and less destructive, so it get’s first choice. The X somewhat resembles a pair of scissors. and the V is almost a down arrow for pulling down off the clipboard, or symbolic of two papers being “pasted” together. By design or accident? I don’t know, but it works. It’s a system that makes sense as a whole. Or brush size in photoshop, mapped to the left and right brackets. Two opposing keys for affecting brush size in opposite ways. Add shift to instead change brush hardness. Once you’ve learned brush size, you can easily learn hardness.

How can we apply some of these principles to gestures on the TV and other devices? How can we design a smart gesture system that makes sense as a whole? We can prioritize a list of the most common commands. We can try to reduce fatigue as much as possible. We can give opposite commands “opposite feeling” gestures, and vice versa with similar commands. We can allow the knowledge of one gesture to inform and enforce other gestures. Clearly we haven’t figured it out yet, but that’s what makes it exciting.