Beyond the desktop: Ubuntu Linux's new Head-Up Display

Beyond the desktop: Ubuntu Linux's new Head-Up Display

Summary: Following on the heels of changing its interface from the GNOME 3.x shell to Unity, Ubuntu is proposing a new, radical change to the desktop Linux interface: Head-Up Display.

SHARE:

A first look at Ubuntu's new Head-Up Display desktop.

A first look at Ubuntu's new Head-Up Display desktop.

Mark Shuttleworth, founder of Canonical Ubuntu Linux's parent company, has announced that Ubuntu will be adopting a radical new change to the interface that will do away with the "menu" in the Windows, Icons, Menus, Pointer (WIMP) interface, which has defined the desktop for the last thirty years.

Shuttleworth states, "The menu has been a central part of the GUI since Xerox PARC invented 'em in the 70?s. It's the M in WIMP and has been there, essentially unchanged, for 30 years. We can do much better!" This new interface, which will first appear as a beta in April's Ubuntu 12.04 release, is called Head-Up Display.

He explains:

Menus serve two purposes. They act as a standard way to invoke commands which are too infrequently used to warrant a dedicated piece of UI real-estate, like a toolbar button, and they serve as a map of the app's functionality, almost like a table of contents that one can scan to get a feel for 'what the app does'. Its command invocation that we think can be improved upon, and that's where we are focusing our design exploration.

As a means of invoking commands, menus have some advantages. They are always in the same place (top of the window or screen). They are organized in a way that's quite easy to describe over the phone, or in a text book ("click the Edit->Preferences menu"), they are pretty fast to read since they are generally arranged in tight vertical columns. They also have some disadvantages: when they get nested, navigating the tree can become fragile. They require you to read a lot when you probably already know what you want. They are more difficult to use from the keyboard than they should be, since they generally require you to remember something special (hotkeys) or use a very limited subset of the keyboard (arrow navigation). They force developers to make often arbitrary choices about the menu tree ("should Preferences be in Edit or in Tools or in Options?"), and then they force users to make equally arbitrary effort to memorize and navigate that tree.

No one would argue with that. So, what does Shuttleworth propose instead?

HUD will use a "vocabulary UI", or VUI. In this you'll start to type or say a command and in 12.04 LTS, the HUD starts a smart look-ahead search through the app and system (indicator) menus. This uses fuzzy matching, combined with a learning function so HUD will prioritize the actions you use do. This works with both the focused app, because that's where you probably want to act, and the global system functionality. So, for example, if, you're always saving your files, you could type Alt-F and instead of the file menu, you'd immediately get the file/save choice. Eventually you'll always be able to simply say "Save" and your word processor, spreadsheet, or what have you will save your current file.

You'll always be able to use HUD to get to other active programs, such as your instant message or Voice-over-Internet Protocol (VoIP) client through HUD, "without changing focus, because those apps all talk to the indicator system.' Shuttleworth claims that after "you've been using it for a little while it seems like it's reading your mind, in a good way."

Shuttleworth believes that voice will be an important part of HUD "Searching is fast and familiar, especially once we integrate voice recognition, gesture and touch. We want to make it easy to talk to any application, and for any application to respond to your voice. The full integration of voice into applications will take some time. We can start by mapping voice onto the existing menu structures of your apps. And it will only get better from there."

Still, Shuttleworth says "even without voice input, the HUD is faster than mousing through a menu, and easier to use than hotkeys since you just have to know what you want, not remember a specific key combination. We can search through everything we know about the menu, including descriptive help text, so pretty soon you will be able to find a menu entry using only vaguely related text (imagine finding an entry called Preferences when you search for "settings")."

It's not just speed though. Another goal, as it has been "in much of the Unity design has been to return screen real estate to the content with which the user is working; the HUD meets that goal by appearing only when invoked. Instead of cluttering up the interface ALL the time, let's clear out the chrome, and show users just what they want, when they want it."

The ideas for this, Shuttleworth continued, have been "inspired by many works of science, art and entertainment; from Minority Report to Modern Warfare and Jeff Raskin's Humane Interface. We hope others will join us and accelerate the shift from pointy-clicky interfaces to natural and efficient ones."

Although Shuttleworth doesn't say so, part of the reason why Ubuntu is making these changes is to make Ubuntu a more attractive option to smart TVs and smartphones and tablets. On these systems, the WIMP interface doesn't work that well. If HUD is successful, users will ultimately be able to use the same interface on any device from PC to smartphone with voice commands.

If that sounds more scary than neat, worry not. Shuttleworth isn't going to leave you in the lurch. "We'll resurrect the (boring) old ways of displaying the menu in 12.04, in the app and in the panel. In the past few releases of Ubuntu, we've actively diminished the visual presence of menus in anticipation of this landing. That proved controversial. In our defense, in user testing, every user finds the menu in the panel, every time, and it's obviously a cleaner presentation of the interface. But hiding the menu before we had the replacement was overly aggressive. If the HUD lands in 12.04 LTS, we hope you'll find yourself using the menu less and less, and be glad to have it hidden when you are not using it. You'll definitely have that option, alongside more traditional menu styles."

Related Stories:

TV for Human Beings: Ubuntu Linux

Can Ubuntu Linux win on smartphones and tablets?

Ubuntu Linux will try for the business desktop

Ubuntu Linux 11.10: Unity comes of age (Review)

New Desktop Interface Flops

Topics: Linux, Hardware, Open Source, Operating Systems, Software, Telcos

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

96 comments
Log in or register to join the discussion
  • pretty cool, but...

    what happens if the speech recognition doesn't work? That has been the big problem to date with voice recognition command systems.
    Mac_PC_FenceSitter
    • If something doesn't work.....

      ....use the old fashion menu or command line options! lol
      :-)
      kd5auq
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        @kd5auq
        or shortcut?
        or gestures?

        But question is really important. And should sound something like:
        Ubuntu have many localizations, and even more non-native speakers. Will it be able to provide proper recognition in all of supported languages? And in situations when person speak in supported language but with different accent?

        Short answer:
        Yes but...

        Long answer:
        I know at least one open source project that provide speech recognition implementation that can be thought by user how to recognize spoken words. And it worked quite good. But...
        It was good for already learned words. If you did not thought word, results where not good. HUD will need way more flexibility.

        But again Ubuntu have HUGELY LARGE community and if Canonical can crowd-source this problem onto it (just like it done with translations), then it can be solved.
        przemoli
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        przemoli

        Not a bad reply, but we can tell English is not your native language. If I understand you correctly, I would have written this:


        But the question is really important. And should sound something like:
        Ubuntu has many localizations, and even more non-native speakers. Will it be able to provide proper recognition in all supported languages? And in situations when the person speaks in the supported language but with a different accent?

        Short answer:
        Yes, but...

        Long answer:
        I know at least one open source project that provides speech recognition implementation that can be taught by the user how to recognize spoken words. And it worked quite well. It was good for already learned words. However, if you used words it had not been taught, the results where not as good. HUD will need way more flexibility.

        But again Ubuntu has a HUGELY LARGE community and if Canonical can crowd-source this problem onto it (just like it done with translations), then it can be solved.

        Dr_Zinj
        English-Nazi

        And no, I can't write worth a damn in Farsi.
        Dr_Zinj
    • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

      @rbethell Also, what happens when you have a whole room full of computer users talking to their computers? I like Siri, but I don't use it in a crowded office. Oy! Then again, since speaking probably means fewer typos, lines like "...HUD will prioritize the actions you use do." will happen less often. Unless you've been drinking.
      JoeFoerster
    • But it does.

      I have a mongrel accent, hard for aussies to understand, but Dragon dictate is now giving me 99% accuracy on a Android tablet, a 2nd decade 21st century computing and comms device.
      For all the WIMP fans, XEROX Menus were supposed to be popup and pasteable, like the right click GIMP button does, not the dropdown variety invented by Apple for their tiny screens.
      For all the keyboard fans, the oral interface was considered by 80's XEROX and others of the period, to be the logical next step to a universal useful human interface.
      stomfi@...
  • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

    I'm going to try it just for kicks, but I'm sticking with Xfce/Xubuntu for now.
    statuskwo5
    • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

      @statuskwo5 finally, someone who knows that Ubuntu is Not synonymous with Unity/Gnome and is smart enough to use a different version of an OS that works for them because of something called Choice.
      Win8AnUglyDisaster
    • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

      @statuskwo5 yep, me too
      bpedman@...
  • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

    Windows 7 already has it - you start typing a command, as in "note" and it gives you an option to start notepad.exe.
    ForeverSPb
    • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

      @ForeverSPb you are mistaken. Search functionality has been with Linux for a long time. Synapse, Gnome-Do etc do the things which you talk about. But this is something else, can you 'search' for menu options of a particular application as well? Can you search for.. lets say 'line spacing' option in word? No, not really. This is a first on ubuntu.
      xeptf4
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        @xeptf4
        How is he mistaken? this is a NEW feature to UBUNTU. Windows 7 already has the function and yes you can type in an apps names as well as voice which was in windows before 7. Sorry man windows is way ahead of the game here. Now if it searches a programs functions like you state search for "line spacing" in word. all we have to say it launch word with its voice commands and it pops up all ready to go. Thats certainly not a killer feature that will make people switch to linux lol
        Stan57
      • Windows 7 already has the function

        @Stan57

        You didn't watch the video did you.
        guzz46
      • Anything New About It Is New to Windows as Well

        @Stan57
        What you are talking about in Windows is not new in Ubuntu or Linux in general. You can argue that this isn't really all that new, but it's at least as new for Windows as it is for Linux.
        CFWhitman
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        @xeptf4

        You can't search for 'Line Spacing' in Word like this, nor have you ever been able to (this is the first time I've ever seen such a feature).

        But... what happens if you DON'T know what you're already looking for, or know what you want to do but don't know the command to type?
        I don't think that the menus will ever go away for that reason alone.

        Though this is way better than the Ribbon Microsoft thought up for Office since you still have the standard interface, and an overlay that lets you get at stuff really fast.

        I still like how many modern features Linux had way before they were copied by Windows/Mac/Mobile (all of them)- a prime example of this is the package manager (which all App Stores are shameless copies of).
        R220
    • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

      @ForeverSPb
      I have to agree. This sounds like a different presentation of the window 7 search box, which actually works now, and does so in the same way.
      Sqrly
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        @Sqrly No it isn't! The Windows search functionality gives you the ability to search for an application and i actually works now in Linux as well! The Head-Up display gives you the ability to search for functions within applications based on patterns such as: I know there should be a delete function although I don't know where it is or what it is called.<br><br>Please read the article next time...
        bleddie
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        @Sqrly No it isn't! The Windows search functionality gives you the ability to search for an application and it actually works now in Linux as well! The Head-Up display gives you the ability to search for functions within applications based on patterns such as: I know there should be a delete function although I don't know where it is or what it is called.<br><br>Please read the article next time...
        bleddie
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        @Sqrly No it isn't! The Windows search functionality gives you the ability to search for an application and it actually works now in Linux as well! The Head-Up display gives you the ability to search for functions within applications based on patterns such as: I know there should be a delete function although I don't know where it is or what it is called.

        Please read the article next time...
        bleddie
      • RE: Beyond the desktop: Ubuntu Linux's new Head-Up Display

        @Sqrly No it isn't! The Windows search functionality gives you the ability to search for an application and it actually works now in Linux as well! The Head-Up display gives you the ability to search for functions within applications based on patterns such as: I know there should be a delete function although I don't know where it is or what it is called.

        Please read the article next time...
        bleddie