Project Sikuli, a machine-vision research project from MIT, allows users to write scripts that automate UI-driven tasks. This tool, while very powerful and simple, can cause a number of headaches if you’re not careful. In this post, I’ll talk about some issues you might encounter, as well as how to avoid them. If you’ve never used Sikuli before, you can see demos here, or download the IDE here.

5. Not wait()ing

If you’re a long-time user of a particular program or operating system, you can probably describe many common tasks from memory. For example, I might describe how to access the “Uninstall or change a program” dialog in Windows 7 as follows:

1. Click on the start menu
2. Click on “Control Panel”
3. Click on “Uninstall a Program” under the “Programs” heading

If you’re new to Sikuli, you might be tempted to script this simple process like this:

You could then run your script and not notice a single issue for a long time. Then, one day, your computer is busy running multiple background tasks and you run your script again. The start menu takes a few seconds to display its contents, and your script raises an exception. The problem is that Sikuli doesn’t automatically wait for an image to be visible on-screen before trying to click() it, so if it doesn’t see “Control Panel” on your screen very soon after clicking on the start menu, it will raise an exception. In order to force it to behave properly, you’ll need to insert wait() statements:

If you’ve added wait()s and you’re still having problems, try setting a longer wait time.

4. Having trouble with context-sensitive/popup menus

If you tried to follow along with the previous example, you likely ran into a problem when you tried to capture the “Control Panel” option in the start menu. After opening the start menu, switching focus back to the Sikuli IDE will cause the start menu to close, thwarting your effort to capture an image. Now, you could use PrintScreen while the start menu is open, paste the image into an image processor, and then use Sikuli to capture the image from the image processor, but thankfully, there’s a better way.

Sikuli installs hotkeys for common tasks like capturing an image (CTRL + SHIFT + 2 by default), and they don’t cause the current program to lose focus. So you can simply open the start menu/context-sensitive menu of your choice and use the hotkey to capture the screen. That way the menu won’t disappear in the process.

3. click()ing when you should be type()ing

Sikuli’s pretty good at finding stuff on-screen, but it’s still a costly and error-prone process. If you can navigate a user interface by emulating keystrokes rather than clicks, you’ll save yourself a lot of trouble. Typing also has the benefit of sending events sequentially to the current program’s event queue, so if your program is stalling on something, your type() commands will wait for that task to be done, meaning you’ll run into problem #5 a lot less often. If you were to instead use click()s, you would have to tell your script in advance how long to wait() before it can take its next action, giving you inconsistent results if your machine is running slower than expected. Keep in mind that Sikuli can emulate key modifiers, function keys, arrow keys, etc., so you can specify some pretty complex interactions using only type() commands.

2. Forgetting that you’re using Python

Sikuli is a powerful and flexible tool, but remember that Sikuli scripts are written in an incredibly powerful and flexible language. Before writing that fancy UI-driven script to do something simple like change the system date/time, consider that the same task can be done in about three lines in Python or a shell script, saving you a great deal of time and headache. Whenever a task seems unnecessarily complex in Sikuli, ask yourself if it might be better solved programmatically, rather than visually.

1. Not knowing where to find help

Since Sikuli is still in the early stages of development, finding online resources to help you can be very difficult, to say the least. Here are some of the pages that I find most useful when I’m writing Sikuli scripts:

• Documentation / Guide – Perhaps the best Sikuli reference out there. Complete documentation of Sikuli, along with some example code.
• Bug Reporting / Tracking – Sikuli is still in beta. See if that problem you’re having is really a bug in Sikuli, not your script.
• Blog – Contains useful code examples and news
• Q&A – If you’re having an issue, there might be someone else who’s been there before.

[Mario is a summer intern with Baydin, and he is spending part of the summer automating functional tests for Boomerang using Project Sikuli]

Back in our previous jobs, we had begun to put all of our team files on SharePoint. As the site filled up with project schedules, business plans and customer reports, it became difficult to find the correct file. We used it as a collaborative dumping ground. Team members were alerted with an email describing the new file and its location. This created a problem: we filled each other’s inbox with “junk mail” so that we could keep track of our team documents. The second problem was that it was still hard to find the right file if we ignored the notification and did not know the title of the document or its location.

Microsoft is solving this problem with Microsoft Office SharePoint Server 2007. They have incorporated fast, effective search. The problem is that many companies have not yet upgraded to MOSS 2007. So we figured we could show you a way to still find documents while using an older version of SharePoint, which is what most of corporate users have. To do this you will need Outlook 2007 and Microsoft Desktop Search installed. You may know it as Instant Search in Outlook. This is the search bar that let’s you find emails and calendar items as you type.

What we are going to do is synchronize SharePoint with Outlook 2007. The files that are in your SharePoint site will be searchable through the “Instant Search” bar. This will save you a lot of time if you want to search for files using key words instead of having to navigate through the SharePoint site to locate a file.

Instructions to Search SharePoint without MOSS 2007 in 3 Easy Steps

Pre-requisite: We assume you have Outlook 2007 and Windows Desktop Search running.

Step 1: Go to your SharePoint Site and go to the location where all the files are kept. In this example, we have it in “Shared Documents.” So we go to that location.

Step 2: Click the “Action” button and select the menu option that says “Connect to Outlook.”

Step 3: You will see a listing of the files from SharePoint in your Outlook panel. These are synchronized with Outlook and can be searched through the Instant Search bar.

Great! You’re done. Now you can search files in SharePoint from your desktop without having to launch a web browser, find the bookmark and navigate through SharePoint.

If you want to be even more productive, you can run Baydin. It works within Outlook 2007 and will display relevant files from SharePoint, shared network drives, your local computer and inbox by analyzing the context of your email. Think of it like a really intelligent recommendation engine that shows you related documents in a side panel. Our early users have found files they didn’t know they had. Others have sped up product development because they discovered someone else in their company with relevant experience had published documents related to their project. Knowing is half the battle and Baydin shows you what you don’t know.

As recently as 2000, the deluge of emails, files, podcasts, blog posts and everything else that we have to keep track of was more like a drizzle.  The average hard drive held about 8 GB of data and we averaged about 7 non-junk emails per day.

As of 2009, those numbers look pretty different.  My laptop’s hard drive is a relatively tiny 160 GB; most computers come with at least 320 GB.  The way we work with email has changed too.  We now average 25 emails per day (almost a whopping 10,000 per year!) thanks to a lot of mailing lists and a lot of CCing.

Of course, we’re not suddenly 60 times more productive than we used to be.  Instead, we just get more of other people’s content.  Before Gmail made email quotas obsolete, CCing large files to everyone who might want a document wasn’t practical.  In 2000, blogs didn’t really exist, and the number of pages that interested each of us on the Internet was orders of magnitude smaller.

The problem only intensifies if we think about it from a corporate perspective.  How many gigabytes of data does your entire company have?  Where does it live?  At our former company, many groups had internal wikis, all of them had internal sharepoint sites (at least three, and as many as fifteen per group!), we had a document management library, we had personal websites with documents attached; everyone cared more about getting the job done than setting it up for other people to have an easy time finding what they created.

So there are now a lot more fragments of information in our brains and a lot more places that the rest of that information could be.  We spend a lot more time asking ourselves “where did I see that again?”  That translates into a lot of time and money. Bill Gates says that the average knowledge worker spends 11 hours a week looking for information, costing his/her company $18,000 per year in lost time.

The future looks like it is going to be even more chaotic – we will not only access more information in more places, but on more devices as well.  We will see some content on our computers, some on our $200 Netbooks, more on our iPhones or BlackBerries, and even more on our Kindles or Sony Readers.  And as we see more content on more devices, remembering where we saw the content we need NOW is going to get even harder.

A lot of productivity gurus are challenging us to “take charge of our Inboxes!” and implement a regimen that will help us manage the information.  But technology caused this problem.  Why isn’t it fixing it?

Fundamentally, the way we look for information hasn’t changed a lick since 2000.  Whether searching our computers or the Internet, we try to figure out what we want to find and we type it into a search box.  We get results that we hope are good enough – they often are.  When programmers have tried to improve on the search box, they’ve come up with some terrifying things.

I’ve attached a screenshot of the MIT Simile Seek project’s implementation of what is called faceted search below.  It’s a programmer’s dream.  I think I am wired to love driving tools like this.  It feels like piloting a starship.  If I know i want the 2nd top level domain to be .mit.edu because I know it came from someone at MIT, but I don’t know which lab, faceted search puts that power right at my fingertips.

But when I showed faceted search to anyone who doesn’t program computers for a living (like Electrical Engineers), they did not share my enthusiasm.  Other search improvements yielded similar gnashing of teeth.  The search box remains the search box.

So we’ve got a lot more content than we’ve ever had before, located in a lot more places than it’s ever been before, and we access it on more devices than we’ve ever used before.  And we still do pretty much the same things to find it that we did in 2000, when we had a lot less content, all on one hard drive, all on one computer.

So there’s a lot to fix.  And we’d love to fix all of it!  But for now, we’re trying to siphon off just one aspect of the problem where we think our technology can make a big difference.  In a few days, we’ll talk more about how we’re going to do it.

There are two major advantages to a modern desktop search experience: the first is that searching for a document is a lot faster than it used to be, and the second is that in virtually all file types, the text inside the document is searchable, instead of just the filename.

Think back to the file search in Windows 95.  It was pretty terrible.  All it could do was search for filenames, and it took the better part of eternity to find anything.  Here’s why: when someone searched for a word, Windows opened the file system and looked at every single file it had.  It compared the search query with the filename for each file, and as it found matches, it added the files to the results listing.  Every time a new search started, Windows had to look at every single file, which is why the results trickled in over a period of a few minutes.  If the search term were somewhere in a document or in an email rather than in the filename of a physical file, we were pretty much out of luck.

Searching the full text of documents was beyond the pale.  To do that, Windows would need to open every single file as it came across them and extract the text.  It would have been slower than slow, it would have required every piece of software that saved any kind of document to provide hooks for Windows to extract the text, and it probably would have made the computer rottenly unstable. 

Searching through email in Office (up until 2003) used the same method, but since every email had a known structure, Outlook could search through the full text of messages.  When a user started searching for something, Outlook opened the most recent email and compared the search terms against each word in that email.  If there was a match, it would add the email to the result list in real-time.  When it finished with the most recent email, it would move on to the next, then to the next, then to the next.  Searching through email was a slow process, but it would eventually yield results where the terms were found only in the text of emails.

A real innovation happened, though, when software developers realized that the same technology that powers web search engines could be applied to the desktop.

When someone clicks the search button on a web search engine, the search engine responds in a totally different way from Windows 95-style search.  Google does not crawl every page on the web, word for word, comparing the search terms for a match.  Instead, Google just looks in a previously-generated database where they already have prepared a list of all the web pages that contain the search term (and a bunch of other information that helps them order the results!)

Instead of sifting through every word ever written on the Internet in real time, Google crawls each page on the web only every few hours, days, or weeks depending on how important a site is and how frequently its content changes.  When Google crawls a site, its crawler looks through every page, processes every term, and updates the database. 

Very crudely, that index looks like this:

All Google has to do when you search for “chicken” is find that index and list the results.

Of course, that’s a sweeping simplification – it doesn’t address multiple-term searches, result order, or the fact that the index is HUGE and difficult to maintain.  There are dozens of fantastic papers from Google engineers that explains a lot of the details; try http://labs.google.com/papers for a listing, or start here for an overview from when Sergey and Larry were still at Stanford.  But for the purposes of this post, that’s all we need to worry about. 

Creating and maintaining a mapping from search terms to web pages is the critical innovation for desktop search.  The idea extends quite well to our individual computers.  Instead of a mapping from terms to web pages, though, we need to make a mapping from terms to documents. So the problem is a little bit harder in that we have to be able to index a whale of a lot of document types instead of just HTML, but it is a lot easier in that the index size is nowhere near as large as the index for the web.  It can be generated relatively fast (probably under an hour for the average computer) and does not require a lot of space.

Google Desktop Search, Windows Desktop Search, and all the competitors do exactly this.  Their indexer runs in the background, opens every file on the computer, and creates a database in the same format as the web databases above:

When I search on my computer for a word, like the web search engines, all my computer now has t
o do is look in that index and find the already-generated list of files that match my term. 

The key takeaway is that thanks to these indexes, searching through the full text of every file on a computer is now thousands of times faster than just searching the filenames used to be.