Do you need to extract text from Word, Excel, PowerPoint, and Adobe PDF files? We recently added full text searching of uploaded files to ProjectSpaces. Thanks to the work of other hackers, we were able to utilize command line utilities to extract meaningful text from proprietary, and/or binary files. Once implemented, we could index the majority of files uploaded by our users and present matching files for search queries.

catdoc for Word and Excel files

The catdoc package (Debian, Ubuntu, RedHat distros). On a Debian or Ubuntu server you can install the package with

sudo apt-get install catdoc

To install on our RHEL4 server, I had to find and download an RPM file for catdoc, and install tk

sudo up2date tk
sudo rpm -ivh catdoc-0.94.2-3.el4.i386.rpm

Once installed, you will have two utilities for extracting from Microsoft Word (catdoc) and Microsoft Excel (xls2csv) files. Using catdoc is as simple as:

catdoc -w ~/MeetingNotes.doc

If successful, the contents of the Word file will be piped to STDOUT. The -w flag is used to suppress word wrapping, otherwise catdoc will wrap lines at 72 characters.

Simlary you can get the content from an Excel spreadsheet with:

xls2cvs ~/Sales.xls

Executing the command above pipes the contents of your spreadsheet to STDOUT with comma-separated values.

xpdf for PDF files

The xpdf package provides a command line utility named pdftotext, which can parse PDF files up to version 1.5. This is a straightforward package to install on Debian or RedHat servers either:

sudo apt-get install xpdf

OR :

sudo up2date xpdf

The utility pdftotext is a little trickier to use, because by default it wants to save the extracted text to another file. By specifying - as the target file, output is sent to STDOUT instead.

pdftotext ~/DeveloperResume.pdf -

xlhtml for Powerpoint

Although catdoc also bundles a utility named catppt. I didn't not have any success in getting meaningful output from Powerpoint files with catppt. Instead, I settled on using ppthtml. This utility maybe a little harder to find a package for your distribution. On Ubuntu, it can be installed with:

sudo apt-get install ppthtml

For Redhat systems, look for the xlhtml package, I was able to find xlhtml-0.5-2.el4.sme.i386.rpm for our server and installed it with

sudo rpm -ivh xlhtml-0.5-2.el4.sme.i386.rpm 

Usage of ppthtml is straightforward:

ppthtml ~/ConferenceSlides.ppt

The text of the Powerpoint file will be sent to STDOUT and formatted as HTML. Since only text is extracted, images and text contained inside images will be ignored.

Capturing content in PHP

Above, I emphasized how to send output to STDOUT. By doing so, with php we can then use the exec command to extract text and process it further or save it to a database table. A function for working with a word file might look like:

/**
 * Attempts to extracts text from a MS Word file
 * @param string full path to file
 * @return string
 */
function extractWord($word_file) 
{
    if (file_exists($word_file)
    {
        // prevent malicious command execution 
        exec("/usr/bin/catdoc -w ' . escapeshellarg($word_file), $output);

        // $output is an array corresponding to lines of output
        return join("\n", $output);
    }
}

Functions for extracting from Excel, Powerpoint, and PDF files would only require switching the command line tool.

What can we do with such tools?

1. Add them to a mysql table for simple FULLTEXT searching.
2. Use Yahoo's Term Extraction service to extract meaningful keywords
3. Index them with a more advanced full text search engine.
4. Provide a text preview of file contents so that users don't fire up a client application or browser plugin.

I'd love to hear other ideas too!

I guess its because MS doesn't follow the javascript specification, but the following bit of code works in Mozilla browsers but not in MS IE6:

<script type="text/javascript">
    // jquery selectors
    var class = 'div.' + $(jquery_selector).attr('name') + '-error';
    $(class).html(msg).show(); 
</script>

Of course, since one need E.S.P. to debug javascript in Internet Explorer, normal people can waste a lot of time figuring out why it won't work. Simply renaming the variable named class does the trick:

<script type="text/javascript">
    // jquery selectors
    var my_class = 'div.' + $(jquery_selector).attr('name') + '-error';
    $(my_class).html(msg).show(); 
</script>

Finally, in 2007, some people have figured out how to provide email communications as an open service. Despite the popularity of syndication feeds with first-adopters, and tech-savy users, e-mail remains as the medium of choice to reach most stakeholders. This may be changing, as younger users see e-mail as a tool old people use, but that is a post for another day.

Programmable web lists 16 online APIs available for working in one way or another with email. Traditionally, you had the choice of outsourcing it to a proprietary vendor, or keeping it in-house. The former meant your data was silo'd away, frequently with only rudimentary tools for importing or exporting data. The latter, required you have technical expertise on operating and maintaining mail servers 24x7, a full time job for sure.

E-mail APIs provide rich access to your lists.

Programmatic access via one of these services opens up a number of intriguing integration opportunities, both for how you manage communications internally, and how you present email offerings on your own site. You could:

• Integrate sending email blasts to constituents with another contact management application or your membership management database.
• Automate regular email messages, generated from content added to your Website or CMS
• Customize the subscription process to your liking. Visitors would never have to leave your website to a 3rd party site, instead your web server talks behind the scenes with the email service through the API.