The problem, of course, is that not everyone is on the same social network. And you might not want to be on any of the social networks that I’m on. Or you might not trust any of those companies. Enter ping.fm.

ping.fm serves as a central point for me to blast out my navel-gazing, mind-numbingly boring status updates to a variety of services (all of the ones mentioned above and more like Plurk, MySpace, Pownce, LinkedIn, Tumblr). This means when I make a status update at ping.fm, it gets fanned out to all the sites I’ve ping-enabled (of course I have to give them credentials for each site).

All of this is great but not totally innovative, it’s just another layer of madness above the existing, silo’ed madness. And it doesn’t solve the fragmented two-way conversation problem (replies on Twitter stay on Twitter, etc). But what caught my eye this evening was the ability to fan my ping.fm updates to custom URLs. This means I can set up a PHP page on my own website and ping.fm will send the status updates to that page also. This opens up some interesting possibilities.

So I did just that. I now have a microblog/status feed on this site if anyone cares when I brushed my teeth or if I’m late for a train. Be the first to know!

How it works: I use ping.fm to post my status and it is fanned out automatically to Facebook, LinkedIn, Twitter, identi.ca and codedread’s custom PHP. That PHP takes the status and adds it to a static Atom file on my site.

So now that I’ve got a microblog feed that’s disentangled from any one particular social network, I’ll probably add some code to my site to display the latest status from my microblog feed somewhere.

Oh and if you don’t like the funky character in the URL then you can also use this one. They both point to the same location.

One request to ping.fm: Please support OpenID!

Matt from WordPress disses the heavy-handed move to PHP5. I personally like PHP5 better than PHP4, though my site was using PHP4 for everything. I notice Matt says “there isn’t a released PHP app today that isn’t PHP 5-compatible”. But I don’t think this can be true, unless he means PHP apps that matter to him. They point to this page which describes the differences between PHP 4 and 5, but I think something huge is missing from this page: DOM manipulation. There is a potentially big change to web apps out there that use DOM XML.

The only thing I had to do when migrating was to change my DOM XML function calls to use DOM.

• DOM XML will “never be released with PHP 5″.
• DOM is a PHP5-only feature.

These changes affect every line of code that used a DOM XML function, but the changes are pretty trivial, especially if you already know the DOM. XSLT handling is also not backward/forward compatible, from what I understand.

But why does that PHP page neglect to mention this? This was pointed out in 2004 by Justin Gehring, but the page fails to have been updated as an official notice. Alexandre Alapetite has provided a module that, I think, eases this transition (Disclaimer: I have not used this module because I rewrote my PHP for PHP5).

In all likelihood, I am missing something obvious here – so please chime in and put me on the clue-train. Do people typically not manipulate the DOM in PHP?

It’s nothing fancy, I just thought I’d save somebody the time of writing it themselves:

Simple usage to create a <g> element and append it to the <svg> element $svgnode. ($doc in this context is the DOM XML object i.e. the “document” in JavaScript):

To set several attributes, you send an array of attribute name/value pairs to the function:

To append children to the node you’re creating, fill an array with the nodes:

You can use this technique to create a tree of nodes to an arbitrary depth, though the biggest pain with using this technique is getting the correct number of right parentheses, make sure you have a good editor to match them up.

At first glance, this seems like such a simple thing. Most people out there think it’s as simple as using the <object> tag, and let me be clear: ideally it should be. However, in my experience the Adobe SVG Viewer (especially version 3.0) tends to have a problem including SVG documents in an <object> tag and prefers to non-standard <embed> tag, Of course, other browsers (Firefox and Opera) have various problems with <embed>. Typical web developer frustration.

I found it interesting to see how my practices had changed on this matter in seven or so months:

• My first approach, like this guy, was to simply use <object> tags because I didn’t know at that time that ASV only worked perfectly with <embed> tags.
• Upon learning this, I wrote some JavaScript that would scan the page and transform any object tags referencing SVG content into embed tags if ASV is detected as installed. This has the benefit of keeping the web page author out of the sticky mess (except for including the JavaScript in the document’s onload event handler) but it resulted in reduced page loading time. Furthermore, I later discovered that ASV acted funny with this sometimes (i.e. it wasn’t a perfect solution). Another problem is that the embed tag does not provide any mechanism to provide fallback content. This means that if a user hadn’t downloaded ASV, their page would be broken.
• In the fall of last year, I learned about IE’s “conditional comments” from Stefan Oskamp who devised a technique of including an SVG image in a HTML document so that I could drop the JavaScript mentioned above. I published that technique here. Using conditional comments, I can supply an embed to IE and an object with fallback content to all other browsers. It has the benefit of providing a static document (i.e. no scripting involved that changed the document at load time) but it has the disadvantages of 1) lengthier HTML code and 2) embed still doesn’t provide a mechanism for fallback content
• At the end of last month, I finally put most of the pieces together and used some JavaScript with the above solution to provide fallback content dynamically. Unfortunately this lengthens the code considerably and the author has to write the fallback content in THREE different places. It also requires scripting be enabled to see the SVG in IE (though I consider this to be acceptable).

So now we arrive at yesterday where I realized the obvious: Since my sites are PHP-based, I will simply provide a PHP function that constructs all the HTML+JS code for me. If Adobe releases a viewer that handles object properly, I can update my PHP function to simply spit out an object and forget about embed forever.

Click here to view the PHP source

To use this PHP library, you include the file once at the top of the web page like so:

Then to “inlay” an SVG document into your HTML you use this PHP fragment:

This will inlay myfile.svg into your HTML page via an object/embed frame (depending on the viewer’s browser) with id=’some_id’ and dimensions 500×375. If the viewer’s browser does not support SVG, fallback content is provided in the form of a link to download the SVG Viewer.

One caveat is that the fallback string must be properly formatted PHP string. This means that if you start the string with single quotes in PHP, then you should use double-quotes within the fallback string to represent quotation marks. In other words, the string has to be valid PHP and the string contents have to be valid HTML.

The HTML code that this PHP function call produces is:

You can see the reason why the PHP file is a good idea. The above is a pretty hefty stretch of code to simply include an SVG document with fallback content. The PHP file keeps your code much cleaner and you can centrally maintain the inlaying functionality.

Now if you don’t have PHP you can still use this technique, it just requires you to manually enter all that code yourself on your HTML pages whenever you want to inlay some SVG.

Now while the results aren’t as pretty as the static rasters, I did provide some minimal interactivity/dynamism that folks like Jon Udell might appreciate.

Without further ado, my live web statistics SVG image is located HERE. It is constructed using a variety of techniques:

• Include a small PHP file to record each document request on my site and blog. Each result is written into a XML file.
• I have another server-side PHP script to crunch each day’s worth of hits into summarized daily statistics XML file.
• I have yet another server-side PHP script that reads in all daily statistics XML files and crunches them into the SVG you see.
• The interactivity is provided via SVG and JavaScript. Basically mouse and keyboard listener events handle moving the “day cursor” around the time-based graph to affect the pie chart

Go ahead and play with it. It seems to work for me in Firefox 1.5 and Internet Explorer with Adobe SVG Viewer. It also displays in Opera 9 TP1 but the dynamic/interactivity part does not work in that browser (hoping to see it fixed in the next TP). Can someone with a nightly build of Safari tell me whether anything shows up?

As with JavaScript, creating SVG using PHP’s XMLDOM library is tedious at best. If I knew XSLT I would have given that a shot, but I don’t (yet) – so I didn’t.

Nonetheless, I’m quite happy with the result because now that it’s written it requires very little manual action on my part to keep it up-to-date. I could hook up calls to my server-side generation scripts to a cron-job if I had a server-machine under my control (mine currently has a fried motherboard), but I actually prefer to ensure that the XML is being properly generated, so for now I’ll just invoke the server-side crunching scripts every few days to keep the SVG fresh.

Click here to see my SVG web statistics in a full page.

NOTE: One challenge was to properly embed the SVG image into this blog entry with fallback content. For IE+ASV, I can serve an <embed> tag via a <script> with the fallback being written by a <noscript>. For all other browsers, I can serve an <object> tag to Firefox and Opera the way it should be. However, it seems that WordPress mangles things to a degree such that it will not load inside WordPress. I eventually gave up on this, though I have successfully used this technique in my own HTML pages.

My design philosophy in the last year or two has switched from “Big, Massive Projects That I Can Never Finish Given My Diminishing Spare Time” to “Small, Constantly Evolving But Always Fully Functional Projects”. The paradigm shift was a result of simply not having enough time, energy, and enthusiasm to commit to a large project and constantly feeling the pain of never finishing what I started. Now, I can play with a project, add a few new features to it and put it away for some time. At all stages of “release”, the project is fully functional, yet I’ve always got a laundry list of future features which I’ll eventually get to. So far, it’s led to a couple interesting little applications, the most complete of which is the afore-mentioned instant messaging web application.

Anyway, the Instant Messenger was unique in my experience so far because there were multiple users who, at any point and time, may be using the latest version of the application.

Deploying the new version into the same directory was not a good idea because the instant messaging application relied on asynchronous JavaScript calls to fetch conversation data that was uniquely identified. In short, adding in new features had the potential to break existing sessions.

Thus, I ended up developing and testing the newer version in a private web directory, then broadcasting to all participants: “Ok, I’m going to deploy the next version of the chat program so everybody log out for a couple minutes”. I’d then scurry around in my website file manager, setting up some config files, rename some directories and then log back in and wait for the other participants to rejoin.

Not the best deployment plan, I know.

Tonight I struck on an easier way for me to deploy the application without having to force people to log off.

First, I set up the directory structure like this:

/chat/
/chat/1.0/
/chat/1.1/
/chat/1.2/
/chat/WIP/

All versions of the chat program code are located in subdirectories inside /chat/. The most recent “live” version in this example is located at /chat/1.2/. The 1.2 directory houses all the PHP, JavaScript, user information, conversation and state information for that version of the application. For instance, /chat/1.2 might contain:

/chat/1.2/index.php
/chat/1.2/some_scripts.js
/chat/1.2/serverside_stuff.php
/chat/1.2/rss.php
/chat/1.2/users.passwd
/chat/1.2/chat0000.xml
/chat/1.2/chat0001.xml
…

Similarly, the /chat/WIP/ folder is my “Work In Progress” folder that allows me to experiment freely without upsetting anyone.

Now it’s kind of a hassle to ask people to browse to http://www.mydomain.com/chat/1.2/index.php and then when I update to 1.3, tell them they have to update their bookmarks accordingly to http://www.mydomain.com/chat/1.3/. Such a scheme would hardly be “user friendly” and it wouldn’t scale properly to an application with a large user base. Same goes for the RSS link (http://www.mydomain.com/chat/1.2/rss.php).

So what I did was a simple PHP trick. I created the following files

/chat/
/chat/deploy.php
/chat/index.php
/chat/rss.php
/chat/1.0/
/chat/1.1/
/chat/1.2/
/chat/WIP/

The content of these files are as follows:

/chat/deploy.php:

<?php
    $DEPLOY_MYCHAT_CUR_VER = "1.2";
?>

/chat/index.php:

<?php
    include 'deploy.php';
    $urlstr = "http://www.mydomain.com/chat/" . $DEPLOY_MYCHAT_CUR_VER . "/index.php";
?>
<html><head>
    <meta HTTP-EQUIV="Refresh" content="0; URL=<?php echo $urlstr; ?>" >
</head><body/></html>

/chat/rss.php:

<?php
    include 'deployment.php';
    $filestr = "./" . $DEPLOY_MYCHAT_CUR_VER . "rss.php";
    include $filestr;
?>

Now what’s the benefit of this? Well, if a user browses to http://www.mydomain.com/chat/ they will end up at http://www.mydomain.com/chat/1.2/index.php. i.e. the most recently released version of the application and all their doings-and-goings-on will be isolated to the /chat/1.2/ folder.

Meanwhile, I finally finished up feature XYZ in the WIP and want to deploy the new version. What do I do?

I create a /chat/1.3/ folder and copy the contents of the WIP into it. Then I modify /chat/deploy.php such that $DEPLOY_MYCHAT_CUR_VER equals “1.3″.

Now anyone currently logged into the chat program can safely stay on 1.2 for as long as they want. However, the moment they log out, when they try to log back in, they will automatically end up at http://www.mydomain.com/chat/1.3/. What I like about this is: 1) there is no requirement for any user currently using the application to log out; they are safe in their “1.2″ buckets and 2) there is no requirement for me to rename existing folders, mess with configuration files, etc. Deployment is a simple two-step process.

To end this entry, I’ll discuss a third thing I like about this technique. I could centralize the data (i.e. the conversations, the user information such as passwords) into the /chat/ directory and allow any version to access this common data. This means that someone using version 1.2 would be able to chat with someone using version 1.3. As long as the backing data store is in a format that easily downgrades for older versions (i.e. XML elements/attributes or database tables/fields that would simply be ignored by older “less-smart” versions). Of course doing so might require some type of synchronization across deployment versions and may end up causing me another deployment nightmare all over again.

The above will not produce “5″. Without explicitly telling PHP that gSomeVariable is a global variable, it will assume it is a local variable and currently undefined.

Do you know how many time this has tripped me up? The correct PHP code is:

For all the times I made this mistake: AAAAAARRRRRGGGGHHHH!!!!!!!

Properly Set the Content Type of the Returned Document

I stumbled around for a bit and couldn’t figure out why the XML document returned by the PHP server script wasn’t being parsed into walkable DOM objects at the client side by my JavaScript. Finally it dawned on me that the client must receive the XML content type in the header. I used the following PHP code (before any output):

And suddenly everything started parsing from the client side!

Preventing Browser Caching of the URL

When I finally got an XML document returned, I noticed that Internet Explorer would not retrieve a newer document from a subsequent request (though somehow Firefox was able to get a new XML document). Once I determined that different browsers were experiencing different results, I had an inkling that it was a caching problem on the client-side. When doing periodic “get” via the XMLHttpRequest document, apparently Internet Explorer was checking its cache and noticing that the same URL had been fetched previously and just using that stale document. IE had no way of knowing that the retrieved document could by dynamic content that might vary over time.

The way around this is to modify the URL slightly so that IE believes it is a new document. I did this using the following JavaScript code:

This inserts an (unused) URL variable into the string and fools the browser into thinking it is a new document (which, in fact, it is!). I have no idea if this should be considered a bug from the IE side or the FF side…

PHP File Locking

Since my little application uses the server’s filesystem for its backend storage (and not a database) and transactions can occur in parallel and asynchronously, I took some care to make sure I locked the server-side files before accessing them. I used PHP’s “flock” function:

Though I never experienced any odd conflict scenarios even before I implemented file locking (due to the relatively low network traffic that my app experiences), I’m convinced that this is a necessity for any real application (and a very persuasive argument to using a database as the backend storage mechanism).

Here’s what I wanted to do:

1) User types in some text on a form and submits the form
2) JavaScript mangles the text so that it is not easily readable and sends the disguised text to a server-side script
3) Server-side script writes any text received to the end of a “conversation” file as XML
4) Server-side script mangles the XML document and sends back to all attached clients
5) Client receives the mangled text, un-mangles and generates an XML document, extracts the text, and displays on the screen

The overall process is pretty simple and is represented by the following diagram:

The frustrating part came when I started encountering the quirks of the various languages/syntaxes that I was using coupled with the requirement to be able to pass any text to the server. Things got confusing fast, and as they always say, the devil is in the details.

Sending Text from JavaScript to PHP

The transition from user text to mangled text is a simple function (written in both JavaScript and PHP) that takes an input string of N characters and returns an output string of N characters. The function works transitively (such that mangled text becomes un-mangled and vice versa) for convenience. This is the really simple part of the process:

However, it is when we want to actually send this output string as a URL argument to a server-side PHP script that we encounter our first problem. For example:

Just as with text entered from the form, the mangled string could contain any printable ASCII character. But the ampersand character (“&”) is used to delimit URL variables in PHP and valid URLs cannot have spaces (” “) in them. There are other characters that have special meaning in URLs as well. Thus, if we tried to put our outStr into the URL it would become:

The portion after the space (i.e. “A;(z;+z“) is removed from the URL. Also, the $TEXT variable from the PHP side would contain only “ubW(” and the remaining string is interpreted as another PHP variable called “2v“.

What we want is for the PHP script to see one variable called $TEXT with the complete mangled string. Thus, it is important that we escape the mangled string such that spaces become “%20″, ampersands become “%26″, “=” become “%3D”, and other characters like “%” become “%25″. i.e. any character that might be interpreted in a special way inside a URL Thankfully we can just use the JavaScript escape() function for this. Handy-dandy:

Now we can send this string to the PHP script, right? Almost, but not quite! It seems that the “+” character in a URL is automatically translated into a space by PHP. Thus, if we really want to send the + character in our URL we’ll also need to escape it as “%2B” (and only AFTER we escape the rest of our string, otherwise its “%2B” would become “%252B”!!):

Is the string safe for PHP now? Yes, it would seem so.

[NOTE: When discussing this topic with a friend he told me about the encodeURIComponent() function. However, I had already gone through the above learning experience and I also noted that encodeURIComponent() is not supported in MacIE or Safari...poor Mac users.]

The nice part is that once passed to PHP, the PHP interpreter automatically un-escapes any %xx characters such they magically return to their equivalent single-character representations again.

But! Apart from unescaping the %xx characters, PHP can also do some mangling of its own. This depends on whether or not the “magic quotes” feature is enabled or not. If it is, then PHP automatically adds backslashes to specific characters (single-quote and double-quotes) presupposing that you’re going to try and shove the text into a database. This is NOT what we want, as extra characters changes the content that was originally typed when you un-mangle things. i.e. we’ll have extra characters in our returned string. You can avoid this on the PHP side by either configuring PHP to not use magic quotes (via php cfg files or .htaccess) or use some simple PHP code:

Now $TEXT should contain “ubW(&2v A;(z;+z” again. We’re finally back to our mangled version of the text from the JavaScript.

However, the fun doesn’t stop there. In fact, we’ve only made it from (1) to (2) in our diagram.

Creating XML in PHP

Now we need to un-mangle the text and shove it into an XML document:

There, now we have un-mangled text, but before we can blindly shove it into a XML document, we have to make sure that the original text typed by the user doesn’t violate the allowed characters of an XML document (namely < and > characters). As you may know, XML documents rely heavily on markup tags and thus, characters such as < and > are verbotten within the XML document contents itself. Since the receiving clients expect a valid XML document we have to take care that any of the data we’re shoving into the XML document does not invalidate the syntax required. PHP provides a simple function for this called htmlspecialchars():

We are finally ready to format $thechatmsg into a XML document and write to a file:

We use xml:space=”preserve” to indicate that whitespace within the XML document should be preserved.

Now we’ve made it safely from (2) to (3). We’ve sent the mangled text across the network to a PHP script and the PHP script has un-mangled it, written it into an XML document to the filesystem. Next, we have to send all new messages back to the JavaScript call in the form of an XML document (as a response to a XMLHttpRequest).

Sending Text from PHP to JavaScript

This is actually pretty straightforward PHP code:

Now we’ve made it from (3) to (4). The only thing left for the client to do is to un-mangle it, reformat it into an XML document, walk the document nodes and extract the information.

Creating an XML Document From Text in JavaScript

Since browsers have different means of doing this, I decided to use Sarissa which abstracts these differences and lets me just deal with the XML document itself.

The mangled text comes back as the responseText field of the XMLHttpRequest object (not as valid XML). I take this string, un-mangle it, and create an XML document out of it:

The one caveat I had is that regular HTML does not (by default) preserve whitespace. To get around this, I used the CSS rule “white-space: pre”.

Overall, there are two main (and obvious!) problems with programming for the web that I’ve experienced so far:

1. Lack of Cohesion

   I’m coming from an environment where all I do is program in C++. To do anything non-trivial for the web requires a solid understanding of at least 5 different technologies: (X)HTML, CSS, JavaScript, PHP and the DOM (I separate this from knowing the core JavaScript language). If you want to use a database, you’ll also need to know some SQL and how to interface PHP (or ASP or whatever) to the database. If you want to have some interactive graphics/animations, you’ll likely need to know Flash or SVG. Now we’re up to seven different technologies.

   Learning all these new languages is great, at first. You can plop those babies onto your resume and sit back and watch the job offers roll in. However, actually working on a web application can either be tackled in two ways: 1) design up all your interfaces between the various technologies beforehand and then tackle each side in turn (do all your PHP, then work on the client-side bindings, then work on the presentation code, etc) or 2) start to cobble together pieces of things as you work in the hopes of getting something you can watch evolve into your final app.

   I’ve got (maybe) an hour or two to spend each day on something like this. Which way do you think I work?

   Of course, the problem with working on everything at once is that it is really disorienting to switch gears. I think I’m getting better at the switch between PHP and JavaScript but it still leaves me a little breathless when I do.

   The other problem is that it takes a ridiculous amount of time to do everything you want to do because of the steep learning curves. Being a man with other priorities (aka “my family”) means that the amount of time I can spend learning anything has been drastically reduced since my single days (not that I’d ever trade my life for that). Expecting to learn seven new technologies and come up with something “killer” is highly unrealistic (and, well, just plain sad, really). The best I should be hoping for is to focus hard on a one or two areas and partner with others to cover everything. Alternatively, I can continue creeping along the seven paths and deal with the whole thing as a hobby to fill the time.

   Along with the “developing-everything-at-once” syndrome, there’s also the matter of debugging/testing your app. Basically to test an end-to-end application requires a web server running PHP and potentially a database (MySQL) set up and configured on any machine you want to test it on. The other option is to constantly FTP your server-side files to your “live” web server into a sandbox location (which is what I do, since I tend to work on the same things from different machines throughout the day).

2. Lack of Standards Compliance

   The server-side, I can deal with, and I can deal with it pretty well. The environment is fixed and even though I’m not fully in control of it, once I get things straight I really don’t need to worry too much about it.

   The problem comes with all the different browsers (user agents) out there that might chance upon my application / website. Each browser is made by a different set of developers, each interprets standards a certain way, each has various levels of standards compliance, each has its own sets of bugs and each has special (aka “non-standards compliant”) features they have chosen to implement. Getting even a few browser configurations right when programming up the client-side has proven to be quite a learning curve for me (considering I’m also learning JavaScript and the DOM). At this point, I’m targetting very modern browsers: IE 6.0 and FF 1.0. I may venture into Safari, Opera, and Netscape at some later stage long after things are stable and well-tested but I don’t think I’ll ever try to be compliant with older browsers (Netscape 6-, IE 5-, etc).

Now the problems I’ve listed above may be very obvious to a web developer, but what amazes me is that everyone is ok with them. Web developers seem to thrive in this environment where they have 8 different (and very sharp!) knives instead of one big bazooka. I can’t say I can see a solution either: 8 knives it is…they all do different tasks (and they do them pretty good).

I’d like to make at least one suggestion: Someone come up with PHP for the web browser, will you? By this, I mean being able to write a client-side script in PHP in a web page that can access the DOM. At least if we have this then we can use one scripting language on both ends: server-side we have the core PHP and the massive PHP libraries invoked by the server and client-side we have the core PHP language and the DOM invoked by the browser. At least then I don’t need to remember whether my variables need dollar signs in front of them or to be declared with “var”…