I just wrote a script to easily download your FULL friendfeed stream. I mean: every post with comments, likes and even images and/or files. Enough for the good news: the bad news is that this a PHP command line script and you have to know how to run it. On a recent  OSX or Linux environment you should be ready to go. For Windows-ers it could be a little more difficult. Sorry.

Anyway, here is the features list:

• can fetch a private stream, giving your remote key
• locally downloads images and files (detects already downloaded things)
• upon subsequent runs, it tries to get only the newly added posts
• the output file is JSON
• it scales well!
• it’s free software (MIT license)

Download the scripts from Github (the link points to the project page, where you can read the README too. In this case you’re interested to the ffexp.php script).

You then may run it like:

php ffexp.php somefilename.json

As you may infer from the output file extension, the output is plain good JSON. This format is very handy for whatever thing you may need to do with the export file. But this is up to you (or your programmer friend).

As an example of JSON postprocessor I also wrote a converter that given your export file will render a fully functional HTML file, looking just like a Friendfeed page. It also automatically embeds the necessary JavaScript, CSS and even icon images. Download the ffexp2html from Github, open it, read the fine documentation and run it like:

php ffexp2html.php somefilename.json > somefilename.html

Now you can open the somefile.html file with your browser (no internet connection needed). You may find a generated output here.

Limits? Yes there are:

• No download of your DMs (don’t know exactly why, maybe a limit of the API. But I don’t care)
• You can download even entire rooms (just use the room name as the username), but it seems that there is a limit of how back you can go. Very high, but it exists.

I wrote a little script in PHP that converts your bookmarks file exported from Delicious in some other formats. The script, if you’re in the mood, is even easily extensibile if you provide it another “driver” to convert in another format (you write it and then plug it into the script. Pretty straightforward).

The script is ready to be used if you need to convert the file in CSV.

Run it like this:

php deliconv.php delicious_export.html > delicious_export.csv

Another idea on how to use the script would be to check every URL for its existance. It’s a pretty easy task if you know how to use the curl extension. I’ll leave it as an exercise for the reader :)

So, if know how to run PHP from the command line, grab the script here (remove the .txt extension).

There is also a Delicious XML exporter, out there.

Update: my friend Nicola D’Agostino has collected some other useful informations and tricks on the matter.

Another login, another password checking, another… Boooring. Given the nature of the application I decided that this time I could cut the mess down and go only with the «Use your Google account» “one click” login. Why not? Lot of people have an google account already and it’s OpenID!

The usual Google documentation page is this one. The good news is that if you follow this guide you probably will not need to read that whole page :) Later on you should integrate this very document with the Google one.

The first thing to know is that you are going to write an OpenID client/consumer. OpenID 2 protocol is indeed the system behind the whole client/server scenario from your application (the client) and the Google accounting system (the server). I have to assume that you know what that means (at an high level, at least). The examples here are in PHP and below I will provide some sample code for you to play with.

OK; so here are the relevant points:

• This one is theoretical, but it’s how Google deployed the OpenID protocol on top of his already existing authentication system. Normally, in a “pure” OpenID environment, during the login process the user will give you his/her openid identifier (an URL); using that identifier your program will discover (via the Yadis protocol) which is the corresponding server party to talk to for the authentication process. In the Google scenario you don’t know which is the user’s OpenID identifier (the user, of course, is not even aware that there’s something like “openid” involved in the way s/he logs in). You have NOT to ask her for her “google account”; you have to make him press a button or follow a link to “Login with your Google account”. But you DO know who you have to ask for protocol informations and not have to infer it from the user’s OpenID (technically, a discovery process is needed anyway). Upon a succesfull login you WILL have the user’s OpenID identifier; it’s something like a generated string, not choosen by the user and crafted by Google itself.
• You need a recent OpenID library that supports OpenID 2.0 XRI and Yadis discovery. This excludes (as for now) the Zend Framework OpenID component (it does support OpenID 2.0, but not the discovery part in the version Google uses). Too bad. You then need to go straight to the JanRain library
• Don’t bother thinking about Simple Registration (SREG): Google does support the more advanced (and useful) Attribute Exchange (AX) extension only
• Speaking of profile data, there is nothing much you can get: email, name and surname. I noticed that unless you specify each one as mandatory (within the AX request), the user is not even asked – by Google – to give them to you. There’s probably somethng I misunderstood, but this is it.

On your side, the login process will consist of two distint phases: one that initiates the login process the and one that finalizes it.

Normally you point your “Login with Google…” link to the script that implements the initiation phase (let’s say “login.php”). The finalize script (let’s say “finalize.php”) will be called by the Google server as per protocol. There is a lot more to know about the OpenID process, of course, but the library should hide the gory details for you.

The login script will immediately and trasparently forward the user to the real Google account login page. The finalize script will have the logic that checks if the authentication has been succesfull or not and act consequently.

I’m about to show you some code, but keep in mind that this is NOT valid PHP code; I want you to understand what’s going on rather than being able to just copy/paste. To further limit clutter I did not even add basic error checking. Please check the JanRain library examples for full fledged (and working) code.

So, for the login process:

# Requires the needed JanRain scripts
require_once "Auth/OpenID/Consumer.php";
require_once "Auth/OpenID/FileStore.php";
require_once "Auth/OpenID/AX.php";

# Setup the library (leaved blank on purpose).
# Will also create the $store variable used below
...

$consumer = new Auth_OpenID_Consumer($store);

$auth_request = $consumer->begin("https://www.google.com/accounts/o8/id");

# Create attribute request object (this is where you prepare the Attribute Exchange request)
$attribute = array();
$attribute[] = Auth_OpenID_AX_AttrInfo::make('http://axschema.org/contact/email', 2, true, 'email');
$attribute[] = Auth_OpenID_AX_AttrInfo::make('http://axschema.org/namePerson/first', 1, false, 'firstname');
$attribute[] = Auth_OpenID_AX_AttrInfo::make('http://axschema.org/namePerson/last', 1, false, 'lastname');

# Create AX fetch request
$ax = new Auth_OpenID_AX_FetchRequest();

# Add attributes to AX fetch request
foreach($attribute as $attr){
  $ax->add($attr);
}

$auth_request->addExtension($ax);

$return_to = sprintf("http://%s:%s%s/finalize.php", $_SERVER['SERVER_NAME'], $_SERVER['SERVER_PORT'], dirname($_SERVER['PHP_SELF']));
$trust_root = sprintf("http://%s:%s%s/", $_SERVER['SERVER_NAME'], $_SERVER['SERVER_PORT'], dirname($_SERVER['PHP_SELF']));

$form_html = $auth_request->htmlMarkup($trust_root, $return_to, false, array('id' => 'openid_message'));

# This will actually redirect to Google
print $form_html;

This is what I have in my finalize.php script:

# Requires the needed JanRain scripts
require_once "Auth/OpenID/Consumer.php";
require_once "Auth/OpenID/FileStore.php";
require_once "Auth/OpenID/AX.php";

# Setup the library (leaved blank on purpose).
# Will also create the $store variable used below
...

$consumer = new Auth_OpenID_Consumer($store);

$return_to = sprintf("http://%s:%s%s/finalize.php", $_SERVER['SERVER_NAME'], $_SERVER['SERVER_PORT'], dirname($_SERVER['PHP_SELF']));
$response = $consumer->complete($return_to);

$msg = '';

switch($response->status) {

  case Auth_OpenID_CANCEL:
    # This means the authentication was cancelled by the user
    $msg = 'Verification cancelled.';
    break;

  case Auth_OpenID_FAILURE:
    $msg = "OpenID authentication failed: " . $response->message;
    break;

  case Auth_OpenID_SUCCESS:
    # "$openid" is the user openid given by Google
    $openid = $response->getDisplayIdentifier();
    $ax_resp = Auth_OpenID_AX_FetchResponse::fromSuccessResponse($response);
    # Here you could save the user data in session for further references...
    ...

    # Redirect the user to her dashboard
    header('Location: /account');
    exit();

}

As you may imagine this scraping process is tedious to set up and prone to errors in the medium/long period. I have to read through the HTML of the page and try to find the patterns that must be interpreted to obtain the needed information (as for the moment, the bots rely on a regular expression match system). But what if the project owner(s) redesign the site? Yeah, the bot miserably fails and calls for help (me, as usual). For example: recently Apache added an “(httpd)” string to the release page of its httpd server, and for the 2.2 version only :) The 2.0 branch still has no “httpd” string in it…

Now I’m wondering if a system could be implemented for the project site webmaster to give a more reliable system to give project like mine a way to gather this (all in all) very little piece of information they need.

What could be the best way to describe and publish the information about VERSION_ID, RELEASE_DATE, CHANGELOG, DOWNLOAD_URI for a project? What do you suggest? This is a short list of ideas, so far:

• META element(s) on the home page HTML
• HTTP header(s) to be requested via HTTP HEAD method
• a common, stated web service via a REST URI like http://someproject.info/info
• an RSS dialect
• a “standard” already exists that I’m not aware of :)

Your comment would be very appreciated :)

Updates:

In the comments void points out the DOAP project. Really interesting.

Ecco cosa devi fare:

• devi usare solo la strftime() e non la date(), visto che quest’ultima non è sensibile al locale del sistema
• devi verificare che il tuo sistema abbia il supporto per il locale italiano (altrimenti il nome dei mesi da dove se lo prende? Il PHP non ha mica tutti i mesi in tutte le lingue “dentro sé stesso”). Verificalo con il comando locale -a. Ti verrà visualizzata la lista di tutti i locali supportati sul tuo sistema
• se il locale che ti serve non è presente nella lista, lo devi installare. Il comando è locale-gen e il parametro è il locale che ti interessa (es: locale-gen it, installerà tutti i locale italiani conosciuti, compreso quello standard e quello svizzero)
• prendi la stringa ESATTA che identifica il locale che ti interessa e la usi come argomento della funzione PHP setlocale() prima di usare la strftime. Esempio: setlocale(LC_TIME, “it_IT.utf8″)

From time to time I have to svn switch, jumping from the tagged branch to the trunk (testing, merging, usual staff…). And you know what? I’m keeping committing on the tagged branch! And that is bad (more on this with an appropriate future post. I promise).

How beautiful could be having a bash prompt that will costantly display the branch I’m in and the svn repository revision of the current directory?

So, here we are:

append this code to your ~/.bashrc, logout/login and enter a svn managed directory (beware: I’ve tested it only on my Ubuntu box… the key here is to use the PROMPT_COMMAND env variabile, passing it the function that does the “sniffing”. The PROMPT_COMMAND command will be executed every time, just before displaying the shell prompt).

PROMPT_COMMAND=prompt_command

prompt_command() {

  if [[ -d ".svn" ]] ; then
    local info rev url ver
    info=$(LC_MESSAGES=C svn info 2>/dev/null)
    rev=$(echo "${info}" | awk '/^Revision: [0-9]+/{print $2}')
    url=$(echo "${info}" | awk '/^URL: .*/{print $2}')

    echo ${url} | grep -q "/trunk\b"
    if [[ $? -eq 0 ]] ; then
      ver=trunk
    else
      echo ${url} | grep -q "/tags\b"
      if [[ $? -eq 0 ]] ; then
        ver=tag-$(echo ${url} | grep -o "/tags.*" | awk -F/ '{print $3}')
      fi
    fi

    echo -e "\e[00;33m[svn:r${rev}@${ver}]\e[00m"
  fi

}

The svn information will be displayed in yellow (is the 33 in the final echo line). Change it at your liking.

UPDATE: Giovanni Bajo gives the hints for the "-q" switch and the LC_MESSAGES trick to avoid locale inconsistences.

Giovanni has gone a step forward, giving us the code for a git enhanced prompt too.

Here it is:

  local gitout
  gitout=$(git branch -v --abbrev --no-color 2>/dev/null)
  if [[ $? -eq 0 ]]; then
    local full branch sha1
    full=$(echo "${gitout}" | grep '^*')
    branch=$(echo "${full}" | awk '/^* \w+ \w+/{print $2}')
    sha1=$(echo "${full}" | awk '/^* \w+ \w+/{print $3}')
    echo -e "\e[00;33m[git:${sha1}@${branch}]\e[00m"
  fi

Thanks to Uqbar on Freenode IRC for the awk hints!

example:

claudioc@enebish:~/Sites$
claudioc@enebish:~/Sites$ cd scrive2/
[svn:r452@trunk]
claudioc@enebish:~/Sites/scrive2$ svn switch $myrepos/tags/R2.4
[svn:r431@tag-R2.4]
claudioc@enebish:~/Sites/scrive2$

Ti scarichi la versione che ti interessa dal repository delle versioni di Thunderbird (che non è facile da trovare, inspiegabilmente).

Te lo scompatti da qualche parte (io uso ~/opt).

Ti accerti di avere installati i seguenti pacchetti (Ubuntu)

Te lo configuri con il seguente comando:

./configure --enable-application=mail --with-system-bz2 --with-system-zlib --with-system-jpeg --enable-calendar --disable-ldap --enab
le-official-branding --disable-accessibility --disable-ogg --disable-wave --disable-inspector-apis --disable-installer --disable-upda
ter --disable-tests --disable-parental-controls  --disable-logging --disable-necko-wifi

Ovviamente dai un’occhiata ai parametri perché potrebbero non essere tutti di tuo gradimento (hint: ./configure –help | less).

Esegui make e attendi una quindicina di minuti.

Il binario di Thunderbird te lo trovi poi in ~/opt/thunderbird-3.0/comm-1.9.1/mozilla/dist/bin/thunderbird

That is all what I needed, from Facebook: I do not need any fancy “get your friends here” or “post in your wall” rubbish. Simple and plain “single sign on”, among Facebook and my application.

After a quick search, the tutorials I found here and there were very disappointing (just a point of view, no offense), with a lot of useless PHP code. So I give you just one more tutorial that focuses more on the process than in the copy & paste kind of help.

I will use PHP for some examples, and FB for “Facebook” so bear with me and give it a spin.

What you need:

• a FB account, if you don’t own one already. Unlikely; I’ll skip this step :)
• a new FB application need to be created
• the FB PHP library, to easily talk to their RESTful API
• Some code refactoring of your own

Creating a new FB application.

Two paths, here:

• the long and detailed multistep form
• the super easy wizard

Note that, whichever way you’ll choose, you will end up creating the exact same thing: an FB application (all your applications are stored here).

We’ll use the wizard:

• the website name is just the name of the application (the handle by which you’ll refer to it in the application list).
• the website url is self explanatory. Note that you could use ANY kind of URL, even something like “http://myapplication.localdomain”. In fact, FB will not need to contact the site, so you can use an development URL as well. For the test purposes I created two applications: one for the “real thing” and one for the mydomain.local devel environment.

The next step will ask you to download a file. You can put the file WHEREVER you want. The root directory of your site is the first choice, and it’s where the wizard will try to find it. There’s no need for the wizard to find the file, though: it’s only a test. You can download the file and skip the test pressing “Upload later“. Two words about that particular file: the file is part of the system that permits our application to be proxied to FB (and vice-versa). I will not enter in details here, but think about this file as your “gateway” to and from the FB requests (more info). The other thing is: there is NOTHING in that file that identifies you application. Every FB application of the world could use the same xd_receiver.htm file. The file name itself is not important… it’s just the default. More on this later.

The third step is the landing page for your newly created FB application. Just a comfy place to pick things up. You need to write down the API KEY and the SECRET strings. You’ll use them from within your application.

What you also need to take from this page is the JavaScript snippet at point 1) and the FBML markup for the “Connect with Facebook” button at point 2).

Enough for the FB setup.

The FB PHP Library and the FB API

This is a set of helper PHP classes that will give you a simple PHP API interface to the RESTful FB API. You could even NOT use the library, and talk directly to the REST endpoints with some code of your own. Your choice.
The FB API are documented here, while you can download the latest version of the PHP 5 Library via SVN

svn co http://svn.facebook.com/svnroot/platform/clients/php/trunk/ fb

(more info on FB Client libraries)

We are finally ready to get our hands dirty.

Hands on the code

We first need to focus on how things works, at an higher level. What will happen when a not logged-in user will click the “Connect with Facebook” button? Using some sort of magic Facebook will test if the user is currently logged in Facebook. If it’s not, it will spawn a popup window asking two things: the user has to agree to give your application access to her data, and after that Facebook will ask her for credential (username/password, as usual). Upon a succesful login, FB will redirect the user to your application toward an url of your choice (let’s call it the “FB Connect endpoint“).
If the user is already logged in Facebook and has already given permission to our application to read some of her data, the login process will be completely transparent. The user clicks on our FB blue-ish button, and he will be instantly logged in our application too. Oh, joy.

As for the HTML part, you will place the FBML markup for the login button you got beforehand wherever you want. The FBML have to be programmatically converted in HTML by JavaScript, and for that to work you need to include the JavaScript snippet you already have. Put it at the end of your <body>, for example, and then run the FB.init() function. This function will be passed two parameters: the API KEY and the name and location of the “gateway” file. As said before, if the second parameter is not passed, the default is “xd_receiver.htm” on the document root.

  <!-- The login button, somewhere in your page -->
  <fb:login-button onlogin="window.location='/fbconnect.php';" v="2" size="medium">Login with Facebook</fb:login-button>
  ...
  <!-- At the page end of the page, presumibly -->
  <script src="http://static.ak.connect.facebook.com/js/api_lib/v0.4/FeatureLoader.js.php/en_US"></script>
  <script>
    FB.init("b4bbr43b4bb3b3b3");
  </script>
</body>
</html>

The interesting part, here, is in the highlighted row. The onlogin attribute contains what we pretentiously called the “FB Connect endpoint”. The onlogin attribute value is a JavaScript snippet that will be automatically run after a succesfull FB login. What we say here is: after a login, please redirect the user to our “fbconnect.php” page.
The fbconnect.php page is where OUR magic will happen. There, we will use the FB Client API to ask FB some questions: “Is this guy a Facebook, logged in user?”, “Can you please give me the picture of him?”… and so on.
The code, here, is given only as an example; think about it more as a pseudocode, really. The actual code I use is somewhat more complex, and I needed to dumb it down a little for clarity.

require("fb/facebook.php");
$fb = new Facebook("b4bbr43b4bb3b3b3", "supersecret1234");
/* If it's present, ask FB for the ID of the current user */
$fbId = $fb->get_loggedin_user();
if (empty($fbId)) {
  /* No FB user, redirect to home page */
  header("Location: /");
}
/* Now we know the FB user id.
   It's our responsability to ask our Database if we already know him */

You need to have a way to find one user of yours given its FB user id. For instance I used a “fb_id” column in my Users table.

What happens now is up to you. There will presumably be two major cases:

1. the user already exists: I programmatically login him into my application, and redirect to the home page as a logged in user.
2. the user does not exist: I give her the choice to merge her old account or to create a new one

For the merge part, I need him to login with his credentials and then update the Users table with his FB id.

For the registration part, I luckily do not need anything else that FB can’t give me; so it’s a matter of pushing a button. The code is something like that:

$fbInfo = $fb->api_client->users_getInfo($fbId, array('name', 'pic_square', 'proxied_email'));
$model = new User();
$name = $fbInfo['name'];
$tries=1;
# find a unique username
do {
  $user = $model->fetchRow($model->select()->where('username=?', $name));
  if (!$user)
    break;
  else
    $name = $name . ($tries++);
} while(TRUE);
$user = $model->createRow();
$user->username = $name;
$user->avatar = $fbInfo['pic_square'];
$user->name = $fbInfo['name'];
$user->fb_proxy_email = $fbInfo['proxied_email'];
$user->fb_id = $fbId;
...
$user->save();
...

Finally, let me complain a bit: I find writing code that interacts with Facebook a very frustrating experience. The information you need is scattered all over the web (and the Facebook site itself, even), and examples quality is very poor. The technology behing FB is very sophisticated, though, and no matter what they try to do, you need to grasp some concepts that are sometimes quite difficult to master or a nightmare to debug.

Good luck.

Further readings

• The official Facebook Connect page (and the developers Wiki one)
• How Connect Authentication Works
• 10 Great Implementations of Facebook Connect
• What is behind Cross Domain Communication and even more
• Facebook API documentation