On The Long Tail of Music, Metrics and Recommendations

Responding to Chris Anderson's Bring tha Noise post with some commentary, some metrics from my music collection and pointers to artists who inhabit this Long Tail of music...

• On The Long Tail of Music, Metrics and Recommendations
• Metrics From My Music Collection
• Full Force in the Long Tail

On The Long Tail of Music, Metrics and Recommendations

Some lunch break ego-surfing prompts this piece. Chris "Long Tail" Anderson tries to use real world data to validate some of his theorizing about music trends. Yours truly is one of the guinea pigs for this exercise in Bringing tha noise as he, and Public Enemy and Anthrax would have it. He's trying to figure out the signal to noise ratio in the fringe.

I seem to be a good source of metrics for this kind of thing because my tastes are not too mainstream and also because I augment my writing with lots of allusions and links to music and books that I champion or lean on to underline the slightest point. "Mining the cultural zeitgeist" is how I have described this tendency of mine when Jon Udell's metrics leaped out to me. I also tend to just use Amazon links which helps those seeking statistics. So then, what to say about the Long Tail of music?

First this: there's a difference between

1. the things that I write about, and quote or link to (i.e. what you'll see if you mined this blog)
2. the things I actually listen to or read
3. the things I've bought - my entire collection of music and books

I recently wrote a piece On Recommendation Systems that is in much the same area as this Long Tail discussion.

On the matter of recommendation systems, a couple of comments are in order.

• Leonard Richardson felt a little guilty that he hadn't spent time enhancing Ultra-Gleeper but that's the way things go, we're all stretched thin. Like he suggests, sometimes it's the idea that is important and not the embodiment in code, we see that all the time in technology. If one manages to give some intern somewhere an itch to scratch, as that paper should do, that is a contribution to knowledge and we are all the better for it.
• Robert Jamison seems to have a made a great contribution in ideas (Searching, Sharing and Stumbling), if not in code (kindakarma.com) to the recommendation systems debate and points to a few others of note. I will be playing with said code and services in coming weeks

But back to Anderson and Udell's search for metrics...

One thing I didn't comment on in Jon's post was that he was looking at statistics of bloggers he reads who most frequently cite books on Amazon.com. The problem was that many of the links from this site are about music rather than books. This may have skewed his statistics a little.

In other words, what was required was some sort of link classifier to figure out what a link refers to (whether book, music cd, dvd etc.). The classifier would be like a regexp pre-processor that would figure out that, for example, a link to IMDB is probably about a movie in much the same way as the LibraryLookup project maps ASINS to APACS. I assume All Consuming uses the Amazon api to figure out what type a given item is (book, music, dvd, other). I continue to wonder if one can have a classifier that will determine when a link points to a person... All Consuming is in flux as they transfer to 43 Things and their old REST apis are orphaned in the transition but that product has an interesting take on link classification and tagging.

Items 2 (what I actually listen to) and 3 (what I've purchased) in the above list are also interesting.

I have a partial pass at item 2, what I actually listen to. You can check out my statistics at Audioscrobbler. I installed Audioscrobbler on my home machine and have been playing with it to figure out if its recommendation engine will prove useful, I'll report back in a couple of months. After a couple of weeks, it has me in quite good company on the basis of the 769 tracks I've played on the home computer.

With a music collection as large as I have, my listening habits tend to be

• One third playlist-driven (I have numerous playlists for every mood as befits someone in the grip of musical obsession - thanks Gardner by the way for your kind comments and Autumn song playlist suggestions)
• One third shuffle serendipity
• The other third typically comes into play during shuffle; I'll hear something which will make me think of some other song and I'll do an ad-hoc thematic playlist, the problem is that I don't tend to save those tangential playlists borne of serendipity to feed back into the first category. I'm too lazy for that.

I'm a bit of an audiophile bigot and "can hear the difference" between original cds and even the 224 kbps VBR mp3 encodings I've done of my collection. Also some things have to be listened to loud (and one has to keep the neighbours on their toes every now and then). My computers haven't been hooked up to the big speakers and gleaming stereo system; the computer speakers are some generic $30 muffled fuzz boxes. Thus I need to get a Squeezebox, Audiotron or something although I'm a late adopter in such things. Thus, what is missing in the Audioscrobbler statistics is the music I listen to on my stereo. Still Audioscrobbler is a good proxy for much of my actual consumption of music.

On the topic of playlists, I'll be sharing some shortly so that others can get a sense of my beat matching juxtaposition insanity.

One problem with dealing with statistics from the blog is that for example, even though I'd estimate that 40 percent of my collection is jazz, I write about jazz only occasionally, like when I was Vibing with Abbey Lincoln or reminiscing about A Soul Jazz Thing with a belated appreciation of the late Jimmy Smith, or say alluding to Kamal (of The Roots) and his Ahmad Jamal keyboard stylings. There has only been one "proper posting" on jazz otherwise the allusions are all incidental say comparing Miles Davis's First Quintet to Rokia Traoré's band.

Jazz is less quotable and difficult to write about so perhaps that is why it wouldn't show up more readily in the statistics from the blog. Still the jazz idiom is a big part of my musical taste and outlook on life.

Metrics From My Music Collection

Anyway here is the data for number 3: I give you my music collection for your forensic analyses.

collection.m3u and collection.pls

My digitized music collection stands at 10,346 songs and amounts to 70.8 GB (about 36 days of continuous music). Note that this is larger than the largest iPod (hence my pause in not adopting that ubiquitous platform). I only ripped things that I liked so presumably one should add a good 50 percent filler for a fuller picture. I've been revisiting that notion since the Best Left Unread piece with a newfound appreciation of the bad things in my collection; disk space is cheap and I've just ordered a 250 GB spare hard drive to mitigate future disasters, that disk could be put to work.

Per contra I actually believe human psyches require a little imperfection.

Of the 1,527 artists therein, there were about 15 that were problematic (e.g. I'm sure there's no musician called Hawaii 5.0 or James Bond 007, to take some of the theme songs I occasionally play to spice things up, and similarly Ghana Highlife 1 to 10 are simply proxies for the unknowns I've picked up). I haven't digitized the 4 crates of records that I have lying about. That's a weekend project for this summer like my photo-digitization project of a few months ago (see Cultural Sensitivity in Technology)

Apparently there are 1,540 albums in the collection. I haven't quite diligent about tagging albums so 2,000 songs don't have album information. Handwaving a 10 song per album ratio that's another 200 albums, so lets say 1,750 albums). I'll fix that and update those files in the next few days.

Generating metadata for albums is difficult: what album would 50 Cent's blistering Jay-Z diss that was the talk of mixtapes in 2002 fall in? And there are also things that come from Greatest Hits, or Live albums as opposed to the original album, and I typically have multiple versions of things. I can't remember which version I actually ripped.

I am actually crazy to have spent cash money on most of these albums instead of downloading so if you assume that's $14,000 to $27982.50 (at $8 per album or $15.99 if I were to pay what record companies would prefer). Luckily I got a lot of these for free when I used to DJ and was on record company mailing lists, also I used Columbia House and BMG music zealously. Still that's a lot of money, I bought my first cd in 1992, previously it was all vinyl and tapes. I can't say that I've downloaded much from file sharing networks (I use Gnucleus). At the higher end of the scale of the cost estimates, $30,000 is a mortgage downpayment (well it was before this insane real-estate bubble that even leads to riots). Music companies should love me. I justify it by saying that my only hobbies are music and books; no drink, fast cars (I'm a public transport prole) or other flash.

I've tagged songs by genre but here you really want to be able to specify multiple tags just like in any of the new social software.

I suppose I should switch to iTunes to get an xml representation of these playlists. One of the bad versions of Winamp, the bloated one that which caused a backlash, used to export playlists in an xml format. That would have been better than m3u or pls. Still that is nothing that a capable regexp and Unix pipeline wizardry couldn't fix.

I'm still in disaster recovery mode but once I've got Cygwin set up, I'll be sure to download and install PlaylistManager and slice and dice the data with that Linux tool.

Anyway I hope this helps...

Full Force in the Long Tail

The obligatory musical note... While writing this note, I just rediscovered the wonderfully fresh sounds of Adriana Evans whose 1995 album for some reason didn't blow up (stupid record companies). Adriana Evans is the name that you drop when you want to put someone talking up Ashanti or Brandy in their place. Love is all Around and Seein' is Believing are some of the most laidback soul jams of all time.

Speaking of Seein' is Believin', how about Cheryl "Pepsii" Riley's (erstwhile of Thanks for my Child fame) song of the same name from 1988's Me, Myself and I album, produced by Full Force. That song is soul/funk perfection, much like Jerome Prister's sublime Say You'll Be that I recently found after 18 years.

And speaking of Full Force, what about that Hall of Fame production crew and House Party funkers? In recent years have been souping things up for Britney and Christine (you've got to pay bills). Their roots, like Jam and Lewis, are in pure soul and a hip-hop/funk aesthetic. Guess who's coming to the crib? is my favourite album of theirs although others preferred their eponymous debut album

My picks in the Full Force canon

• Your Love Is So Def
  My dad lost all respect for me when I used to sing this because he heard "deaf" in my loud singing rather than the 80s hip-hop notion of def.
• Love Is For Suckers (Like You And Me)
  Shower crooners at school couldn't stop singing this song for weeks on end. Not to mention the lines we all primed to repeat on cue, their irresistible verbal tics:

  "Come on bust a move."
  "Full Force Get Busy One Time"

• Take Care Of Homework

  "Yo homeboy, your girl looking kinda fly lately, know what I'm saying?"
  "No I don't know what you're saying."
  "What I'm saying is that she be like checking me out 'cos you been neglecting. D'you know what I mean?"
  "No I don't know what you mean!"
  "What I mean is that you better take care of homework or a dude like me will push up and take your place!"

  On the basis of the irreversibly funky break beat from Take Care of Homework, these brothers proved they were funky enough to revamp James Brown in his sadly neglected I'm Real album. The title track is a ferocious call to arms by The Godfather.
  
  
  
• I'd also note that their Smoove album is no spring chicken. It contains such gems as Ain't My Type Of Hype, the Smooved out title track and that phenomenal ballad, Kiss Those Lips. These are cases in point about the versatility of the crew
• They gained fame with Alice, I Want You Just For Me with the famous opening.

  Testing, Testing One, Two
  Testing, One Two
  In The Place To Be...
  Girl I Want To Shower You With Diamonds And Pearls
  And When Were All Alone I'll Take You For A Trip Around The World
  Yes Indeed I Like Your Style
  Ooh You're Worth My While
  Baby, I'm Your Carpenter,
  Please Me Lay Your Tile
  I Don't Want To Share You With No Else
  Alice Be My Girl
  Can't You See?
  I Want You Just For Me?
  Full Force Get Busy One Time

• Their repeated collaborations with Lisa Lisa & Cult Jam are also essential Spanish Fly.
  
  
  
  
  
  I Wonder If I Take You Home will still fill a dancefloor to this day.

  "Would You Still Be In Love, Baby?
  Because I Need You Tonight."

• Their work with UTFO especially on Roxanne, Roxanne also gave rise to the legendary The Complete Story of Roxanne, those 103 responses to UTFO's 1985 novelty hit.
  
  The Story of Roxanne is one of the best stories of of the Long Tail of Music (the other great case in point is All Roads Lead to Apache).
  
  Roxanne, Roxanne brought to prominence The Real Roxanne and more importantly Roxanne Shante, possibly the best female MC of all time, Roxanne's Revenge will go down as the best response record of all time. One of my favourite songs of hers is Live on Stage which is Marley Marl's production at its best.
  
  
  
• Full Force consolidated their fame with collaborations with Kid 'N Play in House Party and its two sequels. The movies were the inspired brainchild of the Hudlin Brothers, a decade before Barbershop.
  
  
  
  
  
  
  
  
  
  Kid 'N Play were no slouches themselves even if they tended towards the pop side of the hip-hop spectrum. They were fun, their beats and lyrics focused on putting a groove in your step. More to the point, they could dance; at a time when Hip Hop was heading into inaccessible navel-gazing, they reminded everyone about the dance in Kool Herc's dancehall that was in the founding mythology of the music. As historians of Hip-Hop would write, break dancing (the B Boying and B Girling aesthetic) is an essential element - some would even term it the most important of the 4 elements of hip-hop, even more than MCing (rapping), DJing, and that maligned "social nuisance" element, Graffiti. Gittin Funky and Rollin' with Kid 'N Play will cause circles to form and Soul Train breakouts on the dancefloor. Funhouse is their best album and that's the spirit. And those flat tops!

Full Force have since cashed out of the soul side of things except to give some "urban" substance to blue-eyed pop, and who can blame them: "black" audiences are so fickle.

File under: technology, design, architecture, software, web, systems, social software, complexity, services, music, soul, funk, jazz, hip-hop, glue, recommendations, algorithms, heuristics, machine-learning, long tail, Full Force, Adriana Evans, Lisa Lisa, Cheryl "Pepsii" Riley, James Brown, toli