Photo Time: Simulated Depth of Field, blurred in GIMP

Simulating Shortened Depth of Field : Making a mask and applying it to layers to blur the background.

In the old days, most quality cameras could produce nice portraits with the background brlurred. These days the smaller digital cameras create a vert deep depth of field because of the short distance to the sensor chip from the lens, so the background is often very sharp. On dSLRs too, some lenses don't «stop up» enough to give an open enough aperture to «throw the background» either.

However help is at hand because t you can make up for this in PS or GIMP etc, by drawing out a mask and using it with layers to paste a blurred background onto your original sharp foreground. In the image above you can get the idea!

I had a go at the GIMP GURU's excellent tutorial, but found for GIMP v2.6 that there were a couple of functions that either differed, or were desirable to use.

WHAT YOU ARE GOING TO DO: Work-in-porgress Images are at the bottom of the blog: you are going to make a couple of copies of the original, and then on one make a "mask" which will be the area the original, sharp background will shine through, so to speak. You will blurr the edges of the white mask so it merges to the soon to be fuzzy background, and then invert the colour to black. This mask is then laid ontop of the other copy which is first blurred in the software options. Finally you merge the mask to the blurred layer using " apply mask" and then stack this ontop of the original so the person or object appears sharp on a blurred background.

HARD STUFF:

Small errors in the Layers Dialogue box can cost all: you have to take care to do things in the right order, with the correct settings for the "apply mask" and remember to stack them in the right order so the mask is on top. : worth saving WIP copies as you go as xcf files.


"toggeling" between selected area after cutting and background on the mask build, then both in order to go through "cutting" the mast needs to be done fully awake!

It can also be difficult in cutting the mask and then blurring it's edges to get a natural fade into the background, with no halo and no sharp, unnatural edge or vignetted outline areas. Also, getting the right amount of blurring on the background

WORK FLOW

> Copy Original > Lasso Select Sharp Object e.g. person!

> Blacken bacground with paint can fill

> Work up mask in white with an underlying orig' image as guide in BW >

> Blur the edges very slightly to 4 or 5 on gausian, invert to black on white [SAVE A COPY!!! ]

> Open a copy of the original image and blur it to more than '20' on gausian

> [Add Layer Mask] to the blurred image ; NB as ['white opaque']> paste on the mask to this layer in layer's dialogue box

> Open the original as a layer in the window> move it down the stack order in the dialogue box

> merge down the stack to end with. [SAVE as jpeg]

In Detail: On GIMP v2.6

1. Open image File, then CTRL D to make a duplicate GIMP window as a jhandy back up / reference

2. Make a new layer as follows: [Pull Down]Layer>new from visible> duplicate this layer with CTRL D while in the layers dialogue box > rename the duplicate "Mask"
   

3. Use the lasso to clip out the foreground or objects to be sharp. Go to a black and white view or a colour channel ( [Decompose], see GIMP GURU) if you need a clearer outline to work on ie more contrast between subject and background making it easier to lasso correctly
   

4. Once the area is drawn around, double click/click on the lasso in tool bar. The area should now be selected inside a hatched line. Press [CTRL-I] to select background.

5. Select FILL tool ( paint can) on black and fill the background

6. Save a copy once you are here as an xcf file

7. Now we want to try to force the outline of the subject to white. Right-click in the image and choose (Image/Colors/Threshold). Drag the middle pointer on the 0-255 scale left to extend white over the white areas

8. If you have the image (copy from start) open as a layer, then make it grayscale: otherwise make a new layer and paste ALL of the original image onto it, anchoring the floating layer with the little anchor icon in the Layers dialogue window. Choose this new layer and reduce opacity to about 50 -60% such that you can see a ghostly effect : now you can check the mask fits and captures the area you want sharp. Once done, delete the mask guide layer by right clicking and selecting delete layer.

9. Now apply a gausian blur to the mask layerAT ONLY 4 to 6 pixels: this blurs the edges of the mask such that it will run into the to-be-blurred background and avoids obvious lines. Once computed, choose COLOUR>INVERT to now make the image Black on White as the actual mask. Save this and save an xcf or jpeg copy.

10. In the Layers dialog, select the original image in the drop down box if you have it or paste and anchor from the other original open. Then click the Duplicate Layer buttonDouble-click on the name of the top layer. In the Edit Layer Attributes dialog, rename the new layer “Blurred”.

11. I (Filters/Blur/Gaussian Blur )). Experiment to find a value that works well for your desired “depth of field”. Generally you won’t want to go too crazy with this or the look will be all wrong.Try 25 for a partial blur and over 40 for quite unsharp.

12. SAVE A COPY of the WIP at this stage such that you can rework the blur if you feel it is not blurred enough or vice versa

13. Go back to the Layers dialog and right-click on the “Blurred” layer; select “Add Layer Mask”.NB: At this point the next radio button selection is vital ! In the Add Mask Options dialog make sure White (Full Opacity) is selected. Then go to the black mask and select all and paste onto the blurred layer ONTO IT IN THE DIALOGUE BOX. Anchor it and deselect the eye on the mask itself

14. You should see that there is a mask marked out in grey checker pattern over the area you want sharp and a nicely blurred background. Work back with CTRL Z if it is not to your burry liking!

15. Open the original as a new layer. Move the orignal down on the list in the dialogue box so it is under the mask-blurred, and therefore will appear as the layer beneath on the final image. This shouuld now fill the masked area but be written over outside by the nice blurr you have created ! Jobs a gud'n.
    

16. (delete the «mask»). Select the top layer, Blurred with mask, and then Choose Layers: Merge down save the image as a jpeg.

TIPS! When practising, reduce the image size and resolution to a web snap shot: this will make all the rendering and saving of your trial and errors a lot faster while you learn, and then you have a web ready series of shots!

TIPS! Be careful with the lower edge in portrait shots. When "cutting" ie outling the area to be the mask, check there is no foreground object to be blurred: short depth of field means it is just a slice into the distance which is in focus.

TIPS! It can be wise to give a margin to the figure for the edge blur: but if say it is a close up of a head with some diminishing depth of field on the person being good, then you can cut to the edge. This will vary with images : for instance, I find those with shadow from flash on one side need more there so that the shadow is softened. Trial and error will show you what works best, but it is difficult to extend a mask once «cut out».

TIPS! In GIMP 2.6 at least, when you have outlined the mask, and chosen to invert to background [CTRL I ] if the paintcan fill does not fill with black totally, then you can select a huge brush size and just go over it all in black: the foreground selected object will not be painted on as if it was "waxed". To fill in the mask itself, reverse with CTRL - I and then use white of course to remove any small specs or detail.

TIPS ! With similar images and a good contrast / edge to the background you do not really have to bother with an underlying black and white "ghost" to guide your fine working of the edges of the mask.

Footnote: Following on from that last tip:: According to GIMP GURU though " The best contrast between items can often be found in the green channel. It’s a good place to start looking, in any case. So the first thing we will do is decompose to individual RGB channels." as follows:

"Right-click in the image and choose (Image/Mode/Decompose). In the Decompose dialog box, choose RGB and click OK, Examine each channel to find which has the best contrast for isolating the subject. In this case the green channel image was the best. Close the other two images that you aren’t going to use. With a little work this green channel image will become our mask."

ISSUE HERE IS THAT I DO NOT SEE MULTIPLE COLOUR CHANNELS; ONLY ONE AND THIS DOES NOT EDIT:

Images

Too sharp

: too much detail in background to make the people stand out. The father is a little lost because the eye/brain places him more with the sharp sofa in the background

Mask WIP with blurred background. As you can see, the mask is not terribly detailed on the edges, but in this case it is quite sharp into the bodies with only a 3 or 4 pxl blur on the mask edge.

NB I have also been careful to exlcude foreground objects from the people so the foreground is also blurred, and been careful with the table edge and books: a lower edge should be checked for this in head-shoulders shots.

Voila!!!

The eye now picks out the girls much quicker as being that to spend time on looking at, and the father is then pulled out more from the background detal.

The more you look at this shot, the more your eye-brain brings out the people from the background it now considers uninteresting!

I have put on an info dot to track any missuse of this photo. Pleas PM me before you copy this image for use.

Final with Crop

I like to crop! Here I could have used GIMP to "clone" out the light cabel/switch into to the beige wall in order to have a really clean image.

Note also: the fine detail on the LHS girls hair is not actually cut to a very fine mask detail: the eye/brain make the detail stand out more from the background now the rest is blurred but if when this is enlarged, the rough work shows in the hair and on a slight halo round the man's head.

Original cropped for comparison

All images (c) 2010 author. All text not quoted from GIMP GURU is original (c)2010 author

Hard Work! Repeat ! Work Hard"

Now I note that it is time for a new intake of masters students to the 2011 MSc Marketing course.

Firstly congratulations on landing a place on a course which will give you a head start in your career in marketing or business otherwise, in providing you with knowledge, skills and not the least attitudes to suceed.

In terms of your career, marketing is not for the faint hearted. It requires a lot of percieverance just to get a career started, even to get interviews or work experience to get half way up to the first rung on the ladder. You have to be very tough or very lucky to get on and in my experience all too many Msc graduates have vaguely wandered into marketing because it sounds interesting, without knowing that it is very, very hard to get into once you finish.

The freddy blog seems to have been read a lot by previous graduates and I will stress again upon the new students, that marketing is a very, very, propostorously competitive profession to get into and then progress within. Please read back over my previous blogs on job hunting and how to get on: your job hunt preparation begins now and will start in earnest in October if you want to make the best possible head way against the larger brand companies.


Marketing is so extremely tough to get into because so many people want to work in it, not just qualified marketeers. Many of those general business graduates want in, many sales people want to move in(degree or no degree) and then you also have all and sundry BA, BSc, for example many psychology graduates, wanting a piece of the action. Then of course, there is the whole oxbridge bunch of medieval historians or theoretical economics who seem to be able to slide into plumb jobs. They have a head start because so many previous brand managers came from the UK's own "ivy league" and follow the "model for success" by recruting back from Oxbridge, Durham and so on.

You will need your luck, but the harder you try in your studies, job hunt and network building, the luckier you will get.. My career languised until one day I took a call and it was the golden opportunity to get a real start. I seized it, although it was pretty mediocre pay, and worked darn hard to learn the practical aspects of marketing on the shop floor of a "through the line" ad agency. After two years, the world so to speak was actually my oyster because the big agencies had stopped training new graduates and prefered experienced people from even much smaller agencies. I moved on to a decent wage and company car and later had the option to move "client side" for fewer working hours.

Read back through my advice and take it as highly opinionated, personal stuff, but remember it is based on the realities of job hunting and building a career. I have worked for 15 years with internet marketing for example, so new technology does not take over from the human hum-drum of pressing flesh and getting yourself a job these days.

Get Some Ink

Ink is fast becoming a thing of the past: even my local book handlers have a range of computer peripherals, GPSs, digital charts, software and interactive CD/DVD ROMs...and I live in the back end of the sticks.

However there will always be a market for books, magazines and stuff to hang on your wall without any power supply!

In the olden days, photo print qaulity was determined by ISO and resulting grain of the image, with even some 200 ASA transparancies being good enough for enlargements of several feet on photographic paper. If you didn't like the grain, then like being in the Louvre, take a step back and be impressed.

Boots the chemist or Jessops aside, there was also a certain directness when dealing with "ink" printers and the whole colour process: I came into advertising at the very end of the era of direct lithographic printing, "analogue" if you like, where each stage of printing had a different print prep or finishing company. Plates were produced one place often to get the best quality to a price, and the proofs, the large negatives of each colour litho process, were kept by us in a most guarded way! You felt a certain tactile and direct connection to what you could hold in your hand when compared to the mouse jockey world of photoshop which came in soon after.

HOW MANY PIXELS DID YOU SAY ??????????????

Today though, what quality can you expect and what should you deliver for home use?

Take my own Olympus DSLR: 10 Megapixel camera


===========
Largest Image = 3648 x 2736 pixels
===========


::
This is actually 9.8 mpx and is a ratio of 1.33 height to width: four thirds !

large size : fine means best jpeg compression on the display.

If we have a very high quality black and white print at 330dpi then we can utilise all those pixels to give a size in inches thus:

11 x 8 1/4 inches at 330dpi: in other words an A4 side! 210x270mm

We would then want to print this on a 600dpi home or shop printer to get the absolute best tonal range and depth in B/W for a delivered 330 ppi (pixels per inch). Alternatively, if it was a once-in-a-life time photo, you could either get it printed to silver-chemistry ie photographic paper at 1000-2000 dpi or if you wanted to sell it, go to a photo lithographers and get it done on a plate of maybe 660 dpi or higher.

To have an absolutely best qaulity print in colour though, the size would be one quarter this, because each dot on older printers can only have one colour in "dots per inch" so three or four dots are needed to reproduce each pixel to a degree of highly accurate hue (tonal / colour range) and tonal depth (brightness/saturation).

However, printer technology cheats: the high quality 330 dpi - 600 dpi home printer is available to reproduce large PPI quite well to an acceptable quality. The printer driver compiles the pixels into a best colour match for it's hue, saturationand tonal depth and recreates this as points composed of many dots, utilising a best compression and half tone ink saturation to process the image to a best appearance.

The Practice of Relativity


Relative Size:

An A4 sheet is 210 by 280 mm, that will say 21cm by 28cm or 8.25 x 11inches by

A HD screen for now, has 1920 x 1080 pixels, so most digital cameras now can fill at least twice that size with an image without any "pixelation" ie a very good image!

A good quality print can be achieved at relatively low dpi, as to be seen at http://www.smugmug.com/help/print-quality. Here an image is printed out to a large size to convince photographers that the eye actually does not percieve more than 80 dpi as poor, quite the opposite. PC screens are 72dpi, while macs used to be at least, 100dpi or lines equivalent. Another reason MACs were better for graphic work.


The image shown, a racing car, lends itself to an easy reproduction though: it has good contrast, sharp details but no very thin lines or patterns, a thrown background and bright colours. It is printed at 80dpi on a large format just to show how good you can get from precieved lower than usual print quality.

So for a colour print you may expect to be able to get a perceptably high quality at 200 PPI priting out at max resolution for the printer 300 dpi or so. This means that :

18.25 x 13.66 inches

When we say high quality in this instance it means for print material seen from a normal reading distance, not under special scrutiny.

For a potrait to be hung on a wall and viewed from a greater distance then a far lower DPI may be acceptable: even 72ppi or normal screen resolution may be practical for printing:

Thus:

50 x 38 inches @72dpi

Professional Printing
________________________

Printers use either lithographic plates or screen print by in large : ink jet printing is still not cost effective enough for publishing and laser -toner printers are not there yet either.

Lithographic means the process of chemical etching the colour separations from the original negative : in the old days when I worked with printers, these were best achieved from wide format camera negatives or positive transparencies (slide film) using a series of different filters to create the plate for each colour ink.

Chemicals were used to mask and etch from exposing the plate to light,
and plates for say A4 book printing at 4 sides a plate plus "bleed area" were between £300 and £1000 per set.

The print plates are then loaded up on the press and in use, coated with ink and pressed to the paper which is moved very accurately through the different parts of the print line. Hence the registration is very good as time is taken to ensure problems are eliminated by good preparation or stopping the press asap.

A usual "screen" or lines per inch equated to 330dpi and images in photoshop were handled at that level of ppi, with some leewaywith a "positional" associated to the quark express file such that the printer could use their expertise in preparing : often delivered up to 20% larger than needed to a quality printer image before and during the production of colour separations.

Litho' as it was affectionately called, was at one time too expensive for reproducing ordinary print. Books as late as the 1970s when I was a nipper, often had sections of colour "plates" bound in at convenient points in the book : these were often a pain to refer to from the story or reference as the convenience was only for the printer to sew in the better quality paper such that each insert spanned four pages.

You can still see this today in many biographies in paper back, where images of exploits in the SAS or whatever are collected in the centre or at two points in the book. Paperbacks, like newpapers and most large catalogues are screen printed.

Screen print works by having a mesh screen over a drum with ink being rolled on continuosly and another drum compressing the paper roll onto the print area. The paper like a gigantic toilet roll. Thus it is continuous for huge quantities of thin and therefore cheaper paper than could be used in plate presses.

Originally screen was just in black and white with maybe an extra colour, red most often, for the banner line being used. This is where "half tone" and other screen adjustments came into use to produce reasonable quality black and white photo reproduction by creating a better tonal depth and contrast in a print image than the dpi or Lpi as they call it, would permit. However colour separations came into use in the 1980s: some seem to use RGB for colour areas with a usual black screen being used in both images and text, others CYMK.

The great advantage of screen printing is the huge volume of paper that can be printed in a short time: however it is limited in dpi and also is prone to poor registration: the fuzzy or double images you often see in colour images or graphics. It is very expensive to reset a whole print run if this arises or even to scrap a minutes production, so these are sent out anyway for sale at newsagents etc. Usually you would need to travel to another newsagent chain or even town to get part of the run which was okay!

To save on costs and reduce issues with registration in colour screen printing, many catalogues are printed in three colours : black and two special corporate or style pan tone matched inks. Even some high qaulity art books and magazines use just one colour in addition to black : this can be very dramatic in landscapes with say only the blue or green being there on top of the grey tones.

Today, I would say that all plates on litho and screen is done with laser etching : the image is either purely digital in jpeg format or scanned from a negative or transparency by coloured lasers : the separations are then produced in digital format directly from the latter and by computation from jpegs. The famous "heidelberg" machine for A4 x 4 print plate laser-etching fits nicely in half a squash court with room to walk around it!

Even in 1999 when I was up for moving back to the Ad' industry in Scotland, I went to a printer who still used the grand daddy of them all: type set printing which has it's roots in the wood cut presses used to produce the first printed paper money, and the Lutherin bible in German and English.

The university printers a decade before still used these presses and the prepared litho plates for BW logos, drawing or other images. These lacked any real grey tones!

To cut you data nuts down to size in a little Luddite fashion, the term "upper case" refers to the case containing capital lead letters the type setter would reach into when required in assembling scentences onto the type face for printing!!

COLOURs

The CYMK was industry standard but by the mid nineties all printers I knew could have atleast a fith colour or technical process, like a UV spot filter or spot varnish. K was used for black : to do with not being confused with
"Blue" which was used before cyan was the colour chosen.

The trouble with CYMK is that it does not produce all colours equally well, eg orange, and where printers used to mix specific inks to match hues in a book say on birds by James Bond with colour plates in the 1950s, that art is no longer seen as cost effective. Blue skies often have a purpely appearance with notable red dots and this is due to the clash between the cyan and the magenta fighting to make a blue which is near cyan but not near enough.

Orange is never well reproduced as it is a mix of yellow and red: magenta is not red enough and some cyan comes in the litho process or digital separation: thus orange on any CYMK standard pallette if you like, is a mucky brownish orange.

Correspondingly, Orange telecom in the UK were the first company I heard of who stipulated a fifth colour process with of course, the proper corporate pan tone orange for their block logo and fonts.

The extra processes, colour or treatment, use the same basic component parts of the four colour process; a machine unit which very accurately places the paper from the last print onto the next plate. Thus software skills, investment and pure physical factory area are the external limiting factors, while there are limits to how much ink a paper will take.


Even in CYMK, when printing black areas with photos supposed to be frameless, mysterious "chinese window frames" would appear where a positional frame around the image became "super saturated" and shiny with layers or ink.


Many of the top printers for corporate material went over to seven colour process print lines for litho printing, which may seem a bit like "being the last to make buggy whips: darn good buggy whips" while everyone goes new media.

Colour; future:

On the topic of new media, jpeg is in RGB which suits of course LCD monitors very well. However now four colour TV Flat screens are available with a yellow which gives a more pleasing and realistic colour than RGB. Although this may take some time to come to the general market for computers, the technical challenge of "re-separating" an RGB signal to a fourth colour is now practical for better reproduction!

Technically RGB in an LED screen should be perfect colour reproduction as the transmitted, addative colours are bright enough to give hue, saturation and BW tonal depth is better due to the "black ness" of the LEDs which are not transmitting.

A pixel contains a given value for both the hue and the tonal depth: the shade if you like. The LAB set up is actually far better at handling images for print and absolute quality (even when considering CYMK as final out put) but because RGB is the standard for monitors then this is the prefered handling.

I would expect , or wishfully think, a move over to image production using the LAB system where the BW is it's own separation: red green and blue yellow are the other two separations so the fourth colour would be a yellow, making probably for better judgements from eye to screen in what is a good colour adjustment in the "ring" of images shown in most software packages.

I would look to Mr. Jobs' job when that frickin iPaddy makes the stock fall and they move back to their home land of graphic arts.


Picture Content

The requirement to produce a better quality image nearing the ppi x 4 required is driven in part by the content of the image.

Obviously if the subject of the image contains a lot of fine detail or pattern which the eye is lead to, then a high dpi will produce the best results.


Conversely, lack of detail and contrast can require experimentation for best print outs.

Both high light and low light images have subtle tonal differences in BW alone, and any noise from the CYMK compression / conversion detracts from the qaulity: Also the tonal depth will need to be enhanced due to this subtle changes over areas of the picture such that they don't appear "posterised" with harder changes between areas appearing as bands or lines not present in the original.

Hence you need to have four colours per pixel to make an acceptable print quality and thus on a 330 dpi printer you need to divide your megapixel size by 1320 to get how many inches would be absolute best.

Best Fit ; ppi: dpi and image scaling

Practically it would be quicker to use some trial and error, by first printing out a small scale size at 330dpi in colour, or lower quality like 100dpi to see the areas which may be problematic in percieved image qaulity.

Then take a crop of these potentially troublesome areas and vary the size of the image output, writing a text comment with their details so you can keep track of them after printing : view then from a distance and determine which size is acceptable. Calculate out your total size achievable: it will probably be bigger than your printer's width and height!

A picture with less tonal depth and more primary colour composition, more contrast and stronger colours in other words, would maybe be a direct conversion to 330 ppi to dpi or lower upon some scrutiny.

At this stage it would be far easier to move away from using ppi and the whole pixel thing, and use PS's, or light room or Corel's etc etc, image scaling function which gives you dpi and actual inches size or metric E-Qs.
==========================================================================


Notes on sizes

For printing:


YSF YF YN YB
X (Middle)
3200 x 2400
2560 x 1920 XSF XF XN XB

1600 x 1200
W (Small)
1280 x 960

WSF WF WN WB
1024 x 768

For
small-sized
prints and use
on a web site
640 x 480

Social Media Monitoring Goes Bust

A Brief History

To start with a brief history of social media: from cloudy beginings in nerd talk and security services personal info boards, social media came to the public internet in the late eighties and early nineties as the newsgroup: a simple text based repository for messages and early blogs, with circulation by e-mail subscription or internet boards.....and these were indeed monitored then, mainly by police in connection with the pædo rings and other unsavoury comms.

Newsgroups quickly went HTML on the World Wide Web and evolved into the forum, which is still the best repository for accessible, high value and trackable information on the web. This is where most consumer involvement with products and brands takes place. Although the micro blogging / glue services like Twitter are rapidly becoming a larger, faster resource they are still less useful for examining consumer attittudes over months of time or in detail.

Little Brother is Watching You

Now there are a host of agencies offering "social media monitoring" and these currently run the risk of creating a bubble soft and venture capital will feel the implosion of. Currently the most advanced indexing tools are not much better than stringing together several specialist hot shop functions, a core data depository and crawler and monitor and "deck" resourcest which are free on the internet. The larger agencies, like Meltwater are able to attract the larger brands while the major players of Trad' market research are still testing the water and holding off on acquisitions.

Why is Social Media Monitoring Going Bust?

The whole market for social media analytics, is in peril of undervaluing it's core pricing. Put simply, the barriers to entry are rather low given that comp'sci' students are often given crawler-indexer or deck meta data projects in undergrad years. Most of the new emerging agencies are student shops: comp sci and MBAers slung together to play at business. Problem being that they want to set experience before value and are cutting each others throats by going into brand names on "loss leaders".

Another key issue is the propagation of free services: "amsterdam hooker windows" as one software engineer called them in another area. The problem here is that you can begin to string together enough free services to make a picture of your brands' position in SM and then just go into the key forums and twitter yourself with your short-suffering marketing interns. The initial "Brand X status in CGM" is devalued as are tha later tracker reports: the key value for agencies is in presenting the statistics and sentiment mapping, but this is so relative and subject to the media surface changing itself that you really have to question the value of it for brands with less than 2000 hits per month in SM.

Scalability ....lack of it.

Although a few of the new starts may have strong crawler-indexer technology, one major problem is that the technology has to monitor a diverse range of sources out there on the internet, and those sources are not always too keen on being crawled. Programmer time is used in not just attaining sources, but retaining them. Also indexing for relevance and speed takes programmer time,and unfortunetly a new arena for a new brand may not be as fast or inclusive/exhaustive as the last indexing which had been optmised.

Put on top of this the awful costs of account management and new business development in acquiring and maintaining brand name clients and the issue of scalability just in this one area, is the one which will kill off most of the new starts.

Lumpy Custard

This economics of SM monitor agencies is actually nothing new: most small 1970s-80s advertising agencies failed because they tried to scale and could not make the leap from the core owner-manager team to an expanding, system driven agency.

Most of all in marketing services, it is horrible risky business with huge over reliance on a small number of customers ( clients in agency land) . "lumpy custard" as I used to describe it. In the 1980s the expression "lose a client, lose your job" was the mantra of account directors, while today it is more likely to be "lose a client, lose your VC funding".

To give some detail on this, when a new project is taken on then it inevitably comes with new sources or indexing demands. Also it comes with a new set of expectatons and because Market Research is still cinderella to the communications side of on line marketing, then new demands of the clients are usually out of line with charging the baseline 85€ per man hour to even break even in business services.

Winners and Losers

The largest, most comprehensive indexers will probably win over and then get the third level funding or even IPO / alternative exchange floatations. I'd expect these technology and brand become acquisition hungry and buy up the hot shops with key expertise in delivering more from fewer man hours. They in turn will want to either exit from VC or the stock exchange, by being eaten by the MB/TNC or Frosts of the world.

This would be the current exit model for the numerous university spawned start ups in SMM.

However, another huge issue they have is in defending their IPR: outside the US very litte will be patentable and a copyright can either be worked around or be somewhat irrelevant in a David-Goliath situation. So rounds of acquisition will come down to " can we get there cheaper ourselves? Do we head hunt out the key techies? What value do they really add ? How will they integrate technically and cutlurally to us?" for the potential bigger fish.

Acquisitions rounds will become window shopping with all the above questions firmly in mind once they get a look behind the scenes. Small companies would be wise to limit their exposure in terms of their "black box" code and indeed the identities of their staff.

These days you are open to headhunting through facebook and linked-in, so Social Media monitors may be eaten up by their own poison.

On the Road To Damascus

I met with a personnel consultant this week, who turned out to be a former priest.

He advised me to both jack myself down ( as they say here, by the chin) a lot , become modest and more honest maybe about how it has gone on the troubled road here: things haven't suited me., we have tried different places and jobs, I have been too "larry ;: was I too big" yes I was too big!

He told me about the town constables of the middel and pre renaissance period, around the black death here in Norway to make his point about

They would hold wathc outside the towns and at the gates with a spike adorned stave. WHen someone would have access to the town, they would always ask how things had been in the last place. If the arrival said, "yeas there was a lot of okay things and I got along but would like something new" then they would maybe let them in, When the person replied " oh, full of terrible people. It was no goo d for me and I am glad to be shot of the dump" The watcher would always reply " well my friend I am afraid to say it is just the same here and you would be better to move on to another place"

We know the road to damascus is a way upon which we come to some realisation, but really it is more a fundamental. What happens when you take that road is a major shift in the philospophy your life is run by.



According to the interweb, the Road to Damascus is the conversion of actually an arch enemy of one belief to another thought system. Saul, aka Paul of Jerusalem, was sent out by the Rabbis to persecute Christians in Damascus and return some for torture,test-trial and death in Jerusalem under Judiast law for blaspheming . All this by some charter with Arabia he should execute this persecution beyond the realms of Israel. However on the way, without any human interferance, Jesus spoke to him in a blinding light. Upon arrival in damascus he was healed for this blindness by perhaps christians and converted.

Hence the road is to travel out of the usual daily work or family life towards something new yet with the full faith of your previous convictions owning your soul. You expect to carry on in your ways, but the journey is turbulent and you are confronted by a forces which make you blind to your past and acceptant of the new philosophy you will live your life by.

I feel I have been on the road for six years now. God, four years should have been enough time to make it here to at least some semblemce of stability with jobs. But we are here due to chance and it is not for me. I get some small , wee life here while to my kids it is there WORLD.

IF this relates to anything in my career and maybe yours, then it is to maybe working in purcahsing instead of sales and marketing. Shopping is cooler than selling

New Paradigm Career Making jobs

The new paradigm of the post recession world: it will be hard to get full time marketing work and growth will be slower in the economy. To get the head start on your career, or like me on the next phase then you need to consider the new paradigms, which I will blogg about a few times in the forthcoming weeks:

1) forget job, think project, think task, think opportunity in the box
2) forget work place: think availability for the key calls and key pressing of flesh
3) forget "administration": move away from positions which have a lot of filler: concentrate on the higher value mental and personality oriented activities

When I say "forget job" I don't mean you should lose your concept of working for a brand-name, blue chip employer to get it on your CV. In fact if you eventually do want to forget being a delivery bitch at a company and go self employed, then you almost MUST have some decent employers on your CV. What I mean by this is that you should think about what the opportunity is for you and the company to engage in work of higher intellectual value.

Now this means examining the position you are in or being interviewed for or alternatively, identifying opportunities for virtual promotion within the company or value chain "nodes" you are dealing with on a totally pre-emptive basis. You want to avoid being a delivery bitch for long, not that you ever stop needing to do some hard, boring work until you become an MBA God at a firm, but that you need to reduce, avoid and displace menial tasks off your desk and out of your job description.

In outset for you as an MSc Marketing graduate, this means being wary of job offers with vague "marketing executive" titles and enough of a mix of low level BS that you wonder what you will end up doing. This could be sales work, or menial database work etc. If the job is at a lesser B2B brand or whatever, then this is worse: brands sell product, and sales people just work out the discount-volume and what dinner / golf tour the buyers want to "seal" the deal. PWC is a brand, Ernst and Young, McCanns, Blackberry, Zantac, Viagra, Oracle etc etc. Little plucky clever unbranded company does not need an MSc marketing graduate as canon fodder sales material.

It is more important for you, and funnily enough for me now, to define what tasks are high qaulity in a potential job and to try and work around that in discussions with employers. You want to persuade and jagole the employer into giving you responsibility for projects and tasks of higher value, while they also declare how much of the job is menial. Then think opportunity- reduce the job to it's elementary core added intellectual value: marketing strategy.

You may not formulate a marketing strategy yourself, but contributing to NPI, market research, acquisitions, investor relations and so on adds value to your CV rather than monkeys on your back in a low level job.

So dare to push the employer to identify these tasks. Take a week or two's work trial with them if you get an offer. This is thoroughly recommended for any entry level job with a diverse list of tasks. If you don't like the job don't worry: just say it sucks! They aren't going to grass you up to the dole. Be direct and tell them it is boring for you after the two weeks are underway, and tell them what you would rather work with. Look out for opportunities: points of touch, e-business, marketing purchasing, new markets & customer segments, new products/e-marketing ( social media strategy anyone?)


I am back in this position actually because of the recession and the general miss mix of my bloody minded, male and pompous attitudes ! Had I been in touch with my female side (Ha!) then I could have thrived in doing the manual tasks and sucking up to my aggressive bosses.

I need to assert my space in a job now, and so do you because it is really important to get a "career making" job within 2 or 3 years of finishing your MSc, otherwise you need to seriously consider working in another area or getting new skills, like I did when I got some IKT skills : perfect timing! HTML programming in 1995. I never really did much, but it gave me the edge in interview and also performance in some jobs.

I digress: you need to assert your space in the job and this is absolutely best before you start because otherwise you are on the back foot as a would-be-eager-bunny. ( I am SOOO NOT an eager bunny in new jobs with BS to do. I mean maybe a week or two before my nose is out of joint and I am all anarchic again.) In the interview phase you have leverage and they are on the back foot having chosen you. If they really push you though, and won't allow you to actually come in for even a couple of days and look at the tasks you will be doing, then check the work contract and it may have only a weeks notice which will make it very easy for you to move or threaten to move.

Salary is actually a lot less important for you guys luckily, and this is why you can afford to do a two week work trial, or to turn down a 100% position with "blah blah, 9 to 5" and take a part time job where you focus only on those higher value activities: or at least those which are bone fide marketing.


Some administration is however actually sufficiently high value for you to put your back into: Some high value admin will give you the edge in getting a better job, becoming manager for someone replacing you or if the axe falls, you have a "business critical" admin set to do: for example product administration through ISO and accounts and logistics systems.

As against, cleaning databases with calls, telesales itself, sales support admin.

The reverse situation is more dangerous - I know I have been there- you take a job which is "marketing" in title only or very low grade and bang your head off the wall to try and get a year's work experience on your CV. DON'T DO THIS! AAAAAH! Negotiate tasks worthy of you, and accept a part time job if that is all the time it will take, or just a project based job.

I can hear the cogs turning now, some of you are sceptical to becoming "perpetual projecteers" without ever a taste for operational management. Yes this is a danger! But MSc students are overlooked for the majority of blue-chip APM roles or top 3 agency Account Exec roles so you need to be able to add value to yourself with the smaller brands. Small brand with big strategic impact you made is really where it is going to be at over the next few years of recovery, with new small brands becoming the next FaceBooks or Kindles. (I say brands rather than company because if you are at a company with no brand other than flesh-pressing then forget a career in marketing with them!)


I'll blog again on this area, but I hope you leave with the ideas of being more assertive in defining what you do with your valuable time while you are young enough to work hard and work smart!

Mointoring Consumer Generated Media- The new King

The other day there was a programme on the idiot box about measuring IQ: Kalahari bushmen have been measured as the lowest IQ of any homosapien. 61 or the like. POS that IQ tests are for measuring actual ability to learn, perfect crafts, hunt and so on. THis was driven by a fairly racist appearing "researcher" who had done some "convenience samples" and some meta analyses to draw up these intelligence quotients for different ethnic groups.

Another researcher pointed out the validity of IQ tests as they stand: to get by in the modern world and use your brain to forward your well being and reproduce, you need to be able to relate to the westernised IQ test! There is little point in being a hunter gatherer in the 21st century, you are on to a loser.

Now this may seem a bit removed from doing social media monitoring: but in future will consumer opinions and groups who DO NOT make CGM about brands not become largely irrelevant?

I think that market research will become completely web and IP telephony based. And we will use anonymsed IP MPEG streaming to sample into street level consumer behaviour, and into those social strata who don't do CGM yet are important enough to some brands of sports clothing .....yeah the underclass buying their bling rags....

Interviewing your Prospective employer

Now I have to come back to two points about careers and making the right choices, because I am painfully remined of how bad things can go with a company in the "patient capital" late start up ( F-up) phase. They are running out of cash and I am bailing out, having noted that they have actually a pretty inefficient core "technology" from which it wil be hard to compete while making any decent gross margin.

1) You are interviewing the firm as much and more than they are you!

People forget this: we are nervous and want to match our best abilities against the job decscription AND offer some more, often superfluous info to them. We then get given a standard corporate blurb back at us an in the end of the day, none of us is all that much wiser about the possible exchange of values. We both know it will happen though, and given you are the least bad candidate you will get the offer.

Now, you actually have learnt rather little about the job and they have learnt just enough to know you will be able to function at the hands of the politics and flagrant lack of trianing you will get.

When I was a self respecting masters graduate, I considered a two week work trial ( as supported by the dole office, BA or whatever they are called now) as being a total sell out of my skills and abilities. Now I see it, even at first job level, as a really powerful means to gather info, understand the job and lay down some assertions about what you actually want to do.

In a two week period you will get the core tasks of an entry level graduate job and get to work ( or maybe only fleetingly) with your manager. Thus you will get a real insight into the job and if it is at Aardvark Galactica selling widgets to nobody's then you get the chance to say no.

A MUCH BETTER situation than I have now: the company's technology takes too much manual time to produce results and is therefore destined to be over run by faster, leaner competitors who are more customer quick-win oriented. I knew this within two weeks and that my bosses were aresholes but I perciveered because CGM monitoring is pretty trendy and I need a lift. So I coulkd have spent two weeks with them, as I suggested actually come to think of it, and turned round and said the job was far too junior and played to only weaknesses in attention to detail and motivations. Instead I am six months in, they are going belly up and I am stressed to hell.

So a two week trial is pretty good actually.

2) You need to get the RIGHT first job or know you need to move to the next RIGHT job asap!

I have been in the wrong jobs for all the right reasons, and had maybe two trials or . contracts both of which lead to jobs. The trouble for MSc-ers is that you have no real networking route or trail blazed into the big FMCG brands and you must languish in 2nd rate public service and SME marketing in Scotland if you will not move. In this market it is very, very easy to get a marketing job which turns into a nightmare of doing either sales donkey work or having all the monkeys in the department put on your back: ie all the shit no one else wants to do gets queued for your desk from day 1!

The MSc is a double edged sword because really it prepares you for MBA graduate jobs, while people do not respect it as much. So while being qualified to do strategy work you are actually too sharp for the jobs often on offer and cannot compete with "full MBAers" for those jobs you should aim for. I mean I worked with some prick recently who kept on refering to "fluffy" in the marketing department, AS the marketing department actually: market strategy being too important for the likes of marketers to touch !

A work trial of two weeks can allow you to swing in, get a grip on the company's failures and possibilities and then present yourself as a more strategic analyst and planner than the original "marketing exec' " role was specified as.

More often though, you will be able to see if the job is really just a sales admin type job or if it has very little of interest. You may be able to then play anouther card, often played actually, taking a part time position to cover only those functions you feel are relevant to your qualifications. This is really cunning because it buys you the RIGHT experience to shine in AND TIME to look for other jobs, start your own consultancy or take a convenient second job in.

DIY Reports in Consumer Generated Media

I wrote a bit of a philosophical rant on why and how you could DIY CGM (social media/consumer generated medi) research and longditudinal analyses.

The key tool within this is to develop good enough search strings ( list of keywords in a boolean expression) and I will get back to a very simple, artithmentically based emprical method which will make you efficient in "cutting to the quick" of any topic or brand-tracker in CGM.


1) follow the path of least resistance: everyone has trodden there!

- If you get your search strings sorted in your preliminary build, then you will follow a route which all the sheep do, and therefore come upon the forums and bloggs which have the highest postings and the highest reads for your topic or brand.

Forums are for the moment, still the best source of consumer insight for many areas. Remember that google/yahoo etc only index about 15% of the web and probably less of total "live" content so far, heavily biased to some forums in some topic areas. This means you will follow the heard, but remember to go and look at the forum indexes once you identify which are the most ranked in google.

2) Once in the place, read the biggest threads:

subscribe to those which have not become off topic rants. Use the forum search functions, advanced if possible, on your topic of interest. ( go back to your snowballing query build - coming soon on a blogg from DF!)

3) Small can be beautifully ugly!

When looking for say, NPI problems, you want to do the opposite: find small threads or recent ones near a launch date for example, and then see if this has propagated out on bloggs, forums and twitter searches. If you see a smattering of discontent, then this can point to a failure in QA or just a bad product.

4) Follow the opinion leaders: the forum matadors.

Often you can find all posts by an author who seems to be an opinion leader, early adopter or influencer and follow your topic using this route. This is often valuable insight, and also you have a weigthing for importance based on them replying in threads.


5) Use "5 bar gate" counting on hits
-
for various opinions, co-mentions / brand comparisons and opinion polling: often called sentiment rating. Look for ratings for posts and any polls which appear in the area. Also make a note of "reads" on forums and bloggs. All these are useful stats and you are adding a quality filter which computers cant!

6) Too many results! ??

use an nth page technique: however, for a given date range or from the latest listings, read the first few pages to see if the same threads are getting multiple listings, or bloggs are getting extended replies. An nth page sampling strategy plus using quick hit counts onto a sticky note is just as good as a computer doing a census. For 100 pages, an every tenth page would be adequate, or every 5th if the results are often not in true CGM .

Use this also within huge threads- read the first few pages completely, then do every say 5th before moving up to every 10th until the latest 5 or first posted five (depending on rank order by date ascending/descending is default) thoroughly. Then you know the "currency" and "providence" of the thread and can rate it's value.

7) Linky?
Follow links to other forums and other CGMs from your top relevant forum postings. In this way you snowball out and can gain a picture of the prominence on the different SM platforms, and the interconnectivity- itself an interesting meta-metric to consider.


For example there may be a youtube video or a discussion on a related article on the BBC or CNN or so on, which means the prominence in "views" is really very high for a brand, product or topic.

8) Issue tracking:

now of course you can subscribe to a thread, but you can also do a bit of detective work: look back over time and follow external links. Expand your search strings to include.


9) Running statistics:

this is a little bit tediouos but can be done using repetitive searches on advanced SE forms over different date periods, or using five bar counts. Once again for very long results lists you can sample nth pages and get an idea of date-distribution.

A short cut is to know results per page / total pages and when you are in threads, the same for posts per page. It can be a little tedious to get to the last page for the count if there is no "skip to last" button.

10) summarise your stats and findings: priorities around the more prominent media spaces, BBC, CNN; the biggest forums while also pointing to those specialist forums which maybe have the highest QUALITY of insight - early adopters, beta testers, industry insiders, and innovators who are taking your product to bits. From your bar counts you can do a rough guide to opinion ratings for products and illustrate this with quotes and whole conversatipns. You may want to make a couple of case study threads on inlfuence by opinion leaders or on how prominent a single thread has actually been and how "linky" and " tweety" a page is. You also want to include the prominent blogs and the prominent Facebook groups, applications and fanclubs. From all this you could produce a very insightful report on a brand, an NPI issue or on a developing consumer need and this report would likely be of high quality compared to the big hit counter type products.

Show me the Money! Job offers in the Credit Crunch

...or the importance of getting a good deal before you accept the job offer

I haev ignored this three times in the last ten years and it is well worth your effort in NOT accepting a low offer despite your circumstances. Look the gift horse in the mouth! Make an informed decision and play some poker!

I cannot stress enough, I will not say too many times and I dare not shout too low that you must get what you need from a first (or second ) job before you accept the offer. Usually this boils down to money, and YOU MUST be happy with the balance between training, projects and salary, because that is all you are going to get while in the job.

Work experience per se is meanginless if you accept a "marketing assistant" job blindly or the nice tasks are only 10% of the time and maybe telesales, database admin, photocopying etc are 90% of the actual job. I wasted a crucial year in such a job! I managed to play catch up with a heck of a lot of hard work in my next job though. It is in fact much better to be paid virtually nothing on a 2 month work experience project where you actually conduct quality marketing tasks than the department GOFER for 12 months.

You need to avoid this by negotiating before you start. Negotiating is always a subtle play of information, and probing for a little bit more and then using it as a little bit more lever, while giving only a little bit out which gives a poorer lever to the other side. This is how you win, you subtlely win tip the balance in your favour while accepting some sort of concession compromise: a few prawns fall but you put them in check-mate.

But firstly, back to green-backs: Companies will palm you off with " we can't afford" or " we have current wage frameworks". This is rubbish, they want to screw you especially in a recession. " If you want people to work harder, pay them less" ...this is true in bum jobs or non graduate jobs......but how effective this stressed, undervalued work period is for both parties is quesitonable. The cost between a market going rate ( admittedly low -I know in the credit crunch hangover period) and a lower offer boils down to a few hundred pounds per month, maybe £70 quid per week for you as a graduate. Can't afford? they could save this by using a better travel agent for business flights, or missing one useless client dinner a month.

Facts are bosses like to screw you over and say they got you cheap, it makes them look good.

So what is on the table to get your information war going in your favour
1) salary
2) training
3) projects and defined responsibilities
4) relocation expenses
5) a 50 to 80% job so you get time to look for something better
6) promotion prospects and turn over in the firm/sector
7) external prospects when you have a year experience
8) the company brand name for your CV ( or client brand names !)
9) start date
7) legnth of trial period
8) Legnth or termination period
9) possible contract engagement only

How can you bargain? Well your only chip is to withdraw your offer of labour. If you are on benefits, this is high risk - you can get them stopped for refusing a job. But given the job is high risk and has financial implications ( expense of rented property). But it is still a poker game and you can't show your hand too soon

How do you bet this chip then, play this card?

Well first of all, you should from outset know what the "wage framework" would be for this position, and know that you as an MSc Marketer have five years of uni to offer and a very good marketing masters under you belt. This should NOT be turned back on to your expectation: some jobs have been 18K outside london for a while. Play dumb, say you have only started looking for work and have expectations to work hard and get on over money: what would they suggest as for me in this job? Don't push too hard, but do revisit the wage BEFORE you bother to go to final interview. Also don't go to employers who dont refund expenses and prefably choose those who book everything for you: if they do this, then put on a bloody good show as they will look after you as an employee!

Now when you get a disappointing offer in an interview, you need to not look too disappointed. It is more likely you will get this on the phone, same here , move quickly on from the money to the responsibilities. Now between this and the eventual decision you must buy time and get concrete DECISIONS not just information, but agreements on tasks, training and projects you will get. You need this in writing to refer to if y our job starts going down the pan soon in.

The important thing with a low offer and poorly defined work responsibilities is to stall in an assertive, positive looking way. Comms mus tbe very positive but not fully committed.

The pieces then come in the list come to play: low pay versus excellent experience or low pay because it is a crap job a graduate in medieval history could do. What tasks, repsonsibilities are up for grabs? Remember you have an MSc, they have not made their minds quite up, so help point them away from the photocopier and mail list towards the high valuie customer satisfaction survey, the social media PR campaign, monitoring competitors, the effective review of the marketing budget and so on....What training courses are internal, who will be your mentor, what courses do they offer externally, will they support your time and materials in doing the CIM courses?

The next thing is of course relocation expenses. These were , a while ago, tax deductable for employers but funnily enough NOT employee personal tax. Employers will put new senior managers up in penthouse suites with hired 4x4 executive wheels! But they will try it on with you as a new graduate in need of work experience. Don't let them if the salary stinks. If it just an average salary, negotiate even a lower salary for the first six months relocation on the back that they find you afirst a hotel and then pay rented accomodation. I have not done this on one notable occiasion and really got stung for the outlay when the job didn't work out ( they put a directors daughter in my job) As a graduate you will have enough debts.

So now you have the balance of information: if the job now sounds poorly paid, poorly trained, low level tasks without relocation, then you may as well ask for say 20k and tell the m you have an offer at that level: never tell them where ( esp recruitment consultants, they are desperate for leads and to close deals! )

Now given the pay is low, but there is relocation and training, you need to find out more about career prospects. Suggest a day's visit to meet the team and finalise the contract. This is a positive thing for the employer. Ask other employees how often people leave - is there a turn over- ask why the job is vacant from your prospective coworkers . Ask how your coworkers started out in the firm. Ask what they studied.

One thing too, what type of cars are in the car park? This is a very clear indication of salary prospects later. Audis, beamers, volvos, 4x4s good...., old fords and citroens very bad...a subsidised canteen is nothing much really as a benefit but a gymn, studio , pool? .

Now you have enough information to make a decision: take it only with more pay, reject it outright, ask for silly money as above or there is another option: shape something new based upon your own needs: a more focused project 3 days a week or for just three months on say expenses only as a work trial. Or a 60% position covering only relevant, higher value marketing tasks. Or they pay for your CIM proffessional development and two triaing courses of your choice within industry average cost of courses.

In negotiating you need to push them to the wire: stall as long as you possibly can if the wage is low: get as much information out of them while remaining positive, but non committal verbally amnd on paper. Then if you have managed say two weeks stalling for a job where there are lots of questiona and a final visit to meet the team, then you can pull your card out and say the offer is not attractive enough and thank you but no thanks

Then you can let them do the talking and you should not let them bully you, just repeat the pay is very low, you have a better offer, or you have interviews for better paid jobs.

If you have been looking for a job or working temp for a long time, up to 8 months say, and you feel forced into accepting a job, then a final end game is to stall and ask for a later starting date. Use the time and security of something to do anyway to look hard for other jobs. The fall-back and interview-offer experience will be good. When you get a better offer you will only have a months termination, which may be even before your start date or you can turn up to work with an offer, see what tasks and training you will actually get and then demand them or you will just call it all off.

Once again you could ask for a 60% position focusing on the real marketing tasks and thus avoiding being given rubbish to do. You could probably expand it to 100% if it goes well or at least have two days a week to look for other work.

DIY Consumer Generated Media Survey

DIY Market Research in Consumer Generated Media

At this instant, many universities around the world are spawning out small start ups and VC are raising eyebrows as angel captial invests in a new type of market research and intelligence firm.

The new enterprise opportunities are based on the sheer volume of CGM and the vogue for the big brands on the web in this area: Twitter, Facebook, Digg, and the latest brave new entry, google-buzz. Statistics, as I discussed below, are a little hard to use in reality and the cold world of market-movements and quantitative , conclusive, inferential and the numerically indicative is somewhat removed from CGM at the moment. What the meat-of-the-dinner is in fact, remains qualitative research with some utilisable methods for stat's which help describe the parameters and prominent qualities within voice-of-public or "Buzz" in social media.

Now from the arena I have seen, there is quite a smidgen of the "emperors new clothes" around in social media-monitoring. Take for example what is all dressed up and being used and no doubt abused: In areas like sentiment with some pretty flimsy algorythms out there, or little if any statistical significance to confirm the relative changes over time or differences between brands.

Also hit count statistics: for reasons of the prevalence and magnetism of the big sticky threads I discussed earlier, these can in fact populate a large amount of your hits in a topic, and if a topic has become google-rooted ( search engine ranks are high for that forum on the given free search in the topic area) then these get alot of noise about nothing other than one place to look. You see where I am going? If you want to open a kosha sandwich deli, then you will soon realise that most of the current world market is in new york.

It is actually pretty easy to follow the path " he who hath shall have a cup which over floweth, and he who hath not shall go without for ever" : the sticky sites and the sticky threads suck in a lot of the numbers and within this lies some of the really good qaulitative insight. You don't need large indexing or meta crawling tools to get the same qualitative result: but you do need sound judgement and the "corner pieces" of your social media space and range of consumer expression.

The opposite is also true: very small postings or postings which are very similar over a range of web forums and other social media, can point to a lead indicator or early problem alert after NPI. New users posting in a period after product launch are worth picking up: they are often the tip of the iceberg of customer dissatisfaction!

Until a few years ago, search engines did not want to index "live content" for various reasons best known to them selves! So any php , asp or cfm pages where ignored as perishable and not to be indexed. This had me stuck on a few forums we ran for clients a decade and more ag- Iit became a bit tedious because back then the big-thread magnet phenomenon, and ettiquette (discussed two blogs ago). However the corporate bosses were hanging on every word written down in awe and fear of libel suites or some tumultuous disclosure ( which did happen actually)

So your start point should be to follow the well trodden path like a wolf amongst the sheep who go google, and then like the idea of CGM forum rather than reading the corporatised blurb or sanitised PR bloggs. The doors to the crime scene are all open and there are hundreds of footprints.

So your tools are the search engines. Beware being all google centric: some may be more prominent nationally or within a specialist niche of global or national citizens ( academics always used Alta Vista and then moved over to FAST all the web for example). Now you add google buzz, google blogg search, twitter search, youtube, tweet deck etc and you start to have a powerful set of doorways to be able to set out and build a report like "attitudes the the bumble bee brand amongst international english speaking consumers in Social Media"

A few weeks ago, Google announced they would be indexing public content on FaceBook which will make both some opportunity , and a big stick to beat yourself with. As with analysing tweets, it can be a torturous route of reading conversations or following links to actually make sense of hit results.

It is a little difficult to get meaningful statistics in DIY SMMing but some clever use of search string arithmetic will help. More on this , making your google etc advanced or multiple searches efficient and exhuastive in a later blog.

You can meta-track launches, from rumour mill to unboxing and consumer adoption. You can track political issues, viral news story discussion...anything that affects several hundred thousand people in a western country, and you can bet it will be posted on, blogged, tweeted or have it's own fan or hate club on FB.

On some topics you will find a fairly concise set of mega-threads, a smattering of blogs and a pitter-patter of small threads and comments around the various social media nodes. Other topics you choose to research will be huge, sprawling and broad in both their appeal and the spectrum of opinion which is expressed.

Larger topics are usually worth sub categorising by sub topic, geography or forum-colour. Alternatively you can try to see the amenability of searches which find a type of segmentation based around a more qualitative factor: like consumer intention to purchase ; polars of sentiment ; brand or feature comparative posts and pages.

When you find page hits ( times 10 for post hits on average!) which run into the hundreds then it is worth using a very simple, well validated sampling methodology. First ensure that the page listings are exhaustive and you know the total number. Then take this and take it as every tenth page to counts of 100 or 500, and every 25th page for over 500 and so on. This will mean opening everything in "new tab". Most pages on forums will have 10 posts, but some may list the entire thread or hundreds. Then you can apply the same rule of thumb: every nth post: 5 for 100 would be a more quality result. The point of this discipline being that you sample from the whole distribution, (population of posts as species if you like) and you don't follow "interesting routes". In other words, you are forced to take a wide angled shot so you understand the landscape before you can decide which features are actually representative, prominent or meangingful in light of the whole spectrum.

From this approach you can do some surprisingly quick five-bar gate counts of keywords, brands or even sentiment. Many forums have sentiment ratings, and if you include comments and reviews on places like Amazon as CGM then you can start to do sample based sentiment ratings - which in fact can be pretty much as accurate as the latest AI driven ratings- if you have enough time.

All is not equal, as discussed in the sticky threads blog below. Some threads which are large or have topical subject lines, receive many more hits than others. Also some medias are more prominent and perhaps carry more status: like the BBC web forums and comments boxes. Forums with high SE rankings tend to have the most traffic. Retweet rate /total is another meta-metric .

From a knowlegde of prominent forums for a product type, brand, band, author, lifestyle or political view point you can then consider the sub set of consumers who are most interesting to follow up: the innovators, the early adopters, the opinion shapers, the self-appointed authorities, brand champions ( fan boys / fanboi's) ..brand terrorists....and follow their posting to gain a high level view of the discussion: see if indeed they are influencing people or if generally people make their own minds up and buy that pink coloured laptop anyway!

So you start to get a feel for how a report may be structured, using simple hit counts as a top level introduction and then results from your measurements within the samples. Finally you get into the qualitative observation with the prominence of the media and the activity of the opinion leaders, and the sentiment tallies from the different samples to give some kind of summative opinion poll for the topic. The conclusions you may draw should therefore be based upon prominent information, a knowledge of why it is prominent and what else lies in the spectrum, a handle on the polarity of sentiment expressed and the average point for consumers, be it neutral or not! When you make a conclusion which points to a useful management insight, then go back and check the prominence: check the hit coutns relative to other topics or shades or opinions etc, check your sample is exhaustive and re-check your search strings ( a little more on this latter below and then another blog , coming soon to a soggy-spot near you!)

There are plenty of kid on numbers you can put around these things. For computer scientist graduates, metacrawling or re-indexing can be a way forward to producing statistics based aroung the single post as the "Unit of selection". Different sampling strategies based on random and temporal dips can be useful when confronted with 50 million tweets per day!

For the very numerate amongst you as marketers, sociologists or computer scientists, you should be aware than CGM is in such large numbers that a topic such as a fairly common brand name or product, will have a "normal distribution" of opinion if you like, and this can be captured in a correspondingly 2 SD centric list of keywords: the first six search strings capture the first two or even three standard deviations .

There is a bell curve : x axis rating versus Y axis volume. The majority of opinion/keywords etc, will be within the first two standard deviations. When you do a nth sampling you really get to check that the bell curve is covered. If you do manage to plot data, sentiment or keywords, and you find that there are more peaks and troughs than one bell curve then you have either too small a sample size, a poorly defined opinion-keyword-etc scale, or in fact you are measuring two different things: either from two destinct populations with some degree of polarity to each other on your scale ( OOPS! you sample tory and labour forums ( republican / democrat) and not general political discussion!) .

When you know you have a nice bell curve then you can be very safe in using nth sampling or random statistical sampling and that your comparisons can be shown to be statistically significant: FOR THIS DESCRIPTIVE DATA SET. You cannot use this as inferential statistics, primarily because you cannot accurately capture social demographics in CGM and there fore you cannot make any extrapolations to the population as a whole.

If you combine an offline survey which identifies people's demographics in relation to their interaction with CGM, it can be possible to make some tentative inferences based on the knowledge that your large sampling base is composed of a cross section of society idenitfied in this CGM interaction survey . Even then you have to tread very carefully, statistically speaking, because your "Hits" are by a decided number of authors, some using several handles over forums, some using multiple identities to stimulate discussion on the same forums ! In other words your actual "n" for the study group is too small. Is the post more important than the author? Hmmm well people tend to be consistent and only change opinion after some degree of cognitive dissonance so really your "n" is authors and not posts.

Inference to the general population soon evapourates when as you get into small number of authors per posts, and some of my "sticky, syrupy threads" are very much dominated by a gang of less than 10 key proponents. But then again a knowledge of what cross section are reading those forums and the thread rankings on the SE's means you can start to make a judgemental call on the importance of an issue or the opinions around a topic.

Sociologists and psychologists are very taken up with not interfering with the subjects:not introducing experimental method source errors, researcher interference or interpreter bias. If it is purely observational, then just a simple permission disclaimer is all that interfere, or in focus groups, skilled moderators stimulate debate and keep it on topic while being allegedly carful not to introduce biases (observers are usually in other rooms and should not confer on their notes themselves! )

But in the area of CGM you can be a little more anarchic. Having identified and qualified your CGM sources as "prominent" then you can set out to interact a little by starting threads, or tweets, yourself. This is a purely qualitative approach, but it can help you gain insight in an area where you found many tangenital conversations, unclear opinion or forum leader-or fanboy -bullying ( shutting out opinions, topics , alternative products/ solutions etc) previously skewing the area you are researching. Tread a little carefully and pick those forums or social networks where you have established that "noobs" ( newbies...first time or low count posters) receive a positive welcome and a range of replies and are not shut out when they post sensible . This means you can pose a question within a subject which is tenable : this could indeed include concept building around latent demand and unmet needs.

I hope this has stimulated some ideas for just going out and doing some DIY research from your desktop. This approach deals not only with qualitative observations, but you may also pick up some qualitive ideas on what would work with a crawling-indexing system, or a new type of social media platform!

=================
Perspective
=================

To show how long in the tooth I am, and just how jaded I am by the industry, market research is a be-whoared cinderella within marketing. Of course it should be the lead violin, the first on the dancefloor but instead it is the working girl who turns up in her best frock only to have a hand put up her skirt! They want her knickers off, just to get as quick as they can to what makes them happy: to drop the analogy, product managers have often made their own minds up about what makes a good campaign and where they are going and only want market research which will support that or their plan B. They have sales and national account managers to keep happy and they need to steal a bit of limelight by doing something unique.

This is true of research in social media, and it there is a danger for observer bias in generating keywords and search strings, and the in choosing themes or summarising the spectrum of opinion. Conversely, any-road-will-take-you-there-if-you-dont-know-where-you-are-going, so it is easy to follow seemingly prominent themes and paths of arguement which take you down blind alleys. Avoid the critical path approach, and keep it broad and objective.

In a later blog I will discuss how you create an objective set of search strings which are both exhaustive enough while being efficient in "containing" a topic, and as mentioned making sure you are within the first couple of standard deviations for a given distribution with the majority of your efforts.

Why Monitor Social Media ?

Perhaps you are new to the world of social media monitoring at work, or are embarking on your studies at university in this area, and you are wondering just exactly "what is in it for me ?"

Maybe your background is in traditional marketing or perhaps you work in a technical or customer services function and have heard that this new area is indeed something worth looking into further . How can monitoring social media help you make decisions and deal with problems and opportunities?

Well it is of course e-Marketing and web masters are the most interested and are sitting glued to the analytics and consumer opinions. Of course it is not just for the geeks: communications and customer services who are aslo getting engaged with SM. The benefits reach wider and deeper into the organisation though. Everything from new-product-introduction & tracking, or competitor pricing, to issue containment is amenable in almost real time : the feedback loop from action-consumer reaction is drastically shorter.

Statistically Speaking

One burning question is how can information in SM be used to make conclusions about the wider consumer population? Well speaking mathematically, usually we cannot make inferences to the general population or produce actual statistics - yet.

The day may come in the near future though, when the numbers of consumers engaging in discussion will be so high that we can draw inferences to the wider population, such as intent-to-purchase or brand awareness, and put some hard numbers behind this with SM a quantitative source for extrapolation to consumer behaviour in the market as a whole.

Even then it will most likely be from a definable cross section of different geographical- or network-societies: age and education related. It would be dangerous to draw inferences on anything but the first three standard deviations from one of those sub populations who are engaged with the internet and SM. However we will be able to utilise statistical probability based sample methods within any accessable or stored data set to produce smaller data sets which are make analysis more efficient within given parameters of accuracy and inferential significance.

Social Media Monitor Should Be Qualitative

For the moment though, reporting is focuses rather on the valuable qualitative insights to be found, "straight from the horses mouth". These are the root causes of issues, the actual verbatim opinions, the dissatisfactions, the real point-of-touch customer experiences : I could go on! These help illustrate findings from a company's quantitative reports and data-sources, as well as pointing to new insights which uncover consumer opinion hidden or distorted by the very interactive nature of surveys, depth interviews or focus groups. Also they can uncover uncomfortable truths which are hidden by line managers, front line operators or re-sellers.

Everything is Relative

Despite the qualitative output of reporting, descriptive statistics can be used within the domain of social media to illustrate the relative prominence these qualitative observations This includes the relative prevalence (or you could say "share-of-voice" ) of brand names, consumer opinions and for instance problems with newly launched products.


In combination with ever-more-accurate sentiment algortyms running with AI (artificial intelligent) systems, this area of descriptive statistics will be used more and more to give a picture of the cross-section of society using SM to discuss your brands and customer support. The value will increase if it can be shown that the "listen-learn-decide-react" loop on the web connecting to SM, is functioning.

Then our little world of SM becomes a market in it's own right, and this is already happening with companies engaging in different campaigns and communications which are built around information from social-media-reportage. Some companies in future may only interact with consumers through this interface and connections form SM to their web-services.


Numbers and graphs are all very fine and nice to present and talk about, even with the provisio that this is a little and twisted version of the world at large. But even a very low number of "hits" within the latter, for example, can reveal invaluable insight into potential challenges in production lines or further back in the supply chain which are creating problems not detected earlier in the testing and launch programme.


Tracking New Product Introduction

This qualitative approach has been of particular value in tracking new devices launched on the market, which have a plethora of features and most likely diverse internal software. However, this is equally valuable in tracking a new service, or immaterial product from a financial institution or a mobile network operator. Or in defining an unmet need or latent demand out there in the market place.


Within the world of gadgets- consumer electronics like mobile phones, PDAs, laptops or digital cameras - it is in fact often the lead consumers who are the real experts: they can be tracked individually, from maybe a sample of 10, as they try and buy diverse gadgets and report their experiences on the web. Often they seem very informed on how the technical features, like processor, touch screen, GUI, actually deliver benefits in use and how much better this performance is to earlier products or competitors offerings.

In fact these lead consumers seem to have a more wholistic view of the product's perfromance than the head of R&D and most likely the CEO at the manufacturer! Worth listening to SM?

Sticky, Syrupy Threads

Discussion forums continue to be the most useful social media space to gain consumer insight from, for now at least.

Within these forums you can follow consumer's problems, desires, experiences etc and see how discussions evolve around out clients products, services and brand values. One species of thread which has always very been prominent on forums is the mega-thread. Why are they so "sticky"?

These threads are way longer than average in number of posts. Some run to many thousand entries. It seems they break down into both serious, often heated discussion, diverse postings around a topic - like a new mobile telephone - , or pure trivia like jokes or chain stories.

One reason that they are sticky (ie attract users and keep them coming back) is that they often have prominent placings when forum threads are ranked in order of date on topic listings page, or on the more recent "hot topics" side bars on home pages and top level forum category indexes. It would be interesting to see statistically when such threads attract a critical mass, and how they develop from there. This could be a useful metric and watch alarm-trigger. Rate of growth, number of lead influencers, ranking: some algorythm could be made to work and fish them up to a dashboard panel of "Hot stuff"

Consumers coming to a forum see these mega-threads as the " tall oaks" amongst the grasslands of granular postings. By their stature, they demand respect and people will often come on board a product topic and post their " 2 cents" before they would consider creating a new thread, even if their "2 cents" is a little off topic or outside the current line of arguement. In this way they appeal to the conformist consumer.

These threads then, are not purely a result of some natural magnetic force , or social diffusion through the ethers of the internet. Most are nurtured, and some are not only fertilised with fresh content, but have their space cut clear for weeds, these being small competing threads. Users who can be identified as opinion -leaders, early adopters, expert-insiders or collectively lead influencers, will steer a thread they like and by carefully timed posting and replies to users comments, they will keep the thread up there in the top 5.

Lead influencers vary in how often they initiate threads, but they turn up like clockwork on the hot news threads or major theme threads relating to the forum's raison d'etre. They are in fact instrumental in coaxing the threads to gain critical mass.

Nurture also happens unfortunately perhaps, from forum owners and appointed "moderators". Some forum owners will post on new, related threads, stating rather rudely that the users should refer to the long running topic and "this is closed". They even delete competing threads apparently. Scornful lead influencers will also pounce on unsuspecting thread-starting-newbies, and stamp their forum authority by refering the user to an old worn arguement they should join the gang on the proper big thread, or just have used the "search" function to find the info' on old threads.

Some lead influencers like to demonstrate their boundless knowledge and articulate debating skills, while others can appear very helpful and down-to-earth, sometimes though outright patronising to those with lower post number seeking advice. In fact some lead influencers post almost exclusively in big threads and never start their own.

Usually as rule-of-thumb, one can consider those users with over 1000 posts as a start point to identify lead influencers within a topic of interest.

After this start point, you can delve further into their posts and behaviour, to assert if they are leading discussion and influening others to change opinion, be informed or of course buy something.

Mega-threads have another type of gravity: Often they attract a disproportionate number of "reads" to their actual post number, relative to smaller threads in the same forums. In effect they become the headline pages, or new-channels within the forums. People go there first, they grab attention for read-only "lurkers".

This read-count makes them even more important for companies to gain insight and summary of which direction the group of big threads on say, a product launch or a service problem are going.

One explanation for their disproportionate is their prominence on the forum as mentioned. But also these are the threads which the Google/Yahoo type spiders actually come upon and index. The threads live longer and are earlier on the index-crawl. They have bigger clusters of keywords and have by pure virtue of size, more links out and eventually IN to. Hence they are search engine friendly and score high on relevance and hits. In conjucntion with good web site SEO, the forums get quite high index listings on the SE's depending on the search terms. Also consumers set a bigger price on their own generated opinion! They would rather read 100 different user opinions than one PR story regurgitated, neigh, re-tweeted 100 times.

SE listings then helps the threads gain extra critical mass and keeps them "bumping" back with new posts even some time after they seem to have bruned out.

As we know though, yahoo and google only index an estimated 15% of the web, so it is pretty much hit or miss for actually finding these threads. Choose a good SM monitor company with either full indexing on main sector forums and general consumer forums, or those who can sample effectively from these for the big issues.