Showing posts with label facebook. Show all posts
Showing posts with label facebook. Show all posts

Thursday, March 24, 2011

DIY Social Media Sentiment "Barometer" Techniques

Contention: . Traditional hard and boring market research techniques can save your bacon when you are looking to turn-around fast, accurate and most of all PROFITABLE reasearch reports or data mining portals.

I've been lucky to be in the doors of a couple of Social Media Monitoring hot-shops (SMM) and also have a friend who has programmed both back- and front-end for screen-scraping and data mining in web monitoring resources.

I see that today's technology means the usual IT smoke-and-mirrors dissolve away into what is today pretty standard undergraduate project work, just lots of it. Even sentiment rating tools are at a level of final-year project for a Harvard Comp Sci Major.

In this blogg let us look back under the bonnet (the last blogg) and maybe chuck out the v8 single carb for a nice little supercharged injection engine.

A Heavy Handed, Verbose Approach to Social Media Monitoring

What the media-monitor companies seem obssessed with is having accurate census of all posts in consumer forums, and now with access to FB and twitter, all hits on the brand names. This is because, like Everest or the Great Wall, they can build it. Not that they really need 'go there' but because they are used to huge data resources, it seems the right thing to do.

Now this is by no means a labor-of-sisyphus, it probably just feels like that, espeically when major forum web sites kick your crawler's IP address off their triage-server. Up until now, server resources have not been an issue, but with capture and indexing for Twitter, Digg and not the least FB indexing and retrieval navigation start to become sluggish and resource heavy on both vertical, horizontal and labor in maintainance and expansions.

So faced with the practical, economic and client-patience issues of providing huge data mines, what is the alternative "blue sky" out there? Well of course come on guys, don't re-invent the wheel.

Comp'Sci'Bachelor's Myopia

There is a cultural issue with computer science "majors" because they race through math, especially statistics and only listen to the more web oriented parts of any ( if at all) marketing course units. Like many people outside the profession, it seems all comms and "fluffy", not a serious branch, just a bunch of luvvies making hot air.

What they all want to do now is either be a hot java programmer, admirably, and/or do phone apps or APIs for web sites. What they do not want to spend time on is "aligning" to the harder side marketing. The web is the new patch, keep off with your Taylor Nelson 1950s stats and punch cards.

Well post-pubescent-programmer, er sorry, software engineer/app developer, you are wrong. Traditional hard and boring market research techniques can save your bacon when you are looking to turn-around fast, accurate and most of all PROFITABLE reasearch reports or web.

Sample Dear Boy!

The first population cenus was perhaps conducted and documented as long ago as 3400BC according to Wiki. Census means getting ALL the possible data points in the bag. Even in the infancy of market research, the census approach was rejected for consumer reports because it is just too combersome and expensive, and if you make any mistakes in the methodology, like ambiguiities in questions, then your entire results are biased and not accurate. Also there is just a lot of irrelevant data capture points.

Market researcher managers, the hard-men of the industry, soon adpoted sample-set techniques from statistical science used earlier in biology for example, and went on to develope their own methodoologies for creating such smaller sub groups. The key here is that the mathematics gives you a known accuracey or probability that your results from the sample group represent the population - as whole say of a country- or as a set of say, people visiting Wallmart on tuesdays.

Researches uses Random Probability Sampling and went on to use stratified-random-sampling and nice temporal techniques like "sample-resample" where time is also a random factor in combination with geographical location as a random or defined set.

In outset all these techniques mentioned ( apart from sample-resample where you define the time period) rely on a knowledge of the size of the total set, maybe the national population or the geographical locations of all supermarkets. However, you can use educated estimates or other statistical techniques to produce a number.

Dangers and Benefits of Statistical Sampling
These techniques are reliant on a few assumptions, and often some manual intervention as to common-sense sample techniques. Here in lie two dangers though:

1) If you have say one model with assumptions for the population total and there are of course some assumptions in things like the t-test and Chi Squared test, then you can create massive sampling errors by the combination of the two levels of assumptive error.

2) You can become overly confident in the results based on a very good (and expensive) sampling methodology, while being completely let down by "non sampling error", usually a crappy questionnaire.

The big benefits are that you get a senisble and often very small sample size to then look into with GOOD study methodology ie you can put the man-hours into design, execution and results and not the shoe work. Also to the delight of many accountants or numbers guys with MBAs in senior management, you can present the margin of error expected from a sample size ( always " n ") and thus make cost-benefit decisions based on need for accuracy.

How Does this Apply to SMM?

Sampling is great for SMM because if your crawler is working to 99% or better, and your clients are only interested in the web sources you index, then you can spend time doing far better sentiment analysis, manually, than the algorythms will EVER manage to produce on larger sets of data.

You have two approaches in using sampling: sample the entire set of data BEFORE you index it or sample into results (or index-strata) for a given topic.

So for Twitter for example, with maybe 100million global tweets day to the minus one, it would be an efficient strategy to work on total per day ( you hear about it in the news, or you maybe can pay for the info, on a country basis too) and if you can geographically restrict your base, then you can work on the main number being in accessible languages and sample to a much smaller figure.

Comes with a Health Warning

Now here comes two dangers, the first one relates to the latter instance of twitter sampling: if you are looking to drill down to tweets on a brand name, in one country or language, over a short time, then you run the risk of NOT sampling enough records to give statistical robustness at this level of "cross tabulation" aka drill down. This is actually a big problem with many standard market research studies, because when you drill into say: Age by location by salary by single then suddenly your stats on that sub sample to compare them or predict them, evapourate into inprobabilities. Hence you stratify your sample: you use a common sense or pre-filtered method to include those sub populations or you study those sub populations against a sample of the General Public.

You see where I am going with reference to your itemised consumer posts being the "population", not the individual users.

Also the other problem is in this stratificaion: you choose a criteria which is not exclusive enough or exhaustive enough to satistfactorily capture that sample or that population to sample into. This would be the case with building a simple query, a search-taxonomy to dig out records in a certain topic area or a time frame, or geography. You don't capture accurately so all the stats-techniques in the world applied on that sample - population relationship won't save you from GIGO.

Beauty in Eating only Some of the Elephant

However the beauty of it there as a tool is also apparent: you can run very quick manual dips into either pre-indexed data (indexing takes time!) or into query results from your indexed data, or into your indexed data set as a whole. Then you have a back up too in delivering automated sentiment results and taxonomy query reporage: in other words you run the sentiment algorythm on total data for the period you want, and then sample into it and compare your manual ratings ( based on the same principles at least as the algortythm) done carefully on the sample. This gives you both a whole new strategy to add value to client reports; it also gives you a means of building taxonomy with a security you have enough threads to take key words out of; and it gives you a QA on your taxonomy or automated sentiment reportage.

DIY Social Media Sentiment Rating

Worse than that for an SMM company: it gives the average biology graduate, let alone stats' major, a tool to go and do it all manually by knowing forum sizes or tweet rates, and then looking up the random sampling tables to give them their sample points or periodicity.

It then comes down to how objective the human can be in allocating sentiment on say a five point scale to a post. Some people would prefer to leave this to an algorythm and I have sympathy for that!

Monday, January 24, 2011

The Road to the Next "Facebook": Start Ups

When establishing your next venture to rival FB for popularity and success, then you have to start somewhere buddy, and that is going to be soft capital and the business angle network.....

Business Angel and First VC Round Funding


Business Angels can be roughly divided in two: individuals with resources who are prepared to invest in maybe only one venture; and the other portfolio "angels" who are actually self made high risk fund-managers in effect. Now both have their value in bringing not only money but well needed guidance and goal setting for the real world ie 'window dressing' the company for the next round of funding.

Angels fund of course, but also most usually match other resources to the start-up's needs as well: most often that means people with talent, as employees on the initial management team, or consultants to help the team, and often the key members of the board of directors. Some will also marry premises, or actually are property owners with suites ready for expanind businesses.

Start up enterprises should be sensitive to conflict of interests in both these areas above: human resource allocations and premises:

1) croni-ism: It is all very well for the business angel to have given the seal of approval in bringing someone in, but by not advertising generally then you do not actually know if this person is best suited in experience to your company and your market. The Angel may well shoe-horn someone into business development or engineering who has been successful in their portfolio, but who may not be ideal: On the other hand at least their work capacity and motivation will be QA'ed which is a large part of the battle

2) In terms of property and services, some business angels (and management "guns for hire") are far from angelic: they look for ROI in terms of direct revnue back to them, and subsidiary capital gains: so they may push you into buying property, so they enjoy you paying it off with revenues eventually, or actually rent you property they have an interest in. Also they may want consultancy fees or a direct salary.

3) Stand alone business "angels" who have limited experience, and maybe not even a portfolio, will probably want more of the action than they should: they will want to be directly on the board themselves or even actually be an operational manager or internal consultant. This must be weighed up with 2. above as to their intentions, and also their own skill base and personal network : is their market and resource access really good enough?

4) Other personal biases: Angels will have personal biases which may fall on the down side of all the above potential conflicts: furthermore they may have a bias to a market player, partner, university or IPR source which is actually not the ideal route for the company. The route may be completely counter productive because the initial positive personal contact mediated by the "angel" dries up to being a deal which could be of higher value with a competitor to them.

Show Me The Money......

These days in either software, hardware, consultancy or business services a highly profitable company with high growth potential can be hitting its KPIs and milestones in Y1 and Y2 on first round Y1 funding of half to a million dollars/ Euros and one to two millions in the Y2.

Angels will often want to pick up on the next round too, because they see the potential overlooked by VC or the VCs are not "in" that industry / sector of industry at that time. VC invests in industries and high growth sectors by in large, and not individual companies per se. They will have expertise in a segment or buy that in when the KPIs look more rewarding than others. So VC can be high value-niche product shy, because they have bet on other growing sectors as attractive and are hunting for companies who are in there.

Alternatively one fortuity of portfolio Angels is that VCs trust them, and when they are looking to invest extra monies or sectors they have looked into go sour, then they may back a wild-card horse from the trusted portfolio. Some VCs in Silicon Valley probably just take companies "screened" by their favourite portfolio angels.

Business angels without portfolios, say with just a few companies, are going to be more hands on, while those with portfolios are going to be more dogmatic and prescriptive: the latter will want to see the management team and board being of a proven calibre, and this has its expenses in terms of salaries and arrogance of managers coming in to ride and steer the founding entrepreneur's ideas.

Backdoor references: are often looked for, unsolicited networking to assess the qualities of personnel. So there are ongoing issues for the entrepreneur and core "engineering" team in accepting new personnel and being assessed for their own "personal brand" off the record.


Business Angels and Venture Screening : Why the "The Elevator Pitch" ??

The two groups of investors behave on the ground floor of things, quite similarily: they both screen potential companies to invest in based on their "elevator pitch" as the first major screen, and even then there has probably been a "triage" whereby the company pitching is
recommended or screened by an associate.

So the elevator pitch has to explain the product or service, the USP and the size and growth of the market in a very concise and attractive way.

VC will also be doing a further job in finding what industries to invest in: where there is growth and margin, where there is need for supply. Whereas the Angel is probably entrenched in the industry or just has a more open mind to high risk backing a good concept.

Practically a portfolio manager or angel will see between 5 pitches a week and 5 a day. Any pitches that get through the "elevator" exec summary stage then get screened by looking at those unsolicted references, the CVs of the founders, the supply chain, and an idea of what people pay for products and services already in that area. The knock out rate here is therefore high.

This means though that even with a good pitch, you are actually only as good as 1) the attractiveness of your industry to the VC or Angel 2) how good the other pitches were that day, week, month or quarter.

So the elevator pitch is just a function of human buying behaviour: we browse, we feel branding, we pick the best from the shelf to match our thinking on that day. We don't spend time reading the best Harvard MBA business plans, risk assessmetns and financial scenario algorythmic results : we take short cuts based on highly summarised information, a concise presentation of the idea, and the quality of the concept itself.

Attrition and Success Rates for Angel Supported Companies

50 to one ? A third go bust, a third return their original investment plus interest maybe, while the top third make the large multiples in ROI: In terms of the 100% (2x) , 500% or 1000% Y2 ROI in their valuation when they are sold further or floated by FPO. The large ROI on the one third stars must of course pay for the fail-third and the "stake back on your bet" third.
An independent angel without a portfolio may conversely, have such a good grasp of the concept and be able to offer so many open doors and resources that they choose to invest and the marraige is successful. Not only there by, but because the Angel is prepared to put time and shoe leather into supporting the business.

Nimble and Lean Burn is Important to Angels and VCs

Being "nimble" means being able to move quickly to take advantage of changing conditions in your supply chain: It means most of all, being open minded to morphing the business to a new direction. This can include:

  • Picking up on new potential revenue streams quickly and converting them
  • Redefine who and where the customer is in the supply chain and in the world
  • Being able prototype rapidly and test market: Slide dot com tests 4 to 6 ( NPI ) products A DAY!
  • Open mindedness of moving the shell of the business into a completely new direction based on fortuity, or rationalisation of the current route being unprofitable or low in ROI.
  • Dropping poor routes or products and carrying on with the good ones or looking for the direction presented above, to use the shell to do something new they have encountered potential within.
  • Capability to "breath", in expanding by using right-sizing and outsourcing such that projects can be delivered without committment to overhead. This could be even short term, in a matter of weeks in terms of hiring and firing costs.
  • Low Base Cash Burn: being able to return to a low base cash burn rate while still holding and building value in the proposition.
To give an example of the latter two : a company may require to prototype and implement an ISO system for a potential customer. This takes more resources and tooling thant the company has, so they can of course hire, but it would be better to outsource based on sound CDAs. After the prototyping, there may be a protracted evaluation period or changes to the specifications which slow time to payment up. These can be done at a lower head count and the company returns to a base cash burn while they wait the milestone payment or revenue.

Alternatively a company may reach a bottle neck: There may be no properly experienced Java programmers in "the valley" at that time to implement the product. Therefore it may be worth putting the company on ice while the labour market loosens.

Low burn rate in itself enables nimbleness, because companies can survive when trying out new directions. So trial and error can be a positive experience which the company comes out stronger from.

Nimbleness appears the exact opposite of strategy: it is more tactical and reationary than earlier concepts of "core business" and "mission statements". Indeed it seems that strategy is now both a wider direction, while also a focus on the abilities and will to win of the team. Overall you are likely to have a strategy in a market, and that may begin in one niche or direction. You therefore have a strategy to build the initial team of 5 to 8 people and take them in the market and client groups they aim their bow towards.

Right Sizing and Out Sourcing : Risks and Rewards for being Nimble
The problem with outsourcing or using contract labour is that your business concept and IPR are revealed to the personnel who have no definable loyalty to the company or knowledge of IPR laws even. In software for example, they may be able to implement a very similar solution in C++ you have done in Java, thus circumventing copyright and US software patents at least. However some suppliers specialise in secretive projects for start ups or companies needing "breathing " capacity.

Looking for suppliers in this way, addressing top management under CDA is also a very good means of approaching merger and acquistion potential partnerships. It may be a jobbing engineering workshop who do a lot of one-offs and prototypes would want to move into steadier high value production. Or a Java hot shop may be looking to partner to the next "facebook" entrepreneur.

Milestones Should Be Market and Quality Oriented

Businesses should indeed set NPI or earlier prototyping and proof of concept milestones: sometimes with no contact with the market which could expose the IPR too early. However soon the company should be setting alternative milestones and KPIs: breadth of client and partner contact, and depth of business development penetration.

  • Number of presentations at CEO Level,
  • number of call backs.
  • Ranking contacts and meeting outcomes for their quality, and the match between the customer.
  • Number of escalations
  • Conversion Rate in the Funnel

For a consumer oriented business: it should be rapid prototyping and feedback from significant numbers of potential users on as wide a geographic basis as practical.

The level of benefit over other competing products and services must in fact be assessed by customers and not internal staff. This is a key ground for that high tech companies fail: they are myopic to their own solution as being unique and better, while missing the concept of value-increment and adoption-risk pay off for customers.

So a key milestone must be that the NPI does deliver x% better, and that this is more than a just-noticeable-difference. It must be a real threshold which will take early adopters and not just innovator "geeks" at the front edge of introduction and word-of-mouth marketing.

What is Better? What do consumers like in NPI?

In fact the big brands on the internet do really very little that was not done by the mid to late nineties: Twitter, you can trace back to newsgroups and even before, perhaps even phone phreaks and early pre TCPIP notes.

The Facebook concept has had various guises since "personal home pages" became first popular in Universities in the mid ninetees: friends reunited amongst others. DIgg and My Space too, could all essentially be found in the 1990s, and "tweet / Blogg" Decks relate back to the 1990s "Jump Stations" .

Alta Vista and FAST had leading, performant search engines which rivaled Yahoo and Google in the mid ninetees. Why did they not win a normal market share ?

Why are the big brands then, all massive successes?

On the "Product" side, they provide a more cohesive user experience which is in fact, simple and often this means faster to both use and grasp the facets of the service. Also you could say the products were in the right place right time for the mobile device explosion into media rich web sites.

Perhaps many people had gained experiences with the earlier products and so were warmed up to adopt a tighter and neater execution, funded soley by discreet advertising. The last point cannot be stressed enough: intrusive advertising and pay-for-contact social web sites were doomed to reach their limits in utility for building a social network and communicating to them on line.

Certainly there is a lot to be said for the interplay of early adopters, good-simple-branding, a performant NPI concept, and a level of familiarity and readiness for the wider market to adopt these new social media platforms.

Branding is important: we trust FB not to itself, cause issues for us, to change radically, to stop being reliable, or pack up shop.... or just sell out to the latest Javascript pop-over banner ads. People trust twitter to be fast and accurate, and not glue them into spam but rather the extended social network and information constellation out there. We recognise FB's graphics, we are comfortable with the core offering, the layout is consistent, we accept "share to FB" from just about anywhere and we trust it to work and be a resource for us in future.

What Chance the Start Up?

One thing all the big internet brands have in common, is that they all have got funding and personnel in Silicon valley. At that time it was the only place where a critical mass could be achieved. Like Microsoft and IBM in Seattle before, and the big brands and Madison Avenue in the hey days of TV and Press advertising.

So place to seed in is going to be important, and for any given industry a labour supply is also key

People and ideas have become more important than numbers and projections. Perhaps you need the "excuse" of starting up to do X, Y and maybe not Z right now in a fast growing market.
But wait up here, before facebook came, off search engine advertising was a scarey business model? How did FB swing the balance?

Well the cost of gaining a critical mass to enable the huge social-word-of-mouth marketing snowball to roll, was actually relatively inexpensive and once it attained exponential growth, the advertisers wanted on, as an alternative to Google.

So for the start up there is hope but you have to get on the train perhaps at the right station.

Tuesday, October 26, 2010

Facing the FaceBook

I look forward to seeing the film, "the social network" and I suppose it will be something to get clicky on Facebook about.

Not that I am a social network addict: okay I am a forum and blog addict which puts me in a minor league, with most likley just crawlers reading most of my hits. FB I can kind of take or leave and right now I am going through FB fatigue, although that is not as from any huge overdosing on it-. just boredom. I actually delete people from FB and keep my friends numbers to under 50. There are still a couple of people I would like to connect to, God excluded, and some I should maybe connect to, but I am pretty determined to keep to under 50, at least as my private profile goes. Maybe DF will develope his own profile, alter ego as he is.


(links to founders) The founders were, or have become, somewhat shy types who started the whole concept as a beauty rating project: very peri pubescent of them too. However, they soon saw the social value of a simple "face" on the internet with connections round Harvard and later of course, were lucky to ride the storm and catch the money wave.

As I wrote in my last ranting blogg, I am more interested in the beast than we fleas upon its' back. Or rather how we fleas feeding on the blood of the social network, organised to join up.

The beast itself holds some fascination for me, in its quintessential simplicity. It looks like any live content portal I would have worked on in y2000-2001: Conservative, text and thumbnail based. Simple, clear, familiar, and stable. The simplicity lies in "getting it" quickly,what it does for your wild social betworking imagination proposes to you, more on that soon. But the latter two are pretty important: it is familiar, like the php/cgi/cfm pages I'm thinking about. It has levels of permissions, like the extranets I designed a decade ago. You have to join to get in, and you have to be a good little boy once in.

So it is a familliar and therefore safe environment, and unlike many other predecessors, has no half-hidden agenda of fleecing you to make further progress in your sociableness. The stability is vital to this feeling of security. FB evolves very slowly and is currently no doubt have an internal security review after the spate of FB virus, spy-ware and spam "apps" which have plagued us like chain letters did in the 1970s.

Well, well. FB fatigue: I am a little bored, perhaps I got my personal branding wrong and don't have a little band of the cleveries with good reparte. I just find it a bit static, and fully expect a merger and take over YT-FB! Video with video replies, limited tweet wise to 4 Mb or the like.


So there it is, a totally unstructred, short rant for the evening. But what will become of FB, what is the next move? WIll there arise a new beast from the east or something else capture our imagination? What FB could do, in the mind of DF, will be the topic of a forthcoming rant.

Wednesday, March 17, 2010

Why Monitor Social Media ?

Perhaps you are new to the world of social media monitoring at work, or are embarking on your studies at university in this area, and you are wondering just exactly "what is in it for me ?"

Maybe your background is in traditional marketing or perhaps you work in a technical or customer services function and have heard that this new area is indeed something worth looking into further . How can monitoring social media help you make decisions and deal with problems and opportunities?

Well it is of course e-Marketing and web masters are the most interested and are sitting glued to the analytics and consumer opinions. Of course it is not just for the geeks: communications and customer services who are aslo getting engaged with SM. The benefits reach wider and deeper into the organisation though. Everything from new-product-introduction & tracking, or competitor pricing, to issue containment is amenable in almost real time : the feedback loop from action-consumer reaction is drastically shorter.

Statistically Speaking

One burning question is how can information in SM be used to make conclusions about the wider consumer population? Well speaking mathematically, usually we cannot make inferences to the general population or produce actual statistics - yet.

The day may come in the near future though, when the numbers of consumers engaging in discussion will be so high that we can draw inferences to the wider population, such as intent-to-purchase or brand awareness, and put some hard numbers behind this with SM a quantitative source for extrapolation to consumer behaviour in the market as a whole.

Even then it will most likely be from a definable cross section of different geographical- or network-societies: age and education related. It would be dangerous to draw inferences on anything but the first three standard deviations from one of those sub populations who are engaged with the internet and SM. However we will be able to utilise statistical probability based sample methods within any accessable or stored data set to produce smaller data sets which are make analysis more efficient within given parameters of accuracy and inferential significance.

Social Media Monitor Should Be Qualitative

For the moment though, reporting is focuses rather on the valuable qualitative insights to be found, "straight from the horses mouth". These are the root causes of issues, the actual verbatim opinions, the dissatisfactions, the real point-of-touch customer experiences : I could go on! These help illustrate findings from a company's quantitative reports and data-sources, as well as pointing to new insights which uncover consumer opinion hidden or distorted by the very interactive nature of surveys, depth interviews or focus groups. Also they can uncover uncomfortable truths which are hidden by line managers, front line operators or re-sellers.

Everything is Relative

Despite the qualitative output of reporting, descriptive statistics can be used within the domain of social media to illustrate the relative prominence these qualitative observations This includes the relative prevalence (or you could say "share-of-voice" ) of brand names, consumer opinions and for instance problems with newly launched products.


In combination with ever-more-accurate sentiment algortyms running with AI (artificial intelligent) systems, this area of descriptive statistics will be used more and more to give a picture of the cross-section of society using SM to discuss your brands and customer support. The value will increase if it can be shown that the "listen-learn-decide-react" loop on the web connecting to SM, is functioning.

Then our little world of SM becomes a market in it's own right, and this is already happening with companies engaging in different campaigns and communications which are built around information from social-media-reportage. Some companies in future may only interact with consumers through this interface and connections form SM to their web-services.


Numbers and graphs are all very fine and nice to present and talk about, even with the provisio that this is a little and twisted version of the world at large. But even a very low number of "hits" within the latter, for example, can reveal invaluable insight into potential challenges in production lines or further back in the supply chain which are creating problems not detected earlier in the testing and launch programme.


Tracking New Product Introduction


This qualitative approach has been of particular value in tracking new devices launched on the market, which have a plethora of features and most likely diverse internal software. However, this is equally valuable in tracking a new service, or immaterial product from a financial institution or a mobile network operator. Or in defining an unmet need or latent demand out there in the market place.


Within the world of gadgets- consumer electronics like mobile phones, PDAs, laptops or digital cameras - it is in fact often the lead consumers who are the real experts: they can be tracked individually, from maybe a sample of 10, as they try and buy diverse gadgets and report their experiences on the web. Often they seem very informed on how the technical features, like processor, touch screen, GUI, actually deliver benefits in use and how much better this performance is to earlier products or competitors offerings.

In fact these lead consumers seem to have a more wholistic view of the product's perfromance than the head of R&D and most likely the CEO at the manufacturer! Worth listening to SM?