Hi again, it’s been way too long since I last updated this blog, and it’s for a good reason. As a few of you know, I have been heads down developing a system to data mine keyword and social media data. Yes, I know the subject name sucks – any suggestions? What seemed like an easy thing to do had turned into a total beast and development feat. I am here to say that we have accomplished most of the Phase 1 specifications and are now ready to bring it to market as a commercial product. We will add features in 3 phases, which I will talk about in more detail later.
The application does the following:
Aggregates all of your keyword related data into a single repository stored on the cloud. Then allows you to quickly identify critical issues and missed opportunities.
Why did I do this? Throughout my years in search marketing, one of the biggest challenges has been managing keyword data. At IBM, I built a keyword de-duping application just to find and manage duplicates between 30 business units doing paid searches.
It was only when we had this tool that we discovered that 21 business units were all bidding on the exact keyword phrases. Last year, while working with a large multinational in Europe, I requested their keywords, and it took eight days. I received 21 different Excel worksheets from the Search manager that represented their words. In another case, a multinational company with thousands of products had selected only 25 keywords for any optimization efforts, meaning tens of thousands of products were going without any effort. All of these led to my desire to create a tool that would eliminate problems.
Over the past few months, I have conducted a few surveys of companies and from this research, I have identified a number of keyword management challenges.
Keyword Management Challenges:
- Companies struggle to manage keywords in Excel (99.8% of companies surveyed)
- No lens into collaboration of Paid and Organic data (99.87% of companies surveyed)
- No way to see where keywords are under-performing against KPIs \’s (98.75% of companies surveyed)
- No way to monitor if the correct page is ranking (99.25% of companies surveyed)
- No way to understand keyword performance by category or buy cycle (99.98% of companies surveyed)
- No way to understand performance based on the Searcher’s Intent (99.98% of companies surveyed)
- No way to leverage searcher interest data to prioritize content in the organization (100% of companies surveyed)
If you want to add to the survey, simply go and add your responses to the Keyword Management Current State Survey.
What can we do today?
1. Aggregate all of your keyword data into a single searchable repository with role-based login access by different roles in the company.
2. Conduct paid and organic co-optimization analysis – are they cannibalizing or complimenting each other.
3. Preferred Landing Page Analysis – is the page you “want to rank” the one that is ranking?
4. Rank Analysis – same as everyone else but we allow you to sort by priority words, line of business and any keyword cluster or classification
5. HitWise Integration – if you have a HitWise account, we can pull in the API feed and compare HitWise trends to your actual data
6. Data mine and Report on any of 55 different keyword variables
7. Develop Searcher Intent and Persona Segmentation – using any of 50 performance or segmentation factors cluster keywords into logical segments
8. Store, sort and report on data across business units, lines of business or countries
9. Understand performance by keyword length, position and paid and organic assists
10. ROI Modeling based on multiple variables
What has been the outcome?
In my first generation, I used Microsoft Access and Excel Pivot Tables to test the theory, then moved into MVP “Minimum Viable Product” mode to quickly develop functions that allowed us to scale data, resulting in the following success stories:
Success Story 1: UK travel site matched keywords to top ranking pages and found less than optimal pages ranking. Fixing just five pages resulted in $ 60,000 in incremental revenue within 25 days.
Success Story 2: PC Maker found significant searches but no traffic for “End of Life†products that they had no web page representation for – They added new content and PPC campaigns, generating $400k
Success Story 3: Fortune 50 company realigned keyword ownership and budgets based on segmentation analysis for maximum opportunity resulting in a decrease in PPC spend of 12% but a 300% increase in sales
Success Story 4: Travel site identified 50 keywords in top 5 positions with less than 5% share of clicks – optimized snippets increasing click rates from 5% to 15% resulting in 85% increase in revenue
Where are we at today?
This is not a mainstream consumer site product. It is designed for a site with a large base of keywords, typically more than 500,000 of them that want to get more out of the product. We are looking to develop a mainstream version of the application, but it seems those customers want something cheap that does a lot of automated analysis and does not require them to think.
We are not quite there yet, and when we are, we will roll that version out.
What is is not?
While the tool does a lot, it does not or will not do any of the following:
1. A bid management tool – there are plenty of them that will work great for you
2. A SEO Automation tool – there are plenty of them that will work great for you
3. A Search Analytic tool – sort of but does not replace Omniture, Google or Web Trends
4. Keyword Research tool – sort of since we can mine data but we typically are not looking for new words. Great tools like Keyword Discovery, Wordstream exist for this function.
The challenges:
Where do I start with this one? This project has taken every ounce of patience and sanity I could muster not to abandon it along the way. Fortunately, I have been working with a great team at OC4 on managing this in the cloud. We have nailed most of the big issues and are just fine-tuning.
Data Integration – The biggest challenge was integrating and managing the data. Readers, there is some messed up data out there. I found that there are a lot of agencies and people who should be fired for incompetence, if not fraud. This was the biggest challenge of integrating the data. There are large volumes of it and we needed to suck them in and align them.
Data Clean-Up – People use some unusual words to find your products, and there is a lot of bot activity. We has to write a data import and cleaner tool just to parse log traffic data. Omniture does a decent job of sorting these into “Small Elements,” but other tools do not. We had to develop a routine to process 286 different types of data contamination before we could import CSV files into the system. We had whole paragraphs coming in, multiple commas, and ton of scrapping strings looking for pricing or other elements that all had to be cleaned out. Not to mention beginning and trailing white space and other issues just to normalize the data. In one case, there were 854 different misspellings of the company name, which we leave in the tool but don’t necessarily want to gather other data for.
Product Naming – Another significant challenge is finding a suitable name. My original concept was to call it “VOCDMS” – Voice of the Consumer Data Management System – but clearly, that does not roll off the tongue, so we are working on a more suitable name. We have decided to keep it aligned with Back Azimuth, as that remains the foundation of what we believe – helping you get back to your consumer.
Development Teams – I am now on the 3rd iteration of a development team. The current team from Exadel are fantastic. They have fixed many of the bugs and problems from the previous India teams. I have tried local developers but they were way to expensive, too distracted or wanted a large share of the company while only offering mediocre skills.
Market Interest – This one has been strange. When I’ve told people about the tool, they are skeptical, assuming there’s already one like it. Then, they realize or ask their teams to find that they’re like the rest of the companies using Excel or maybe an in-house database to manage it.
What’s next?
We currently have a few clients and pilots in place, and we will begin marketing the product to a broader audience by the end of the year. If you have a wish list of items, send them my way.
