(originally posted here)
To make all this process as simple as possible, a variation of the third approach (Rack middleware + Selenium Webdriver + no caching) is available here as a Gem. Drop it in your project, have the dependencies installed, and may the SEO gods bless you!
Much has been said about how hard it is to build a single-page app that responds well to crawlers - most crawlers don’t support javascript, which means all they get are blank screens, when crawling a web-app entirely assembled and rendered on the browser. Luckily, there are several approaches one can take to circumvent the lack of faith from certain crawlers - there’s obviously no “one size fits all” approach, so let’s take a minute to go through three of the most commonly used approaches, highlighting pros and cons of each one of them.
This strategy consists of rendering your app normally, BUT with pieces of “static” content already baked in (usually inside a <noscript><div> block). In other words, the client-side templates your app servers to be rendered will have at least some level of server-side logic on them - hopefully, very little.
Although it’s a somewhat natural approach for a developer used to rendering templates and content on the server side, it leads to a scenario where everything has to be implemented twice - once in the javascript templates/views, once in the pages, making everything hard to maintain (and potentially out of sync) real quick. That Product detail view now includes a “featured review”? No problem, have its text rendered on the server-side (and returned by the JSON endpoint your client app will use to render it again in a Javascript template).
This DOES work well for mostly-static, single-content apps (eg.: blogs), where even the javascript app itself would benefit from having the content already pre-loaded (the javascript views/snippets/segments would effectively fiddle with blocks of content, instead of fetching them from the server).
It’s worth noting that you should NOT rely on rendering just bits of content when serving pages to bots, as some of them state that they expect the full content of the page. It’s also worth pointing that Google also the snapshots it takes when crawling to compose the tiny thumbnails you see on search results, so you want these to be as close to the real thing as possible - which just compounds on the maintenance issues of this approach.
This technique is supported by Google bot alone (with limited support by some other minor search bots - Facebook’s bot works too, for instance) and is explained in detail here.
In short, the process happens as follows:
The search bot detects that there are hash parameters in your URL (eg.: www.example.com/something#!foo=bar)
The search bot then makes a SECOND request to your server, passing a special parameter (_escaped_fragment_) back. Eg.: www.example.com/something?_escaped_fragment_=foo=bar - it’s now up to your server-side implementation to return a static HTML representation of the page.
Notice that for pages that don’t have a hash-bang on their URL (eg.: your site root), this also requires that you add a meta tag to your pages, allowing the bot to know that those pages are crawlable.
<meta name=”fragment” content=”!”>
Notice the meta tag above is mandatory if your URLs don’t use hash fragments (which is becoming the norm these days, due to the amazing adoption of html5 across browsers) - analogously, this is probably the only technique of these three that will work if you depend on hash-bang urls on your site (please don’t!).
You still have to figure out a way to handle the escaped_fragment requests and render the content these are supposed to return (like the previous approach), but at least it takes that content away from the body of the pages served to regular users (reducing, but not eliminating, the duplication issue). This works somewhat well on sites which part of the content is dynamic - not so much on single-page apps. No universal search bot support is also an obvious downside. Plus, you still have to pick a strategy to render the content without Javascript when the _escaped_fragment_ request arrives. Which leads us directly to the third approach…
Although this seems counterintuitive at first, this is one of the approaches Google suggests when you’re dealing with sites whose majority of the content is generated via javascript (all single-page apps fall on this category).
The idea is simple: if the requester of your content is a search bot, spawn a SECOND request to your own server, render the page using a headless browser (thankfully, there are many, many options to choose from in Ruby) and return that “pre-compiled” content back to the search bot. Boom.
The biggest advantage of this approach is that you don’t have to duplicate anything: with a single intercepting script, you can render any different page and retrieve them as needed. Another positive point of this approach is that the content search bots will see is exactly what a final user would.
You can implement this rendering in several ways:
before_filter on your controllers checks the user-agent making the request, then fetches the desired content and return it. PROS: all vanilla-Rails approach. CONS: you’re hitting the entire rails stack TWICE.As for how to store the server-side rendered content (again, from worst to best):
There are a few obvious issues you have to keep in mind when using this approach:
EVERY iOS developer should give this a try - if nothing else, testing a Rubymotion app is AMAZING!
It has been exactly a year since we launched RubyMotion. Yep, this is right, RubyMotion is one year old!
We released over the year a total of 35 software updates fixing countless bugs reported by our beloved users. That’s almost 3 updates per month on average!
We shipped significant…
[Details about the company and sender obviously omitted to preserve their privacy. You can skip straight to the benefits section, which is downright priceless. Gotta admire the guy for trying, though (I guess).
—
Hi Herval,
I found your information on XXXXXX. I apologize for the long email or if I am mistaken about your potential interest.
About Us: We are a young start up called XXXX. We are a XXXXX for XXXXX who are XXXXXX and we solve that by XXXXXX for XXXXXX.
What We Need: Though we have a site and system setup, it needs to be upgraded and redone completely. It is essentially a Frankenstein site with much room for error and inefficiency. It is not intuitive and not built to scale. We’re looking for someone to redo the whole site (design+dev) from scratch with a focus on intuitive/appealing design.
What we need: yes, there is actually a very detailed, 20 pages spec attached to the email. And the product exists!
The Fine Print: We are still very much bootstrapping the business and have only approximately $3,500 to spend on this project. I am well aware that this is NOT a lot of money for high quality coding and i’m not one of those douches that doesn’t value the effort/talent of a good programmer; we’re just simply not in the financial position to pay you what you deserve.. So why would you even think of taking us on??
If you read this far, thanks! If you’re at all interested or have any questions, please let me know. Would really love to work with you.
Hope to hear from you soon, X](null)
My third company came almost five years after the second one (not counting the “pet projects” that happened in this period). After coming back to Brazil from a less than pleasant experience working in a large, soul-sucking company in Europe, I felt it was time to try something on my own again - an idea my wife promptly embraced (and joined as a founding partner).
Being in love with “smart hardware” for quite some time (I used to build half-baked “robots” and electronic boards back when ISA was the top-notch standard for computer boards), I decided to give a shot at a hardware-centric idea this time. We spent a couple of months researching markets - everything from smart houses (too small and fragmented market) to public camera systems with face identification (too tricky to sell) and even smart watches (to the likes of the Pebble - too expensive to build and we had zero expertise on the whole supply chain). We ended up settling on the “interactive environments” niche, which is supposedly cheaper to sell and usually boils down to software running on hardware you buy on local shops.
Interactive environments are basically an attempt to augment the “real world” with digital, either by bringing online presence to physical environments or by using sensors to detect people and serve them with interesting digital information: think interactive windows, billboards you can send tweets and the like. Not knowing the market from inside out and knowing exactly which tech could we focus on, we took a few weeks building quick proofs-of-concept, talking to customers in different segments and all in all getting to know the marketplace in order to take a well-positioned first step.
And boy, did we pivot. After attempting all sorts of different “MVPs” - bluetooth marketing (contextually sending images and ringtones), touch-screen kiosks, projecting twitter-feeds in walls for events, little GSM boxes one could use to make “interactive SMS polls” and using Kinects to detect purchasing intentions inside supermarkets - we eventually settled down on the more basic “digital signage” package, licensing our platform for media companies that would like to roll out their own indoor marketing networks.
Digital signage (we’re talking a niche inside a niche, now!) is not a new thing: it compromises all forms of digital communication using billboards, indoor TVs or projectors, which can be used to convey information, entertain people while they wait in line, etc. Think the flight information screens you see at airports, or TVs on bars and supermarkets. The big pain points we were determined to fix here were the deploy complexity of these systems (updating content requires physical access to each machine) or the absurd lack of features they provide (most digital signage “solutions” out there are just recorded video or powerpoint decks playing non-stop). And thus was born BlooBox: our 100% online, one-click install, remote manageable digital signage solution that supported not just displaying text, photos and videos on TV screens, but also polling for live tweets on any topic or hashtag, splitting screen with several different kinds of content (want to put a ticker over a video stream with an announcement on the side? No problem!).
One thing we quickly learned the worst possible way was that building rich desktop applications is surprisingly hard. There are just no options out there to build the kind of thing we had in mind on (a client app that could self-update, display dynamic content, run on any platform and allow the users to build their own content with tools they know how to use, like Flash or HTML) - the only two contestants being Adobe Air and Microsoft Silverlight, both “mature” by market standards, but frustratingly restrictive in every way possible. There are memory leaks everywhere (making the app crash every X hours). You can’t do two things at the same time (Adobe Flash is single-threaded, meaning your entire UI freezes when you try to download a piece of content for later displaying, for instance). There’s no documentation whatsoever on things such as loading modules/content, or automatic updating. The list of problems and pain points kept creeping up by the second, and for every bug we squished, another two were born. We even got in touch with the Adobe Air team itself, helping them debug a couple of memory leaks on foundation components of the platform (which led us to use even flimsier beta versions for long periods of time).
Despite all that, with a bit of glue and a lot of spit, we were able to put together a functional product in a few months.
Along with lack of adoption, pricing is the number one killer of companies/projects. Not knowing exactly how to make customers pay, we toyed with all sorts of sales models, from pay-per-use to pay-per-plugin, as well as the overly popular “ad supported” model that works so well online (but was a complete disaster in our offline context - more on that later). The big problem, in our case, was due to the horizontal nature of the market: the customers included everyone from bars (which will pay nothing for a product) to franchise networks (great clients, but with a very long sales process) and single-venue businesses, such as hospitals or clinics. Given our location (yes, here we were again, trying to start a product in the worst possible market. At least it was supposed to be “just during the beta period” this time…), we had only a handful of possible clients on each of those verticals, so specializing was not an option.
Driven mostly by what others were doing, we ended up settling on the “pay per TV” model, with a low monthly value (~48 USD/each) and free installation. The customer would have to buy his own TV and computer, though, which was costly enough for most small ones, and the support cost was an obvious red herring: the more hardware you have distributed in a wide area, the more people you need to keep everything working. You’d be surprised how often customers call in just so that you find that the system “is not working again!!!” because they’ve unplugged it from the power plug or simply forgot to turn on the TV…
Less than a year in from the day one, we were finally breaking even. But the outlook was not good: we had to make more money per customer, if we wanted to be able to keep growing at all. Our final big mistake was, ironically, trying to “eat our own dog food”: determined to not just offer the software for third-parties, we decided to go the extra mile and actually manage our own indoor media network.
Looking back at it, this was possibly the dumbest decision I did in my entire life. The final change on sales model we did on the company would require a big swing in the way we presented the product to everyone: no longer would bar owners be asked to buy TV sets or pay monthly fees (we’d give them all that for free). No longer we would spend time energy and countless trips trying to convince every single layer of management on a franchise network that they should get our product and roll it continent-wide. Our sole sales would turn on trying to convince mid-to-large sized companies to advertise on our TV screens, positioned on half a dozen public spots. If we really wanted anyone to look at our network and see some potential for advertising, we’d have to quickly expand too - and so we did, growing from 6 paying customers to almost 20 free ones in less than 2 weeks (which positioned us as the second biggest ad network in the region).
That single decision that costed a fortune on hardware acquisition, incurred an even larger overhead on support and deviated us too far from what should have been our sole focus: providing software as a service, get paid for it, rinse and repeat. Selling advertising space proved itself a tedious, impossible task and undermined the morale of the team to a point that most of us simply started hating to get out of bed for yet another day of trying to stay afloat.
It’s worth noting that we didn’t just fall on this trap out of the blue: the entire ad-supported digital signage idea was a trend worldwide, with big-name networks being acquired all over. It was obviously “the way to go”. Today, a lot of those companies, even large ones, have also died out, victim of the same uneven expectation about a market trend that didn’t materialize. If there was any money to be made on the ad-supported networks, it was made by very few, deep-pocket giants. The money we spent on this “wild goose hunt” was sorely missed, in the end, and could have kept us afloat for another year.
It still saddens me to see that no one has ever done the things we pioneered in those little MVPs. The whole “interactive environment” idea is a very exciting one - if only the associated cost wasn’t so big, the tools to build it so archaic. Maybe in a few years - until then, here we are, stuck with clunky kiosks, useless interactive floor-projections and TVs displaying ads that no one pays attention to…
In the end, the company wasn’t a total failure: we managed to sell it, a mere couple of weeks before running out of cash, and decided to move on to something less hardware related, more viral. Time for some buzzwords!
From the useful to the borderline insane, a huge bunch of tips and tricks collected by James Edward Gray II (epic name!). Worth every second of your time.
I’m specially awed by all the amazing collection methods I never used (zip, chunk, partition, take, drop, cycle… the list goes on and on), the various tricks to access Ruby’s context, interacting with processes and pretty much every dark corner of the Ruby language I thought I knew all about.
Go read it. NOW!
Yet another dumb mistake, but apparently one very few people have ever faced. While writing a simple Resque job, I started getting a weird “Jobs must be placed onto a queue” error every time I tried enqueing it.
The catch is obviously simple: if you enqueue a job using a string instead of a class instance, it can’t find the class, and obviously not a queue associated to it:
Resque.enqueue(“MyJob”) # this fails!
Resque.enqueue(MyJob) # this works
Obvious at first, but quite hard to spot at first (the class name in question was getting passed as a command line argument).
Oh, the little joys of dynamic typing :-)
A bit of a hack that I happened to stumble upon lately: if you use Capistrano to deploy your stuff and, for whatever reason, need to issue a “nohup” command (i.e. start a process on the server that should keep running once the SSH session is over), you’ll probably start trying it like this:
run(“cd #{deploy_to}/current && /usr/bin/env nohup rake my_beautiful_job > log/my_job.log &”)
Although this would work perfectly on a bash session, it will hang Capistrano forever. The reason here is that Capistrano will wait for the exit of the command to return something (a ‘stop’ signal), which will never happen since the nohup command cuts out the IO for the command.
It’s possible to make Capistrano move on by setting the :pty parameter:
run(“cd #{deploy_to}/current && /usr/bin/env nohup rake my_beautiful_job > log/my_job.log &”, :pty => true)
That will make Capistrano move on after the nohup return - nice! But still, there is a little problem: since the command will be ran in a pseudo-tty, the ‘pseudo session’ that Capistrano opens will close before the rake task is put in background.
The final solution (and the one that worked for me) is shoving a sleep time at the end:
run(“(cd #{deploy_to}/current && /usr/bin/env nohup rake my_beautiful_job > log/my_job.log &) && sleep 1”, :pty => true)
The extra ‘sleep 1’ at the end will give just enough time for the command to be forked. And life deployment goes on!
My second attempt was born still inside the walls of the first company - in 2003. While playing with the “cutting edge” devices of the time (wow, phones with COLOR screens!) and the rudimentary JavaME 1.0 (J2ME at the time), we came up with an idea: what if we invested in mobile… games? Don’t get me wrong: B2B is certainly where the money is, and all… but I mean, just look at those GORGEOUS little screens. Who couldn’t get excited about the prospect of playing on these sexy little devices?

For that 180 degree turn in focus, I brought in two other friends, who worked with me in parallel with the “parent” company (and for free). The simple prospects of having our games running on BILLIONS of devices all over the world kept us going with a never-ending enthusiasm (we all used to make games in the past - I myself learned to program by writing little games on a Z80, as a kid). It was more than enough to make everyone work double shift (besides their regular “job” or company).
The first (and huge) barrier we found was the mobile market itself: back then, it was a nightmare of epic proportions: each telco had their own WAP portal, billing system and pricing scheme. They kept up to 70% of the revenue, meaning our margin would be lower than 30% (after taxes). And the prices HAD to be low, otherwise “people wouldn’t pay”. The second challenge, was, of course, actually BUILDING the games: the tools we had back then were so bad (and the top-of-line devices so puny) most games couldn’t even play sound for the lack of device support. Using the mobile network for anything except downloading the game itself was also a big no-no, given the absurd data rates (and the astonishing speed of the spotty 2g networks).
Despite all this, we pushed on. Managed to close contracts with ALL the telcos in the country (then some more in the rest of south america and even the USA), built some very good looking games, assembled a truly amazing little team, got in the news (and then again, and then again). We even got an investment offer of over $1M for a very favorable stake in the company, after presenting in a VC conference where we were the only ones not wearing a tie (it was also my first experience with VCs, which included going to absurdly expensive restaurants in a BMW and getting mocked because I was asking for too little money).
Despite all this, something didn’t feel right. Despite having around 10 games distributed on the portals of a dozen different Telcos, the sales where simply not happening. It was all somewhat expected: we knew the best phones were yet to arrive (we made friends inside the major handset manufacturers who provided us with prototypes), that the data coverage would improve in a matter of months (we made friends on at least two major telco networks). What we could not predict was how long it would take for this market to bloom - for the sales to start tickling.

We tried diversifying: instead of just building our own games (which was costly and took time), we started to also act as publishers for third-party developers from all over the world. Considered shifting focus to Palm games (which, by the time, could have made us a buck - that’s how PopCap bootstrapped their growth, with the first version of Bejeweled). We considered SMS as a platform, but dismissed it as “too simple to matter” (the only guy I know that moved away from mobile games into SMS services got so rich its grand-grandchildren will never have to consider working at all). Then gradually, over the course of 2 years, as some of the original team members departed, the idea of mobile games as a viable business slowly faded. The major mistake was obviously timing: we were at least 4 year too early, in a market that would only blossom with the rise of the iPhone.
During the entire lifetime of the company, we managed to sell a grand total of $100 (yes, a hundred dollars) in copies of our games, even with some of them being on the top 10 of more than one portal for months. We were not the only ones who got burried in that graveyard: of the 50-something companies we published in our catalog, only a handful survived past 2005-2006. The few that survived (many of which we had distribution rights back then) lived on to become the huge household names of today: Rovio, Gameloft, Digital Chocolate, Glu, ngmoco, com2us. And they all made it after the iPhone.
Ironically enough, all the involved in the company back then (me included) were dismissive when the iPhone and Android trends started to rise. It was “just another Symbian, after all”.
The takeaway: it doesn’t matter whether you have the best product, know the right people and have the right contacts: if the timing is wrong, you’re done.
My first “real” company was founded in 2002 along with 3 partners, all colleagues at the company where I did my internship (a couple of months before I graduated). We started with a grand total of $8k, which basically compromissed all my savings back in the day (the other partners contributed with some computers, a couple of chairs and mom’s car). Being four partners with equal distribution was both our best and worst decision: on the good side, we had very clearly defined responsibilities from day one (I took the commercial role, the most experienced of the partners went for the tech & infrastructure and a third took care of daily operations). The bad side of that equalitarian division proved to be one hell of a problem…
The company was focused on developing mobile solutions for enterprise customers, the leading piece being a sales force automation software for Palm devices (bear in mind the Palm III was the top-notch device at the time!). Services such as salesforce.com were still on their infancy, the term m-commerce had just been coined and the market was ripe for disruption. Better still, we had a well functioning product (which I had developed during university time) AND a paying customer. The sky was the limit - except we decided NOT to fly.

It was all poised to happen, if not for two crucial details: being located in one of the poorest states in the country put us in a very though market for growth. That wouldn’t be so much of a problem if we followed the obvious formula popularized by outsourcing firms: develop the product where it’s cheap and sell where the money is. But there was one small problem with that approach: most partners were absolutely scared of the “big city” (it was Brazil after all - the country where people get shot in the streets… At least according to the news). Plus, we’d require more money to pull that stunt, and although we were selling reasonably well, the money was barely enough to pay our bills.
To make things worse, one of the partners had wife and kids, while the other was on his way into a marriage - as it were, two thirds of the company “decision men” were all about staying where we were and growing like a “family” business (and I didn’t have a dime to invest anymore). That crucial decision played a huge role on gluing the company to the ground when we could (should) have spread wings and flied with all our 20-something strength. Ultimately, we could never reach an agreement to what level of risk each of us found acceptable. Staying local was the only possibility of the moment, so we had to figure out a way to make it big. And that meant selling A LOT of licenses.
Interestingly enough, a change in the business model never even crossed out minds: we sold monthly per-device licenses - a model that scales very well if you have a great number of devices floating around. Problem is: the vast majority of the tech team’s time was spent on integrating the system with all sorts of different back-end software - and we did that basically for free. And then, given the small size of the average local firms, we rarelly saw more than 5 or 10 licenses per customer. The rational was that “licenses would bring in a lot more money in the long term”. Why we never even considered a more consulting-style approach is a total mystery to me. I promissed myself never fall on this “SaaS trap” again (which I promptly did, years down the road. Ouch.)
Given everyone’s reluctancy on taking larger risks in a larger market,the only decision that stuck with all the involved was “to try something bigger, but locally”. We would “try the big city once enough money rolls in”. But what huge customer would we be able to close in the middle of nowhere? Well, the government, of course!
Being the government in question one of the most corrupt in the entire world, it requires but a sidenote to guess how much of a risky path that was. Government was (and still is) the biggest buyer of tech in e entire country - problem is: how do you actually get to sell without having to resort to all sorts of dirty tactics, and after that, how do you actually get paid.
The project in question was huge: we proposed replacing all the paper forms used by itinerant doctors in every city of the state with handheld devices, with all the data consolidated on servers and automatically synced. The savings alone would be enough to justify the upfront cost: the entire health program costed a few millions per month to run, which included a gazillion tons of printed paper forms per month, a huge team of typists, transport crew to move paper boxes around and, of course, the ancient system used to save all that information on an old, dusty box in the server room. The negotiation (which included a couple of gov agents basically asking to “keep a few Palms for themselves as a gift”) was surprisingly fast, and soon enough, we got the contract. A thousand devices. A THOUSAND bloody licenses. But first, we needed to finance all that with the bank - we’d be paid in a couple of months, right? They even ran TV ads and billboards with our product on them!
Then came elections, and the government changed. Our contract (and everyone that ever heard of it) vanished. The devices themselves (I can’t recall wether we got to buy all the thousand ones) disappeared. We were back on square one - seeing any of the money invested back would take a long and litigious process (one which I didn’t stick around to watch).
After two years of struggle, the company was still not making enough to sustain the team. The gov sale fiasco took all my hope with it. A final event came as a sort of “divine answer” to my question to whether or not I should jump ship: while going to one of the final meetings that would close a much more favorable deal on the gov project, one of the partners (driving like a suicidal maniac) blew a couple of traffic lights, then got hit full force by a crossing car. I took the majority of the impact, and although not even a single bone was broken, surviving the 3 rows and exiting the crashing car only worrying about the laptop and the meeting we were to miss got me thinking that was not a path I wanted to keep trailing.
After these events, I gave away my share, leaving with a 15k hole in the bank, and moved on to pursue an entirely different product we were already cooking up at the time. We’d make it big on games…
To some people’s surprise, the company is still alive and well. Not surprisingly, though, it never made past the local market: it’s now a stable little software house, feeding two partners and a couple of developers. Of our competitors at the time, two got sold for very large figures and one became a big market leader in the country.
My takeaways from the experience: picking the right partners will make or break you. Having the guts to risk it all is key, specially if you’re sure the market and timing is right. Never bet all your eggs in a single omelette.
In the past 10 years, I’ve been involved in a lot of different startups - several of which I founded, either alone or with a couple of partners. Perhaps for not having a mentor, perhaps for being plain stubborn, learning had me necessarily falling down on every pothole and taking every wrong turn possible.
Given the startup fever is full rage these days - with basically everyone I know leaving their jobs to chase “the dream” - I decided to write down a bit of my past experience as a founder. Maybe it will help someone, maybe not. In the worst case, it’ll help me contemplating what was done right and what was done wrong, in this long and winding road.