Parsely - request a recrawl

Parse.ly is an analytics platform designed specifically for online publishers. Parse.ly provides digital publishers with clear audience insights to answer questions about how readers are responding to content.

Once you set it up for your website, Parse.ly Crawler will perform a crawl of the pages as the pageviews stream in from those URLs.

The crawl is performed only once for a unique URL - when the article is published and traffic starts coming in.

When the article changes a recrawl would be nice. To do that, you need API key and API secret, that you can access via API Settings page.

In order to request a recrawl, you have to submit a POST request to the following address, appending the to request a recrawl for:

http://dash.parsely.com/<API_KEY>/ping_crawl?secret=<API_SECRET>&url=<URL>

This is how to do it in a Rails app.

Add a ruby http client http.rb to your Gemfile:

gem 'http'

Add your Parse.ly API credentials to secrets.yml:

parsely_api_key: <%= ENV['PARSELY_API_KEY'] %>
parsely_api_secret: <%= ENV['PARSELY_API_SECRET'] %>

Set environment variables (or put those environment variables to .envrc if you're using direnv), so that you don't push them to a remote repository:

export PARSELY_API_KEY=__our_api_key__
export PARSELY_API_SECRET=__secret__

And this is what a crawler might look:

We'll need a worker, that requests the recrawl:

And finally tell the worker to perform the job when a blog post changes, in active admin's resource:

That's it! Now our blog posts are recrawled on every update!

Loading comments…
All rights reserved