Comments on: Some Twitter Bot Service Basics http://dansgalaxy.co.uk/2009/06/27/some-twitter-bot-service-basics/ King Geek (or is it Geek King?) Wed, 30 Jun 2010 07:36:49 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Aaron Bassett http://dansgalaxy.co.uk/2009/06/27/some-twitter-bot-service-basics/comment-page-1/#comment-1661 Aaron Bassett Sat, 27 Jun 2009 13:22:21 +0000 http://dansgalaxy.co.uk/?p=486#comment-1661 You can merge multiple phrases into a single request using OR operators. Wouldn't suggest doing it for hundreds or thousands of phrases but would def work for 3 phrases. So that's only 3 requests rather than 9. Also as you do not need to grab every tweet you could reduce the frequency of your requests. So if you were grabbing the public timeline at a rate of once per minute, but your key phrases are only uttered once an hour you could reduce your requests down to once every 4 days (search brings back max of 100 results @ 1 per hr these 100 results would cover 4.16667 days, extreme example but you get what I mean) But you are correct about it not working if you have hundreds of bots, none of the rate limited services would. But once you reach of those kind of traffic levels maybe you could make a good pitch for requesting access to the 'Firehose' - http://blog.twitter.com/2008/07/twitter-and-xmpp-drinking-from-fire.html You can merge multiple phrases into a single request using OR operators. Wouldn’t suggest doing it for hundreds or thousands of phrases but would def work for 3 phrases. So that’s only 3 requests rather than 9. Also as you do not need to grab every tweet you could reduce the frequency of your requests. So if you were grabbing the public timeline at a rate of once per minute, but your key phrases are only uttered once an hour you could reduce your requests down to once every 4 days (search brings back max of 100 results @ 1 per hr these 100 results would cover 4.16667 days, extreme example but you get what I mean)

But you are correct about it not working if you have hundreds of bots, none of the rate limited services would. But once you reach of those kind of traffic levels maybe you could make a good pitch for requesting access to the ‘Firehose’ – http://blog.twitter.com/2008/07/twitter-and-xmpp-drinking-from-fire.html

]]> By: Dan http://dansgalaxy.co.uk/2009/06/27/some-twitter-bot-service-basics/comment-page-1/#comment-1660 Dan Sat, 27 Jun 2009 13:01:02 +0000 http://dansgalaxy.co.uk/?p=486#comment-1660 Ah but the issue with that is i then have to run a search api query for every phrase for every bot. As this system is already running 3 bots and (currently) 3 phrases each that's 9 requests verse 1 request. And I am hoping to build on this to eventually end up running tens (maybe even hundreds) of bots which as far as i can tell from the rate limits would just be totally undoable :( Also more bots/phrases the % of useful tweets also goes up ^^ Happy to be corrected if I am wrong though :) Ah but the issue with that is i then have to run a search api query for every phrase for every bot.

As this system is already running 3 bots and (currently) 3 phrases each that’s 9 requests verse 1 request.

And I am hoping to build on this to eventually end up running tens (maybe even hundreds) of bots which as far as i can tell from the rate limits would just be totally undoable :(

Also more bots/phrases the % of useful tweets also goes up ^^

Happy to be corrected if I am wrong though :)

]]> By: Aaron Bassett http://dansgalaxy.co.uk/2009/06/27/some-twitter-bot-service-basics/comment-page-1/#comment-1659 Aaron Bassett Sat, 27 Jun 2009 12:55:39 +0000 http://dansgalaxy.co.uk/?p=486#comment-1659 Rather than catch every (or attempt to) tweet via public stream you could use the search API. That way you are only lifting out tweets that matter to the bot. So for example say every 10,000 tweets you grab from public timeline one contains the phrase you are looking for then only 0.0001% of the data stored in your DB is useful to the bot. If you use the search API to only grab those of interest it rises to 100% :) Rather than catch every (or attempt to) tweet via public stream you could use the search API. That way you are only lifting out tweets that matter to the bot.

So for example say every 10,000 tweets you grab from public timeline one contains the phrase you are looking for then only 0.0001% of the data stored in your DB is useful to the bot.
If you use the search API to only grab those of interest it rises to 100% :)

]]>