Installed tweepy API.
It has all the 3 methods in single API:
1. Rest API
2. Streaming API and
3. Search API
Streaming API is useful to monitor live post with some keywords and/or hashtags. This will prove useful for government to report some tweets as soon as they get posted.
I put porn as a keyword and saw approx 2 posts per second being posted.
Also, I was looking through all the withheld tweets and did not find any single tweet that is missing from chillingEffects. So, I think, twitter is perfectly transparent.
I also found a nice article: http://thenextweb.com/twitter/2012/01/27/twitter-isnt-censoring-you-your-government-is/
which says, “Twitter isn’t censoring you. Your government is. ”
Is there a way to crawl through all the tweets till now and see “withheld_in_countries” parameter in that?
I have two updates:
1.
I wrote a third script using the REST API v1.1- tweet / GET statuses/show/id API
I ran the script by providing one of the withheld tweet ID: 357419942754267136 tweeted July 17th and requested withholding 10/1/13 on CE ( the same one I took snap shot last week see this doc)
I dumped the content to a text file and visually scanned the content here is some interesting fields [note the text field is the tweet content]
…
“id”:357419942754267136,
“id_str”:”357419942754267136″,
“text”:”http:\/\/t.co\/YsQlxs874J \u043a\u0443\u043f\u0438\u0442\u044c \u0433\u0430\u0448\u0438\u0448 \u043d\u0430 \u043c \u0442\u0435\u043a\u0441\u0442\u0438\u043b\u044c\u0449\u0438\u043a\u0430\u0445″
“screen_name”:”askajra”
“location”:””
…
“time_zone”:”Arizona”
“geo_enabled”:false,
“withheld_in_countries”:[“RU”]
I decoded the test field to char:
http:\/\/t.co\/YsQlxs874J купить гашиш на м текстильщиках
using google translate:
http: \ / \ / t.co \ / YsQlxs874J buy cannabis at m Tekstilshchiki
Comment: the user tweeted a link to drugs related stuff which was the reason for the withhold
Conclusion: we don’t have to be logged from the withholding country to find if the tweet was withheld. the “witheld_in_countries” will be returned
To do- write another API call to retrieve account info to see where the account was created (fyi: the name is russian)
Just to confirm my results, I ran the script against id: 389753553410981889 (one of my tweets I know they are not withheld) I looked for the withheld_in_countries field and sure enough it did not exist (output stored in text file (tweetbyid2Rtanash) on my machine
Conclusion: the witheld_in_countries field is sent only if the tweet is withheld in certain countries (this will be good to know when writing our robust crawler)
2.
Twitter transparency report
Cited from twitter:
• To minimize potential risk to our users, we’re not including specific numbers if fewer than 10 users / accounts are specified and we’ve not withheld any content in response to the particular request; instead you’ll see ‘< 10’ in the relevant cells.
• Where permitted by law, Twitter has published copies of the removal requests which have resulted in withheld content to Chilling Effects.
Which law are they referring to “when permitted by law”?
https://transparency.twitter.com/removal-requests/2012/jul-dec
Notice that the number they report here is much higher than what we found on chilling effects:18 (by June 30, 2013) when running the international query
We should have retrieved at least 102 international withhold requests which adds up to 135 withheld tweets total by July 2013
conclusion: not all international requests are reported on chilling effects. The Twitter transparency report is not specific and only includes graphs and some exmples see below
I changed my country setting to be Saudi Arabia, searched for several sex related hashtags in both English and Arabic. In all cases the pages viewed from Saudi were the same to the pages viewed from the US. This could mean one of several things.
1) Twitter is more sophisticated than we give them credit for in determining your location and the country setting is halfway a red herring
2) Twitters in depth process to requesting the withholding of each individual tweet is too much of a burden for the Saudi government to go through
3) The tweets are just being taken down instead of withheld
As it is, I’m willing to bet it’s more of 1) or 2), as there was still plenty of content for these hashtags.