The reason behind LJ temporarily banning Tor

[charmian]

For awhile, LJ banned Tor because a site which had been banned from accessing LJ was using Tor to circumvent the ban. What was this site?

It was a site known as Lj2rss. This site would take your LJ name and password and I think, output an RSS of your friends page. The site was banned for two reasons a) making too many requests (hitting the servers too much, I guess), and b) providing a paid service to users, which is against TOS.

At first, I was confused by the news that an RSS of the friends page is considered a paid service. I was aware that you could use an S1 hack to get an RSS of your friends page if you were a paid user, but I wasn't aware that this was considered a paid feature which LJ really doesn't want external services to provide.

From the email that LJ sent the site owner: "we have determined that your service provides a feature and function in violation of our site policies. As stated in the following FAQ, RSS feeds of a users' friends page are only available to Paid and Permanent users [http://www.livejournal.com/support/faqbrowse.bml?faqid=149]. Your service is therefore bypassing an account-level restriction for a feature provided by LiveJournal, and is in violation of our Terms of Service"

Huh. I didn't know this was considered so important a feature. You can of course read your flist as individual authenticated RSS feeds; however, this is rather unwieldy and it is more efficient to have one feed. But why is one banned and the other not? What's the big deal about that? Also, how was the circumvention managed? No matter what, a non-paid account can't access custom layers, right? So how did they do something which only paid users can manage? (This seems to be the code, but I don't know programming so I don't understand it)

Also, what does this possibly mean for Dreamwidth's planned authenticated friends-list reading feature? Hopefully it will not fall afoul of whatever strictures LJ has related to this.

Comments

[afuna]

They appear to have been screenscraping http://www.livejournal.com/mobile/friends.bml, which is available to all users :-)

ETA: I'm not sure where DW is in relation to LJ with authenticated friends-list reading, but we'd be using a different approach.

[charmian]

Ah, I see... and then they turned it into an RSS? I don't really understand LJ's logic there then, because aside from the server abuse, if this page is available to all users, then where is the problem? Couldn't anyone conceivably create this kind of script?

Yeah, especially since if you say the approach of this site is rather different from the paid feature, although it gives similar results (but aren't there a lot of problems with screen-scraping?)

[afuna]

Yeah, the point about RSS feeds seems a stretch, especially since, from the FAQ, "Paid and Permanent users can create a custom style that functions as an RSS feed for their Friends page. LiveJournal does not offer a built-in RSS feed for your Friends page." -- the paid feature is being able to use S2 to create a hacked together RSS feed, and not having your friends page available in RSS form per se.

It's purely academic at this point since the server abuse is more than enough to make LJ decide to rate limit the service, but the second point is pretty... odd.

Screen scraping is horrible because you load up a lot more than you need, and don't obey any built in rate limits that might be imposed by a proper data API. On the flip side, there could have been worse pages to use to grab the data.

[charmian]

Considering it was such a different mechanism, I also thought it was a stretch. And also, IIRC other external apps also can give you paid features. LJ sec allows you to make all of your journal entries friends-locked at once, which is otherwise a paid feature, and that features seems like one more used that RSS.

Yeah... but they didn't rate limit it, they banned it entirely right? I mean, theoretically couldn't the programmer have just caused it to hit the server less and thereby not caused problems? But the way the letter makes it seem, the second point was rather decisive.

[afuna]

Oops, I meant to say ban (ban, rate limit, they are so close together on the keyboard...)

I am slightly worried from the wording of the letter that this is what LJ will use to block other services (say, DW ;-)) that \would allow users to access their friends page from other services -- though if their argument is primarily format, then I guess I should be less worried.

[charmian]

Yeah... By format, what do you mean, RSS? So in your reading the whole problem is that from the screen scrape, the data was turned into an RSS feed, but DW won't be using RSS, and therefore there is less likely to be a problem?

[afuna]

The format thing seems to be what they're concentrating on now, so I can hope that this doesn't mean they'll shut down a reading page feature from DW -- but I can't say if they'll continue to focus on that, with whatever DW comes up with.

[pne]

the paid feature is being able to use S2 to create a hacked together RSS feed, and not having your friends page available in RSS form per se.

That's what I thought, too, so quoting the FAQ as reason for banning them seems odd to me -- the paid feature isn't the friends RSS per se.

[elena]

Note that there is an OPML file at http://www.livejournal.com/tools/opml.bml that you can import at your feed reader. It would be nice if LJ publicized it a bit more instead of banning hacks and workarounds that try to reinvent the wheel.

I'd also be wary of any service that asks you to provide the username and password of another one.

[charmian]

I've done it using Gregarious (import OPML), but honestly, it's pretty unwieldy calling say a hundred feeds instead of one. In the end I decided it wasn't worth it, especially since a lot of people on my flist had turned off their feeds.

It would be better if done with OAuth, but I'm not sure LJ supports that.

[elena]

I've used OPML files moving feeds around with Bloglines and Google Reader, and it works quite well (except for Google reader not letting me rearrange the feed floders, but whatever). YMMV.

Then again, I use Google Reader mainly for reading blogs, and I'm not that concerned about not getting friendslocked LJ entries in my feed list.

[charmian]

I've found those are faster than desktop readers, actually. Probably the reason Gregarious was slow was because I was running it off my own webhosting.

Google reader and bloglines don't allow authenticated RSS, so that doesn't help folks. If it weren't for the authenticated feeds, I think it would be relatively trivial to create an aggregate RSS feed of your flist (or filters thereof) using Yahoo Pipes or some similar service.

http://www.tothepc.com/archives/10-tools-to-combine-mix-blend-multiple-rss-feeds/

The problem really is the authenticated RSS feeds and the lack of support for that.

[elena]

Gregarius pulls all feeds in a row only when you tell it to update, so yeah, it is really slow. Online feed readers like Google Reader set scheduled jobs to update feeds according to the frequency they are updated (IIRC), and cache the results, apart from hardware/software capabilities that you don't get at a non-professional webhosting (server cluster, load balancing, whatever). That's why you don't see the lag you get with Gregarius.

I have Gregarius hosted at my domain, too, but I guess any day now I'll delete it and read everything from Google Reader. It's somewhat redundant now.

[charmian]

Yeah, and that's why people don't feel that anything other than Google Reader, Bloglines, etc., is up to snuff. (and besides, the trend is to do everything in the cloud nowadays.) That's why you have people willing to even do dangerous things like strip feeds of their authentication in order to read using Google Reader. I've been googling and it seems that's the solution people are generally using; using freemyfeed or Yahoo Pipes to aggregate the authenticated feeds.

[elena]

I think that defeats the very purpose of friendslocking things Ô_ó

[charmian]

Well, this usually gives an secret URL which isn't easily derivable, and if they set it as non-shared on google reader, probably no data will leak out, but that's a probability. It's sort of like the method posterous uses for the RSS of its private blogs.