The Bishop and the Butterfly: Murder, Politics, and the End of the Jazz Age
    Michael Wolraich's picture

    Lagblog

    By Michael Wolraich on Mon, 10/08/2012 - 12:15pm | Potpourri

    Hey folks. As some of you may have noticed, dagblog has become somewhat sluggish in maturity. When she was brand new, she zipped along like a peppy new sports car. But over the years, she has filled out a bit. The server is groaning under the weight of some 8,777 blog posts and 96,105 comments, and dag's reaction time has slowed to a crawl.

    In the past, we made some quick fixes to keep her running, such as disabling the search feature. But we're out of quick fixes.

    The obvious solution would be to get a new server capable of carrying all that weight, but those servers don't come cheap, and we run a pretty tight budget around here. When I get a chance, I'll look into cloud hosting options that might boost the power without boosting the cost, but that process will require a fair amount of time and the end result might not be much better.

    In the meantime, we'd like to propose the technical equivalent of a tummy-tuck. If we archive all the old comments from before 2012, it would take a lot of pressure off the server, and I expect that dag would run lighter. We would still leave up the blog posts themselves, but we would take down the comment threads and store them so that we could access them if needed and bring them back to life if we ever get a faster server. 

    Please let us know how you feel about this proposal. We're not enthusiastic about removing all that content from the great World Wide Web, but doing so will make room for a lot more comments in the future.

     

    Regards,

    The Management

    Topics: 
    Potpourri

    Comments

    I really appreciate this explanation.

    I thought it was just my damn computer!

    I'll back your decision; for sure.

    Since I have over a thousand blogs, I love going back and reading them for ideas and such and I do love to review comments at times and this also gives me ideas. and many of those older blogs would have been lost forever (prior to 4/11/11)

    Whatever you decide, I aint gonna complain.

    I suppose a small donation or subscription is not out of the question. Time costs me fifteen a year for access all the way back to the '30's but there are a few folks here who do not even have monies for extra pc/internet costs.

    Just thinkin'


    by Richard Day on Mon, 10/08/2012 - 12:43pm

    For a mere $50 per year, you can join DagPrime, your backstage pass to everything dag: in-depth commentary, members-only discussions, and comments that live forever. ;)

    Seriously, I appreciate your flexibility, Richard. The best thing that you can do for dagblog is to continue to write and to share the stuff you like. Doc's last post has almost 100 likes on Facebook. It's a good way to broaden the community.

    PS And buy my next book when it comes out!


    by Michael Wolraich on Mon, 10/08/2012 - 1:02pm

    Oh you know for a fact that the other Prime is offering comments forever? cheeky (Hard to tell, as no sampling, no tours, you pays your money and you takes your chances.)

    More seriously, I was under the impression from one of the myriad of forums where I have been a member, that one of these forum softwares offered the option to lock down old threads as static single pages including the comments, but the individual comments not having interactivity or links anymore?

    I seem to recall Marshall's geeks doing this inadvertantly when switching from like Scoop to Drupal (before you were there,) the older threads on the old software ended up static single pages--sometimes with comments out of order-- a real "archive" as it were.

    What happens when you close a thread to comments? All the comments still pull on the system individually? Dumb non-geek asks master geek meekly: might there be some other "switch" you don't know about?


    by artappraiser on Mon, 10/08/2012 - 4:08pm

    My immediate reaction is: is it possible to stretch the comments accessible to something like two years rather than one?

    I have some suspicions about this whole problem in general as regards forums and behavior of participants, and it's a Catch-22 kind of thing.That if people know comments are not as long lasting as blog posts, they will have a tendency to start a new blog post of their own about something they want to discuss that they just saw on a thread, when they otherwise might have done it on a comment on that other person's thread. Interactivity and communication suffers, while the proprietor ends up with a lot more blog posts on a topic to store, defeating some of the purpose.

    I saw this principle involved in action when TPMCafe instituted the new software in early 2008 (at which time many of the participants here started blogging there.) It didn't have tracking of comments or users at first, so everyone that had something to say about Obama vs Hillary, no matter how trivial, started a new blog post. The blog log ran so fast and furious, and blog entries disappeared so quickly, that people got viciously competitive about getting recommends so their poistwould be up a bit longer on a visible menu. (I remember Andrew Golis put up a "scream" post that basicially said: "can someone please blog on something else besides Hillary Clinton?" It was like all of the blog input for several days could have gone on a single thread!)

    An aside: Even though they have tracking, I've noticed that at Firedoglake, the competitiveness and bitterness problem about threads staying up and visible is very common. And they also have a lot of people creating new blog posts on the same topic rather than chosing to enter an existing discussion. Though I haven't quite figured out what it is about their system that makes this happen, it's something about posts going to a place quickly where they are not easy to find.

    Back to your questions. You've probably noticed I like to update news posts with comments, sometimes over periods of months. For continuity, to have the full story down, the follow up. Not having people just rely on memory for what was reported or what they said or thought. Longer than that down the road, or if there's a big development in the story, I would make a new post, but refer back to the old thread with a link. In your new plan to deal with this, all my comment updates on the old thread would be obliterated, and there would just be the initial "breaking" news post.

    I'm not saying all of this to expect you to be a S$ulzberger with a full archive. I don't expect that here, I know something's got to give.

    But I'm bringing it up as a bigger meta issue (insert Genghis sigh here cheeky) that I think is important about the world we are living in now and the future. If someone doesn't solve this storage problem, and the ADD-like symptoms it causes, we're not using the net any different than the systems we had in the past: discarded daily newspapers where you forget what happened yesterday, and radio talk where the emphasis is on ginning up tak of the hot topic of the day with all other news buried by the flood of emoting.  All kinds of bad results from this, like an emphasis on breaking news rather than follow-up, smears becoming conventional wisdom, people ranting egotistically rather than trying to communicate with each other, everybody talking on minutae of the day while Rome burns, etc.


    by artappraiser on Mon, 10/08/2012 - 1:10pm

    It occurs to me that I should furnish another example besides my own of long-term use of posts and comments. Jolly Roger is one example that comes to mind, he'll throw an outrageous thought out there in a blog post, then others comment "yes, and here's more ideas on that," or "no, you're thinking wrong, read this." And he changes his mind. and adds that in a comment. And he often refers back to a post like a year later in a new post; with the comments gone, without the full developed discussion on the thread, it's not the same thing, the whole is lost, only the inccurate part is left.


    by artappraiser on Mon, 10/08/2012 - 1:18pm

    We could start with a two-year limit and see if it helps sufficiently with the speed. I don't foresee the kind of comment-to-blog inflation you're talking about though. The examples you offer from other sites involve content that would disappear within hours, which is a much different timeframe.

    I think that the main drawback is the inability to go back and reference old threads, as you mention. I suspect that it's rare for people to go back more than a year, but I put up this post because I can only speak for myself. We want to find out if its worth the sacrifice for better performance.

    As for the broader problem you mention, keep in mind that people used to store very little of the data that we do now. All these blog comments would have been delivered verbally, if they were produced at all. Libraries maintained archives of the major newspapers, but many have of the smaller ones were lost to the wind. These days, we continue to keep everything we used to keep plus a hellava lot more. That creates new problems, but even if we haven't hit our full storage potential, we still have a much larger capacity for historical memory than we used to have. (I mean we as a society. Individuals human are as history-deficient as ever.)


    by Michael Wolraich on Mon, 10/08/2012 - 1:35pm

    I don't understand the problem - people ignore my comments real-time. If you archive 2 years of my comments, their ignoring them is no different than if you erase them.

    Come to think of it, they ignore my diaries too. Case closed, shut it all down. Let's go out for pizza instead.


    by PeraclesPlease on Mon, 10/08/2012 - 2:10pm

    I'm sorry to hear that, Peracles. I have consolidated all your comments below. I encourage people to please respond.


    by Michael Wolraich on Mon, 10/08/2012 - 3:35pm

    Can't be mine - it says "responsible" and doesn't mention "motherf*cker". 

    But it is a bit like seeing Solaris for the first time.


    by PeraclesPlease on Mon, 10/08/2012 - 4:15pm

    I don't have a problem with this.  I understand what AA is saying about losing the whole, and just getting the first part of something much larger (and in some cases more accurate).  I wonder if there could be an exception request created.  If there is one blog I think the comments are of particular interest, I could ask to have it placed in a special "history of Dagblog" section that do not have the comments archived.  (this might create its own competition).  It would be interesting to go and look at say blogs from 2010.


    by Elusive Trope on Mon, 10/08/2012 - 1:50pm

    Come to think of it, if we removed places where you & I argued, any mention of "The Confederacy", and any threads both rmrd & I were on, we've probably cured Genghis' problem right there.

    But Southerners had a point....


    by PeraclesPlease on Mon, 10/08/2012 - 4:11pm

    Maybe we could have a "best of Dag" section like "Best of Craigslist?

    Alternatively, what about letting everybody go in and move the comments to the tail end of each blog? (You'd lose the individuality of the comments but the whole would be preserved intact.) It would be easier for bloggers like me--I don't write that much or have all that many comments--much harder for the more prolific bloggers to do this.


    by erica20 on Mon, 10/08/2012 - 4:12pm

    One possible answer, of course, would be to curtail or eliminate the "Creative Corner" and return Dagblog to an exclusively political site.   This would be a sad thing for me, but thoroughly understandable, as I'm sure my weekly 'haikulodeon' postings must be taking up a lot of space, and in the future, I could simplify things and merely post a link to my blog-site: www.thehaikulodeon.com.  Perhaps if everyone that cross-posts their blogs simply posted links to their sites and require all comments to be made on their site, rather than here, that would save Dagblog a lot of space.   


    by MrSmith1 on Mon, 10/08/2012 - 2:24pm

    I appreciate the sacrifice, Smithy, but I assure you that neither your haikus nor the Creative Corner are not bringing down the site. There are almost 100,000 comments in the database.

    More to the point, we don't want to inhibit anyone from writing. In fact, we want to make it easier with a more responsive site. The old comments seem like the biggest culprit and also the most expendable.


    by Michael Wolraich on Mon, 10/08/2012 - 3:42pm

    Haikus by the pound,

    pulled the ancient server down;

    Wet snow breaks the tree.


    by moat on Mon, 10/08/2012 - 7:00pm

    Brilliant, Moat!  I wish I wrote that one!


    by MrSmith1 on Tue, 10/09/2012 - 11:44am

    I'm fine with this, Genghis.  With the exception of this particular comment, by me.  Please don't think it's my ego talking.  I just honestly believe that if you archive this comment that the entire Internet will erupt into riots.


    by Michael Maiello on Mon, 10/08/2012 - 2:50pm

    You know that I delete all your comments after two days, right?


    by Michael Wolraich on Mon, 10/08/2012 - 3:45pm

    And this, Genghis, is why every day I print up all my comments and mail them to Josh Marshall.


    by Michael Maiello on Mon, 10/08/2012 - 3:49pm

    You too?


    by Michael Wolraich on Mon, 10/08/2012 - 3:58pm

    Go do whatcha godda do.


    by Bruce Levine on Mon, 10/08/2012 - 3:13pm

    I feel grateful that you've been willing to store all our posts for so long...getting rid of the comments is unfortunate (because they often elevate the original post,) but it sounds like it's necessary, so I'm good with it. In reality, how often do we actually go back and read the WHOLE posts, comments and all?


    by stillidealistic on Mon, 10/08/2012 - 4:10pm

    Maybe if we didn't double post, that would be helpful!


    by stillidealistic on Mon, 10/08/2012 - 4:18pm

    Genghis, I don't know if this would help, but I'll throw it out there--my bf and his brother, along with this pretty old academic guy, developed an indexing system a few years back. It's different than search; it has something to do with Noam Chomsky and the syntactic web. (Blah blah blah, my eyes glaze over, but some people love this stuff.) It sort of sits around with nothing to do, waiting to go through some enormous document somewhere and assign order to it. Maybe Dag comments are that enormous document. Let me know if you think there might be something to this idea. I'm sure they'd be happy to discuss it with you! (Just don't ask me to explain it.)


    by erica20 on Mon, 10/08/2012 - 4:33pm

    Thanks, Erica. The problem is in the database. It's not really an indexing issue.

    (There is actually an indexing issue in the search, which is why I disabled it, but integrating a third-party search solution would take more time than I have available.)


    by Michael Wolraich on Mon, 10/08/2012 - 7:56pm

    Darn. I think they would love a chance to play around with that thing.


    by erica20 on Tue, 10/09/2012 - 10:48am

    If they want to create a drupal module that I can plug in, I'd be happy to try it out.


    by Michael Wolraich on Tue, 10/09/2012 - 11:51am

    Since I cross-post most of my stuff, anyway, it's really the comments that are unique for me.  I can understand why you need to do this but if you could start by hiding all but the past two years' worth of comments, that would be the most prudent way to do it.

    And if you see that there is still a lag, then you could make it one year only.  But whatever you decide is fine with me.  I'm just grateful to have my posts here at all.


    by Ramona on Mon, 10/08/2012 - 4:52pm

    I'd be willing to volunteer all comments made by Trope for oblivion. Also, Peracles. And Jolly. Ok, pretty much everyone. If we're gonna make this place look good for posterity, I'm afraid we pretty much need to can em all.

    Except for me and Wolfrum. That Bristol Stool Chart should live forever. Bloggers thousands of years from now will wanna know the origins of that classification system. 

    Oh yeah. Did I mention Trope?


    by Qnonymous (not verified) on Mon, 10/08/2012 - 5:00pm

    Well, that does remind me that since my handlers back at the agency maintain nice transcripts of all my discussions, I know that any thread in which I participate will never see oblivion, but rather live on forever in the universe of bureaucratic memory, read and re-read and analyzed and summarized for all of time. 


    by Elusive Trope on Mon, 10/08/2012 - 5:08pm

    Your handlers at the "agency" called - seems your new batch of Thorazine was mislabeled, they expect the feeling of being Ming the Merciless or captive of Transfalmadore to wear off in another 20 hours or so. Meanwhile, rest assured, they will re-read and analyze your comments and thoughts - a breakthrough for modern science perhaps.


    by PeraclesPlease on Tue, 10/09/2012 - 2:33am

    I agree with the two years (even 18 months). If preserving a certain blog's comments is important to an author, he/she could certainly copy and paste to save. 


    by Aunt Sam on Mon, 10/08/2012 - 5:07pm

    Oooo, I like that.  How about every blogger be responsible for saving, (archiving), the comments from their own blogs?  Then, if someone wants to go back to the old blog, and are interested enough to want to read the comments, they simply contact the original blogger who posted the blog.


    by MrSmith1 on Mon, 10/08/2012 - 6:27pm

    And I thought I had mis-remembered a Search box.

    It's your dagblog, do what you gotta do --- but before doing it will you please remind me how to export my posts with comments like we did from TPMCafe so I can import them to blogspot?

    As for comments to others' posts, will you please reinstate the Search long enough for us to e-mail, print or save them entirely or at least an index of them?  Why?  Probably just a useless carry over of correspondence filing from pre-internet times.  What can I say.  I am that old.

     

     


    by EmmaZahn on Mon, 10/08/2012 - 5:36pm

    Per my comment below, you can access your comments at http://dagblog.com/comments_by/emmazahn. Does that give you would you need?


    by Michael Wolraich on Mon, 10/08/2012 - 7:25pm

    Instead of a DagPrime how about a SubDag, and you can pay us and we will remove everyone else's comments that we don't like.

    Hah,

    Sounds like you have  plan G. good luck!


    by tmccarthy0 on Mon, 10/08/2012 - 6:40pm

    Dear Genghis,

    I have been a wife and mother for over 30 years and in all that time no one has ever listened to a damn thing I've said whether it was spoken aloud or digitalized on the web. So go ahead and delete my comments to your hearts content. It'll be just like home.

    Love,
    ~flowerchild~


    by wabby on Mon, 10/08/2012 - 6:56pm

    Ouch!  Can't wait to see what that danged Genghis has to say about that!  lol


    by Ramona on Mon, 10/08/2012 - 7:07pm

    Well, I hope he has a laugh and knows I don't disagree with his solution. smiley ... Shift+R improves the quality of this image. Shift+A improves the quality of all images on this page.


    by wabby on Mon, 10/08/2012 - 7:13pm

    Maybe if you wrote in ALL CAPS...


    by Michael Wolraich on Mon, 10/08/2012 - 7:16pm

    I just checked the numbers. Two years won'to help that much. It would only reduce about 20 percent of the comments. But if we go to June 1, 2011, we can reduce over 50 percent.

    Please keep in mind that we wouldn't delete anything. If you should have an urgent need in the future, I could retrieve the comments. And if we get a faster server, we could bring them back.

    FYI, I created a page for you to access all your comments: http://dagblog.com/comments_by/[your screen name]. e.g. http://dagblog.com/comments_by/genghis. If you have spaces in your screen name, use hyphens.

     


    by Michael Wolraich on Mon, 10/08/2012 - 7:14pm

    Hey folks. I just found some more tweaks that I could make. I'm caching everything for one minute now, so there might be a slight delay between when you submit a comment or post and when it shows up in the sidebar. Let's see how much this helps.


    by Michael Wolraich on Mon, 10/08/2012 - 7:54pm

    This is a great blogsite and speeding it up makes all the sense in the world.  Go for it!


    by HSG on Mon, 10/08/2012 - 8:56pm

    Thanks a lot, Hal. The changes I just made seem to be helping a lot. Maybe it won't be necessary to archive the comments.


    by Michael Wolraich on Mon, 10/08/2012 - 9:14pm

    Just what I expected, I am not surprised, beeeecause you are a such a frigging genius at this stuff, I thought if I whined a little and then some others joined in, you would find a way. (I'm figuring Ramona's plaint was the one that turned on the lightbulb--thank you Ramona.) cheeky

    Hey a thought--would it help if people stopped posting embedded videos in comments and just put a link to the video? It's something I've noticed on other websites, that they don't enable that in comments. Maybe pix too? I'd rather give up pix in comments than give up having comments available. Like this--if you want to embed a video or pix, start a new post, otherwise, in comments, link to the video or pix instead?


    by artappraiser on Tue, 10/09/2012 - 12:42am

     Nope.  Wasn't me, aa.  I think it was you.  But I like your idea of only linking to videos and pics.  You're right that most websites don't enable them in the comments, and that probably makes the most sense.

    I never could figure out how to add them here, anyway, so it makes no nevermind to me. 


    by Ramona on Tue, 10/09/2012 - 4:55am

    Pretty sneaky, sis!

    Don't worry about the pics and videos. Those aren't stored on our server, so they don't add to the load. I suspect other sites just don't want the clutter / spam / porn / poop.


    by Michael Wolraich on Tue, 10/09/2012 - 11:49am

    I can't believe you'd dis poop like that.

    May you suffer from Bristol Type 1 for the next week.


    by Anonymous (not verified) on Tue, 10/09/2012 - 12:07pm

    I have no problem with you deleting as much as you like.  I don't like the idea of leaving a paper trail for posterity anyway.


    by Dan Kervick on Mon, 10/08/2012 - 9:46pm

    Genghis I copied my comments but noticed they only go back to 4/22/2011. I thought I'd been hanging out at Dag for longer than that. Does that seem right?


    by erica20 on Tue, 10/09/2012 - 11:55am