Lurking tool: ConceptArt.org image scrapper! (Edit: Version 2.2!)
Join the #1 Art Workshop - LevelUpJoin Premium Art Workshop

Page 1 of 2 1 2 LastLast
Results 1 to 30 of 32

Thread: Lurking tool: ConceptArt.org image scrapper! (Edit: Version 2.2!)

  1. #1
    Join Date
    Nov 2007
    Location
    Paris France
    Posts
    28
    Thanks
    4
    Thanked 33 Times in 6 Posts
    Follows
    0
    Following
    0

    Lurking tool: ConceptArt.org image scrapper! (Edit: Version 2.2!)

    Hi all,
    I'm a bad artist but a long time CA.org lurker. I've just spent a few hours making a tool that will make my lurking easier. I find the message board a bit cluttered and hard to navigate on a small screen, so this page is my solution.
    It displays the most recent images posted on the forum and loads more as you scroll down. I know the tool is a bit slow, it's ok for me but if there's any interest I'll see what I can do to optimize it.

    http://www.antoinegersant.com/misc/lurk/
    Name:  lurk.jpg
Views: 1254
Size:  261.3 KB

    EDIT: Version 2.1 is up and running!

    -- Changelog Version 2.2 --
    · Added tooltips with thread names

    -- Changelog Version 2.1 --
    · Filtering can now be turned ON or OFF (default: OFF) and its defaults setting filters everything out

    -- Changelog Version 2.0 --
    · Much faster
    · Filter pictures by forum
    · Keeps one month worth of pictures
    · Updated every hour

    Last edited by fil_razorback; December 27th, 2012 at 06:22 PM.
    Reply With Quote Reply With Quote  

  2. The Following 26 Users Say Thank You to fil_razorback For This Useful Post:

    + Show/Hide list of the thanked


  3. #2
    Join Date
    Dec 2010
    Location
    Capital of TX
    Posts
    794
    Thanks
    247
    Thanked 262 Times in 202 Posts
    Follows
    0
    Following
    0
    This is so neat man! Thanks for sharing and please if you can work on it more!

    Sketchbook Help me improve! Animations
    “To be an animator you have to have a sense of the dramatic, a feel for acting. You have to be a storyteller.”-Marc Davis
    "It's not the size of the dog in the fight, it's the size of the fight in the dog.” ― Mark Twain
    My peeps
    Oghren Care
    Lakai
    Etern
    Reply With Quote Reply With Quote  

  4. #3
    Join Date
    Mar 2008
    Location
    Falmouth, Cornwall, UK
    Posts
    315
    Thanks
    182
    Thanked 93 Times in 76 Posts
    Follows
    0
    Following
    0
    This is so awesome. Thanks Fil I've already found a few really interesting threads I probably wouldn't have found otherwise.

    My Portfolio

    My Sketchbook
    Please feel free to add a comment, I appreciate all feedback!
    Reply With Quote Reply With Quote  

  5. #4
    Join Date
    May 2012
    Location
    Buenos Aires, AR
    Posts
    269
    Thanks
    82
    Thanked 97 Times in 74 Posts
    Follows
    0
    Following
    0
    Nice! That's a pretty good tool for discovering new threads. I think it'd be much be better if it was part of ca.org, or at least hosted in these servers. That way ca doesn't have to spend bandwidth on it.

    Btw, you could try caching the first set of images every hour so it loads faster initially.
    Great idea razorback

    "Great job guys! I love you. You're fired."

    Sketchbook! Me vs Anatomy (and other things)
    Reply With Quote Reply With Quote  

  6. #5
    Join Date
    Jan 2006
    Location
    Brazil
    Posts
    553
    Thanks
    201
    Thanked 80 Times in 66 Posts
    Follows
    0
    Following
    0
    Very cool!

    My website

    Facebook Page

    New CA Sketchbook

    " The scientific and generally accepted-in-art term for this is "You're fuckin' screwed, dude..." " Ilaekae - May 16th, 2009
    Reply With Quote Reply With Quote  

  7. #6
    Join Date
    Nov 2008
    Location
    North Carolina
    Posts
    759
    Thanks
    656
    Thanked 367 Times in 244 Posts
    Follows
    0
    Following
    0
    That's great! Yeah I would like to see what you could do to optimize it. Sometimes its better just to look at the pictures and not be distracted by all of the text.

    Reply With Quote Reply With Quote  

  8. #7
    Join Date
    Aug 2004
    Location
    Tennessee
    Posts
    1,287
    Thanks
    219
    Thanked 167 Times in 106 Posts
    Follows
    0
    Following
    0
    I love it. I wonder if it would be possible to make it just for "Finally Finished" or "Sketchbook" sections?

    Reply With Quote Reply With Quote  

  9. #8
    Join Date
    Nov 2007
    Location
    Paris France
    Posts
    28
    Thanks
    4
    Thanked 33 Times in 6 Posts
    Follows
    0
    Following
    0
    Thanks for your feedback! I'm working on an updated version =)

    Reply With Quote Reply With Quote  

  10. #9
    Join Date
    Dec 2008
    Location
    SEA, WA
    Posts
    867
    Thanks
    260
    Thanked 175 Times in 152 Posts
    Follows
    0
    Following
    0
    DUDE! Awesome!

    I love it, and will be checking back to see what you can do to optimize it. Good luck, and thank you!

    -I often post from my phone; so please excuse the typos
    Sketchbook
    Reply With Quote Reply With Quote  

  11. #10
    Join Date
    Jan 2008
    Posts
    474
    Thanks
    120
    Thanked 194 Times in 69 Posts
    Follows
    0
    Following
    0
    This looks like something that could be a major addition to CA. If this was a part of the site (with some more improvements like faster loading time, etc.) I could possibly be more active here. Hmm.

    Reply With Quote Reply With Quote  

  12. #11
    Jason Manley's Avatar
    Jason Manley is online now Administrator Level 17 Gladiator: Spartacus' Dimachaeri
    Join Date
    Jul 2002
    Location
    Austin TX
    Posts
    9,066
    Thanks
    1,965
    Thanked 10,567 Times in 2,176 Posts
    Follows
    2
    Following
    1
    Hey Razor...Thanks for making that. Plz check your PM's when you get a chance. Awesome.


    JM

    LEVEL UP! - ConceptArt.Org online workshops 25% off registration right now!
    Reply With Quote Reply With Quote  

  13. #12
    Join Date
    Nov 2007
    Location
    Paris France
    Posts
    28
    Thanks
    4
    Thanked 33 Times in 6 Posts
    Follows
    0
    Following
    0
    @Jason Manley > Replied
    @Others > Thanks!

    Version 2 is up and running!
    Changelog:
    - Much faster
    - Filter pictures by forum
    - Keeps one month worth of pictures
    - Updated every hour

    I hope you like it, I certainly had fun making it. Here it is.

    Reply With Quote Reply With Quote  

  14. The Following 3 Users Say Thank You to fil_razorback For This Useful Post:


  15. #13
    Join Date
    Nov 2011
    Posts
    289
    Thanks
    104
    Thanked 66 Times in 58 Posts
    Follows
    0
    Following
    0
    Very awesome!
    I can also sense some work coming your way, huh ? That's great, man!

    I'm looking forward to seeing updates with this program/site. Very useful!

    Reply With Quote Reply With Quote  

  16. #14
    Join Date
    Aug 2012
    Location
    Australia
    Posts
    214
    Thanks
    117
    Thanked 31 Times in 29 Posts
    Follows
    0
    Following
    0
    Brilliant work!
    2.0 is definitely faster than the previous version.

    The filtering probably needs a bit of work IMO.

    Example:
    When page is opened, all forums are selected as one checkbox.
    Along with all the other forums/sub-forums available deselected.
    Enabling people to select what they want to see specifically without having to uncheck all the checkboxes.

    Then, for example if this does get included into the site, have it so that upon login it can remember what forums you visit so once the page is opened it goes straight to your "favorites" instead of all selected.

    Just my two-cents.

    Again, great work!

    www.travismillar.com

    Ever just woken up and gone "shit, does the world around me exist"?
    Reply With Quote Reply With Quote  

  17. #15
    Join Date
    Aug 2012
    Location
    Australia
    Posts
    214
    Thanks
    117
    Thanked 31 Times in 29 Posts
    Follows
    0
    Following
    0
    Brilliant work!
    2.0 is definitely faster than the previous version.

    The filtering probably needs a bit of work IMO.

    Example:
    When page is opened, all forums are selected as one checkbox.
    Along with all the other forums/sub-forums available deselected.
    Enabling people to select what they want to see specifically without having to uncheck all the checkboxes.

    Then, for example if this does get included into the site, have it so that upon login it can remember what forums you visit so once the page is opened it goes straight to your "favorites" instead of all selected.

    Just my two-cents.

    Again, great work!

    www.travismillar.com

    Ever just woken up and gone "shit, does the world around me exist"?
    Reply With Quote Reply With Quote  

  18. #16
    Join Date
    Nov 2007
    Location
    Paris France
    Posts
    28
    Thanks
    4
    Thanked 33 Times in 6 Posts
    Follows
    0
    Following
    0
    Thanks for the suggestion, I think it's a lot more convenient now.
    Version 2.1

    Reply With Quote Reply With Quote  

  19. The Following User Says Thank You to fil_razorback For This Useful Post:


  20. #17
    Join Date
    Aug 2004
    Location
    Tennessee
    Posts
    1,287
    Thanks
    219
    Thanked 167 Times in 106 Posts
    Follows
    0
    Following
    0
    I just noticed the "Filter" option. That's amazing! Thanks for this.

    Reply With Quote Reply With Quote  

  21. #18
    Join Date
    Feb 2012
    Location
    living in England
    Posts
    82
    Thanks
    12
    Thanked 18 Times in 16 Posts
    Follows
    0
    Following
    0
    Wow, awesome. This is really handy since you can pick and choose quickly. It would be great if somehow you could see the thread name by hovering over the picture, which could quickly put the picture into context.

    Reply With Quote Reply With Quote  

  22. #19
    Join Date
    May 2012
    Location
    Buenos Aires, AR
    Posts
    269
    Thanks
    82
    Thanked 97 Times in 74 Posts
    Follows
    0
    Following
    0
    Great update. Feels snappier now

    "Great job guys! I love you. You're fired."

    Sketchbook! Me vs Anatomy (and other things)
    Reply With Quote Reply With Quote  

  23. #20
    Join Date
    Nov 2007
    Location
    Paris France
    Posts
    28
    Thanks
    4
    Thanked 33 Times in 6 Posts
    Follows
    0
    Following
    0
    Thanks all! Having actual users make me really happy =)
    @Shirobon > Done! It's not *exactly* the thread names (because I haven't been crawling them) but a readable version of their URL. The difference is subtle so I think it will do.

    Reply With Quote Reply With Quote  

  24. #21
    Join Date
    Dec 2008
    Location
    SEA, WA
    Posts
    867
    Thanks
    260
    Thanked 175 Times in 152 Posts
    Follows
    0
    Following
    0
    I'm speechless! This is truly badass.

    I like the "hover" title aspect. I don't want to see a bunch of title text over all of these images as I am scrolling through; only the ones I hover over. The whole appeal of this site is that it cuts out all the clutter of text and brings you just the images.

    Again, truly awesome.

    -I often post from my phone; so please excuse the typos
    Sketchbook
    Reply With Quote Reply With Quote  

  25. #22
    Join Date
    Dec 2008
    Location
    SEA, WA
    Posts
    867
    Thanks
    260
    Thanked 175 Times in 152 Posts
    Follows
    0
    Following
    0
    I'm speechless! This is truly badass.

    I like the "hover" title aspect. I don't want to see a bunch of title text over all of these images as I am scrolling through; only the ones I hover over. The whole appeal of this site is that it cuts out all the clutter of text and brings you just the images.

    Again, truly awesome.

    -I often post from my phone; so please excuse the typos
    Sketchbook
    Reply With Quote Reply With Quote  

  26. #23
    Join Date
    Dec 2008
    Location
    SEA, WA
    Posts
    867
    Thanks
    260
    Thanked 175 Times in 152 Posts
    Follows
    0
    Following
    0
    PS: if there is a way to extend the backlog of images to more than a month, that would be great. I was just lurking the Tutorial section, and it looks pretty skimpy because not a lot has been uploaded in the last month.

    Maybe you can keep the bigger subforums (Sketchbook, IFF, WIP, etc) at one month, but increase the smaller/less visited forums?

    -I often post from my phone; so please excuse the typos
    Sketchbook
    Reply With Quote Reply With Quote  

  27. #24
    Join Date
    Dec 2008
    Location
    SEA, WA
    Posts
    867
    Thanks
    260
    Thanked 175 Times in 152 Posts
    Follows
    0
    Following
    0
    PS: if there is a way to extend the backlog of images to more than a month, that would be great. I was just lurking the Tutorial section, and it looks pretty skimpy because not a lot has been uploaded in the last month.

    Maybe you can keep the bigger subforums (Sketchbook, IFF, WIP, etc) at one month, but increase the smaller/less visited forums?

    -I often post from my phone; so please excuse the typos
    Sketchbook
    Reply With Quote Reply With Quote  

  28. #25
    Join Date
    Apr 2012
    Location
    Ireland
    Posts
    122
    Thanks
    99
    Thanked 34 Times in 23 Posts
    Follows
    0
    Following
    0
    Wow! This is awesome!
    The only recommendation I have is maybe you could have a "search-by-keyword" function as well as the filters.

    I know that CA has an image search function but I prefer the nice, clean interface you've created to browse pictures
    Can't wait to see how this progresses!



    Feel like kicking some artistic butt this year?
    Come check out the new subforum
    Beginners Challenges and Activities
    Reply With Quote Reply With Quote  

  29. #26
    Join Date
    Nov 2007
    Location
    Paris France
    Posts
    28
    Thanks
    4
    Thanked 33 Times in 6 Posts
    Follows
    0
    Following
    0
    @Syle > I've adjusted the settings, the tool will now keep pictures for 6 months (that will occupy about 1GB on my server, just with thumbnails). However, I cannot crawl data older than a month (because the activity stream doesn't go this far) so it will fill gradually from now on. While I was at it I also started crawling actual thread names: pictures that are already indexed now display their slug (URL friendly name) on rollover but future ones will have their name exactly as it appears on the forum.
    @The-Idle-Elk > Thanks. I'm not too sure about a search function…It is doable but it feels a bit out of scope and I'm not sure I could improve a lot on the experience you get from the original CA.org there.

    Reply With Quote Reply With Quote  

  30. The Following User Says Thank You to fil_razorback For This Useful Post:


  31. #27
    Join Date
    Nov 2008
    Location
    North Carolina
    Posts
    759
    Thanks
    656
    Thanked 367 Times in 244 Posts
    Follows
    0
    Following
    0
    Its working great. You've got the speed issue pretty well fixed. Now all I have to do is scan through the thumbnails for something interesting. Before you had to dig through all of the threads and you never knew if you were missing out on something cool. Thanks!

    Reply With Quote Reply With Quote  

  32. #28
    Join Date
    Mar 2008
    Location
    Michigan
    Posts
    437
    Thanks
    36
    Thanked 101 Times in 86 Posts
    Follows
    0
    Following
    0
    Yeah I'm not surprised Jason hopped on this... this would be really cool to see integrated into the site somehow. There are portions of this forum that I am hardly aware of and I rarely click through more than a few sketchbooks. Very cool!

    Reply With Quote Reply With Quote  

  33. #29
    Join Date
    May 2010
    Location
    Stockholm/Sweden
    Posts
    758
    Thanks
    372
    Thanked 468 Times in 233 Posts
    Follows
    0
    Following
    0
    Reply With Quote Reply With Quote  

  34. #30
    Join Date
    Mar 2009
    Location
    Philadelphia, PA
    Posts
    481
    Thanks
    539
    Thanked 395 Times in 174 Posts
    Follows
    0
    Following
    0
    This is fantastic!!

    "Contrary to the belief of the layman, the essential of art is not to imitate nature, but under the guise of imitation to stir up excitement with pure plastic elements: measurements, directions, ornaments, lights, values, colors, substances, divided and organized according to the injunctions of natural laws. While so occupied, the artist never ceases to be subservient to nature, but instead of imitating the incidents in a paltry way, he imitates the laws."-Andre Lhote

    Web, FineArt, Sketchbook
    Reply With Quote Reply With Quote  

Page 1 of 2 1 2 LastLast

Members who have read this thread: 1

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •